Not sure how different it is on xbone, but have you tried setting it up like I did? I leave the calibration on default (100) and do all the adjustment in my TV settings. I think the biggest problem with their implementation is that increasing that setting makes the entire scene uniformly brighter and more washed out, and they're telling people with OLEDs to jack it all the way up to 300. I turn up the HDR brightness on the TV and it just makes the bright areas brighter/more vibrant while leaving dark areas dark.
It's totally different from my setup for GoW, Spider-Man, Tomb Raider, etc. but it still looks alright in the end. Nowhere near as good as GoW did, but still much better than with HDR off entirely.
FYI for other people like
Tarrant
who have a TCL 6-series, there's advanced calibration settings for the TV that can only be accessed via the phone/tablet app. The color stuff you probably don't watch to touch unless you actually do a professional hardware calibration, but the gamma setting is pretty useful all the time and really should have been in the TV menus.