Bug: Rounding discrete distributions to their nearest discrete point after undoing the SVD in the TPE might pick the wrong point

[guillaume-chevalier]

Describe the bug
Upon doing an SVD and applying the TPE ratios within the SVD's space and then undoing the SVD space to get back to normal space, then the discrete points next to the found best point may not match.

To Reproduce

1. Fix #465
2. Optimize a quantized log-distributed hyperparam
3. Observe that it may round most of the time to the lowest value instead of the highest value

Expected behavior
The quantized point to be properly rounded to its nearest quantized value upon exiting the TPE to return the optimal point.

I believe that to fix this, it'd be as easy as:

1. round up and round down
2. pass the up and down rounding back into the SVD space
3. evaluate the distance in SVD space of this point v.s. the center point. In SVD space, everything is linear as per #465, so this means that the distance will be the good one
4. pick the nearest point
5. get back to normal space by undoing the SVD transformation
6. re-round the point to its nearest point (just to eliminate numerical errors when back in the original space)
7. you now have the good discrete point after having passed through the Orthogonalized (SVD) TPE.

Additional context
#464

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in the next 180 days. Thank you for your contributions.