bug with `deep_ensemble_trajectory` internal state

[hstojic]

I have observed strange behaviour with internal state of deep_ensemble_trajectory.

When using acquisition function similar to ParallelContinuousThompsonSampling with multiple models/outputs there are two undesirable behaviours, both observed using the ask-tell interface and using diversify option in deep_ensemble_trajectory, relevant state variables being indices and seeds:

1. If we fix TF random seeds to have reproducible experiments, deep_ensemble_trajectory for all models ends up with the same state. This happens only in the first BO step, when prepare_acquisition_function is called, after that I guess when update_acquisition_function and internal state is updated they become different and all is fine. It's not a really big issue as these are models of different outputs so same network or quantiles used doesn't mean the same input to the acq function, but its really not ideal.
2. If we don't fix TF random seeds something else happens. In the initial step state of all samplers is different but then in the following steps they become identical. One potentially important detail is that we save the optimization state in the previous step and load it back in (and have to recreate the acq rule in doing so as it is not saved).

@uri-granta could you perhaps look into this?

[uri-granta]

I've managed to reproduce the first problem but not the second: resampling two samplers with no random seeds set still gives me different states afterwards.

The first problem is to do with how TF global seeds interact with tf.functions:

Note that tf.function acts like a re-run of a program in this case. When
the global seed is set but operation seeds are not set, the sequence of random
numbers are the same for each tf.function.

The best solution I've found is for DeepEnsemble to generate a fresh operation seed for each sampler by default, which then gets incremented whenever we resample. However, it's still quite ugly and I'm not totally sure the extra code complexity is worth it given the low impact of 1. See #625 for discussion.

[hstojic]

2nd issue was a weird one, not sure how it came about really, but its great yo have managed to deal with the 1st