Efficiently parallelize hyper parameter evaluation

[sarin1991]

• BTB version: 0.2.4
• Python version: 3.7.1
• Operating System: Amazon Linux (4.14.88-88.76.amzn2.x86_64)

Description

I am trying to speed up hyper parameter tuning by parallelizing the training and scoring using an architecture as shown below:

The problem with the above approach is that sometimes btb proposes the same pipeline or similar hyper parameters to one being currently computed in the worker process. Is it possible to avoid this? Maybe one option would be to store the proposed parameters and ensure that the new ones proposed aren't close by?

[csala]

Hello @sarin1991

A similar error has been already reported in #105, so I'm adding the duplicate label here, as this will be resolved there.

However, I would like to warn you that some of what you report is an expected behavior:

On one side, getting hyperparameter proposals that are close to previous proposals is highly expected, especially around proposals that got a good score, as the tuner will tend to give priority to proposals that are in the regions that are known to be good.
If this is not the case and you are getting proposals that are close to previous ones that were very bad, it could be because the tuner does not have enough data yet to make a better selection.

On the other side, BTB should be expected to discard previous proposals that were already scored and added back to the tuner (this is what #105 will cover), but it should not discard previous proposals that are still running, as there is no way to know if they were successfully evaluated.

So, if you are running multiple workers in parallel and hit this problem, I would suggest that you keep track of the proposals that you are currently scoring and, if a duplicate shows up, discard it and ask the tuner again for a new proposal.

[pvk-developer]

Hello @sarin1991

We have made a major reorganization to the project and a lot of improvements, refering to the issue #131

I have answered how the new Tuner proposes on the following comment: #105 (comment) .

However, if you are using multiprocess for calling multiple times the tuner, we can't ensure that the combinations will not match, because it will discard combinations that were recorded. This is why I would recommend you the following:

• Work with batch proposals. Assuming that you have a cardinality of N, create N/16 or even smaller batches depending on how many workers you have and your N amount.
• Make the workers read / evaluate the proposals from the batch.
• Record the obtained scores inside the tuner.
• Propose new batch .

[micahjsmith]

We've closed this issue as your question about duplicate proposals has been resolved.

If you come up with an approach to parallelizing the evaluation of hyperparameter proposals that you think is flexible enough that others may find it useful, please do chime back in and we can see whether a PR would be in order!