facebook/Ax

[GENERAL SUPPORT]: Parallelism and arbitrary parameter type support.

muazhari opened this issue · 5 comments

muazhari commented

Question

Does the library support parallelism & arbitrary parameter type support like in pymoo?

And, can you reference the related documentation and tutorial? I can't find any of it.

Please provide any relevant code snippet if applicable.

# parameter type example:
class Edge:
 id: UUID
 node_1: Node
 node_2: Node
 weight: float
 data: Dict[str, Any]

Code of Conduct

  • I agree to follow this Ax's Code of Conduct
bernardbeckerman commented

@muazhari for parallel evaluations, we rely on an abstraction called the Scheduler, which orchestrates multiple trials simultaneously in asynchronous fashion. Please see the Schedule tutorial to learn more (link).

As for "arbitrary parameter type support" I'm not exactly sure what you're requesting, but Ax offers a few types of parameters, each of which has values corresponding to python primitives (i.e., int, float, bool, or str):

  • RangeParameter: must be int or float (link)
  • ChoiceParameter: can be int, float, bool, or str (link)
  • FixedParameter: can be int, float, bool, or str (link)

I hope that helps!

muazhari commented

As for "arbitrary parameter type support" I'm not exactly sure what you're requesting, but Ax offers a few types of parameters, each of which has values corresponding to python primitives (i.e., int, float, bool, or str)

I mean, does it support like this?

# parameter type example:
class Edge:
 id: UUID
 node_1: Node
 node_2: Node
 weight: float
 data: Dict[str, Any]
 
values = [
Edge(id=uuid.uuidv4(), node_1=Node("a"), node_2=Node("b"), weight=1, data={"something1": 1}),
Edge(id=uuid.uuidv4(), node_1=Node("b"), node_2=Node("c"), weight=1, data={"something2": 2}),
Edge(id=uuid.uuidv4(), node_1=Node("c"), node_2=Node("d"), weight=1, data={"something1": 3})
]
p_1 = ChoiceParameter(name="name1", parameter_type=Edge, values=values)
bernardbeckerman commented

Unfortunately not! You can address this in a few ways:

  1. Decode primitive types using a dictionary, e.g.,
param_value_to_edge = {1: edge_1, 2: edge_2, ...}
p_1 = ChoiceParameter(name="name1", parameter_type=int, values=[1, 2], ordered=False)
# then at the usage site
my_edge = param_value_to_edge[parameters["name1"]]
...
  1. Serialize your non-primitive types to string when creating your parameter, then deserialize them at the usage site.

Let me know if that makes sense.

muazhari commented

Unfortunately not! You can address this in a few ways:

  1. Decode primitive types using a dictionary, e.g.,
param_value_to_edge = {1: edge_1, 2: edge_2, ...}
p_1 = ChoiceParameter(name="name1", parameter_type=int, values=[1, 2], ordered=False)
# then at the usage site
my_edge = param_value_to_edge[parameters["name1"]]
...
  1. Serialize your non-primitive types to string when creating your parameter, then deserialize them at the usage site.

Let me know if that makes sense.

So, we can still use these methods?

  • Only use the id of the values then reference it back to data outside the evaluation function.
  • Encode any objects to string as values (even though the type of each object is different in the same parameter) then decode again to be used in the evaluation function.

Do those methods affect the optimization performance? It bit hacky though.

bernardbeckerman commented

Do those methods affect the optimization performance?

The serialization/deserialization method has no inherent effect on optimization performance, however it matters how you parameterize your search space. For instance unordered ChoiceParameters are harder for Ax to search over than ordered ChoiceParameters or RangeParameters (assuming the ordering corresponds to some natural order among choices; for instance ["small", "medium", "large"] are naturally ordered whereas ["apple", "orange", "banana"] may not be). To this point, it may be easier for Ax to learn your search space if you split up various attributes of this function. For example, if you'd like to tune the edge weight specifically, it would be easier for Ax to model if you provide it as a RangeParameter, like

RangeParameter(
    name="edge_weight",
    lower=0,
    upper=1
    parameter_type=ParameterType.FLOAT,
)

instead of an unordered ChoiceParameter corresponding to different edge objects.

It bit hacky though.

I'm not sure I agree with this characterization when it comes to referring to dict values by key. It does put a very minor implementation burden on the side of the evaluation function, but avoids significant complexity in serialization/deserialization, which we support throughout our library, without sacrificing any ability to optimize systems efficiently.

Closing this out for now, but please message here (at-mention me so to be sure I see this) or create a new issue for further assistance.

  • 👍1