SymbolicML/DynamicExpressions.jl

Possible extensions

Opened this issue · 9 comments

AlCap23 commented

Hey @MilesCranmer !

I've been thinking about possible extensions and wonder how you think about this:

  1. Parametric Operations
    In short, I want to be able to supply function with hold tuneable constants, e.g. f(x, y , p) = x / ( y + p) where p is a constant with a tuneable value. This could maybe be done with predefined patterns of nodes, but also might be doable somewhat different?

  2. N-ary Operations
    This one seems a little harder, but might be doable as well ( major refactoring ). My reasoning for allowing this is basically steering more into the field of program synthesis and allow chunks of expressions to be parsed. Possible applications for this might be inference of systems of equations based on "building blocks" with contain typical patterns. I know that in theory this is also possible using just binary ops, but the chance of discovery might increase based on the structural prior.

  3. Arbitrary Expression Graphs
    This plays along with 2. and reuse of patterns is key here. In many (natural) systems, reoccurrence of features is common ( in classical mechanics this would be the sin(q) and cos(q) for describing the translational movement of an angular joint ). Within a classical binary tree, each expression needs to be sampled individually while in a DAG ( possibly "just" a topological sorted DAG ), the influence of a given node could be extended beyond its direct parent.

I've made some prototypes using Symbolics / MTK for this, but it its rather slow ( given that I need to build the function for each candidate ).

Something of a parallel part:
I've been working on some alternatives to GA/GP to solve symbolic regression, partly motivated by the latest trend of using RNN to sample the graph and also related to MINLP. If you're interested we could have a chat about this :).

Cheers!

Edit Uh, I just noticed that this might all be doable quite easily given that you generate dispatches using the OperatorEnum! Nice.

MilesCranmer commented

Hey @AlCap23,
Thanks for reaching out!

  1. I think this is already possible, it just would require 2. to allow for operators with multiple parameters. Note that in SymbolicRegression.jl, you can set arbitrary constraints, so I commonly set a constraint that a power law operator (^) only has a constant right child – which is basically a parameter.

  2. This requires a bit of work but in theory its perfectly doable. Within DynamicExpressions.jl it shouldn't be too bad; it would get a bit harder in SymbolicRegression.jl because you need to change some of the formula involved in generating mutations/crossovers, etc. I've never had a use-case that justified the effort required to get this working though, as most operators can just be binary.

  3. This is already possible, see MilesCranmer/SymbolicRegression.jl#135 (now moved to this repo). When copying a tree, you would pass preserve_topology=true to preserve a graph structure.

Always interested in discussing alternatives to GPs, happy to chat whenever! I think it should be reasonably easy to just stick a learned sampling strategy here: https://github.com/MilesCranmer/SymbolicRegression.jl/blob/51d205c518eb3e99cfd45ac6a2d3dbbbd1944f32/src/Mutate.jl#L63. In general you don't want to do "too much" with neural nets inside the search process of a GP because they don't generalize well outside of a training set of expressions (it's simply too large of a space).

Cheers,
Miles

AlCap23 commented

Thanks for the answer!

Awesome, I didn't know that 3. is already doable! I'll add you to a repo I've adjusted for DynamicExpressions yesterday, so maybe my intentions will be clearer ;) .

Additionally, the PR #15 could be useful. This is basically the ability to reuse certain parameter patterns within equation ( fix the number of unique parameters ). An example would be :

$$ f(x, p_1) = p_1 * x - \frac{x}{p_1 + x} $$

Where we want to add the constraint that all parameters are linked.

MilesCranmer commented

Thanks. For that use-case, I think the existing struct already works for it (without named nodes). Basically just have a single node and point to it from multiple parents => that parameter is thus linked.

AlCap23 commented

Hm, my test case was something like:

op = OperatorEnum(binary_operators = [-, *], unary_operators = [sin])
param = Node(; val = 3.0)
feat = Node(; feature = 1)
ex = sin(feat*param) - param
get_constants(ex) # Returns [3.0, 3.0], so each Node is still treated as an individual
set_constants(ex, [2.0]) # Error which makes sense

And I tried to constraint it by using the hash map, but this did not work out either. Another approach would be to just store the parameters and set them, which could also be done ( I think ).

MilesCranmer commented

Oh, for constants in particular, I actually didn't merge the update yet. But a working implementation is here: MilesCranmer/SymbolicRegression.jl@master...shared-node-utils#diff-dbc5c3e1c445f26c200c1ffef22b8451d9c6d30f2d02d51ff1a982e5789e39cfL127

MilesCranmer commented

I wonder if it could also be a good idea to have the Node type itself indicate whether nodes are to be shared or not. Then there would be no need for making an explicit indication that one wishes to copy topology or not.

MilesCranmer commented

Hey @AlCap23! I'm cleaning up some of the utilities for working with shared nodes: #17. I think it should be easier to add other functionalities now. I will try to keep the syntax for as much as we can preserve_topology=false as a keyword argument. Setting that to true will assume that shared nodes are in the tree.

MilesCranmer commented

@AlCap23 - Graph like expressions are working in v0.14!! 🎉

julia> operators = OperatorEnum(;
           binary_operators=[+, -, *], unary_operators=[cos, sin, exp]
       );

julia> x1, x2 = GraphNode(feature=1), GraphNode(feature=2)
(x1, x2)

julia> y = sin(x1) + 1.5
sin(x1) + 1.5

julia> z = exp(y) + y
exp(sin(x1) + 1.5) + {(sin(x1) + 1.5)}

Here, the curly braces {} indicate that the node
is shared by another (or more) parent node.

This means that we only need to change it once
to have changes propagate across the expression:

julia> y.r.val *= 0.9
1.35

julia> z
exp(sin(x1) + 1.35) + {(sin(x1) + 1.35)}

This also means there are fewer nodes to describe an expression:

julia> length(z)
6

julia> length(convert(Node, z))
10

where we have converted the GraphNode to a Node type,
which breaks shared connections into separate nodes.

  • 🎉1