Example: Expression graphs with quoted expressions

[jimbaker]

Let's create a worked-out example in about 1 page of code or less of an interesting computation graph that can be re-expressed as an expression graph.

The heart of PyTorch is the ability to define the following:

• Computation graphs of tensors. Tensors are more or less like Numpy multidimensional arrays, but they can optionally remember how they are computed (with autograd=True option).
• Support computation of the gradient (autodifferentiation, via backprop).

Tensors are eagerly evaluated, although there's at least one PR for lazy evaluation. Everything else in PyTorch is making this run as efficiently as possible, while also making it easy to build these computation graphs for neural networks, such as for CNNs (where sharing is very important!). See for example https://pytorch.org/tutorials/beginner/nn_tutorial.html that discusses this in more detail - and shows that deep learning is not at all magical.

Like other computation graph schemes, there's an alternative computation strategy which is built up by a graph of objects, then executed. Doing so allows for retaining the computation steps. This is a graph (specifically a DAG) because intermediate steps are reused.

With that in mind:

• Is it possible to readily express the computation graph instead in terms of an expression graph that could be created with quoted functions, as in #6?

• Would this at all be useful, while retaining for example the autodiff capability? (No need to actually write autodiff, just show it's possible by being able to introspect the graph.)

If so, create a toy variant (maybe with wrapping NumPy? but it really doesn't matter too much, it could be just integer variables) to show what this might look like, especially with respect to ergonomics.

[jimbaker]

The distinction in terminology between "expression graphs" (eg with assignment expressions, memoization) and "computation graphs" (such as building out my own AST and writing an interpreter) is not ideal. There might be better terms, but we can stick with these for now.

[gvanrossum]

I think expressions are usually trees, right? I think of ‘x+x’ as a tree, and ‘f()+f()’ too.

Computation graph makes me think of something that can be distributed across multiple machines, and the graph (presumably DAG) aspect seems relevant there.