Why the globals().update(locals()) hack is needed

[tbodt]

This is in response to @3b1b's last video, which featured the hack of writing globals().update(locals()) in IPython, and asked the audience if anyone could explain why or how to do it better.

This happens because: In a repl, variable reads and writes first go to the locals() dict. But an important dirty secret of python is that local variables within functions don't do that - instead the bytecode compiler creates an array for local variables and uses indexes (because a list is faster than a dict). This means that, for an inner function to access a local variable from an outer function, python generates code almost like this, without involving locals() at all:

def outer():
	var = 0
	def inner(): return var
# becomes
def _inner(var):
	return var
def outer():
	var = 0
	inner = functools.partial(_inner, var)

The consequence is that capturing variables is only possible if the compiler actually sees that one function is inside another. So, if you define a function a repl spawned in a function context, it will only inherit global variables and not function/repl local variables.

Why does this work in IPython normally? Ultimately it is in fact because this repl is spawned from inside a function. When started normally, globals() and locals() are the exact same dictionary, so it doesn't matter that variables writes go to locals() and function variables reads come from global(). (Turns out this is equally true for code at the top level of a module!)

What to do about it? Without changing IPython or Python, the best thing I can think of is to just make a more sophisticated version of the globals().update(locals()) hack. Maybe write a dict subclass to replace globals() that's able to fall back to locals() for lookups, then use that as the globals only while running the IPython repl!

[akshatsinghania]

helpful

[fofoni]

Hi!

I also wrote about that and submited a PR to fix it: #2180

I liked yor explanation of cell variables! I don't think that's what's happening in the issue in the video, though, because cell variables only work like that when you have textually nested functions: The compiler needs to see the "def inside a def" syntax in a single piece of source code in order to generate cells.

I think this was on the right track:

Why does this work in IPython normally? Ultimately it is in fact because this repl is spawned from inside a function. When started normally, globals() and locals() are the exact same dictionary, so it doesn't matter that variables writes go to locals() and function variables reads come from global(). (Turns out this is equally true for code at the top level of a module!)

That's the problem: in the top-level of a module, and also in vanilla REPL shells, globals() and locals() are the same dict instance, but in the shell spawned by manimgl --embed they're distinct. (Which is exactly what my PR changes.)

As for "it is in fact because this repl is spawned from inside a function", I wouldn't word it that way. The shell executes user code by passing that code to Python's exec/eval builtins, and it also passes globals and locals namespaces to those exec/eval calls, and what's important is simply guaranteeing that those namespaces are the same. Now, the implementation of Scene.embed in manim attempted to "recreate" the scope inside a function by passing different locals and globals namespaces to the shell. But that doesn't really work: since the compiler doesn't see the "def inside a def", there are no cells, so the nonlocal keyword doesn't work; but still, variables defined in the shell scope are local to that scope, not global, so the global keyword doesn't work either. Anyway, I explained all of this (or at least, my understanding of it) in more detail in my PR.

[fofoni]

Oh, also, something I forgot to mention in my PR: the fact that list comprehensions were previously affected, but somehow are not affected anymore, is PEP 709: Inlined comprehensions.