Stale output when python file changes during runtime

[danr]

I'm writing a live-reloading library on top of python which compiles and executes the same file each time it is saved. When running snoop inside such a file we see that snoop keeps referring to the first saved version of the file even when it has been changed. I'm attaching code that reproduces this:

Setup

open('./live.py', 'w').write('''
import snoop
with snoop:
    x = 1
    y = 2
    print(x, y)
''')

exec(compile(open('./live.py').read(), 'live.py', 'exec'), {})

open('./live.py', 'w').write('''
import snoop
with snoop:
    x = 4
    y = 5
    print(x, y)
''')

exec(compile(open('./live.py').read(), 'live.py', 'exec'), {})

Observed output

17:10:01.04 >>> Enter with block in <module> in File "live.py", line 3
17:10:01.04 ...... __builtins__ = {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other object...nil' object; Ellipsis represents `...' in slices.", '__package__': '', '__loader__': <class '_frozen_importlib.BuiltinImporter'>, ...}
17:10:01.04 ...... len(__builtins__) = 152
17:10:01.04 ...... snoop = <class 'snoop.configuration.Config.__init__.<locals>.ConfiguredTracer'>
17:10:01.04    4 |     x = 1
17:10:01.04    5 |     y = 2
17:10:01.04    6 |     print(x, y)
1 2
17:10:01.04 <<< Exit with block in <module>
17:10:01.04 >>> Enter with block in <module> in File "live.py", line 3
17:10:01.04 ...... __builtins__ = {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other object...nil' object; Ellipsis represents `...' in slices.", '__package__': '', '__loader__': <class '_frozen_importlib.BuiltinImporter'>, ...}
17:10:01.04 ...... len(__builtins__) = 152
17:10:01.04 ...... snoop = <class 'snoop.configuration.Config.__init__.<locals>.ConfiguredTracer'>
17:10:01.04    4 |     x = 1
17:10:01.04 .......... x = 4
17:10:01.04    5 |     y = 2
17:10:01.04 .......... y = 5
17:10:01.04    6 |     print(x, y)
4 5
17:10:01.04 <<< Exit with block in <module>

Expected output

# ... same as above until:
17:10:01.04    4 |     x = 4
17:10:01.04 .......... x = 4
17:10:01.04    5 |     y = 5
17:10:01.04 .......... y = 5
17:10:01.04    6 |     print(x, y)
4 5
17:10:01.04 <<< Exit with block in <module>

[alexmojaki]

This is similar to #27

I don't see a way for snoop to efficiently ensure the cache is up to date regardless of who might be doing this kind of magic. But if you want to clean it in your own library, you can run this after the file contents are changed before executing them:

snoop.formatting.Source.for_filename(filename, {}, use_cache=False)

[danr]

Thanks Alex! Your fix works and it is reasonable that snoop users explicitly clear the caches if they change the file live.

I was hoping that there would be some way to clear all executing caches but the _cls_local function ironically prevents this:

https://github.com/alexmojaki/executing/blob/13995ba3a165687e04018aba1dc9c37d9faea0c5/executing/executing.py#L313-L314

Is it feasibe to add a cache clear method to executing which would clear all subclass caches as well?

[alexmojaki]

Clearing all executing caches would be ideal, yes.

Based on https://stackoverflow.com/questions/3862310/how-to-find-all-the-subclasses-of-a-class-given-its-name, here's a snippet:

import executing
import snoop
import stack_data

class S(stack_data.Source):
    pass

def all_subclasses(cls):
    return set(cls.__subclasses__()).union(
        [s for c in cls.__subclasses__() for s in all_subclasses(c)]
    )

print(all_subclasses(executing.Source))
# {<class 'stack_data.core.Source'>, <class 'snoop.formatting.Source'>, <class '__main__.S'>}

[alexmojaki]

executing now clears caches in all Source subclasses after an importlib.reload(module). Hopefully that helps with this case too.