Accumulate decorators and decorate myself like @send_self@classmethod

Question

Accumulate decorators and decorate myself like @send_self@classmethod

Closed this issue 4 years ago · 5 comments

At first glance, I did not know what yield do in Python, but after having a hard time studying it, I finally understand the idea. But having calling this = yield and later passing the mysterious this.send as callback, does not seem clear than right away defining some lambda or closure. However, you should ask this to a python beginning which do not know neither closures nor what yield does, because I am already used to lambdas and closures. Or perhaps I am not seeing the big picture because the example is too simple or I am too dumb, therefore in a more complex scenario this could have great advantages. Then you could add also a complex example using closures vs resumeback.

I got that linear order, however I do not see how that can easier than just use closures. My problem is against the declaration of the first this = yield and the second yield function(this.send). If I look both codes:

With closure

def main():
    arbitrary_value = 10

    def on_done(number):
        number = str(number)
        print("Result:", number * arbitrary_value)

    ask_for_user_input("Please enter a number", on_done)

With resumeback

from resumeback import send_self

@send_self
def main():
    this = yield  # "this" is now a reference to the just-created generator
    arbitrary_value = 10

    # Yield pauses execution until one of the generator methods is called,
    # such as `.send`, which we provide as the callback parameter.
    number = yield ask_for_user_input("Please enter a number", this.send)
    number = str(number)
    print("Result:", number * arbitrary_value)

I can clearly understand when I what is going on with closures. But when looking at those two yield's with resumeback, I do not see it working clearly. Basically, I would say that closures are more intuitive than the this = yied plus the second yield function(this.send) call.

I think it would help, if the first this = yield call could not required, so I can forget about it. Hence, only the second yield call could be done like this:

from resumeback import send_self

@send_self
def main():
    arbitrary_value = 10

    # Yield pauses execution until one of the generator methods is called,
    # such as `.send`, which we provide as the callback parameter.
    number = yield ask_for_user_input("Please enter a number", send_self)
    number = str(number)
    print("Result:", number * arbitrary_value)

Can the this = yield call be embed inside the decorator @send_self and formerly to "send myself" I could just call yield function(send_self)?

I would say that it would look much better than having to remember both to do the first call this = yield and yield function(this.send).

@FichteFoll I could do this:

from resumeback import send_self

@send_self
def main(this):
    # Yield pauses execution until one of the generator methods is called,
    # such as `.send`, which we provide as the callback parameter.
    number = yield ask_for_user_input("Please enter a number", this.send)
    number = str(number)
    print("Result:", number * arbitrary_value)

I'm unsure how I would make the above work for methods though. Gonna need even more magic

I would suggest the this to be a function pointer, so, it can be more straight forward. However, I do not know about other usages of it where could not be pointing to a function directly:

from resumeback import send_self

@send_self
def main(myself):
    arbitrary_value = 10

    # Yield pauses execution until one of the generator methods is called,
    # such as `.send`, which we provide as the callback parameter.
    number = yield ask_for_user_input("Please enter a number", myself)
    number = str(number)
    print("Result:", number * arbitrary_value)

For classmethods, perhaps it could be a cumulative sum of decorators:

from resumeback import send_self

@send_self
@classmethod
def main(myself, cls):
    arbitrary_value = 10

    # Yield pauses execution until one of the generator methods is called,
    # such as `.send`, which we provide as the callback parameter.
    number = yield ask_for_user_input("Please enter a number", myself)
    number = str(number)
    print("Result:", number * arbitrary_value)

But I do not know whether or not you can patch stuff like that in Python.

Answer 1 · 2018-02-11T16:12:34.000Z

Perhaps the function pointer could patched instead of the parameter:

from resumeback import send_self

@send_self
def main():
    arbitrary_value = 10

    # Yield pauses execution until one of the generator methods is called,
    # such as `.send`, which we provide as the callback parameter.
    number = yield ask_for_user_input("Please enter a number", main)
    number = str(number)
    print("Result:", number * arbitrary_value)

Or

from resumeback import send_self

@send_self
def main():
    arbitrary_value = 10

    # Yield pauses execution until one of the generator methods is called,
    # such as `.send`, which we provide as the callback parameter.
    number = yield ask_for_user_input("Please enter a number", main.send_self)
    number = str(number)
    print("Result:", number * arbitrary_value)

Answer 2 · 2018-02-13T22:31:41.000Z

So, this reply will be kind of technical because I need to touch on various aspects of how this module actually works. Many parts of it are already micro-optimized and there isn't much room for improvements in some aspects because I'm hitting Python limitations. Anyway, let's begin.

I can clearly understand when I what is going on with closures. But when looking at those two yield's with resumeback, I do not see it working clearly. Basically, I would say that closures are more intuitive than the this = yied plus the second yield function(this.send) call.

I can't deny that, but the reason for closures being more intuitive is because generators are a kind of advanced concept in Python and many people only very rarely run into them. resumeback makes it even harder because here you are even thinking about how generators need to be resumed after they have been paused, which even I have never used except for this module.

That said, understanding what a generator is (at least in a broad sense) is fundamentally important when using resumeback because that's basically what it does: Turning your function into a resumeable routine.

A similar concept that has been added to Python's standard library more or less recently is asynchronous programming, natively suported. It was hacked onto it for web servers and the most prominent implementation was probably Twisted, but now that it's in the stdlib, it has become much more convenient to work with. Regardless, asynchronous programming is not a trivial concept and requires a different train of thought than you would normally entail with threads, and considering threads have dominated concurrency paradigms for years you can't really blame anyone for finding threads more intuitive to work with. However, once you start understanding how the asynchronous stuff works, you will quickly find yourself not wanting to go back to threading unless you really need to (low-level performance critical tasks which you most likely won't see implemented in Python).

Anyway, my point is: Yes, it seems unintuitive, but only because it is unconventional. You have been working with clojures in many places and languages before, most likely, but the idea of pausing execution within a function to wait for something else to happen is gaining traction. Golang's entire premise is basically that.
The entire point of this module is to "backport" this paradigm to callback-based APIs, so can't address your style preferences. I could potentially emphasize the example a bit by adding another 2 or 3 levels of clojures, but I think this task can be handled by the reader, too. I might add a sentence or two about this to the readme/docs.

I would say that it would look much better than having to remember both to do the first call this = yield

This is a good idea, but I am currently unsure how I can remove that. You quoted my proposal on that about passing the, so I won't repeat it here. The problem with passing the generator object in a parameter, however, is that at the moment of invocation, the generator hasn't been created yet. I could, however, wrap the entire generator inside another generator and then wrap it with a yield from. This would make it incompatible with Python 2 though.

Another concern is applying the decorator on a method definition, because methods (and classmethods as well) are special in Python.

I would suggest the this to be a function pointer, so, it can be more straight forward. However, I do not know about other usages of it where could not be pointing to a function directly

The translated suggestion is to make WeakGeneratorWrapper.__call__ be an alias for WeakGeneratorWrapper.send.
The only problem here is that all the resuming functions are properties that create a strong reference to the generator to prevent it being garbage-collected as all references to it would have been removed once it is paused, so I can't make that happen. It would be possible to do this for the StrongGeneratorWrapper, but then you get a cyclic reference instead that the garbage-collector needs to detect before it can remove it.

Perhaps the function pointer could patched instead of the parameter

This would not be thread-safe. I could hack thread-specific return values on an attribute by using the descriptor protocol and defining __get__, but that's a lot more magic than I was hoping to use. But more importantly, it wouldn't work for methods because you cannot add attributes to a bound method object (I've run into that before). And you might have a long-ish function name that you'd rather not type every time you want to pause and resume execution.

I hope I didn't forget addressing any of your or my thoughts on this

Answer 3 · 2018-02-13T23:48:14.000Z

Thanks for the insights. I am nor sure whether this should be closed or it could be kept open about the first this = yield call explicit line.

Answer 4 · 2018-02-14T02:00:10.000Z

I'll close it once I explored a few ideas about that and either found a solution or concluded that it's not feasible.

Answer 5 · 2020-04-09T20:15:13.000Z

Done in 233b418.