Redefine magic methods

[jonathaneunice]

I was able to redefine str.lower very nicely, but int.__add__ failed to be redefined on either 2.7 or 3.3. For example:

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 01:25:11)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from forbiddenfruit import curse
>>> def notplus(a,b):
...     return a * b
...
>>> curse(int, '__add__', notplus)
>>> 5 + 9
14

[jonathaneunice]

There seems to be some "not going through the method dictionary" issue here. Because:

>>> (2).__add__(3)
6

So the __add__ method is being switched out. But in expression form, it's not being used.

[clarete]

That's fantastic. I was working on this exact feature RIGHT NOW. My guess is that the parser might be calling things differently for built-in types!

[fabiomcosta]

wow!

[clarete]

@jonathaneunice thank you so much for your bug report. I think I understood what's going on. Let me try to explain what's going on here, then we can try to find a solution to this problem! :)

When the python is processing some code and finds a sum operation like this:

var1 + var2

The system that evaluates the code (Python/ceval.c) will basically call the PyNumber_Add function, which is part of the python API that handles the object abstract protocols (the magic methods -- you find it in Objects/abstract.c). This function, unlike the regular getattr will check for things declared in the tp_as_number type slot before trying to access the instance dict (our dear __dict__ attribute).

So, basically, we need to find a way to patch the type data and replace the ob_type->tp_as_{number,sequence}[nb_add] method.

tl;dr: Basically, magic methods are resolved in a different way compared to the instance or class methods. There's a C array that contains a pointer to the actual implementation. Wich means that it's not actually present on the __slots__ dictionary.

I'm not sure if I'm 100% right, but it's my best shot so far. Thank you!

[jonathaneunice]

@gabrielfalcao I assume he would not entirely approve. ;-)

As JavaScript and Ruby demonstrate, there's no fundamental reason why the base classes of a dynamic language cannot be extended. In JavaScript, libraries like underscore.js are very popular. But Python seems governed with a less exuberant desire for unapproved extensions, and more of a "monkey-patching considered harmful" view of the world.

[clarete]

My efforts to kill this issue are happening here: https://github.com/clarete/forbiddenfruit/tree/fix/4

[clarete]

Well, I was being kinda silly here. I completely forgot about the code generation. Python does not execute everything every time. For example, given the following python file:

1+1
2+3

The python compiler will generate the following code:

  1           0 LOAD_CONST               4 (2)
              3 POP_TOP

  9           4 LOAD_CONST               5 (5)
              7 POP_TOP
              8 LOAD_CONST               3 (None)
             11 RETURN_VALUE

Which means that no matter what, the integer sum operation will happen before my patch gets any chance to be applied.

[jonathaneunice]

I'm not hugely disappointed that operations on primitive numeric types cannot be entirely intercepted. I expected that, optimizations being what they are. The ability to monkey-patch basic data classes such as dict, str, list, and set is still quite good magic.

[clarete]

Also, @jonathaneunice we still can change all other operations that are not optimized. So I'll keep this bug open and I'll write more tests addressing other cases. Thank you and let me know if you have any other ideas!

[clarete]

Working on getting the segfault fixed on Python 3.x

[rizo]

Do we have any progress on this issue? What's the current status? I'd be glad to help with it.

[clarete]

Hi @rizo, thanks for asking. That's the last thing I did: feae79a

If you're interested in helping, I can definitely provide you all the details you need to keep the train moving! :D

[tomyo]

I'll buy a beer to the first one fixing this.

[jab]

Just found my way here while trying to override slice to use a sentinel object rather than None for the value of start, stop, and step when the user doesn't pass anything (background).

I tried running the following with forbiddenfruit revision feae79a and it didn't work:

from forbiddenfruit import curse

def simpletest(self):
    return 'works'

curse(slice, 'simpletest', simpletest)
assert slice(None).simpletest() == 'works'


def myslice__init__(self, *args, **kw):
    print('myslice__init__')  # never called
    self.start = self.stop = self.step = 42

curse(slice, '__init__', myslice__init__)
print(eval('slice(None).start'))  # doesn't work :(

class myslice:
    def __init__(self, *args, **kw):
        self.start = self.stop = self.step = 42

def myslice__new__(cls, *args, **kw):
    print('myslice__new__')  # never called
    inst = myslice()

curse(slice, '__new__', classmethod(myslice__new__))
print(eval('slice(None).start'))  # doesn't work :(

Is there something I'm missing, or is this expected to not work yet?

Thanks for reading, and thanks for releasing forbiddenfruit, it's super cool.

[jachymb]

I was trying to redefine the __mul__ operator for functions so it's do function composition instead. I was trying with forbidden fruit, but it wouldn't work because of this error. But I came up with a different and satisfactory solution, in case someone needs inspiration, I put it here:

import types
__all__ = ['composible']

# This decorator makes function composible by '*' operator with other functions
def composible(f):
    class ComposibleFunction:
        def __init__(self, f):
            self.f = f
        def __call__(self, *a, **kw):
            return self.f(*a, **kw)
        def __mul__(self, g):
            return self.__class__(lambda *a, **kw: self.f(g(*a, **kw)))
    return ComposibleFunction(f)

# Make all builtin function composible:
for b in (b for b in dir(__builtins__)
            if type(getattr(__builtins__,b)) in (type, types.BuiltinMethodType)):
    exec('global '+b+'\n'+b + '=composible(__builtins__.' + b + ')')
    __all__.append(b)

# New functions need to be decorated
@composible 
def foo(x): return x+1

@composible 
def bar(x): return x*2

# This will output 7:
print((foo * bar * sum)([1, 2]))

[voidworker]

I've found this issue when looking for ability to compose functions like f @ g @ h. While cursing object with redefined __matmul__ method doesn't work, there is still yet another satisfactory solution, inspired by @jachymb :

from forbiddenfruit import curse
def go(arg, *t):
	s = t[0](*arg)
	for f in t[1:]:
		s = f(s)
	return (s,) # Composability: everybody loves monoids!
curse(tuple,"go", go)

usage: (x,y,z).go(f, g, h).go(f1,f2) turns into f2(f1(h(g(f(x,y,z)))))

(3,4).\
go(lambda x,y: x*y
  ,lambda x: x**2).go(print)

144
(None,)

Of course, one can also define composable functions like (some_long_chain).then(f, additional_args_for_f). Iterables may also come into play...
Happy functional programming! :)

[clarete]

@tomyo I guess you owe a beer to @alendit :)

[clarete]

Closed by #24

[davips]

I was trying to redefine the __mul__ operator for functions so it's do function composition instead. I was trying with forbidden fruit, but it wouldn't work because of this error. But I came up with a different and satisfactory solution, in case someone needs inspiration, I put it here:

import types
__all__ = ['composible']

# This decorator makes function composible by '*' operator with other functions
def composible(f):
    class ComposibleFunction:
        def __init__(self, f):
            self.f = f
        def __call__(self, *a, **kw):
            return self.f(*a, **kw)
        def __mul__(self, g):
            return self.__class__(lambda *a, **kw: self.f(g(*a, **kw)))
    return ComposibleFunction(f)

# Make all builtin function composible:
for b in (b for b in dir(__builtins__)
            if type(getattr(__builtins__,b)) in (type, types.BuiltinMethodType)):
    exec('global '+b+'\n'+b + '=composible(__builtins__.' + b + ')')
    __all__.append(b)

# New functions need to be decorated
@composible 
def foo(x): return x+1

@composible 
def bar(x): return x*2

# This will output 7:
print((foo * bar * sum)([1, 2]))

Is there any lib implementing this wonderful?