Python hangs when calling flush_all and doing add or set operation with multiprocess.

[ManofWax]

I'm using memcache 1.6.21 from official docker images.

I tried running this code using pymemcache 4.0.0 and python 3.10.12:

from multiprocessing import Pool
from pymemcache.client.base import Client


client = Client('localhost')
def f(x):
    return client.add('foo', 42, noreply=False)

if __name__ == '__main__':
    client.flush_all(noreply=False)
    with Pool(4) as p:
        print(p.map(f, [1, 2, 3]))

This script hangs indefinitly and when interrupted with ctrl+c this is the output:

python test.py
^CProcess ForkPoolWorker-3:
Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Process ForkPoolWorker-4:
Traceback (most recent call last):
  File "/home/ffoschini/Src/M33/misc/test.py", line 12, in <module>
    print(p.map(f, [1, 2, 3]))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 768, in get
    self.wait(timeout)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 765, in wait
    self._event.wait(timeout)
  File "/usr/lib/python3.10/threading.py", line 607, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.10/threading.py", line 320, in wait
    waiter.acquire()
KeyboardInterrupt

This is the output of memcache server:

ns_memcached                    | <24 new auto-negotiating client connection
ns_memcached                    | 24: going from conn_new_cmd to conn_waiting
ns_memcached                    | 24: going from conn_waiting to conn_read
ns_memcached                    | 24: going from conn_read to conn_parse_cmd
ns_memcached                    | 24: Client using the ascii protocol
ns_memcached                    | <24 flush_all 0
ns_memcached                    | >24 OK
ns_memcached                    | 24: going from conn_parse_cmd to conn_new_cmd
ns_memcached                    | 24: going from conn_new_cmd to conn_mwrite
ns_memcached                    | 24: going from conn_mwrite to conn_new_cmd
ns_memcached                    | 24: going from conn_new_cmd to conn_waiting
ns_memcached                    | 24: going from conn_waiting to conn_read
ns_memcached                    | 24: going from conn_read to conn_parse_cmd
ns_memcached                    | <24 add foo 0 0 2
ns_memcached                    | 24: going from conn_parse_cmd to conn_nread
ns_memcached                    | > NOT FOUND foo
ns_memcached                    | >24 STORED
ns_memcached                    | 24: going from conn_nread to conn_new_cmd
ns_memcached                    | 24: going from conn_new_cmd to conn_mwrite
ns_memcached                    | 24: going from conn_mwrite to conn_new_cmd
ns_memcached                    | 24: going from conn_new_cmd to conn_waiting
ns_memcached                    | 24: going from conn_waiting to conn_read
ns_memcached                    | 24: going from conn_read to conn_parse_cmd
ns_memcached                    | <24 add foo 0 0 2
ns_memcached                    | 24: going from conn_parse_cmd to conn_nread
ns_memcached                    | foo
ns_memcached                    | >24 NOT_STORED
ns_memcached                    | 24: going from conn_nread to conn_new_cmd
ns_memcached                    | 24: going from conn_new_cmd to conn_parse_cmd
ns_memcached                    | <24 add foo 0 0 2
ns_memcached                    | 24: going from conn_parse_cmd to conn_nread
ns_memcached                    | foo
ns_memcached                    | >24 NOT_STORED
ns_memcached                    | 24: going from conn_nread to conn_new_cmd
ns_memcached                    | 24: going from conn_new_cmd to conn_mwrite
ns_memcached                    | 24: going from conn_mwrite to conn_new_cmd
ns_memcached                    | 24: going from conn_new_cmd to conn_waiting
ns_memcached                    | 24: going from conn_waiting to conn_read

If i use Pool(2) everything works fine, every other value greater than 2 hangs the script. (I've a Intel Core i7-1165G7 4 core 8 threads).

If I remove the flush_all everything works as expected.

[ManofWax]

I did some further testing and it seems the issue happens only when flush_all is called from the same pymemcache Client used for other calls.

This code works fine:

from multiprocessing import Pool
from pymemcache.client.base import Client

client_task = Client('localhost')
def f(x):
    return client_task.add("foo", 42, noreply=False)

if __name__ == "__main__":
    client = Client('localhost')
    client.flush_all(noreply=False)
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

[jogo]

The default client isn't thread/multiprocessing safe, instead you should try this https://pymemcache.readthedocs.io/en/latest/getting_started.html#using-a-client-pool

Hope that helps, if not please re-open this ticket.