signetlabdei/sem

invalid argument value when using a lambda / function

MarshallAsch opened this issue · 1 comments

Based on your comment #46 (comment), and your recent WNS3 2021 presentation, I tried using a lambda to set 2 of my parameters to the same value using the following param_list

param_combination = {
    'total-nodes': [40, 80, 120, 160, 200],
    'storage-space': lambda p: p['total-nodes'],
    'buffer-space': lambda p: p['total-nodes'],
...
}

campaign.run_missing_simulations(param_combination, runs=1)

when running that batch of simulations I got an error about the lambda failing to pickle

Running simulations:   0%|                                                                                              | 0/1 [00:00<?, ?simulation/s]
Traceback (most recent call last):
  File "/home/user/Documents/ns3_stuff/experiment/./simulation.py", line 55, in <module>
    campaign.run_missing_simulations(param_combination, runs=1)
  File "/home/user/.local/lib/python3.9/site-packages/sem/manager.py", line 372, in run_missing_simulations
    self.run_simulations(
  File "/home/user/.local/lib/python3.9/site-packages/sem/manager.py", line 292, in run_simulations
    for result in result_generator:
  File "/home/user/.local/lib/python3.9/site-packages/tqdm/std.py", line 1185, in __iter__
    for obj in iterable:
  File "/home/user/.local/lib/python3.9/site-packages/sem/parallelrunner.py", line 22, in run_simulations
    for result in pool.imap_unordered(self.launch_simulation,
  File "/usr/lib64/python3.9/multiprocessing/pool.py", line 870, in next
    raise value
  File "/usr/lib64/python3.9/multiprocessing/pool.py", line 537, in _handle_tasks
    put(task)
  File "/usr/lib64/python3.9/multiprocessing/connection.py", line 211, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib64/python3.9/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f22bfe71940>: attribute lookup <lambda> on __main__ failed

So I tried writing it the other way you suggested without the lambda:

def getNumNodes(param):
    return param['total-nodes']

param_combination = {
    'total-nodes': [40, 80, 120, 160, 200],
    'storage-space': getNumNodes,
    'buffer-space': getNumNodes,
...
}

campaign.run_missing_simulations(param_combination, runs=1)

and got the error

Stderr: Invalid argument value: storage-space=<function getNumNodes at 0x7f4f4d8eb940>
...

Use this command to reproduce:
python waf --run "rhpman-example  --total-nodes=40 --storage-space=<function getNumNodes at 0x7f4f4d8eb940> --buffer-space=<function getNumNodes at 0x7f4f4d8eb940> --RngRun=0"

It appears as though this is caused by the function not actually being called to generate the correct value for the field.
Is there a way arround this error or will I have to specify the values without using the function?
I currently using the latest pip package, version 0.2.3.

It seems this issue is just on that version, I tested it later last night on the develop branch after I remembered that your WNS3 2021 presentation used that and it worked as excepted. :)