Substra/substra-backend

authenticate_worker in Process has a strange behavior (not systematic)

Closed this issue · 0 comments

When using this code :

import time
from billiard import Process
from substrapp.tasks.utils import authenticate_worker

def fetch_auth(node_id):
    time.sleep(1)
    res = authenticate_worker(f'MyOrg{node_id}MSP')
    if f'Secret{node_id}' not in res.password:
        raise Exception(f'MyOrg{node_id}MSP - {res.password}')

def fetch_node_ids():
    models = []
    for node_id in range(1, 12):
        args = (node_id, )
        proc = Process(target=fetch_auth,
                       args=args)
        proc.start()
        models.append(proc)

    for proc in models:
        proc.join()


fetch_node_ids() 

it raises, rarely, this kind of error

2020-11-30 13:51:22,439 -  - INFO - MyOrg1MSP, MyOrg1MSP, selfSecret1                                                                                                                                                                        
2020-11-30 13:51:22,445 -  - INFO - MyOrg1MSP, MyOrg2MSP, nodeSecret2w1                                                                                                                                                                      
2020-11-30 13:51:22,450 -  - INFO - MyOrg1MSP, MyOrg3MSP, nodeSecret3w1                                                                                                                                                                      
2020-11-30 13:51:22,458 -  - INFO - MyOrg1MSP, MyOrg4MSP, nodeSecret4w1                                                                                                                                                                      
2020-11-30 13:51:22,477 -  - INFO - MyOrg1MSP, MyOrg6MSP, nodeSecret6w1                                                                                                                                                                      
2020-11-30 13:51:22,491 -  - ERROR - MyOrg1MSP, MyOrg7MSP, nodeSecret9w1                                                                                                                                                                      
Process Process-106:                                                                                                                                                                                                                          
Traceback (most recent call last):                                                                                                                                                                                                            
  File "/usr/local/lib/python3.6/site-packages/billiard/process.py", line 327, in _bootstrap                                                                                                                                                  
    self.run()                                                                                                                                                                                                                                
  File "/usr/local/lib/python3.6/site-packages/billiard/process.py", line 114, in run                                                                                                                                                         
    self._target(*self._args, **self._kwargs)                                                                                                                                                                                                 
  File "<ipython-input-10-bc3e67016c6e>", line 10, in fetch_auth                                                                                                                                                                              
    raise Exception(f'MyOrg{node_id}MSP - {res.password}')                                                                                                                                                                                    
Exception: MyOrg7MSP - nodeSecret9w1                                                                                                                                                                                                          
2020-11-30 13:51:22,497 -  - ERROR - MyOrg1MSP, MyOrg5MSP, nodeSecret10w1                                                                                                                                                                     
Process Process-104:                                                                                                                                                                                                                          
2020-11-30 13:51:22,500 -  - ERROR - MyOrg1MSP, MyOrg8MSP, nodeSecret5w1
Process Process-107:
2020-11-30 13:51:22,489 -  - ERROR - MyOrg1MSP, MyOrg9MSP, nodeSecret8w1
Process Process-108:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/billiard/process.py", line 327, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.6/site-packages/billiard/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-10-bc3e67016c6e>", line 10, in fetch_auth
    raise Exception(f'MyOrg{node_id}MSP - {res.password}')
Exception: MyOrg9MSP - nodeSecret8w1
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/billiard/process.py", line 327, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.6/site-packages/billiard/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-10-bc3e67016c6e>", line 10, in fetch_auth
    raise Exception(f'MyOrg{node_id}MSP - {res.password}')
Exception: MyOrg8MSP - nodeSecret5w1
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/billiard/process.py", line 327, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.6/site-packages/billiard/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-10-bc3e67016c6e>", line 10, in fetch_auth
    raise Exception(f'MyOrg{node_id}MSP - {res.password}')
Exception: MyOrg5MSP - nodeSecret10w1
2020-11-30 13:51:22,526 -  - INFO - MyOrg1MSP, MyOrg11MSP, nodeSecret11w1

Outgoing node are defined like this :

outgoingNodes:
  - { name: MyOrg1MSP, secret: selfSecret1 }
  - { name: MyOrg2MSP, secret: nodeSecret2w1 }
  - { name: MyOrg3MSP, secret: nodeSecret3w1 }
  - { name: MyOrg4MSP, secret: nodeSecret4w1 }
  - { name: MyOrg5MSP, secret: nodeSecret5w1 }
  - { name: MyOrg6MSP, secret: nodeSecret6w1 }
  - { name: MyOrg7MSP, secret: nodeSecret7w1 }
  - { name: MyOrg8MSP, secret: nodeSecret8w1 }
  - { name: MyOrg9MSP, secret: nodeSecret9w1 }
  - { name: MyOrg10MSP, secret: nodeSecret10w1 }
  - { name: MyOrg11MSP, secret: nodeSecret11w1 }