How to replicate dynamic nested dictionaries
Closed this issue · 3 comments
I am evaluating pysyncobj for state replication in our cluster. We have a dictionary of resource states where state corresponding to each resource is a dictionary again. For example
resource = { # Sample resource state
'id': 'resourceId1'
'maxCapacity': 100,
'allocatedCapacity': 20
'resourceCapabilities': {
}
}
resources = { # Dictionary of resource states
'id1': resource1,
'id2': resource2
}
Background about the service:
We have a resource allocation service to which the resources register dynamically when they come up providing their capabilities. This state needs to be replicated across the cluster. When one of the servers gets a job, it allocates the least loaded resource that can handle the job and also updates the resource capacity utilization.
I have created resources as an instance of ReplDict. This is added as a consumer while creating the pysyncobj. When a resource registers with the service dynamically, I create a resource object which again is an instance of ReplDict. Then I add this object into the resources dictionary with key being the resource id. When I do this it fails with an exception that looks like this
"/usr/local/lib/python3.6/site-packages/pysyncobj/syncobj.py", line 1387, in newFunc
funcName = self._syncObj._getFuncName((consumerId, func.name))
AttributeError: 'NoneType' object has no attribute '_getFuncName'
To me it looks like you can only update a resource thats been registered as a consumer while creating the SyncObj. Is that the case? Is there a way to replicate a dynamically updated nested dictionary?
To me it looks like you can only update a resource thats been registered as a consumer while creating the SyncObj. Is that the case?
Yep, currently there is no ability to add consumers dynamically. For your situation you can write a custom consumer, eg:
class Resources(SyncObjConsumer):
def __init__(self):
super(Resources, self).__init__()
self.__resources = {}
@replicated
def addResource(self, resourceID, initialData):
self.__resources[resourceID] = initialData
@replicated
def setResourceValue(self, resourceID, key, value):
self.__resources[resourceID][key] = value
@replicated
def updateResourceValues(self, resourceID, newResourceData):
self.__resources[resourceID].update(newResourceData)
def getResource(self, resourceID):
return self.__resources[resourceID]
And use it like:
r.addResource('id1', {
'id': 'id1',
'maxCapacity': 100,
'allocatedCapacity': 20,
'resourceCapabilities': {
}
})
r.setResourceValue('id1', 'maxCapacity', 200)
@bakwc That works. Thanks a lot for taking time to suggest this solution.
As there is an alternate way to achieve this I am closing this ticket.