dell/PyU4V

Hangs when multiple calls made to failover_storage_group_srdf() with different SG names

Closed this issue · 4 comments

Sacheen123 commented

Thanks Paul for your reply in advance!!!

I have a following call in my code with different group_name & same rdf_number & opt as:
replication.failover_storage_group_srdf(storage_group_id=self.group_name, srdf_group_number=self.rdf_number, failover_options=opt, _async=False)

I am calling above function at the same time for 6 storage groups. It is giving me exception:
"Bad or unexpected response from the storage volume backend API: Error PUT symmetrix resource. The status code received is None and the message is None."

Also error message:
ERROR:PyU4V.rest_requests:The PUT request to URL https://10.221.172.100:8443/univmax/restapi/92/replication/symmetrix/000297900701/storagegroup/SG_dl360g10-141_142_R6/rdf_group/101 timed-out, but may have been successful. Please check the array. Exception received: HTTPSConnectionPool(host='10.221.172.100', port=8443): Read timed out. (read timeout=120).

Looks like the API call hung caused the problem. I then used API to set timedout as below did not help me.
self.conn = PyU4V.U4VConn(username=self.username, password=self.password, server_ip= self.server_ip, port=self.port, verify=self.verify,array_id=self.array_id)
self.conn.set_requests_timeout(500)

Am I missing something?
NOTE: We have both Gatekeeper & Unisphere available.

Thanks,
Sacheen Birhade

rawstorage commented

Sacheen, you are issuing calls for 6 different groups at the same time, Try staggering the failovers, put a wait in doing them one after the other.

The error code is a generic one coming from the Unipshere server not from PyU4V, I suspect your Unipshere server is busy and replying back with the generic error. Another thing you could do is issue the calls asyncronously and check the job ids after, Check the examples in https://pyu4v.readthedocs.io/en/latest/programmers_guide_src/provisioning.html#a-synchronous-provisioning-creating-storage-for-a-host it's the same concept for all calls that support Asyncronous Execution, this is an ideal candidate for this type of operation. You won't be getting timeouts or overloading the server.

rawstorage commented

@Sacheen123 we have enabled the Discussions feature on GitHub for questions and Q&A, for non PyU4V issues and general discussions if you want to create discussions we can tackle on best effort basis.

Let me know if the Asynchronous calls resolved your issue, I'm confident that this would

Sacheen123 commented

Thanks Paul for your help. I have fixed this issue using exclusive lock through my python code.
In my case multiple calls to the same API were through different thread, so file based exclusive lock helped me to send one request at a time. By that time all other threads were waiting.
flow:
take exclusive lock()
all to api ()
release exclusive lock()

You were really helpful to me. Thanks. best wishes.

rawstorage commented