UGE project addition is not working
Opened this issue · 11 comments
HI Team,
When i try to add a project, always throws an error saying ObjectAlreadyExists.
Please help me to resolve this issue,
Below is the problem snippet,
n [19]: print(pp.to_json())
{"object_version": "1.0", "object_class": "Project", "modified_by": "kmr", "modified_on": "2020-07-31T09:35:54.129700", "data": {"name": "test123", "oticket": 0, "fshare": 100, "acl": null, "xacl": null}}
In [30]: pp2=api.add_prj(pycl_object=pp,name=None, data=None, metadata=None, jso
...: n_string=None)
ObjectAlreadyExists Traceback (most recent call last)
in
----> 1 pp2=api.add_prj(pycl_object=pp,name=None, data=None, metadata=None, json_string=None)
in add_prj(self, pycl_object, name, data, metadata, json_string)
qconf_api.py in wrapped_call(func, *args, **kwargs)
155 def wrapped_call(func, *args, **kwargs):
156 try:
--> 157 result = func(*args, **kwargs)
158 return result
159 except QconfException as ex:
qconf_api.py in add_prj(self, pycl_object, name, data, metadata, json_string)
1470 return self.project_manager.add_object(
1471 pycl_object=pycl_object, name=name, data=data,
-> 1472 metadata=metadata, json_string=json_string)
1473
1474 @api_call
dict_based_object_manager.py in add_object(self, pycl_object, name, data, metadata, json_string)
117 try:
118 old_object = self.get_object(object_name)
--> 119 raise ObjectAlreadyExists('%s %s already exists.' % (self.OBJECT_CLASS_NAME, object_name))
120 except ObjectNotFound as ex:
121 # ok
ObjectAlreadyExists: Project test123 already exists.
any update on this issue is greatly appreciated.
Thanks
Hi,
did you verify that the prerequisite is given, i.e. the project does not already exist ?
On the commandline you can verify with:
source your UGE env
qconf -sprjl
you should not see test123
you can this do programmatically as well by
see the config-api/test/test_project.py example...
Andre
Andre,
Thanks for the prompt reply!
I verified it doesn't exist. I followed exact same steps given in the testsuite.
BTW, i have tried this both in Python2.7 and 3.6 versions.
Thanks
So did you try this one and isn't this working for you ?
cd config-api
nosetests -s test/test_project.py
this works ok for me (8.6.x = V86_BRANCH and 8.7.x master)
HI,
I have UGE 8.6.7 version deployed, i tried using python2.7/3.6 versions to run PyCl, both failed.
I have updated 'test_values.json' with 2 test projects, you can see it complains 'ObjectAlreadyExists' error. I made sure those 2 test projects are unique in GRD.
Even ACL/Queue, addition also complains the same.
Below is the error log,
F.E..E.EF....
ERROR: test.test_project.test_add_prj
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/demo/config-api/test/test_project.py", line 64, in test_add_prj
project = API.add_prj(name=PROJECT_NAME)
File "", line 2, in add_prj
File "/demo/config-api/uge/api/qconf_api.py", line 157, in wrapped_call
result = func(*args, **kwargs)
File "/demo/config-api/uge/api/qconf_api.py", line 1472, in add_prj
metadata=metadata, json_string=json_string)
File "/demo/config-api/uge/api/impl/dict_based_object_manager.py", line 119, in add_object
raise ObjectAlreadyExists('%s %s already exists.' % (self.OBJECT_CLASS_NAME, object_name))
uge.exceptions.object_already_exists.ObjectAlreadyExists: Project 3p4N2H already exists.
-------------------- >> begin captured logging << --------------------
UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprjl]
UgeSubprocess: DEBUG: Exit status: 0
UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprj 3p4N2H]
UgeSubprocess: DEBUG: Exit status: 0
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: test.test_project.test_get_prj
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/demo/config-api/test/test_project.py", line 87, in test_get_prj
assert (project.data['name'] == PROJECT_NAME)
KeyError: 'name'
-------------------- >> begin captured logging << --------------------
UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprj 3p4N2H]
UgeSubprocess: DEBUG: Exit status: 0
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: test.test_project.test_modify_prj
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/demo/config-api/test/test_project.py", line 111, in test_modify_prj
oticket = project.data['oticket']
KeyError: 'oticket'
-------------------- >> begin captured logging << --------------------
UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprj 3p4N2H]
UgeSubprocess: DEBUG: Exit status: 0
--------------------- >> end captured logging << ---------------------
======================================================================
FAIL: test.test_project.test_object_not_found
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/demo/config-api/test/utils.py", line 169, in inner
return func(*args, **kwargs)
File "/demo/config-api/test/test_project.py", line 47, in test_object_not_found
assert (False)
AssertionError:
<uge.objects.project_v1_0.Project object at 0x7fffe9107cf8> = <uge.api.qconf_api.QconfApi object at 0x7fffea2246d8>.get_prj('non_existent_project')
assert (False)
-------------------- >> begin captured logging << --------------------
QconfApi: DEBUG: Configuration: SGE_ROOT=/default, SGE_CELL=tform, SGE_QMASTER_PORT=6444, SGE_EXECD_PORT=6445
QconfExecutor: TRACE: Retrieving UGE version
UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -help]
UgeSubprocess: DEBUG: Exit status: 0
QconfExecutor: DEBUG: UGE version: 8.6.7
UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprj non_existent_project]
UgeSubprocess: DEBUG: Exit status: 0
--------------------- >> end captured logging << ---------------------
======================================================================
FAIL: test.test_project.test_delete_prj
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/demo/config-api/test/test_project.py", line 125, in test_delete_prj
assert (len(project_list2) == len(project_list) - 1)
AssertionError:
-------------------- >> begin captured logging << --------------------
UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprjl]
UgeSubprocess: DEBUG: Exit status: 0
UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprj 3p4N2H]
UgeSubprocess: DEBUG: Exit status: 0
UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -dprj 3p4N2H]
UgeSubprocess: DEBUG: Exit status: 0
UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprjl]
UgeSubprocess: DEBUG: Exit status: 0
--------------------- >> end captured logging << ---------------------
Name Stmts Miss Cover
uge/init.py 1 0 100%
uge/api/init.py 2 0 100%
uge/api/ar_api.py 95 59 38%
uge/api/impl/init.py 0 0 100%
uge/api/impl/access_list_manager.py 49 26 47%
uge/api/impl/admin_host_manager.py 13 0 100%
uge/api/impl/ar_manager.py 16 1 94%
uge/api/impl/calendar_manager.py 16 0 100%
uge/api/impl/checkpointing_environment_manager.py 16 0 100%
uge/api/impl/cluster_configuration_manager.py 34 15 56%
uge/api/impl/cluster_queue_manager.py 16 0 100%
uge/api/impl/complex_configuration_manager.py 45 21 53%
uge/api/impl/dict_based_object_manager.py 191 43 77%
uge/api/impl/dict_list_based_object_manager.py 82 51 38%
uge/api/impl/execution_host_manager.py 30 8 73%
uge/api/impl/host_group_manager.py 17 0 100%
uge/api/impl/job_class_manager.py 16 0 100%
uge/api/impl/list_based_object_manager.py 57 34 40%
uge/api/impl/manager_manager.py 13 0 100%
uge/api/impl/operator_manager.py 13 0 100%
uge/api/impl/parallel_environment_manager.py 16 0 100%
uge/api/impl/project_manager.py 19 0 100%
uge/api/impl/qconf_executor.py 101 55 46%
uge/api/impl/qrdel_executor.py 71 50 30%
uge/api/impl/qrstat_executor.py 93 68 27%
uge/api/impl/qrsub_executor.py 74 53 28%
uge/api/impl/resource_quota_set_manager.py 16 0 100%
uge/api/impl/scheduler_configuration_manager.py 21 3 86%
uge/api/impl/share_tree_manager.py 50 26 48%
uge/api/impl/submit_host_manager.py 13 0 100%
uge/api/impl/user_manager.py 16 0 100%
uge/api/qconf_api.py 408 135 67%
uge/config/init.py 0 0 100%
uge/config/config_manager.py 152 29 81%
uge/constants/init.py 0 0 100%
uge/constants/uge_status.py 10 0 100%
uge/exceptions/init.py 10 0 100%
uge/exceptions/ar_exception.py 32 18 44%
uge/exceptions/authorization_error.py 5 1 80%
uge/exceptions/command_failed.py 14 7 50%
uge/exceptions/configuration_error.py 5 1 80%
uge/exceptions/invalid_argument.py 5 1 80%
uge/exceptions/invalid_request.py 5 1 80%
uge/exceptions/object_already_exists.py 5 0 100%
uge/exceptions/object_not_found.py 5 1 80%
uge/exceptions/qconf_exception.py 32 11 66%
uge/exceptions/qmaster_unreachable.py 5 1 80%
uge/log/init.py 0 0 100%
uge/log/log_manager.py 143 53 63%
uge/log/logger_factory.py 38 3 92%
uge/log/stream_log_handler.py 12 3 75%
uge/log/timed_rotating_file_log_handler.py 12 3 75%
uge/log/trace_log_record.py 13 0 100%
uge/log/trace_logger.py 22 5 77%
uge/objects/init.py 0 0 100%
uge/objects/ar_object_factory.py 68 51 25%
uge/objects/project_v1_0.py 12 0 100%
uge/objects/qconf_name_list.py 23 9 61%
uge/objects/qconf_object.py 251 105 58%
uge/objects/qconf_object_factory.py 154 78 49%
uge/objects/uge_release_object_map.py 49 0 100%
uge/utility/init.py 0 0 100%
uge/utility/uge_subprocess.py 75 38 49%
TOTAL 2777 1067 62%
Ran 13 tests in 6.010s
FAILED (errors=3, failures=2)
Maybe there is a mismatch between the current config-api implementation and UGE 8.6.7. Is it possible for you to upgrade your cluster to a more recent UGE patch version ? We arrived already at 8.6.14 and there have been many issues fixed.
I installed the 8.6.7 bits and proceeded as follows:
- install all packages (including the config-api package)
- install UGE as usual
- add additional python modules (python-nose*, python-decorator, python-xmltodict, not sure if python3 was already completely working for 8.6.7)
- cd $SGE_ROOT/config-api, tar xvzf tar xvzf uge-pycl-8.6.7.tar.gz, cd uge-pycl-8.6.7/
- there are two files missing test/utils.py and test_values.json copy them in from the opensource repo
- start your cluster and source the UGE env
- nosetest -s test/test_project.py runs successful then for me
There seems to be a problem with the packaging of the test dir content since these two files should of course be present...
Thanks for your valuable inputs.
We can't upgrade UGE to latest version at the moment as lot of co-ordination required.
In my case, test dir has all the files required, but it is not letting project/queue addition, not sure why.
How about UGE Cli, can you please through some light on how to use it? so that for missing portions, i will run 'qconf' commands directly.
Thanks a lot!
see AdminsGuideGE.pdf, section 1.6.6 (or search for projects)
HI,
Assuming, you are referring to 'adminsguide' related to UGE PycL, where to find it? please let me know
I could see only developer_guide.md in this repo.
Thanks
It is part of UGE gridengine packages. What do you get if you do: qstat -help | head -1
Usually you find the pdf documentation under $SGE_ROOT/doc. If it is not there you have to install your version specific doc package of UGE. I assume that you are running UGE and no other gridengine distribution or am I wrong ?