Archive/Delete in chunks? 500 Internal Server Error

[TJM]

When I try to run arsa.py, we run into a problem:

[root@ft-spacewalk arsa]# python arsa.py 
Username: myusername
Password: 
Found completed action #52972 ('Show differences between profiled config files and deployed config files')...
Found completed action #52971 ('Show differences between profiled config files and deployed config files')...
Found completed action #52970 ('Show differences between profiled config files and deployed config files')...
Found completed action #52969 ('Show differences between profiled config files and deployed config files')...
(snip)
Found archived action #63063 ('Show differences between profiled config files and deployed config files')...
Found archived action #63062 ('Show differences between profiled config files and deployed config files')...
Archiving actions...
Enabling workaround to archive/delete more than 100 actions...
[65536, 65541, 65543, 65547, 65548, 65553, 65554, 65558, 65559, 65565, 65566, 65570, 65571, 65574, 65576, 65577, 65583, 65584, 65588, 65589, 65590, 65592, 65595, 65601, 65604, 65607, 65613, 65614, 65618, 65619, 65623, 65625, 65631, 65634, 32868, 65637, 65643, 65644, 65648, 65649, 65655, 65661, 
(snip)
72211, 71446, 71447]
Traceback (most recent call last):
  File "arsa.py", line 152, in <module>
    client.schedule.deleteActions(key,tempActions)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1243, in request
    headers
xmlrpclib.ProtocolError: <ProtocolError for localhost/rpc/api: 500 Internal Server Error>

• There are ~5400 entries in the array. I had already archived like 30,0000 through the GUI ;)

Perhaps there are WAY more in the "archive" ? I am deleting them 10,000 at a time through the webUI for now, hopefully we can setup arsa.py to run periodically and not get into this state again :)

I think the problem was encountered during the "delete" step rather than the archive step, where there was ~55,000 entries in my archive (unless I lost count deleting them 10k at a time through the browser)

[TJM]

Note, it works fine today with only ~150 entries.

[stdevel]

Hey TJM,
sorry for the delay. I'm kinda busy currently. I will have a look at your problem in my testing environment.

Which Spacewalk or its flavor are you running? Which version?

I remember having strange issues like that with older versions of Spacewalk and Red Hat Satellite. I was able to fix them by checking/altering the JVM Heap size. See also: http://www.freiesmagazin.de/mobil/freiesMagazin-2015-03-bilder.html#fm_15_03_spacewalk_teil5

(I know, it's german. Maybe you can translate it with Google Translator.)

Best regards!

[TJM]

Hi @stdevel,

No worries, we are using spacewalk-backend-server-2.2.43-1.el6.noarch. I think we are actually due for a 2.3 upgrade. I was able to get the gist of the article and bumped the Xmx to 512m (from 256m). Either way, if the Browser UI limits the number to be archived by 30k, perhaps that is a good limit to use on this script? In addition, I'd suspect an API bug that it allows you to attempt an action which causes a 500 error in the first place, but I am not sure whether spacewalk is under active development. Of course that only happens if you don't run this script for like a year with 100+ servers, and get a lot of completed actions stacked up.

Tommy