Priorised fitting not performing any fits
Closed this issue · 4 comments
Hi Paul,
I am having trouble getting the priorised fitting mode of Aegean to work with Aegean version 2.2.2-(2020-06-19). I am using the following options:
aegean --ratio=1.0 --priorized=1 --input=input_GLEAM_J000120-140410.vot --table=test.vot map.fits
The input catalogue contains 13 sources. No fits are performed:
81407:source_finder:INFO Using ratio of 1.0 to scale input source shapes
81407:source_finder:INFO 13 sources in catalog
81407:source_finder:INFO 13 sources accepted
81407:cluster:INFO Regrouping islands within catalog
81407:source_finder:INFO Performing fits
81407:source_finder:INFO fit 0 components
81407:aegean:INFO found 0 sources total
If I only include one source in the input catalogue, the fitting works fine... Can you please check?
Tom
Hi @tfranzen2 I can't reproduce the error. Can you give me a dropbox link (or similar) of the input map and catalogue?
(either here or via email)
Hi Paul,
After sending you the input map and catalogue, I updated Anaconda on my Mac and re-installed AegeanTools. This fixed the source fitting problem!
However, I now have another issue. When running Aegean (both the stable and latest versions) with multiple cores I get broken pipe errors in 'pprocess.py'. This sometimes causes the pipeline to crash. Here is an example where the pipeline does not finish successfully:
/Users/franzen/opt/anaconda3/lib/python3.7/site-packages/AegeanTools/pprocess.py:410: UserWarning: Removed channel <AegeanTools.pprocess.Channel object at 0x7faf55be6710> due to exception: [Errno 32] Broken pipe
warn("Removed channel %r due to exception: %s" % (channel, exc))
/Users/franzen/opt/anaconda3/lib/python3.7/site-packages/AegeanTools/pprocess.py:410: UserWarning: Removed channel <AegeanTools.pprocess.Channel object at 0x7faf55be6d10> due to exception: [Errno 32] Broken pipe
warn("Removed channel %r due to exception: %s" % (channel, exc))
/Users/franzen/opt/anaconda3/lib/python3.7/site-packages/AegeanTools/pprocess.py:410: UserWarning: Removed channel <AegeanTools.pprocess.Channel object at 0x7faf55be6a90> due to exception: [Errno 32] Broken pipe
warn("Removed channel %r due to exception: %s" % (channel, exc))
Traceback (most recent call last):
File "/Users/franzen/opt/anaconda3/bin/aegean", line 345, in
docov=options.docov, cube_index=options.slice)
File "/Users/franzen/opt/anaconda3/lib/python3.7/site-packages/AegeanTools/source_finder.py", line 1901, in find_sources_in_image
for srcs in queue:
File "/Users/franzen/opt/anaconda3/lib/python3.7/site-packages/AegeanTools/pprocess.py", line 908, in next
self.store()
File "/Users/franzen/opt/anaconda3/lib/python3.7/site-packages/AegeanTools/pprocess.py", line 406, in store
self.store_data(channel)
File "/Users/franzen/opt/anaconda3/lib/python3.7/site-packages/AegeanTools/pprocess.py", line 888, in store_data
data = channel.receive()
File "/Users/franzen/opt/anaconda3/lib/python3.7/site-packages/AegeanTools/pprocess.py", line 141, in receive
obj = self._receive()
File "/Users/franzen/opt/anaconda3/lib/python3.7/site-packages/AegeanTools/pprocess.py", line 125, in _receive
obj = pickle.load(self.read_pipe)
_pickle.UnpicklingError: pickle data was truncated
I don't get these broken pipe errors when I set --cores=1. For now, I can run Aegean fine with a single core but obviously this can be slow for large images. Do you know what might be causing this?
Thanks,
Tom
Given that updating Aegean has fixed your issue, I'm going to say that the bug with not fitting all the sources has been fixed between whatever version of Aegean you were using and the latest version (even though they report same/similar version numbers). I'll close this ticket.
As for the pickle error this is described in issue #108 . The summary is that it is caused by large objects being shared between processes by the pprocess module, it is not something that I have been able to fix directly, but I have a work around in progress that should avoid the issue all together.