Images without original file block linking

[pwalczysko]

It seems that if I have an image in my Project which does not have an original file associated, then the plugin does not do any linking for that project at all.

Workflow:

1. Create an image in OMERO.figure, using the feture "New OMERO Image" under the menu in top-right corner which is most usually used for creating PDFs of the Figures, see screenshot below.
2. This will create a new image with ID 4490453 (ID just to match the example in my error pasted below) in the same dataset as the image(s) in the Figure. (alternatively, you can create an image without original file by running the util scripts > Channel Offsets... script)
3. Try to run the omero transfer pack command - this is apparently successful
4. Try to run the omero transfer unpack command - this ends with an error (click on details below to expand). Also observe that all the images are in the Orphaned daataset, the image coming from the Figure is missing (probably was not "packed" in the first place ?) and no linkage of images (of any images) was performed.

....
Matching source and destination images...
{'/uod/idr/nightshade/OMERO/ManagedRepository/./pwalczysko_4356/2016-01/25/16-41-44.859/Baskydy.jpg': [4072883]}
{'/Users/pwalczysko/all-stuff/./pwalczysko_4356/2016-01/25/16-41-44.859/Baskydy.jpg': [4490454]}
defaultdict(<class 'list'>, {'pwalczysko_4356/2016-01/25/16-41-44.859/Baskydy.jpg': [4072883]})
defaultdict(<class 'list'>, {'pwalczysko_4356/2016-01/25/16-41-44.859/Baskydy.jpg': [4490454]})
Creating and linking OMERO objects...
{'Project:26228': 31575}
{'Dataset:69275': 82318}
{'Annotation:358610': 383125, 'Annotation:358605': 383126, 'Annotation:358612': 383127, 'Annotation:358608': 383128, 'Annotation:358604': 383129, 'Annotation:358611': 383130, 'Annotation:358614': 383131, 'Annotation:358606': 383132, 'Annotation:358613': 383133, 'Annotation:358603': 383134, 'Annotation:358616': 383135, 'Annotation:358607': 383136, 'Annotation:358615': 383137, 'Annotation:358609': 383138, 'Annotation:325403': 383139, 'Annotation:356151': 383140, 'Annotation:358602': 383141}
Traceback (most recent call last):
  File "/Users/pwalczysko/miniconda3/envs/cli37/bin/omero", line 11, in <module>
    sys.exit(main())
  File "/Users/pwalczysko/miniconda3/envs/cli37/lib/python3.7/site-packages/omero/main.py", line 125, in main
    rv = omero.cli.argv()
  File "/Users/pwalczysko/miniconda3/envs/cli37/lib/python3.7/site-packages/omero/cli.py", line 1784, in argv
    cli.invoke(args[1:])
  File "/Users/pwalczysko/miniconda3/envs/cli37/lib/python3.7/site-packages/omero/cli.py", line 1222, in invoke
    stop = self.onecmd(line, previous_args)
  File "/Users/pwalczysko/miniconda3/envs/cli37/lib/python3.7/site-packages/omero/cli.py", line 1299, in onecmd
    self.execute(line, previous_args)
  File "/Users/pwalczysko/miniconda3/envs/cli37/lib/python3.7/site-packages/omero/cli.py", line 1381, in execute
    args.func(args)
  File "/Users/pwalczysko/miniconda3/envs/cli37/lib/python3.7/site-packages/omero/plugins/transfer.py", line 84, in _wrapper
    return func(self, *args, **kwargs)
  File "/Users/pwalczysko/miniconda3/envs/cli37/lib/python3.7/site-packages/omero/plugins/transfer.py", line 124, in unpack
    self.__unpack(args)
  File "/Users/pwalczysko/miniconda3/envs/cli37/lib/python3.7/site-packages/omero/plugins/transfer.py", line 200, in __unpack
    populate_omero(ome, img_map, self.gateway)
  File "/Users/pwalczysko/miniconda3/envs/cli37/lib/python3.7/site-packages/generate_omero_objects.py", line 195, in populate_omero
    link_images(ome, ds_map, img_map, conn)
  File "/Users/pwalczysko/miniconda3/envs/cli37/lib/python3.7/site-packages/generate_omero_objects.py", line 134, in link_images
    img_id = img_map[img.id]
KeyError: 'Image:4490453

[erickmartins]

Yep, I'm aware of this one - I am doing no checking whether an image has an associated file or not source-side, so images without a file are added to the XML file, and then cannot be matched destination-side because omero download cannot download them. Couple options for solving this:

1. Skip images without files source-side. That will allow everything else to work as intended.
2. omero export images without files as TIFFs that are added to the zip file, and manually create an association between those things on the XML. That will allow the image to be reproduced destination-side, and is probably what we should ultimately do here.

[pwalczysko]

Thank you. Definitely voting for Option 2. , this seems to be what we need in the use case which drives our investigation here and is the preferred way to enable this tool to be a reliable data archiving solution imho.

[erickmartins]

So far my proposed solution (at #11 ) is checking which images do not have OriginalFiles, and export each Image ID that doesn't have them as a TIFF inside the package zip file. On the other side, I import the TIFF as a "regular" image, and everything seems to work just fine. Does that cover your current use case? Any suggestions?

[pwalczysko]

So far my proposed solution (at #11 ) is checking which images do not have OriginalFiles, and export each Image ID that doesn't have them as a TIFF inside the package zip file. On the other side, I import the TIFF as a "regular" image, and everything seems to work just fine. Does that cover your current use case? Any suggestions?

Thank you, will test tomorrow or later this week.

[erickmartins]

Solved with #11 .