BigWarp, get transformations back into BDV

[martinschorb]

Hi,

is there a way to get the transformations that I create with bigwarp associated to my selected sources and then back into "our" bdv?

[NicoKiaru]

Not yet, but that's doable. I'll give it a shot

[martinschorb]

I do see a xxxx_xfm when I launch BigWarp appearing in my sources list. Is that it?

[martinschorb]

Also, I noticed that BigWarp's "Export as ImgPlus" does produce entirely empty images.
Before it crashed (see saalfeldlab/bigwarp#88) but I got a new snapshot that fixed at least this...

[bogovicj]

Also, I noticed that BigWarp's "Export as ImgPlus" does produce entirely empty images.

Can you elaborate? Obviously, it's not supposed to do that. What's your setup like?

[bogovicj]

@NicoKiaru
Regarding getting bigwarp transformed images into bdv, can you give this a try

LandmarkTableModel ltm = new LandmarkTableModel( src.getSource(0, 0).numDimensions() );
try {
	ltm.load(landmarks);
} catch (IOException e) {
	e.printStackTrace();
}

WarpedSource<T> srcOut = new WarpedSource<>( src, src.getName() + "_transformed");
srcOut.updateTransform( new ThinplateSplineTransform( ltm.getTransform() ));
srcOut.setIsTransformed( true );

BdvFunctions.show( srcOut );

[martinschorb]

Minimum working example:
https://oc.embl.de/index.php/s/fFxYlIqFmULJspK

• select fm_ch0 as moving and m_120 as target

• display fused image (in fixed image viewer)

• export as ImgPlus (Target,Target)

BdvHandlePostprocessor:BdvHandle found.
BdvHandlePostprocessor:BdvHandle found.
Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
	at bigwarp.BigWarpExporter.<init>(BigWarpExporter.java:98)
	at bigwarp.BigWarpRealExporter.<init>(BigWarpRealExporter.java:58)
	at bigwarp.BigWarpRealExporter.<init>(BigWarpRealExporter.java:70)
	at bigwarp.BigWarpExporter.getExporter(BigWarpExporter.java:639)
	at bdv.ij.ApplyBigwarpPlugin.runExport(ApplyBigwarpPlugin.java:740)
	at bigwarp.BigWarp.exportAsImagePlus(BigWarp.java:1071)
	at bigwarp.BigWarp.exportAsImagePlus(BigWarp.java:818)
	at bigwarp.BigWarpActions$ExportImagePlusAction.actionPerformed(BigWarpActions.java:1012)
	at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
	at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
	at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
	at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
	at javax.swing.AbstractButton.doClick(AbstractButton.java:376)
	at javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:842)
	at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(BasicMenuItemUI.java:886)
	at java.awt.Component.processMouseEvent(Component.java:6539)
	at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
	at java.awt.Component.processEvent(Component.java:6304)
	at java.awt.Container.processEvent(Container.java:2239)
	at java.awt.Component.dispatchEventImpl(Component.java:4889)
	at java.awt.Container.dispatchEventImpl(Container.java:2297)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4904)
	at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4535)
	at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4476)
	at java.awt.Container.dispatchEventImpl(Container.java:2283)
	at java.awt.Window.dispatchEventImpl(Window.java:2746)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:760)
	at java.awt.EventQueue.access$500(EventQueue.java:97)
	at java.awt.EventQueue$3.run(EventQueue.java:709)
	at java.awt.EventQueue$3.run(EventQueue.java:703)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:84)
	at java.awt.EventQueue$4.run(EventQueue.java:733)
	at java.awt.EventQueue$4.run(EventQueue.java:731)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:730)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

This happens in both moving and target windows.

Tested on Linux & Windows.

[bogovicj]

Thanks - I also get this behavior.
I have an idea what this is about, but can you try something to confirm you get the same thing as me:

1. Open the images in the same way as before.
2. Control-click several times in the image (at least 4). (This explicitly creates an identity transformation.)
3. export as ImgPlus (Target,Target), like before.

You'll hopefully not get an error. A resulting image will appear, but it probably will be empty :(

[NicoKiaru]

FYI, if relevant, the pom file has these explicit versions:

<bigdataviewer-core.version>7.0.0</bigdataviewer-core.version>
		<bigdataviewer-vistools.version>1.0.0-beta-15</bigdataviewer-vistools.version>
		<bigwarp_fiji.version>5.0.0</bigwarp_fiji.version>

[martinschorb]

1. Open the images in the same way as before.

2. Control-click several times in the image (at least 4).  (This explicitly creates an identity transformation.)

3. export as ImgPlus (Target,Target), like before.

You'll hopefully not get an error. A resulting image will appear, but it probably will be empty :(

Exactly. Empty image.
The error message appears when I try to export without doing anything.

[bogovicj]

Perfect, I'll get you a patch as soon as I can.

To clarify - you're using this to export an affine-transformed moving image into target space, right?
Is that correct?

[martinschorb]

Both images come affine transformed from @NicoKiaru 's multi-source viewer. I use BigWarp to refine this transformation. This works: the resulting transformed moving image is transformed correctly. I just realized that the export to ImagePlus seems to be broken in this case. When I load the pixel data of the two images into BigWarp the "traditional" way, the export works.

[NicoKiaru]

This repo uses a different 'manual transform' than the one from bigdataviewer.
There are 2 ways to store a manual transform (append and mutate), and the way the 'storage' is done depends on the original source class (TransformedSource and AbstractSpimSource supported).

• Case 1 : mutate a TransformedSource : it concatenate the transform to the current transform of the source -> you cannot retrieve the original transform. The output source is still the initial TransformedSource object
• Case 2 : append a TransformedSource : it wraps the original transformed source into a new transformed source -> you keep the previous transformed source. The output source is a NEW TransformedSource object
• Case 3 : mutate an AbstractSpimSource : it concatenate the transform to the last transform taken contained in the model of the AbstractSpimData object -> you cannot retrieve the original transform. The output source is still the initial AbstractSpimSource object
• Case 4 : append an AbstractSpimSource : it adds a transform to the last transform taken contained in the model of the AbstractSpimData object. The output source is still the initial AbstractSpimSource object

All these modes are in the SourceAndConverterUtils class :

bigdataviewer-playground/src/main/java/sc/fiji/bdvpg/sourceandconverter/SourceAndConverterUtils.java

Lines 441 to 497 in add256d

	/**
	* if a source has a linked spimdata, mutates the last registration to account for changes
	* @param affineTransform3D
	* @param sac
	* @return
	*/
	public static SourceAndConverter mutateLastSpimdataTransformation(AffineTransform3D affineTransform3D, SourceAndConverter sac) {
	assert SourceAndConverterServices
	.getSourceAndConverterService()
	.getSacToMetadata().get(sac).containsKey(SPIM_DATA_INFO);
	assert SourceAndConverterServices
	.getSourceAndConverterService()
	.getSacToMetadata().get(sac).get(SPIM_DATA_INFO) instanceof SourceAndConverterService.SpimDataInfo;
	SourceAndConverterService.SpimDataInfo sdi = ((SourceAndConverterService.SpimDataInfo)
	SourceAndConverterServices.getSourceAndConverterService()
	.getSacToMetadata().get(sac).get(SPIM_DATA_INFO));
	// TODO : find a way to pass the ref of starter into this function ? but static looks great...
	BdvHandle bdvHandle = SourceAndConverterServices.getSourceAndConverterDisplayService().getActiveBdv();
	int timePoint = bdvHandle.getViewerPanel().getState().getCurrentTimepoint();
	ViewRegistration vr = sdi.asd.getViewRegistrations().getViewRegistration(timePoint,sdi.setupId);
	ViewTransform vt = vr.getTransformList().get(vr.getTransformList().size()-1);
	AffineTransform3D at3D = new AffineTransform3D();
	at3D.concatenate(vt.asAffine3D());
	at3D.preConcatenate(affineTransform3D);
	ViewTransform newvt = new ViewTransformAffine(vt.getName(), at3D);
	vr.getTransformList().remove(vt);
	vr.getTransformList().add(newvt);
	vr.updateModel();
	try {
	Method updateBdvSource = Class.forName("bdv.AbstractSpimSource").getDeclaredMethod("loadTimepoint", int.class);
	updateBdvSource.setAccessible(true);
	AbstractSpimSource ass = (AbstractSpimSource) sac.getSpimSource();
	updateBdvSource.invoke(ass, timePoint);
	if (sac.asVolatile() != null) {
	ass = (AbstractSpimSource) sac.asVolatile().getSpimSource();
	updateBdvSource.invoke(ass, timePoint);
	}
	} catch (ClassCastException e) {
	e.printStackTrace();
	} catch (Exception e) {
	e.printStackTrace();
	}
	return sac;
	}

[bogovicj]

Quick update -
I'm confident the cause of the error is fixed, but pretty sure there is another problem which causes the image to be empty (for the test above).

I'm looking into it, and will keep you posted.

[NicoKiaru]

@martinschorb , are sources 2d or 3d ?

[martinschorb]

@martinschorb , are sources 2d or 3d ?

these are 2D.
I can try with 3D as well but due to Fiji currently not starting with BDV Playground (#98) in all except one remote installation it is a bit tricky to test... Will keep you posted.

[martinschorb]

Hi,

It seems that the new sources that are added somehow are screwed up in terms of ViewSetupID...
Wrong Colour. It also does not remember the Projection mode.
When I have Views fused in the BigWarp Window, also the fixed source appears (with assigned Colour and Contrast settings for one moving...)

[martinschorb]

Minimum example:

files:
https://oc.embl.de/index.php/s/KihC2SEFIsCBYB1

• empty BDV (mixed proj)
• fm->Sum ; m120 -> Avg
• Show, change colour fm ->red

Launch BigWarp, fm... moving

landmarks.zip

• transform (T)
• show fused (F)

now I have:

• I have to change Projectionmode of fm_xfm.. to Sum

• Show new source

• AutoContrast,AdjustView does not matter...

[ERROR] Module threw exception
java.lang.NullPointerException
	at sc.fiji.bdvpg.scijava.services.SourceAndConverterBdvDisplayService.show(SourceAndConverterBdvDisplayService.java:181)
	at sc.fiji.bdvpg.scijava.command.bdv.BdvSourcesAdderCommand.run(BdvSourcesAdderCommand.java:31)
	at org.scijava.command.CommandModule.run(CommandModule.java:199)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:228)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)