Occluding Projection mides

Hi,

The occluding projection modes don't seem to behave as I would think they do.

AverageOccluding looks like it behaves as Sum to the other average sources but is not displayed on top of any Sources
SumOccluding is on top of everything.

I think for the CLEM case we should have (maybe the AverageOccluding case) behave such that it is occluding every other Average source but still gets overlayed by Sum sources. (Scenario: high-mag EM source occludes low-mag EM sources but FLuorescence (Sum) sources are still on top).

Arf, too bad. @tischi could you have a look ?

@martinschorb

I had a quick look and could not see an obvious bug.

Could you please have a look at this code and see if you find the culprit (the getArgbIndex method is where things mostly happen)?

The code definitely needs improvement, e.g., I am sorting the sources right now, but I think that's not even used....

Another question: do you think having layers and some alpha-blending mode would do the job? I am thinking, e.g., of iteratively adding images to the display like this: display = image + (1-alpha) * display, where each image has an alpha and alpha=1 would occlude everything that has been added earlier to the display and alpha=0 would be the normal sum-projection; issue with this is that we would to somehow encode the order in which the images are added. I am also not sure this would be enough to cover your use cases or whether we would need additional logic. For example, I do not immediately see how to realize an average projection with this kind of math. Probably it would need to be display = layer + (1-alpha) * display and then within the layers one again does some blending of several different images?

I think I found one bug that I may have introduced recently. I fixed it in the code that is referenced above, please anyway have a look whether you think the logic is correct as it is now.

Here:

bigdataviewer-playground/src/main/java/sc/fiji/bdvpg/bdv/projector/AccumulateMixedProjectorARGB.java

Line 179 in 7d6531a

    
           if ( blendingMode.equals( BlendingMode.Sum ) || blendingMode.equals( BlendingMode.SumOccluding ) )

I would understand the code logic for sources with SumOccluding such that they are considered for summing. Shouldn't it be the other way round (not equals), so they occlude all other Sum sources? The same is true for AverageOccluding.

I am not entirely sure what to do with the alpha channel. We obviously don't use it for our datasets. I also don't know how alpha<255 behaves when the number of underlying sources changes (->the sum of all alphas is no longer 255). For fluorescence, the summing makes a lot more sense. Our average projection in my eyes does exactly the same job as using alpha but taking the number of sources into account.

In general, I don't think that a final (display) alpha of less than 255 makes sense at all, since we have already dealt with all visible sources at the current pixel. I think we can force set it to 255.

I would understand the code logic for sources with SumOccluding such that they are considered for summing. Shouldn't it be the other way round (not equals), so they occlude all other Sum sources? The same is true for AverageOccluding

I understood it such the Occluding just means that they are occluding everything that is not itself , sort of an occluding layer.

For example SumOccluding just means that the source has two tags: Sum and Occluding

In other words:

there is an "occluding" tag and if there are more than zero sources with this tag only the sources with the occluding tag are shown.
there is an "average" tag, sources with this tag are averaged => "averageSource"
there is a "sum" tag, sources with this tag, and the averageSource are summed

Maybe you could try if this logic would allow you to realize your use cases. If not, we have to think of something else.

OK, I see.

This means Occluding supersedes everything regardless of the additional Projection mode tag.

In general, ProjectionMode descrbes the modality type:

Average for Brightfield sources (high intensity background, signal from contrast variations) like EM, Transmission, ...
Sum for Darkfield sources (low intensity background, signal = high intensity) like Fluorescence

Our main use case for Occluding is that there are multiple sources of the same type but different magnifications (mostly EM). These sources will almost always be shown as Average (brightfield information).

Now, we only want to show one of these sources, so we set it as Occluding.
However, we would like that sources from a different modality (Fluorescence, set to Sum) should still be shown (summed).

I see two options to achieve this functionality:

we define something like "source modality" as additional attribute and then and then introduce two different types of "occluding" (intra-modal and global). It will then become tricky to then do the final mixed projector from many modalities.
we keep sum and average for bright- and darkfield sources but try to treat them more separately. We would then also have separate occlusions. When you define the source order (

bigdataviewer-playground/src/main/java/sc/fiji/bdvpg/bdv/projector/AccumulateMixedProjectorARGB.java

Line 98 in 7d6531a

public static int[] getSourcesOrder( BlendingMode[] blendingModes )

), instead of putting just all "occluding" sources to the top, "Average" and "Sum" sources could be treated as two subsets. Within the subsets, we would put "occluding" sources to the top and treat them exclusive. We would then end up with one "averageSource" and one "sumSource" that are summed to give the final mixed output.

Alternatively, we could obviously start thinking of using the alpha channel instead of averaging sources.

forget this post. I did something wrong...

OK,

now changing the projection mode for a source that is displayed in a BDV window does not do anything anymore... I tried a lot of different combination and there is simply nothing happening. This is the current update.

I get

Bdv Playground actions settings File B:\\plugins\bdvpgsettings\bdvpg.tree.actions.txt does not exist.
Bdv Playground default actions settings File B:\plugins\bdvpgsettings\bdvpg.tree.actions.txt.default.txt does not exist.
Bdv Playground actions settings File B:\\plugins\bdvpgsettings\bdvpg.tree.actions.txt does not exist.
Bdv Playground default actions settings File B:\plugins\bdvpgsettings\bdvpg.tree.actions.txt.default.txt does not exist.

when starting up, could this be related? The file names of the defaults.txt look screwed-up...

BTW:
I started fiji in the B: directory, because that is where most data lives. Usually there should be no write access possible at that level and obviously these actions/settings files do not exist there.

Indeed I was assuming that fiji was always opened from its executable location, thus a folder plugins should exist. @tischi do you know if there's a way to catch the location of the the current fiji's executable location ?

I just tried with a brand new Fiji, new PLuigin installation (BDV-PG only) and starting from the Fiji directory.

Still no response when changing a projection mode.

Ah, no sorry, indeed it's not related to the projection mode, it's related to what you get in contextual menus (right click on bdv or on the tree view). I lost the purpose of the thread, but there was no way this could be related to your issue.

@tischi I think there's a test for mode changing. maybe it's worth looking at if it's working or not

Still no response when changing a projection mode.

Just tried it and for me, it works.

@NicoKiaru Do you mind merging this PR to clean up the code there a bit? #207

I can't merge it because it's in draft mode. But you can do it after changing the state of your PR

I don't know.

Here it works for not a single one of my sources. (Could it be windows-specific?)

data is uploaded

Could you try without adding to the occluding layer? I had a bug there at some point.

The first change was simply from sum to average, also nothing changes...

Just to be sure @martinschorb : when you create the bdv widow, so you select Mixed projector ?

Yes, that's the problem.

Does BDV-PG not remember the choice of the BDV window that I open?
I always do mixed, so I hardly ever checked.

Could it be that when I open BigWarp or do the manual transformation, this will enforce the Sum projector and then afterwards the new BDV windows will behave like that?

Also, when I select sources and show them without a BDV window open, it will generate one without asking for Projection Mode.

Could simply all BDV windows be mixed? Is there anything that would speak against that?

I don't think we can control the projector within BigWarp, this can't be changed. Also the output sources from bigwarp (the transformed ones), are set to sum by default, yes, but this could be changed.

Is your issue BigWarp specific ?

No, this happens in the "regular" BDV window.

In fact it is the manual transformation.

#208

This will change the BDV window to Sum and then disables the possibility to change projection modes.

Also, when I close the BDV window and then show sources without explicitely asking for a "mixed" BDV, it will give me a sum window.
Very annoying...

Also, when I close the BDV window and then show sources without explicitely asking for a "mixed" BDV, it will give me a sum window.

That was the other suggestion I had, maybe in the beginning you did not ask for the mixed projector?
Does it work if you use the mixed projector?

Personally, I would vote for making the mixed projector the default.

Personally, I would vote for making the mixed projector the default.

Full support!

However, some actions apparently can trigger the BDV window to change.

From what I remember, RGB images appear black in the mixed projector, so I can't use it + I don't know how to override the projector in BigWarp + for warped sources we had issues in dealing with the 'transparency', using a weird hack.

But let's try to understand first the cause of the issue, because I'm quite confused regarding what the real issue is right now: is it the mixed projector which is not enabled ? is it the sources 'projector' metadata which are forgotten in some operation ?

This needs to work well in all cases if you want to have this as the default.

Step by step:

2 sources with projection mode average
show them in a new BDV window Mixed
initiate manual transform for one of the sources
the open BDV window will become Sum and so do all the shown sources.

close that BDV window
show the sources (it is not explicitely asking for Mixed)
the new BDV window is Sum only again

Mutate or Append or both ? I'm surprised mutate would cause the trouble

I noticed with Append.

Now obsolete because the projection mode is not not handled in this repo anymore.

Closing for now, but please reopen if you prefer