hms-dbmi/viv

Visualizations that use zoom-level precisely are quite off

Opened this issue · 10 comments

ilan-gold commented

Describe the bug
The scale bar is jumpy on the Avivator site (but somehow not on Vitessce). I imaging there is some sort of state management issue. In any case we should be rendering the scale bar in its own view anyway as suggested here

To Reproduce
avivator demo with a scale bar shows jumpiness

Expected behavior
The scale bar should be fixed like vitessce

Environment:

  • Release or git hash:
  • Browser: FF
  • Browser version: 127.0.1
ilan-gold commented

Ok still getting my web development feet wet again. This only happens on FF (of course).

ilan-gold commented

This happens also on google chrome by using the scale bar. So I would venture a guess it has something to do with bounding box calculations.

ilan-gold commented

Ok this also happens with the side-by-side viewer where one side will get thrown off. So I think there is something going on with the zoom level.

ilan-gold commented

It looks like this actually happens all the time, but is only noticed by things that use zoom state explicitly. These numbers should be monotonic
Screenshot 2024-06-28 at 12 35 21

ilan-gold commented

@xinaesthete See here for more discussion. I have a suspicion this is related to visgl/deck.gl#8989

xinaesthete commented

I think I might have another look into this in combination with updating React / MUI / zustand etc as I still think these things may be somewhat related. In MDV we don't have the janky scale-bar issue and can also load views with appropriate zoom, as well as linking viewState between charts. The latter gets confused if the images have different physical pixel size, but apart from that it seems to work reasonably well and most of the relevant code is based on refactored version of Avivator...

There are some warnings from Zustand currently which it'd be good to get rid of and I'm not sure it's in-scope for the other PR, but it'd be nice to get a new release with all this somewhat cleaned up relatively soon... so I think I'll crack on with that now.

xinaesthete commented

I've just been looking at an image that was encoded in a not terribly useful way - it has a few hundred channels which should really be z-slices etc, and no physical size metadata for the pixels... that last point seems to be helping viv keep the side-by-side views in sync perfectly where others go badly wrong.

ilan-gold commented 
import React, { useState, useEffect } from 'react';
import ReactDOM from 'react-dom/client';
import {
  getChannelStats,
  loadOmeTiff,
  MultiscaleImageLayer,
  ImageLayer
} from '@hms-dbmi/viv';
import DeckGL from '@deck.gl/react';
import { OrthographicView } from '@deck.gl/core'
const url = 'https://viv-demo.storage.googleapis.com/Vanderbilt-Spraggins-Kidney-MxIF.ome.tif'; // OME-TIFF

const useImageLayer = true

// Hardcoded rendering properties.
const props = {
  selections: [
    { z: 0, t: 0, c: 0 },
    { z: 0, t: 0, c: 1 },
    { z: 0, t: 0, c: 2 },
  ],
  colors: [
    [0, 0, 255],
    [0, 255, 0],
    [255, 0, 0],
  ],
  contrastLimits: [
    [0, 10005],
    [0, 10005],
    [0, 10005],
  ],
  channelsVisible: [true, true, true],
}
const INITIAL_VIEW_STATE = useImageLayer ? { target: [100, 100, 0], zoom: -1 } : { target: [10000, 10000, 0], zoom: -7 }

function App() {
  const [loader, setLoader] = useState(null);
  const [viewState, setViewState] = useState(INITIAL_VIEW_STATE);
  const [autoProps, setAutoProps] = useState(null);
  useEffect(() => {
    loadOmeTiff(url).then(setLoader);
  }, []);

  // Viv exposes the getChannelStats to produce nice initial settings
  // so that users can have an "in focus" image immediately.

  async function computeProps(loader) {
    if (!loader) return null;
    // Use lowest level of the image pyramid for calculating stats.
    const source = loader.data[loader.data.length - 1];
    const stats = await Promise.all(props.selections.map(async selection => {
      const raster = await source.getRaster({ selection });
      return getChannelStats(raster.data);
    }));
    // These are calculated bounds for the contrastLimits
    // that could be used for display purposes.
    // domains = stats.map(stat => stat.domain);

    // These are precalculated settings for the contrastLimits that
    // should render a good, "in focus" image initially.
    const contrastLimits = stats.map(stat => stat.contrastLimits);
    const newProps = { ...props, contrastLimits };
    return newProps
  }

  useEffect(() => {

    computeProps(loader).then(setAutoProps)

  }, [loader])

  if (!loader || !autoProps) return null;
  const layer = loader && new (useImageLayer ? ImageLayer : MultiscaleImageLayer)({
    id: 'layer',
    loader: useImageLayer ? loader.data.slice(-1)[0] : loader.data,
    contrastLimits: props.contrastLimits,
      // Default extension is ColorPaletteExtension so no need to specify it if
      // that is the desired rendering, using the `colors` prop.
    colors: props.colors,
    channelsVisible: props.channelsVisible,
    selections: props.selections
  });

  return (
    <DeckGL
      layers={[layer]}
      views={[new OrthographicView({ id: 'ortho', controller: true })]}
      viewState={viewState}
      onViewStateChange={arg => setViewState(arg.viewState) && arg}
    />
  );
}
ReactDOM.createRoot(document.getElementById('root')).render(<App />);

is a reproducer without VivViewer and the same thing is still happening. Separately, I don't know why setViewState needs to be used like this but ok.

ilan-gold commented

Some things ruled out:

  1. VivViewer or related react-based mechanisms (so zustand included)
  2. ScaleBarLayer even though it's very janky
  3. Removing signal appears to have no effect so it's not the signal aborting mechanism
  4. This happens both with MultiscaleImageLayer and ImageLayer so it's probably XRLayer specific
  5. Related to the above, this isn't a data loading issue then