google/neuroglancer

Visualize tractography-generated streamlines in Neuroglancer

Opened this issue · 9 comments

kabilar commented

Use case

We are looking to visualize tractography-generated streamlines from a Track File (.trk) in Neuroglancer. See Track File docs and an example Track File. We have a current implementation (see Current implementation section below), but are looking for advice on the optimal implementation (see Open questions section below). Thank you.

Requirements

  1. Add support in Neuroglancer to natively read and visualize a public Track File (.trk). (To prevent data duplication, we would prefer not to save into another format for visualization.)
    1. Transform the streamlines from voxel space to world space.
  2. To improve rendering performance, add front-end component(s) to filter the streamlines.
    1. Add a slider to subselect a random number of streamlines (with a fixed seed for reproducible states).
    2. Add a slider to subselect the number of points per streamline.
  3. Add a front-end component to optionally display a colormap overlay on the streamlines. The colormap reflects the orientations of the streamlines at each vertex.

Implementation details

  1. Track File (.trk) documentation
  2. Track Files store streamlines in voxel space. The vox_to_ras matrix is stored in the header and can convert the points from voxel space to world space.
  3. A publicly available trk file can be found in Dandiset 000724. (Files can be much larger than this example.)
  4. The nibabel Python library can read trk files.

Current implementation

My colleague (@balbasty) has created the ngtools Python package with the accompanying example notebook. The tract.py module performs the following operations:

  1. Reads the streamlines into memory.
  2. Selects a subset of streamlines (to improve performance).
  3. For the subset of streamlines, creates concatenated arrays of the vertices, edges, and orientations. (Orientations are used for a colormap overlay.)
  4. Converts the streamlines into a single, local skeleton layer by passing the above arrays into neuroglancer.skeleton.Skeleton(). A previous version of this package converted the streamlines into a precomputed skeleton format.

After discussion with @balbasty, a downside to this approach is that a user must locally load all the streamlines into memory and perform this (minor) compute operation to convert the streamlines into a format that can be visualized. Thus the state of the Neuroglancer viewer with a Track File cannot readily be passed between users.

Open questions

  1. Is there a reference implementation for visualizing tractography-generated streamlines (from any file format) in Neuroglancer?
  2. If there is not a reference implementation, what is the optimal way of displaying streamlines (Requirement 1)?
  3. Would we need to display only a subset of streamlines to improve rendering performance at multiple scales (Requirement 2)?

cc @balbasty @aaronkanzer @MikeSchutzman @satra @ayendiki

fcollman commented

To make this work as you desire you would need to add trk files as a new kind of data source to neuroglancer, which would mean writing a typescript module capable of reading trk files that conforms to the data source api.

this would then let you add layers with sources like trk://https://myseerver.com/data

relying on python means you need to provide a server with the data eliminating as you point out the share ability of the solution.

  • 👍1
fcollman commented

Neuroglancer is used to display billions of objects, usually this is done by having a concept of segment IDs that represent individual objects and then ways of selecting subsets of them. I would suggest for very large numbers of streamlines that you make each line segment ID and attach sufficient metadata to them to make selecting the ones you want easy.

there is already one format that implants such a metadata loading system with tags and numerical properties and labels, that scales to probably the hundreds of thousands of segment ids. Doing this for more than that would likely require a different approach.

  • 👍1
ayendiki commented

Hundreds of thousands would work, thanks @fcollman ! Even when a dataset has millions of streamlines, we typically display < 1% of them at a time, otherwise the display becomes too cluttered.

  • 👍1
kabilar commented

Thank you, @fcollman. This is very helpful.

aaronkanzer commented

there is already one format that implants such a metadata loading system with tags and numerical properties and labels, that scales to probably the hundreds of thousands of segment ids. Doing this for more than that would likely require a different approach.

Thanks as well, @fcollman -- just had one follow-up question for clarification -- when you say "there is already one format that implants such a metadata loading system" -- what format are you referencing?

fcollman commented
balbasty commented

Thanks for the pointers @fcollman!

Our current implementation does use the precomputed skeleton format. Our implementation can choose between:

  1. Converting our file format to a precomputed skeleton "virtually" and "on the fly" using a local fileserver. This allows our file to be displayed in any instance of neuroglancer, but we need to run a fileserver.
  2. Converting our file format to a local Skeleton class and load it using the python API. This forces us to run our own instance of neuroglancer.

We did give a different ID to each streamline in a first instance, but this made the display way too slow (even on a local instance accessing a local file), since each streamline gets queried by its own GET request. This is why we ended up fusing all the streamlines (or rather, all the displayed streamlines) into a single skeleton.

Another issue with the skeleton format (I think) is that our streamlines typically range very long distances, so whatever the current field of view, there is allways 1-10k streamlines passing through it, and all of the must be queried and displayed. This is very different from the skeletons of cells in EM volumes, which seem to be much more localized. One solution could be to chunk our streamlines into smaller segments so that they don't have to be loaded entirely. But it doesn't solve anything at coarse resolutions, where all segments must be shown.

This is why we feel that having builtin routines for streamlines could be beneficial. And we think that our user would appreciate the ability to change the density of displayed streamlines at different resolutions.

fcollman commented
jbms commented

As Forrest mentioned, to support this format natively in Neuroglancer you would need to implement a Neuroglancer datasource for it in JavaScript/TypeScript. Additionally, though, the datasource implementation is responsible for presenting the loaded data source as a collection of one or more "subsources" of various types --- the current subsource types include volumetric data, single and multi-resolution object meshes, segment properties, segment agglomeration graph, skeletons, annotations, and (not commonly used) individual meshes.

Of the existing subsource types, the only plausible ones are skeletons and annotations, plus segment properties could be used to indicate per-track properties if each track is represented as a separate segment id.

The annotation source type has the advantage of already supporting spatial indexing, unlike skeleton sources, and it also already does a form of subsampling when using the spatial index. You could see if it could be made to work. (Skeleton sources might ideally be unified with annotation sources in the future, and in general you can do everything with an annotation source that you can do with a skeleton source, except that a skeleton source can be displayed by a segmentation layer along with volumetric segmentation data, while an annotation source currently can only be displayed on an annotation layer.)

To represent the tracts as an annotation source, each pair of adjacent points would be added as a line annotation. Any per-point properties would be included, and any per-tract properties would also be duplicated to every line annotation, if you wish for those properties to be usable for rendering. You could also specify as a "related segment" the tract id.

When using the spatial index, subsampling is automatic. There is a UI control that determines the amount of subsampling. The subsampling is deterministic and basically is just selecting a stopping point along a particular ordering of the annotations. You can influence the subsampling by choosing the ordering when generating the spatial index, e.g. to approximately ensure that entire tracts are sampled together. However the annotation layer would not currently provide a good way to subsampling the points within a tract.

In principle a new source type could also be defined --- then it would also be necessary to implement rendering support for it and any necessary UI interactions. This would be rather costly both in terms of initial implementation effort and ongoing maintenance, though, so it would be much better to add any necessary functionality to annotation sources instead.