Streaming analysis for Xenon experiments
Strax is an analysis framework for pulse-only digitization data, specialized for live data reduction at speeds of 50-100 MB(raw) / core / sec.
For comparison, this is more than 100x faster than the XENON1T processor pax, and does not require a preprocessing stage ('eventbuilder'). It achieves this due to using numpy structured arrays internally, which are supported by the amazing just-in-time compiler numba.
Features:
- Start from unordered streams of pulses (like pax's trigger)
- Output to files or MongoDB
- Plugin system for extensibility
- Each plugin produces a dataframe
- Dependencies and configuration tracked explicitly
- Limited "Event class" emulation for code that needs it
- Processing algorithms: hitfinding, sum waveform, clustering, classification, event building
Strax is initially developed for the XENONnT experiment. However, the configuration and specific algorithms for XENONnT will ultimately be hosted into a separate repository.
Documentation is under construction. For the moment, you might find these useful:
- Tutorial notebook
- Introductory talk (aimed at XENON1T analysis/DAQ experts)
- Function reference (TODO readthedocs)
To install the latest stable version (from pypi), run pip install strax.
Dependencies should install automatically:
numpy, pandas, numba, two compression libraries (blosc and zstd)
and a few miscellaneous pure-python packages.
To install extra extensions, do e.g. pip install -e .[mongo] or for all extensions pip install -e .[complete]
You can also clone the repository, then setup a developer installation with python setup.py develop.
If you experience problems during installation, try installing
exactly the same version of the dependencies as used on the Travis build test server.
Clone the repository, then do pip install -r requirements.txt.
The provided demonstration notebooks require test data that is not included in the repository. Eventually we will provide simulated data for this purpose. For now, XENON collaboration members can find test data at:
- Processed only (for strax demo notebook)
- Raw (for fake_daq.py and eb.py) at midway:
/scratch/midway2/aalbers/test_input_data.zip
To use these, unzip them in the same directory as the notebooks. The 'processed' zipfile will just make one directory with a single zipfile inside.