Azimuth, an open-source dataset and error analysis tool for text classification, with love from ServiceNow.
Azimuth is an open source application that helps AI practitioners and data scientists better understand their dataset and model predictions by performing thorough dataset and error analyses. The application leverages different tools, including robustness tests, semantic similarity analysis and saliency maps, unified by concepts such as smart tags and proposed actions. It currently only supports text classification use cases.
Play with Azimuth on CLINC-OOS and a DistilBert model with a live demo.
Published Paper at EMNLP
Our paper was accepted as a system demo at EMNLP 2022. It is available here.
Our documentation contains all the instructions and references to use Azimuth.
- Getting Started contains all the instructions to install and launch the app.
- Key Concepts explains the different concepts and analyses that are provided in Azimuth to perform dataset and error analysis.
- User Guide goes screen per screen to explain the different interactions and visualizations available.
- Reference details the config file and the different contracts which allow configuring Azimuth with different datasets and pipelines.
- Development guides on how to develop and contribute to the repo.
After installing the requirements, as explained in Installation, run the following command to run the demo. More details in Learn Basics.
pip install gdown
make download_demo
make CFG_PATH=/config/development/clinc/conf.json launch
Once the startup tasks are completed, you will be able to access Azimuth at http://localhost:8080.
To learn more about Azimuth's features, refer to
our Key Concepts and
our User Guide.
To use your own dataset and models, please refer to Run on Your Use Case.
Refer to our CONTRIBUTING.md.
At different places in the code, you'll find README files explaining further how the back end works. Don't hesitate to
add other README files in appropriate places, and don't forget to edit them if you change these components.
- In azimuth, explaining the main back-end components.
- In azimuth/modules, explaining how modules work.
- In azimuth/routers, explaining how to add new routes.
├── azimuth # Back End
│ ├── modules
│ │ └── Where core capabilities are implemented
│ ├── plots
│ │ └── Plotting functions
│ ├── routers
│ │ └── FastAPI routers
│ ├── types
│ │ └── Specific pydantic types that this application is based on
│ ├── utils
│ │ └── Extra utilities
├── azimuth_shr
│ └── Custom code that is shareable.
├── config
│ └── Config files for different data/models
├── docs
│ └── User documentation
├── tests
│ └── Unit/integration tests
├── webapp
│ └── Front End
├── docker-compose.yml # Where the config and images are specified.
├── docker-compose-gpu.yml # Extension to use Azimuth with gpus.
└── runner.py # Application entrypoint.To contact us, join our slack or send us an email!
The package is licensed by ServiceNow, Inc. under the Apache 2.0 license. See LICENSE for more details.
Please notify psirt-oss@servicenow.com regarding any vulnerability reports in addition to following current reporting procedure.