Glean Dictionary

The Glean dictionary aims to provide a comprehensive index of datasets generated inside Mozilla for applications built using the Glean SDKs.

This project is under active development. For up to date information on project structure and governance, see:

https://wiki.mozilla.org/Data/WorkingGroups/GleanDictionary

The production version of the Glean Dictionary is deployed at:

https://dictionary.telemetry.mozilla.org

Getting Started

You should be able to create your own local copy of the dictionary so long as you have Python (version 3.8+) and node.js (version 12+) installed. You will also need npm v7 or greater: run npm install -g npm@latest if you need to upgrade.

Assuming those requirements are met, follow these instructions:

# Create and activate a python virtual environment.
python3 -m venv venv/
venv/bin/pip install -r requirements.txt

# Build data needed by dashboard
./scripts/gd build-metadata

# Install npm dependencies and start a local
# instance of the GUI
npm install
npm run dev

If that worked, you should be able to see a local version of Glean at http://localhost:5000

You can speed up the "build data" step by appending the name of a set of application(s) you want to build metadata for. This can speed up the process considerably. For example, to build a metadata index for Fenix (Firefox for Android) only, try:

./scripts/gd build-metadata fenix

Search Service

The Glean Dictionary also includes a search service which enables searching through active metrics. Under the hood, this service is implemented with netlify functions. For example:

https://dictionary.telemetry.mozilla.org/api/v1/metrics_search_burnham?search=techno

You can start it up via the netlify command line interface (assuming you have it installed):

netlify dev

If you have generated metadata as described above, you should then be able to test the search functions locally:

http://localhost:8888/api/v1/metrics_search_burnham?search=techno http://localhost:8888/api/v1/metrics_search_firefox_legacy?search=ms

Storybook

We use Storybook for developing and validating Svelte components used throughout the app. To view the existing list of stories, run:

npm run storybook

Storybook Snapshot Testing

To give us more confidence that changes don't unintentionally break the UI, we run storybook snapshot tests.

You can run them manually as follows:

npm run test:jest

If you intentionally made a change to a component that results in a change to the output of the storybook snapshots, you can re-generate them using the following command:

npm run test:jest -- -u

End-to-End Testing

We use Playwright for our end-to-end tests.

Before testing, download the supported browsers needed for Playwright to execute successfully by running:

npx playwright install

To run the end-to-end tests along with other tests:

npm run test

To run only the Playwright tests:

npx playwright test

ETL Testing

The transforms used by the Glean Dictionary have their own tests. Assuming you've run the set up as described above, you can run these tests by executing:

venv/bin/pytest

Glean Debugging

In order to enable ping logging set the GLEAN_LOG_PINGS environment variable.

GLEAN_LOG_PINGS=true npm run dev

In order to send Glean pings to the debug viewer set the GLEAN_DEBUG_VIEW_TAG environment variable.

GLEAN_DEBUG_VIEW_TAG=my-tag npm run dev

Deployment

The production version of the Glean Dictionary (https://dictionary.telemetry.mozilla.org) is deployed from the production branch on this repository, which usually corresponds to the latest GitHub release. To update the Glean Dictionary to the latest version, follow this procedure:

Do a quick test of https://glean-dictionary-dev.netlify.app to make sure it's working as expected.
Create a new release, typically off of the main branch (use the auto-generated release notes, omitting dependency updates).
From a local checkout, update the production branch to be in sync with the tag you just created, then push to the production branch. After the integration tests pass, dictionary.telemetry.mozilla.org should be automatically updated to the latest version.

A version of the Glean Dictionary running the development branch (main) is accessible at https://glean-dictionary-dev.netlify.app/

Contributing

For more information on contributing, see CONTRIBUTING.md in the root of this repository.

Dexterp37/glean-dictionary