/cms-hlt-parser

Read, parse and provide CERN CMS High-Level Trigger and Luminosity information.

Primary LanguagePython

cms-hlt-parser

Read, parse and provide CMS High-Level Trigger and Luminosity information such as trigger menus, trigger paths, filter names, etc.

The tasks defined in this project are based on the luigi analysis workflow (law) package. For more information, see

Available tasks:

The setup and code in this project use pip3 and python3.

Setup

git clone https://github.com/riga/cms-hlt-parser
cd cms-hlt-parser

# a proxy is required for communicating with DAS and the HLT database
voms-proxy-init -voms cms

# always run the setup (which installs some software _once_)
source setup.sh

First example commands

Before diving into the various tasks listed below, you might want to start with some simple commands to get a feeling of what they do and how the results look like.

1. Lumi information of HLT paths

# command
> law run hltp.GetLumiData \
    --hlt-path 'HLT_IsoMu2?_v*' \
    --print-summary

# output
runs : 478
lumi : 59735.969 /pb
paths: 3
  - HLT_IsoMu27_v14
  - HLT_IsoMu27_v15
  - HLT_IsoMu27_v16

2. Get HLT menus found in a dataset

# command
> law run hltp.GetMenusFromDataset \
    --dataset /TTToSemiLeptonic_TuneCP5_13TeV-powheg-pythia8/RunIIAutumn18MiniAOD-102X_upgrade2018_realistic_v15-v1/MINIAODSIM \
    --print-summary

# output
menus: 1
  - /frozen/2018/2e34/v3.2/HLT/V1

3. Get HLT paths active in certain runs

# command
> law run hltp.GetPathsFromRuns \
    --run-numbers 315640,315641 \
    --print-summary

# output
+------------------------------------------+---------------+-----------------------+
| HLT menu                                 | Runs          | Matching HLT path(s)  |
+==========================================+===============+=======================+
| /cdaq/physics/Run2018/2e34/v1.2.2/HLT/V4 | 315640,315641 | HLT_AK4CaloJet100_v10 |
|                                          |               | HLT_AK4CaloJet120_v9  |
|                                          |               | HLT_AK4CaloJet30_v11  |
|                                          |               | HLT_AK4CaloJet40_v10  |
|                                          |               | HLT_AK4CaloJet50_v10  |
|                                          |               | HLT_AK4CaloJet80_v10  |
|                                          |               | HLT_AK4PFJet100_v17   |
|                                          |               | HLT_AK4PFJet120_v16   |
# ... and many more
# consider adding "--hlt-paths <csv_separated_patterns>" to filter them

Tasks

To execute a task, run law run hltp.<task_name> [parameters]. Please find a list of all available tasks, parameters and examples below.

Also, some task parameter have defaults that lookup values in the law.cfg config file.

The output file(s) of a task can be obtained by adding --print-status 0 or --print-output 0 to the respective law run command. They are mostly JSON files containing the described output information. Also, almost all tasks print a summary after they successfully run.

In case a task is already completed (i.e., its output exists), it is not run again and no summary is printed. To print only the summary, add --print-summary to the law run command.

GetDatasetLFNs

> law run hltp.GetDatasetLFNs [parameters]

Queries the DAS service to get the logical file names (LFNs) for a dataset (e.g. /TTToSemiLeptonic_TuneCP5_13TeV-powheg-pythia8/.../MINIAODSIM). The output is the list of LFNs.

Parameters:

  • --dataset: A dataset as shown in DAS. Patterns not supported.

Dependencies:

none

GetLumiData

> law run hltp.GetLumiData [parameters]

Uses brilcalc lumi to get luminosity data for a trigger path (name or pattern), given luminosity and normtag files. The output is a dictionary run -> {lumi, path}.

Parameters:

  • --hlt-path: A trigger path. Patterns allowed.
  • --lumi-file: A luminosity file. Defaults to the hltp_config.lumi_file config.
  • --normtag-file: A normtag file. Defaults to the hltp_config.normtag_file config.

Dependencies:

none

GetMenusFromDataset

> law run hltp.GetMenusFromDataset [parameters]

Returns the menus found in the n-th file of a dataset using the hltInfo command. The output is a list of trigger menus.

Parameters:

  • --dataset: A dataset as shown in DAS. Patterns not supported.
  • --file-index: The number of the file to query. Defaults to 0.

Dependencies:

flowchart LR
    GetMenusFromDataset --> GetDatasetLFNs
GetMenusInData

> law run hltp.GetMenusInData [parameters]

Uses brilcalc trg to obtain all trigger menus used for data-taking and maps them to the runs the menus they were used in. The output is a dictionary menu_name -> {menu_id, runs}.

Parameters:

  • --show-runs: When used, all run numbers are printed in the summary table instead of only their count. Defaults to false.

Dependencies:

none

GetPathsFromDataset

> law run hltp.GetPathsFromDataset [parameters]

Returns the triggers paths found in the n-th file of a dataset. Internally, this is done by opening the file with fwlite and reading the trigger names using the trigger bits. The output is a list of trigger paths.

Parameters:

  • --dataset: A dataset as shown in DAS. Patterns not supported.
  • --file-index: The number of the file to query. Defaults to 0.

Dependencies:

flowchart LR
    GetPathsFromDataset --> GetDatasetLFNs
GetPathsFromMenu

> law run hltp.GetPathsFromMenu [parameters]

Returns the triggers paths for a trigger menu hltConfFromDB. The output is a list of trigger paths.

Parameters:

  • --hlt-menu: The trigger menu to query. Patterns not supported.

Dependencies:

none

GetPathsFromRuns

> law run hltp.GetPathsFromRuns [parameters]

Returns the trigger paths for trigger menus which were active in certain runs, optionally filtered by a list of trigger path patterns. The output is a dictionary menu -> {runs, paths}. The columns of the summary table are "HLT menu", "Runs" and "Matching HLT path(s)".

Parameters:

  • --run-numbers: A list of run numbers to query. Can be inclusive ranges in the format "start-end". When empty, the run numbers from the luminosity file defined by --lumi-file are used.
  • --lumi-file: A luminosity file. Defaults to the hltp_config.lumi_file config.
  • --hlt-paths: Optional trigger paths to filter the paths obtained by querying a trigger menu (associated to a run). Patterns allowed.
  • --table-format: The tabulate table format. Defaults to "grid".

Example output:

Run paths

Dependencies:

flowchart LR
    GetPathsFromRuns --> GetMenusInData
    GetPathsFromRuns -- multiple --> GetPathsFromMenu
GetFilterNamesFromMenu

> law run hltp.GetFilterNamesFromMenu [parameters]

Gets the names of all EDFilter modules of a trigger path as defined in a trigger menu. Internally, hltConfigFromDB is used to obtain the configuration file of the path, the configuration is loaded, and the filter modules are identified programmatically. The output is a list of dictionaries {name, parameters}.

Parameters:

  • --hlt-path: The trigger path to query. Patterns not supported.
  • --hlt-menu: The trigger menu to query. Patterns not supported.

Dependencies:

none

GetFilterNamesFromRun

> law run hltp.GetFilterNamesFromRun [parameters]

Gets the names of all EDFilter modules of a trigger path used in a specific run. Internally, GetMenusInData is used to make the connection between trigger menu and run number, and GetFilterNamesFromMenu is invoked internally as a dynamic dependency. In fact, the output is identical to GetFilterNamesFromMenu, i.e., it is a list of dictionaries {name, parameters}.

Parameters:

  • --hlt-path: The trigger path to query. Patterns not supported.
  • --run-number: The run number to query.

Dependencies:

flowchart LR
    GetFilterNamesFromRun --> GetMenusInData
    GetFilterNamesFromRun --> GetFilterNamesFromMenu
GatherMCFilters

> law run hltp.GatherMCFilters [parameters]

Gathers information about the EDFilter names of several trigger paths used for MC production. The menus are obtained by querying the respective first files of some datasets. The output is a list of dictionaries {menu, datasets, paths}, where paths itself is a list of dictionaries {name, filters}. The summary consists of two tables. The first table shows which "HLT path(s)" belong to which "HLT menu" as found in which "Dataset(s)". The second table contains the actual filter names. The columns are "HLT path(s)", "HLT menu(s)" and "Filter names".

Parameters:

  • --datasets: The MC datasets to query. Patterns not supported. Defaults to the entries in the hltp_mc_datasets config.
  • --hlt-paths: The trigger paths to query. Patterns allowed. Defaults to the entries in the hltp_paths config.
  • --verbose-datasets: When used, the full dataset names are printed in the first summary table. Defaults to false.
  • --table-format: The tabulate table format. Defaults to "grid".

Example output:

MC filters

Dependencies:

flowchart LR
    GatherMCFilters -- multiple --> GetMenusFromDataset
    GetMenusFromDataset --> GetDatasetLFNs
    GatherMCFilters -- multiple --> GetPathsFromMenu
    GatherMCFilters -- multiple --> GetFilterNamesFromMenu
GatherDataFilters

> law run hltp.GatherDataFilters [parameters]

Gathers information about the EDFilter names of several trigger paths in several trigger menus employed for data-taking. Valid run numbers are taken from a luminosity file. The output is a list of dictionaries {menu, runs, paths}, where paths itself is a list of dictionaries {name, filters}. The summary table has the columns "HLT path(s)", "Runs", and "Filter names", and optionally "HLT menu(s)".

Parameters:

  • --hlt-menus: The trigger menus to query. Patterns allowed. Defaults to the entries in the hltp_data_menus config.
  • --hlt-paths: The trigger paths to query. Patterns allowed. Defaults to the entries in the hltp_paths config.
  • --lumi-file: A luminosity file. Defaults to the hltp_config.lumi_file config.
  • --show-menus: When used, the optional "HLT menu(s)" column is shown in the second summary table. Defaults to false.
  • --verbose-runs: When used, the run numbers in the second summary table are not contracted to ranges. Defaults to false.
  • --table-format: The tabulate table format. Defaults to "grid".

Example output:

Data filters

Dependencies:

flowchart LR
    GatherDataFilters --> GetMenusInData
    GatherDataFilters -- multiple --> GetPathsFromMenu
    GatherDataFilters -- multiple --> GetFilterNamesFromMenu