This Pack enables a variety of functions when LogStream is used to receive data from Syslog senders.
The pack provides the following benefits:
- Provides a pipeline for use as an Input Pre-Conditioning pipeline
- Volume reduction by removing redundant information, such as the human-readable timestamp.
- Timezone normalization, when senders do not include timezone information
- Lookup-based enrichment to set additional meta-information for a given sender. Examples include index, sourcetype, and time zone.
The pack also includes a sample log for use when evaluating which features to enable/disable for a given customer deployment.
This Pack's pipeline is intended for use with general-purpose ingest of data from Syslog senders, by way of an Input Conditioning pipeline. It is used when the sender is using the Syslog protocol to send to LogStream, as opposed to data in a syslog-compliant format being delivered via another delivery method.
Input conditioning pipelines are particularly suited to general actions that one always wants to take for a given input source. They generally should not be used for specific data sets; pipeline manipulation for a given data set should be achieved within a pipeline specifically tailored to that data set.
For example, to process all Syslog data from all Syslog senders on ports 514, and XYZ, use an input conditioning pipeline to normalize processing of Severity and Facility information, and for basic volume reduction.
When processing a specific data set, such as Palo Alto firewall data sent via Syslog, the input conditioning pipeline is used in combination with the normal routes+packs (or routes+pipelines) approaches. The Palo Alto pack would do any additional processing specific to that dataset.
Most Cribl LogStream Packs provide a collection of routes and pipelines that are used after the Pre-Processing phase. This pack is different, in that it provides a Pre-Processing pipeline.
To use this Pack happens in these main phases:
Understanding the pack
- Open the Pack's CriblSyslogPreProcessing pipeline
- Select a saved sample, and review IN and OUT versions
- Read the pipeline's comments and enable/disable functions as necessary for the deployment environment.
- Optional: To capture data from a deployed production environment with Syslog sources, use Sample Data -> Capture New with a filter of
__inputId.startsWith('syslog:') - Proceed to the next phase when the steps above are complete.
Editing the Knowledge Object
- Within the pack, select Knowledge and edit
SyslogLookup.csv - Add host information from the deployment environment.
- Optionally, remove lines 4,5,6
- Click Save
Tying the pipeline to sources
- In the Worker Group's Sources page, select a Syslog source where PreProcessing is desired. (Probably, most of them.)
- In Pre-Processing, select "[Pack] cribl-syslog-input
- Repeat steps 1 and 2 for additional syslog sources as needed.
Initial release of the Cribl Syslog Pre-processing pack.
To contribute to the Pack, please connect with Michael Donnelly on the Cribl Community Slack. You can suggest new features or offer to collaborate.
This Pack uses the following license: Apache 2.0.