Stroom Content
Stroom Content is a repository of the content definition files used to import entities into Stroom. Entities are such things as XML Schemas, XSLT translations, data splitter definitions, pipelines, dashboards, etc. This repository provides a means to share common Stroom content in a form that can be imported into multiple instances of Stroom.
The content is grouped into a number of content packs which can each be built into a zip file that is ready for import directly into Stroom.
If you are looking for visualisation content packs then see stroom-visualisations-dev.
Caution!
Errors/problems will arise if you import an incompatible content pack into your version of Stroom.
If you wish to build a previous version of a particular content pack, you will need to git checkout the appropriately
named git tag.
Content pack contents
The following content packs are currently available. Click on the link for details of what is in each content pack. In each content pack section is a table detailing the different versions of the pack along with details of which versions of Stroom it is compatible with.
core-xml-schemas
The core XML Schemas required by Stroom for basic operation.
Compatibility matrix
| Pack version | Stroom 5.0.x | Stroom 6.0.x | Stroom 7.0.x |
|---|---|---|---|
| v2.2 | Y | Y | Y |
| v2.1 | Y | Y | Y |
| v2.0 | Y | Y | Y |
| v1.1 | Y | Y | Y |
| v1.0 | Y | Y | Y |
event-logging-xml-schema
An XML Schema standard for describing audit events.
Compatibility matrix
| Pack version | Stroom 5.0.x | Stroom 6.0.x | Stroom 7.0.x |
|---|---|---|---|
| v3.4.2 | Y | Y | Y |
| v3.2.3 | Y | Y | Y |
| v3.1.2 | Y | Y | Y |
| v3.1.1 | Y | Y | Y |
| v1.0 | Y | Y | Y |
internal-dashboards
A set of Dashboard entities for displaying various metrics about the state of the Stroom application and the undelying hardware and file systems.
Compatibility matrix
| Pack version | Stroom 5.0.x | Stroom 6.0.x | Stroom 7.0.x |
|---|---|---|---|
| v1.1 | Y | Y | Y |
| v1.0 | Y | Y | Y |
internal-statistics-sql
A set of StatisticStore entities for representing the internal statistics generated by Stroom and record by the SQL Statistics service. This pack was previously known as 'internal-statistics' so the versions in the matrix below refer to either name.
Compatibility matrix
| Pack version | Stroom 5.0.x | Stroom 6.0.x | Stroom 7.0.x |
|---|---|---|---|
| v2.2 | No | Y | Y |
| v2.1 | No | Y | Y |
| v2.0 | No | Y | Y |
| v1.2 | Y | No | No |
| v1.1 | Y | No | No |
| v1.0 | Y | No | No |
internal-statistics-stroom-stats
A set of StroomStatsStore entities for representing the internal statistics generated by Stroom and recorded by the Stroom-Stats service.
Compatibility matrix
| Pack version | Stroom 5.0.x | Stroom 6.0.x | Stroom 7.0.x |
|---|---|---|---|
| v2.2 | No | Y | Y |
| v2.1 | No | Y | Y |
| v2.0 | No | Y | Y |
standard-pipelines
A set of standard data format agnostic pipelines for processing data.
| Pack version | Stroom 5.0.x | Stroom 6.0.x | Stroom 7.0.x |
|---|---|---|---|
| v0.4 | No | Y | Y |
| v0.3 | No | Y | Y |
| v0.2 | No | Y | Y |
| v0.1 | No | Y | Y |
stroom-101
The entities used in the quick start guide
Compatibility matrix
| Pack version | Stroom 5.0.x | Stroom 6.0.x | Stroom 7.0.x |
|---|---|---|---|
| v1.0 | Y | Y | Y |
stroom-logs
Compatibility matrix
| Pack version | Stroom 5.0.x | Stroom 6.0.x | Stroom 7.0.x |
|---|---|---|---|
| v3.0-beta.1 | No | No | Y |
| v2.0-alpha.5 | No | Y | Y |
| v2.0-alpha.4 | No | Y | Y |
| v2.0-alpha.3 | No | Y | Y |
| v2.0-alpha.2 | No | Y | Y |
| v2.0-alpha.1 | No | Y | Y |
| v1.1 | No | Y | Y |
| v1.0 | No | Y | Y |
template-pipelines
A set of template pipelines that other pipelines can inherit from.
| Pack version | Stroom 5.0.x | Stroom 6.0.x | Stroom 7.0.x | Stroom 7.1.x | Stroom 7.2.x |
|---|---|---|---|---|---|
| v0.5 | No | No | No | No* | Y |
| v0.4.1 | No | No | No | No* | Y |
| v0.3 | No | No | Y | Y | Y |
| v0.2 | No | Y | Y | Y | Y |
| v0.1 | No | Y | Y | Y | Y |
* Stroom v7.1-beta.3 onwards is compatible with this content pack, but it must not be used in v7.1-beta.1 or v7.1-beta.2
example-index
A basic Stroom index designed for event-logging XML.
| Pack version | Stroom 5.0.x | Stroom 6.0.x | Stroom 7.0.x |
|---|---|---|---|
| v1.0 | No | No | Y |
Building the content packs
Each content pack is defined as a directory within stroom-content-source with the name of content pack being the name of the directory.
There are currently two methods to building content packs, a Gradle build or a python script. The former has superseded the latter, though the python script is still useful if you want all packs built into a single zip file. For all other purposes the Gradle build should be favoured.
Gradle build
The Gradle build will manage the dependencies between content packs, including transitive dependencies, so if you want a pack that has dependencies on other packs then those dependency packs will get built with it. Running the build requires xmllint and python.
To run a full build of all packs do:
./gradlew clean build
If you just want to validate the pack's source to make sure the UUID references are correct and there are no clashes, then do:
./gradlew validate
To build a single pack do something like:
./gradlew clean :my-pack-name:build
The build will place the following pairs of zip files in ./build/distributions:
-
my-pack-name.zip -
my-pack-name-all.zip
The -all variant contains all dependency packs and is therefore larger.
The other zip just contains the named pack with no dependencies.
Released content packs
To see all the pre-built content packs see the releases page.
Importing the content packs
The content pack zip files can be imported into Stroom by selecting Import from the Tools menu.
Content pack development
The Stroom entities defined within stroom-content-source/ are all serialised forms, with XML as the serialisation format. Some entities consist of two files, one for the core entity and one for any significant payload, e.g. XSLT, JavaScript, JSON, etc. It is not recommended to create these entities by hand. The easiest way to create new content or to amend existing content is from within Stroom and then use the Export function to export the content to its serialised form for inclusion in stroom-content.
IMPORTANT - A word about UUIDs
Each Stroom entity is identified by a Universally Unique Identifier. Where entities have dependencies on other entities, e.g. a Dashboard entity depends on a StatisticStore entity, this dependency relationship is defined by the UUID of the dependency entity. It is possible to import content into Stroom with missing dependencies as this can sometimes be a valid use case, though with missing dependencies the entities will not work correctly until the dependencies are resolved, e.g. by updating the dependency.
All the entities defined in these content packs are defined with hard coded UUIDs and dependencies. It is therefore critical that the correct UUIDs are used in the entity and dependency definitions in the XML.
It is likely that a content pack will contain entities that depend on entities in other content packs. This is so that we can have some common entities, e.g. template pipelines, that can be used by multiple packs, with the intention that only those packs that are needed are imported.
Duplication between packs
It is possible for multiple packs to include the Folder entities with the same path structure, however they must have the same UUID. Duplication of other entity types should be avoided.
Creating new content
New content should be crafted in Stroom, ideally an empty development instance of Stroom. If the new content relies on any entities from other content packs then these should first be imported into Stroom so that the dependencies can be correctly defined. The new content should be created in the chosen folder structure. Once the content has been created, use the Export function to export the folder(s) that contains it.
The Export function will create a zip file containing all the serialised entities complete with the original path structure. You should create a directory under stroom-content-source/ with the name being the name of the content-pack. The content of the zip file should be unzipped into this directory.
Modifying existing content
Minor changes to existing content pack can be made by editing the XML files directly, though testing the pack in an instance of Stroom is advised to ensure the change works.
More complex changes should be done by importing the pack to be changed and any dependencies into a vanilla instance of stroom, making the changes in Stroom and then exporting the packs out.
The exported pack should be unzipped into the pack's stroomContent directory for addition into source control.
It is advisable to remove any of the audit trail elements (createUser, createTime,updateUser & updateTime) that Stroom adds in to the XML file as these are not required in content packs and may result in warnings on import if left in.
Gradle dependencies
The dependencies between content packs are also defined in the Gradle build.gradle file in the root of each content pack directory.
The createSkeletonPack.sh script will create a skeleton build.gradle file for you with an example of how to define dependencies on other packs.
A pack should define all direct dependencies it has in its entities and not rely on transitive dependencies as this is more explicit.
Helper scripts
The following helper shell scripts exist to assist in content pack creation.
createSkeletonPack.sh
This script will create a skeleton structure for a new content pack.
./createSkeletonPack.sh my-new-pack
createNewStroomFolder.sh
This script can be run from within an directory to create a new Stroom Folder (XML definition file and directory). This should only be used to create a directory that doesn't already exist in Stroom as it will have a new UUID generated for it.
removeAuditElements.py
This script (requires python3) will strip the audit elements (e.g. <createUser>...</createUser>) from all xml files.
This is to make it easier to clean any content generated in stroom and exported out for inclusion in a content pack.