jpontdia/common-metrics

The metrics toolkit (formerly metrics accelerator/framework) is a Mule application intended to collect, aggregate and load platform metrics into different visualization systems; providing out of the box integrations and visualization options, including useful dashboards and charts.

common-metrics

The metrics toolkit is a Mule application intended to collect, aggregate and load platform metrics into different visualization systems; providing out of the box integrations and visualization options, including useful dashboards and charts.

Table of Contents

• Table of Contents
• Metrics Toolkit
  • Features
    • Modes
    • Loader options
  • Available Metrics
    • Platform Metrics
    • Platform Benefits
    • SDLC Metrics
  • Installation
    • Requirements
    • Steps
    • Properties configurations
      • SDLC metrics collectors properties configuration
    • Postman Collection
    • Splunk steps
      • Properties specific for Splunk
    • ELK steps
      • Properties specific for ELK
    • Tableau steps
    • MongoDB steps
      • Properties specific for MongoDB
    • SFDC-Einstein Analytics Steps
  • Considerations
  • Some Theory around the Toolkit
    • Business Needs
    • Data Sources
    • Measurements
    • Problem Space
    • Approach - Conceptual
    • Approach
      • Conceptual
      • Detailed
  • Final Notes

Metrics Toolkit

The metrics toolkit (formerly the metrics accelerator/framework) is a Mule application intended to collect, aggregate and load platform metrics into different visualization systems; providing out of the box integrations and visualization options, including useful dashboards and charts. In addition to the platform metrics, the toolkit also extends the capabilities to integrate with external applications like Jira, Confluence, Jenkins, Bitbucket and Splunk to gather SDLC metrics. This is an UNLICENSED software, please review the considerations. If you need assistance on extending this application, contact your MuleSoft Customer Success representative or MuleSoft Professional Services

Features

• Compact Mule application (1 single application)
• Provides more than 100 metrics from 3 complementary domains:
  • Platform Operational Metrics: collected and calculated automatically based on multiple products from Anypoint Platform: Exchange, Design Center, Runtime Manager, Access Management; covering metrics from applications deployed on-prem (Standalone), RTF and CloudHub.
  • Platform Benefits: require manual input to calculate final metrics, crossing information from the "Platform Operational" domain
  • External SDLC Metrics: collected and calculated automatically based on multiple external applications: Jira, Confluence, Jenkins, Bitbucket and Splunk;

Modes

• Poller (Push mode)

  • Collects, transforms and loads metrics in a defined visualization system
  • Configurable - frequency (cron expression and timezone ) and status (enabled/disabled).

• API to manage the asset:

  • API endpoint to obtain metrics on-demand (Pull mode)
  • API endpoint for triggering a specific loader to push data to a visualization system
  • API endpoint for loading platform benefits - manual input required

Loader options

• CSV
• JSON
• Plain Log: in case you forward logs to external systems (e.g using Splunk forwarder)
• Splunk: Including an out of the box dashboard with more than 100 charts
• ELK: Including out of the box, basic, Kibana dashboards
• Anypoint Monitoring: Requires Titanium subscription, dashboard is not provided
• Embedded dashboard: Including an out of the box basic embedded dashboard accessed by running the application offering an UI with a number of metrics obtained
• Tableau: including an out of the box dashboard with current consolidated platform metrics
• MongoDB
• SFDC: The Salesforce Loader will load data into Tableau CRM, formerly known as Einstein Analytics via the Salesforce Analytics connector. Dashboards are not provided, and it is assumed users are familiar with Tableau CRM, data recipes, and Analytics Studio.

NOTES:

• some adjustments in Metrics Toolkit implementation may be required if the loader does not work as expected for your specific scenario.
• By default, CSV loader output uses comma as the separator char. In order to change the CSV output format, modify the output options in loader-csv-build-structure.dwl and/or loader-csv-build-benefits-structure.dwl located under src/main/resources/dw/loader directory. For more information about CSV output formatting options, check MuleSoft's official documentation.

Available Metrics

Platform Metrics

Product	Metric	Dimensions	-
Access Management	Users Total	BG
Access Management	Active Users	BG
Access Management	Inactive Users	BG
Access Management	Active Users Last 60 days	BG
Access Management	Active Users Last 30 days	BG
Access Management	Environments Total	BG
Access Management	Environments Production Total	BG
Access Management	Environments Sandbox Total	BG
Access Management	API Client Applications Total	BG
Design Center	Assets Total	BG
Design Center	API Specs Total	BG
Design Center	Fragments Total	BG
Design Center	Flow Designer Apps Total	BG
Exchange	Assets Total	BG
Exchange	API Specs Total	BG
Exchange	Mule 3 Connectors Total	BG
Exchange	SOAP APIs Total	BG
Exchange	Fragments Total	BG
Exchange	HTTP Proxies Total	BG
Exchange	Policies Total	BG
Exchange	Extensions Total	BG
Exchange	Custom Assets Total	BG
Exchange	Overall Satisfaction	BG
Exchange	API Fragments reuse	BG
Exchange	API Spec reuse	BG
Exchange	Extensions reuse	BG
Exchange	API Specs managed in API Manager reuse	BG, Environment
Exchange	Policies applied in API Manager reuse	BG, Environment
API Manager	API Specs Managed Total	BG, Environment
API Manager	API Instances Total	BG, Environment
API Manager	API Instances Active Total	BG, Environment
API Manager	API Instances Inactive Total	BG, Environment
API Manager	API Instances Versions Total	BG, Environment
API Manager	API Instances With Policies Total	BG, Environment
API Manager	API Instances Without Policies Total	BG, Environment
API Manager	API Instances With Security Policies Total	BG, Environment
API Manager	API Instances Without Security Policies Total	BG, Environment
API Manager	API Instances With Contracts Total	BG, Environment
API Manager	API Instances Without Contracts Total	BG, Environment
API Manager	API Instances With More Than One Consumer Total	BG, Environment
API Manager	API Instances With One or More Consumers Total	BG, Environment
API Manager	API Contracts Total	BG, Environment
API Manager	Policies Used	BG, Environment
API Manager	Policies Used Total	BG, Environment
API Manager	Automated Policies Used	BG, Environment
API Manager	Automated Policies Used Total	BG, Environment
API Analytics	Transactions Last 30 days Total	BG, Environment
RuntimeManager - CloudHub - Networking	VPCs Total	BG
RuntimeManager - CloudHub - Networking	VPCs Available Total	BG
RuntimeManager - CloudHub - Networking	VPCs Used Total	BG
RuntimeManager - CloudHub - Networking	VPNs Total	BG
RuntimeManager - CloudHub - Networking	VPNs Available Total	BG
RuntimeManager - CloudHub - Networking	VPNs Used Total	BG
RuntimeManager - CloudHub - Networking	DLBs Total	BG
RuntimeManager - CloudHub - Networking	DLBs Available Total	BG
RuntimeManager - CloudHub - Networking	DLBs Used Total	BG
RuntimeManager - CloudHub - Networking	Static IPs Total	BG
RuntimeManager - CloudHub - Networking	Static IPs Available Total	BG
RuntimeManager - CloudHub - Networking	Static IPs Used Total	BG
RuntimeManager - CloudHub - Applications	vCores Total	BG, Environment
RuntimeManager - CloudHub - Applications	vCores Available Total	BG, Environment
RuntimeManager - CloudHub - Applications	vCores Used Total	BG, Environment
RuntimeManager - CloudHub - Applications	Applications Total	BG, Environment
RuntimeManager - CloudHub - Applications	Applications Started Total	BG, Environment
RuntimeManager - CloudHub - Applications	Applications Stopped Total	BG, Environment
RuntimeManager - CloudHub - Applications	Runtime Versions Used	BG, Environment
RuntimeManager - CloudHub - Applications	Runtime Versions Used Total	BG, Environment
RuntimeManager - RTF - Capacity	Fabrics Total	BG
RuntimeManager - RTF - Capacity	Workers Total	BG
RuntimeManager - RTF - Capacity	Controllers Total	BG
RuntimeManager - RTF - Capacity	Cores Allocated Total	BG
RuntimeManager - RTF - Capacity	Memory Allocated Total	BG
RuntimeManager - RTF - Capacity	Cores Allocated Per Fabric Average	BG
RuntimeManager - RTF - Capacity	Memory Allocated Per Fabric Average	BG
RuntimeManager - RTF - Applications	Cores Allocated Total	BG, Environment
RuntimeManager - RTF - Applications	Memory Allocated Total	BG, Environment
RuntimeManager - RTF - Applications	Applications Total	BG, Environment
RuntimeManager - RTF - Applications	Applications Started Total	BG, Environment
RuntimeManager - RTF - Applications	Applications Stopped Total	BG, Environment
RuntimeManager - RTF - Applications	Runtime Versions Used	BG, Environment
RuntimeManager - RTF - Applications	Runtime Versions Used Total	BG, Environment
RuntimeManager - Standalone	Mule Servers Total	BG, Environment
RuntimeManager - Standalone	Mule Clusters Total	BG, Environment
RuntimeManager - Standalone	Mule Server Groups Total	BG, Environment
RuntimeManager - Standalone	Mule Applications Total	BG, Environment
RuntimeManager - Standalone	Mule Applications Started Total	BG, Environment
RuntimeManager - Standalone	Mule Applications Stopped Total	BG, Environment
RuntimeManager - Standalone	Mule Runtime Versions	BG, Environment
RuntimeManager - Standalone	Mule Runtime Versions Total	BG, Environment
MQ	Queues total	BG, Environment, Region
MQ	FIFO Queues total	BG, Environment, Region
MQ	Queues In Flight Messages total	BG, Environment, Region
MQ	Queues Received Messages total	BG, Environment, Region
MQ	Queues Sent Messages total	BG, Environment, Region
MQ	Queues ACK Messages total	BG, Environment, Region
MQ	Exchanges total	BG, Environment, Region
MQ	Exchanges Published Messages total	BG, Environment, Region
MQ	Exchanges Delivered Messages total	BG, Environment, Region
OSv2	ObjectStore V2 Request Count	BG, Environment

Platform Benefits

These metrics require manual inputs

Name	Dimensions
Developer Productivity	BG
Platform Benefits	BG
Savings From API Reuse	BG
Savings From Maintenance Productivity	BG
Savings From Reuse in Maintenance	BG
Total Savings	BG

SDLC Metrics

These metrics are optional and can be cherry picked as per your requirement

Name	Metric
BitBucket	Total Number of BitBucket Repositories
Confluence	Total Number of Confluence pages
Confluence	Total Number of Confluence pages created in the last 30 days
Confluence	Total Number of Confluence pages updated in the last 30 days
Confluence	Top Contributors in the last 30 days and associated number of pages created
Jenkins	Total Number of Jenkins jobs
Jenkins	Total Number of failed Jenkins jobs
Jenkins	Total Number of successful Jenkins jobs
Jenkins	Total Number of unexecuted Jenkins jobs
Jira	Total Number of Jira stories in the backlog
Jira	Total Number of Jira stories in the current sprint
Jira	Jira stories categorized by type and associated count in the current sprint
Jira	Jira stories categorized by status and associated count in the current sprint
Splunk	Total Number of Splunk dashboards

Installation

Requirements

• Mule Runtime 4.2.2 or above
• All deployments models are supported: CloudHub, OnPrem hosted Runtimes, Runtime Fabric
• Anypoint Platform credentials - Two options are supported:
  • Anypoint Platform user with the Organization Administrator role, and CloudHub Admin role (or specific permissions - see connected app section). Both roles should be provided in the Master and all Sub Orgs for which you wish to gather data. The CloudHub Admin role is environment specific - therefore should be granted for each environment in each business group.
  • A Connected App (client credentials) with the following scopes (make sure to include all Sub Orgs and all environments you want to collect data):
    • Design Center
      • Design Center Developer
    • Exchange
      • Exchange Viewer
    • Runtime Manager
      • Cloudhub Network Viewer
      • Read Alerts
      • Read Applications
      • Read Servers
    • Runtime Fabric
      • Manage Runtime Fabrics
    • API Manager
      • View APIs Configuration
      • View Contracts
      • View Policies
    • General
      • Profile
      • View Environment
      • View Organization
• (Optional for SDLC metrics) Authorized user with API access to any of the applications: Jira, Confluence, Jenkins, Bitbucket and Splunk for which you want to gather data.

Steps

1. Clone or download the project from GitHub git clone git@github.com:mulesoft-catalyst/metrics-toolkit.git

2. Adjust the properties, run the project and test it - go to your browser and open http://localhost:8081/console/

3. Use the postman collection provided (/postman) to understand the API. The postman collection contains the following requests:

• Platform Metrics:

  • GET Platform Metrics: retrieves plaform metrics
  • POST Platform Metrics - Load - Splunk Strategy: used to load platform metrics to Splunk. For more information, see Splunk steps
  • POST Platform Metrics - Load - Tableau Strategy: used to load platform metrics to Tableau. For more information, see Tableau steps
  • POST Platform Metrics - Load - CSV Strategy: returns platform metrics in CSV format.
  • POST Platform Metrics - Load - JSON Strategy: returns business metrics in JSON format.

• Business Metrics:

  • GET Benefits: retrieves business metrics showing the benefits of using the platform
  • POST Benefits - Load - Splunk Strategy: used to load business metrics to Splunk. For more information, see Splunk steps
  • POST Benefits - Load - JSON Strategy: returns business metrics in JSON format.

4. If you want to run the application using the poller mode, you have to configure some properties

Properties configurations

• Default configurations defined in /src/main/resources/properties/app-{env}.yaml:
• Make sure to encrypt all sensitive data using the Secure Properties Module: https://docs.mulesoft.com/mule-runtime/4.2/secure-configuration-properties.
• Default secure config file defined in /src/main/resources/properties/secure/app-{env}.yaml
• Example mule.key used and configured as a Global Property under global.xml file

Name	Description	Default Value
http.port	The port for exposing the metrics-toolkit API	8081
poller.enabled	Property to enable or disable the poller to collect and load metrics in external systems	false
poller.frequency.cron	Defines the exact frequency (using cron-expressions) to trigger the execution: Recommended to collect metrics once a day	0 0 0 * * ? *
poller.frequency.timezone	Defines the time zone in which the cron-expression will be efective	GMT-3
aggregation.raw	Flag to define the format of the final response False: Won’t provide the raw data but final metrics True: Will provide raw data to be aggregated outside this asset	false
collectors	Comma separated set of collectors that should be executed. Default value: all. Possible values available for all deployment models: core (Core Services) ap (Automated Policies) apc (API Clients) apm (API Manager) arm (Standalone Runtimes) dc (Design Center) ex (Exchange). The following collectors are not available for PCE: amq (Anypoint MQ) apma (API Manager Analytics) ch (Cloudhub) rtf (Runtime Fabric)	osv2 (Object Store V2)
loader.strategy	In the case of using the poller, this property defines the strategy for loading data in external systems, the options are: csv, json, logger, splunk, am, elk, tableau, mongodb	logger
anypoint.platform.host	Anypoint Platform Host. Change to eu1.anypoint.mulesoft.com if using the EU Control Plane or to a private host if using PCE	anypoint.mulesoft.com
auth.mode	Authentication mode. Valid options are: platform-credentials or connected-app-credentials	platform-credentials
auth.username	Anypoint Platform username. Used when auth.mode is platform-credentials
auth.password	Anypoint Platform password. Used when auth.mode is platform-credentials
auth.clientId	Anypoint Platform Connected App Client Id. Used when auth.mode is connected-app-credentials
auth.clientSecret	Anypoint Platform Connected App Client Secret. Used when auth.mode is connected-app-credentials
auth.orgId	Anypoint Platform master org Id
ignoreLists.organizations	An array (comma-separated values) of Anypoint Platform sub-organizations id that will be ignored while retrieving metrics e.g "cdfa4e7d-47cd-n1h1-8f39-6a73fbb9ffcb, cdfa4e7d-47cd-n2h2-8f39-6a73fbb9ffcb"
ignoreLists.environments	An array (comma-separated values) of Anypoint Platform environments id that will be ignored while retrieving metrics e.g "cdfa4e7d-47cd-n1h1-8f39-6a73fbb9ffcb, cdfa4e7d-47cd-n2h2-8f39-6a73fbb9ffcb"
api.securityPolicies	A list of security policies IDs that are applied within the organisation. Should be updated to ensure accuracy of the APIs with/without security metrics	client-id-enforcement,ip-allowlist,ip-blocklist,jwt-validation,ldap-authentication,openidconnect-access-token-enforcement,external-oauth2-access-token-enforcement,http-basic-authentication

SDLC metrics collectors properties configuration

Name	Description	Default Value
sdlc.confluence.enabled	Property to enable or disable application to collect metrics from Confluence	false
sdlc.confluence.host	Confluence server host
sdlc.confluence.port	Confluence server port
sdlc.confluence.path	Context url of the Confluence REST API
sdlc.confluence.user	Authorized Confluence user to access REST APIs
sdlc.confluence.token	User token to access REST APIs
sdlc.bitbucket.enabled	Property to enable or disable application to collect metrics from Bitbucket	false
sdlc.bitbucket.host	Bitbucket server host
sdlc.bitbucket.port	Bitbucket server port
sdlc.bitbucket.path	Context url of the Bitbucket REST API
sdlc.bitbucket.user	Authorized Bitbucket user to access REST APIs
sdlc.bitbucket.token	User token to access REST APIs
sdlc.jira.enabled	Property to enable or disable application to collect metrics from Jira	false
sdlc.jira.host	Jira server host
sdlc.jira.port	Jira server port
sdlc.jira.path	Context url of the Jira REST API
sdlc.jira.backlogPath	Context url of the Jira REST API to fetch stories from backlog
sdlc.jira.user	Authorized Jira user to access REST APIs
sdlc.jira.token	User token to access REST APIs
sdlc.jenkins.enabled	Property to enable or disable application to collect metrics from Jenkins	false
sdlc.jenkins.host	Jenkins server host
sdlc.jenkins.port	Jenkins server port
sdlc.jenkins.path	Context url of the Jenkins REST API
sdlc.jenkins.user	Authorized Jenkins user to access REST APIs
sdlc.jenkins.token	User token to access REST APIs
sdlc.splunk.enabled	Property to enable or disable application to collect metrics from Splunk	false
sdlc.splunk.host	Splunk server host
sdlc.splunk.port	Splunk server port
sdlc.splunk.path	Context url of the Splunk REST API
sdlc.splunk.user	Authorized Splunk user to access REST APIs
sdlc.splunk.password	Password to access REST APIs

NOTE: Please note that each external system collector should be self-contained, it means that all associated configuration must be part of the Mule configuration file itself and must not be externalized inside the global.xml

Postman Collection

A Postman collection which contains sample API requests is included under /postman/postman_collection.json. To import this, open Postman and click File > Import, and select the .json file. The collection contains documentation describing prerequisites and setup. Clicking the collection folder opens the documentation window. A number of Environment variables are needed. A template environment is also included under /postman/environment_template.json. To import into Postman, click Environemts > Import and select the .json file. Populate the variables with accurate values.

Splunk steps

1. Create 2 indexes: metrics and platform_benefits (of type Events)
2. In the Splunk instance configure an HTTP Event Collector (HEC) associated to these 2 indexes, format _json
3. The token obtained will be used as part of the properties of the Mule application
4. Create a new application
5. Load the dashboards, simply copy the xmls provided under /dashboards/splunk to {SPLUNK_HOME}/etc/apps/{APP_NAME}/local/data/ui/views
6. Restart the Splunk instance
7. If you can't copy the dashboard xmls, you can use the UI to create them and using the "Source" option, you can copy & paste the content of the xmls provided

Follow official Splunk documentation: https://docs.splunk.com/Documentation/Splunk/

Properties specific for Splunk

Name	Description	Default Value
splunk.host	HTTP Event Collector (HEC) host
splunk.port	HEC port	8088
splunk.protocol	HEC endpoint protocol: HTTPS or HTTP	HTTP
splunk.token	HEC token
splunk.source	HEC source	metrics-source
splunk.source.type	Source Type	_json (*)
splunk.index.metrics	Index for storing Platform operational metrics	metrics
splunk.index.benefits	Index for storing Platform benefits	platform_benefits

(*): Please note that by default, "Source Types" are created with a limit of 3000 characters. The Metrics Toolkit JSON event will likely exceed this limit. In order to solve that, you must increase this limit adding a new property "TRUNCATE" in the Advanced configuration of the specific Source Type. For example: TRUNCATE = 40000. Depending of the size of your organization, in terms of Business Groups, environments and number of applications and APIs in each environment, this value can be higher.

ELK steps

NOTE: Dashboards were created and tested with Kibana 7.6.2, adjustments may be necessary for other versions

1. The toolkit will load data into the metrics and platformbenefits indexes. Once data is loaded create an index pattern on Kibana for these indexes
2. Set the loader strategy to elk on the app-{env}.yaml file, along with the elk.user and elk.password parameters in the secure app-{env}.yaml file
3. To load the dashboards, replace the <YOUR-INDEX-PATTERN-ID> occurrences on all of the dashboards provided under /dashboards/elk with your index pattern ID. The index pattern ID can be obtained on Kibana under Management>>Index Patterns
4. Log into your Kibana instance, and on the Management>>Saved Objects menu, click on import for each dashboard. This will import all dashboards and visualizations

Follow official ELK documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html https://www.elastic.co/guide/en/kibana/current/index.html

Properties specific for ELK

Name	Description	Default Value
elk.host	Elasticsearch host
elk.port	Elasticsearch port	9200
elk.protocol	Elasticsearch protocol: HTTP	HTTP
elk.user	Elasticsearch username
elk.password	Elasticsearch password
elk.index.metrics	Index for storing Platform operational metrics	metrics
elk.index.benefits	Index for storing Platform benefits	platformbenefits

Tableau steps

Tableau Dashboards use a JSON File Data Source, which is configured to load data from all JSON files present in a given directory, each JSON file representing a Platform Metrics snapshot.

Metrics Toolkit provides the following approaches to generated those files:

• Using Tableau Strategy (poller or API): this option can only be used if Tableau Desktop can can access the filesystem where Metrics Toolkit is running (cannot be used for CloudHub or Runtime Fabric, since Tableau Desktop won't be able to access Mule Runtime local filesystem).
• Using the Get Platform Metrics API operation (check GET Platform Metrics request provided in the Postman collection): suitable if Metrics Toolkit is running on CloudHub or Runtime Fabric, or if you don't have access to Mule Runtime local filesystem.

To learn more about Tableau, follow the official documentation: https://www.tableau.com/support/help

NOTES:

• Tableau Dashboard Data Source will only use JSON files which has the following naming pattern: platform_metrics_agg_*.json
• Tableau Dashboard Data Source will only be able to use JSON files containing aggregated data. Thus, Tableau won't be able to render dashboards if JSON files contain raw data (check Properties Configuration Section for futher details.

Generating JSON Files Using Poller or API Loader Tableau Strategy

When using the Tableau Strategy, Metrics Toolkit will create the JSON files in the directory defined by tableau.outputDir property (poller) or by loaderDetails.outputDir provided in the request (API).

Check POST Platform Metrics - Load - Tableau Strategy Postman collection request if using the API Loader.

IMPORTANT: Make sure aggregation.raw property (poller) or loaderDetails.rawData (API) is set to false

Generating JSON Files Using Get Platform Metrics API operation

1. Use JSON loader strategy via API (check POST Platform Metrics - Load - JSON Strategy in the provided Postman collection)
2. Save the response content as a JSON file in the directory of your choice. All files must be in the same directory and must respect the naming convention (platform_metrics_agg_yyyyMMddHHmmssSSS.json)

IMPORTANT: Make sure the query parameter raw is set to false

Visualizing Dashboards in Tableau Desktop

1. Before opening a workbook in Tableau Desktop, make sure that you have at least one JSON file available.
2. Make a copy of the desired workbook provided under /dashboards/tableau.
3. After copying the workbook, open it in Tableau Desktop.
4. When prompted, edit workbook connection. Select one of the available JSON files and click Ok.

Properties specific for Tableau

Name	Description	Default Value
tableau.outputDir	Directory where JSON files will be written to.

File name pattern by workbook

Workbook	File Name Pattern
current_consolidated	platform_metrics_agg_*.json

Embedded Dashboard steps

1. Enable the dashboard by changing the embedded.dashboard.enabled property to "true"
2. Deploy & Run the application
3. Use a web browser to access the applications base URL (e.g. if deployed locally, use http://localhost:8081)
4. Use the "Login" page to enter your Anypoint platform username, password and organization ID
5. Wait for the dashboard to run the metrics request and once done, navigate through the different metrics taken using the UI

SFDC Steps

Using the sfdc loader option, and initialising Salesforce Analytics Studio with an empty project, will allow you to quickly inject data into Tableau CRM Dashboards to visualise your Anypoint Platform Metrics in different ways. Tableau CRM can also be used as an historic data repository to allow the metrics data to be displayed against time for trend analysis (e.g. vCore growth, API count or transaction growth over time).

1. Create a new application in Data Manager
2. If using the poller functionality, configure the SFDC loader properties in the properties files. If using the API, format the request body with the required parameters
3. Deploy and Run the metrics toolkit application, allowing the poller to execute, or performing an API request
4. Validate the dataset is created in Data Manager
5. Create a new data recipe, setting your new dataset as the source. Transform "value" from a dimension to a measure.
6. (Optional) - Filter out the non-numeric value fields (API Manager Policies Used and CloudHub Runtime Versions Used) and store this in a new 'enum' dataset
7. (Optional) - Create a 'historic' dataset. Add a step to your data recipe to append to this historic dataset More detailed steps can be found in the SFDC specific README.

Note: This is only a high level introduction and it is highly recommended that you become familiar with Tableau CRM through official documentation.

Properties specific for SFDC

Name	Description	Default Value
sfdc.appName	Application name in Tableau CRM
sfdc.dataSetName	Dataset name to be created/overwritten in Tableau CRM
sfdc.notificationEmail	Email adress for notifications to be sent to
sfdc.sendNotification	Occurances which should generate notifications. ENUM values are: ALWAYS,FAILURES,NEVER,WARNINGS	NEVER
sfdc.username	Salesforce username to use for authentication
sfdc.password	Salesforce password
sfdc.securityToken	Salesforce developer security token
sfdc.authUrl	Specific Salesforce auth URL to be used if required

MONGODB steps

NOTE: Data was pushed and tested with Mongodb 4.4.1. Adjustments may be necessary for other versions.

1. In MongoDB you will need to create a Database called matrixdb and a Collection called metrics.
2. Set the loader strategy to mongodb on the app-{env}.yaml file, along with the mongodb.username and mongodb.password parameters in the secure app-{env}.yaml file
3. Uncomment 2 sections from the pom.xml (dependencies and shared library)
4. Rename the file src/main/mule/loaders/loader-mongodb.disabled to src/main/mule/loaders/loader-mongodb.xml

Properties specific for MONGODB

Name	Description	Default Value
mongodb.host	MongoDB host
mongodb.port	MongoDB port
mongodb.database	Database name
mongodb.colleciton	Collection Name
mongodb.username	MongoDB user
mongodb.password	MongoDB password

Considerations

• This application can be deployed in any Mule Runtime (OnPrem, CloudHub, RTF)
• The metrics collection will depend on the features available in each account; e.g if the account has the API Manager add-on, the toolkit will collect and aggregate the metrics related to API Manager, otherwise the values will appear as zeroes; if using PCE, there won't be information about API Analytics
• In order to enable or disable specific collectors, you have to change the property "collectors" if using the poller or add a query parameter "collectors" if using the API, including a CSV string as explained in the properties section

Limitations

• Access Management metrics:
  • Not supported on GovCloud
• Exchange reuse metrics:
  • Not supported for Private Cloud Edition (PCE)
• API Manager metrics:
  • API Manager metrics available only for accounts with the API Manager and Analytics add-on
• Runtime Manager (CloudHub) application metrics:
  • CloudHub is not supported on Private Cloud Edition (PCE)
• Runtime Manager (CloudHub) networking metrics - VPCs, VPNs, DLBs and static IPs usage:
  • Not supported when authenticating with Connected Apps
  • Not supported on GovCloud
• Runtime Manager (RuntimeFabric) metrics:
  • Runtime Fabric is not supported on GovCloud
  • Runtime Fabric is not supported on Private Cloud Edition (PCE)
• Runtime Manager (Standalone) metrics:
  • Runtime Manager (Standalone Runtimes) not supported on GovCloud
• API Platform Client Applications metrics:
  • Not supported when authenticating with Connected Apps
  • Not supported on GovCloud
• Analytics metrics:
  • Not supported on GovCloud
  • Not supported on Private Cloud Edition (PCE)
  • Not supported when authenticating with Connected Apps
• Anypoint MQ metrics:
  • Not supported on Private Cloud Edition (PCE)
  • Not supported when authenticating with Connected Apps
  • Not supported on GovCloud
• OSv2 metrics:
  • Not supported on Private Cloud Edition (PCE)
  • Not supported when authenticating with Connected Apps

Some Theory around the Toolkit

The toolkit is intended to cover the main areas to define and implement metrics using Mule.

Business Needs

There are several business needs around the definition of metrics. The principal goal is to provide visibility on 4 main areas: People/Teams, Projects, Processes and Technology. By having visibility on these areas, the involved stakeholders can make decisions, anticipate technical and non-technical needs and optimize time and resources

Data Sources

Any system containing raw data

Measurements

1. Basic measurements:

Examples:

• Number of projects
• Number of incidents

2. Complex/Composed measurements

Examples:

• Average daily-TPS per application/project
• vCores usage per application/project - monthly

3. Indexes/KPIs

Examples:

• Growth
• Tendencies
• Efficiency / Performance

Problem Space

How to link business needs, measurements and data sources?

Approach - Conceptual

1. Event: Canonical object representing a metric, based on dimensions and facts

2. Collectors: Processes to Collect raw data from different data sources

3. Aggregators/Transformers: Transform (calculate, compose, enrich, build metric/event normalized/denormalized objects

4. Loaders: Processes to send/load events into external storages/systems

5. Visualization: Third party systems for showing charts and dashboards based on metrics-events (e.g Microstrategy, Tableau, Splunk dashboards, ELK, custom dashboards, BI software, etc)

Approach

Conceptual

Detailed

Final Notes

Enjoy and provide feedback / contribute :)

Common Errors

Permission errors

• All 403 errors, specifically, for the endpoint related to RTF deployments /hybrid/api/v2/ are permissions issues, be sure the user/connected-app has the right permissions (Runtime Manager and Runtime Fabric specifically): Requirements