[ARCHIVED] Replace Tokens task

This GitHub project has been archived. Ongoing development on this project can be found in https://github.com/qetza/replacetokens-task.

Azure Pipelines extension that replace tokens in text files with variable values.

Usage

If you are using the UI, add a new task, select Replace Tokens from the Utility category and configure it as needed:

If your are using a YAML file, add a task with the following syntax:

- task: qetza.replacetokens.replacetokens-task.replacetokens@3
  displayName: 'Replace tokens'
  inputs:
    targetFiles: |
      **/*.config
      **/*.json => outputs/*.json

Note: the task will only work on text files, if you need to replace token in archive file you will need to first extract the files and after archive them back.

Parameters

The parameters of the task are described bellow, in parenthesis is the YAML name:

Root directory (rootDirectory): the base directory for searching files. If not specified the default working directory will be used. Default is empty string
Target files (targetFiles): the absolute or relative newline-separated or comma-separated paths to the files to replace tokens. Wildcards can be used (eg: **\*.config for all .config files in all sub folders). Default is **/*.config

Syntax: {file path}[ => {output path}]

web.config will replace tokens in web.config and update the file.

web.tokenized.config => web.config will replace tokens in web.tokenized.config and save the result in web.config.

config\web.tokenized.config => c:\config\web.config will replace tokens in config\web.tokenized.config and save the result in c:\config\web.config.

Wildcard support

*.tokenized.config => *.config will replace tokens in all {filename}.tokenized.config target files and save the result in {filename}.config.

**\*.tokenized.config => c:\tmp\*.config will replace tokens in all {filename}.tokenized.config target files and save the result in c:\tmp\{filename}.config.

Only the wildcard * in the target file name will be used for replacement in the output.
Relative paths in the output pattern are relative to the target file path.\

Negative pattern
If you want to use negative pattern in target file, use a semi-colon ; to separate the including pattern and the negative patterns. When using output syntax, only the wildcard in the first pattern will be used for generating the output path.

**\*.tokenized.config;!**\dev\*.config => c:\tmp\*.config will replace tokens in all {filename}.tokenized.config target files except those under a dev directory and save the result in c:\tmp\{filename}.config.

Files encoding (encoding): the files encoding used for reading and writing. The 'auto' value will determine the encoding based on the Byte Order Mark (BOM) if present; otherwise it will use ascii. (allowed values: auto, ascii, utf-7, utf-8, utf-16le, utf-16be, win1252 and iso88591). Default is auto
Write unicode BOM (writeBOM): if checked writes an unicode Byte Order Mark (BOM). Default is true
Escape type (escapeType): specify how to escape variable values. Value auto uses the file extension (.json and .xml) to determine the escaping and none as fallback. Default is auto (allowed values: auto, none, json, xml and custom)
Escape character (escapeChar): when using custom escape type, the escape character to use when escaping characters in the variable values.
Characters to escape (charsToEscape): when using custom escape type, characters in variable values to escape before replacing tokens.
Verbosity (verbosity): specify the level of log verbosity. Default is normal (note: error and system debug are always on) (allowed values: normal, detailed and off)
Action on missing variable (actionOnMissing): specify the action to take on a missing variable. Default is warn
- silently continue (continue): the task will continue without displaying any message.
- log warning (warn): the task will continue but log a warning with the missing variable name.
- fail (fail): the task will fail and log the missing variable name.
Keep token for missing variable (keepToken): if checked tokens with missing variables will not be replaced by empty string. Default is false
Action on no file processed (actionOnNoFiles): specify the action when no file was processed. Default is continue (allowed values: continue, warn, fail)
Token pattern (tokenPattern): specify the pattern of the tokens to search in the target files. Default is default (allowed values: default, rm, octopus, azpipelines, doublebraces and custom)
Token prefix (tokenPrefix): when using custom token pattern, the prefix of the tokens to search in the target files. Default is #{
Token suffix (tokenSuffix): when using custom token pattern, the suffix of the tokens to search in the target files. Default is }#
Use legacy pattern (useLegacyPattern): if checked whitespaces between the token prefix/suffix and the variable name are not ignored. Default is false
Use legacy empty/default feature (useLegacyEmptyFeature): if check use the old empty & default values features. The new feature/code can now distinguish between an undefined variable and an defined empty variable without the use of a specific "empty" token (Empty value parameter). Default is true
Empty value (emptyValue): with legacy empty/default feature: the variable value that will be replaced with an empty string. Default is (empty)
Use default value (useDefaultValue): with new empty/default feature: if check replace variable not found with a default value specified in Default value. Default is false
Default value (defaultValue): the value to be used if a variable is not found. With legacy empty/default feature: do not set to disable default value feature with the legacy feature; to replace with an empty string set the default value to the Empty value. Default is an empty string
Enable transformations (enableTransforms): if checked transformations can be applied on variable values. Default is false. The following transformations are available:
- lower: make variable value lower case. Example: #{lower(MyVar)}#
- upper: make variable value upper case. Example: #{upper(MyVar)}#
- noescape: disable variable value escaping. (this can be used if you want to inject raw JSON or XML for example). Example: #{noescape(MyVar)}#
- base64: encode variable value in BASE64. Example #{base64(MyVar)}#
- indent: indent newlines with support of 2 parameters; the first parameter after the variable name is the indent size (default 2) and the second is a boolean specifying if the first line must be indented also (default false). Examples #{indent(MyVar)}#, #{indent(MyVar, 4, true)}#
Transform prefix (transformPrefix): The prefix between transform name and token name. Default is (.
Transform suffix (transformSuffix): The suffix after the token name. Default is ).
Variable files (JSON or YAML) (variableFiles): the absolute or relative comma or newline-separated paths to the files containing additional variables. Wildcards can be used (eg: vars\**\*.json for all .json files in all sub folders of vars). YAML files must have the .ymlor .yaml extension otherwise the file is treated as JSON. Variables declared in files overrides variables defined in the pipeline.
Inline variables (YAML syntax) (inlineVariables): A YAML formatted string containing inline variables. Variables declared inline overrides variables defined in the pipeline and in files.
Variable separator (variableSeparator): the separtor to use in variable names for nested objects and arrays in inline variables or variable files. Default is .. Example: { 'My': { 'Value': ['Hello World!'] } } will create a variable My.Value.0 with the value Hello World!.
Enable recursion in values (enableRecursion): if checked token replacement (with transformation) will be run on values.Example: "Say: #{value1}#" with value1 = "hello #{upper(value2)}#" and value2 = "world!" will result in "hello WORLD!". Default is false
Send anonymous usage telemetry (enableTelemetry): if checked anonymous usage data (hashed collection and pipeline id, no file parameter values, no variable values) will be sent to the task author only to analyze task usage. Default is true

Output variables

The task creates the following as output variables:

tokenReplacedCount: the total number of tokens which were replaced by a variable.
tokenFoundCount: the total number of of tokens which were found.
fileProcessedCount: the total number of files which were processed.
transformExecutedCount: the total number of transformations which were executed.
defaultValueCount: the total number of default value used.

Data/Telemetry

The Replace Tokens task for Azure Pipelines collects anonymous usage data and sends them to its author to help improve the product by default. If you don’t wish to send usage data, you can change your telemetry settings through Send anonymous usage telemetry parameter or by setting a variable or environment variable REPLACETOKENS_DISABLE_TELEMETRY to true.

Tips

If you want to use tokens in XML based configuration files to be replaced during deployment and also have those files usable for local development you can combine the Replace Tokens task with the XDT tranform task:

create an XDT transformation file containing your tokens
setup your configuration file with local developement values
at deployment time
- inject your tokens in the configuration file by using your transformation file
- replace tokens in your updated configuration file

Release notes

New in 4.6.0

Task 5.3.0
- Add support for indent transformation with indent size and indent first line parameters (326).
Task 4.4.0
- Add support for indent transformation with indent size and indent first line parameters (326).
Task 3.12.0
- Add support for indent transformation with indent size and indent first line parameters (326).

New in 4.5.0

Task 5.2.0
- Fix recursion cycle detection (#308) (contributed by Chad Smith).
- Add support for YAML multiple document files in variable files and inline variables (#287).
- Add support for JSON with comments variable files (#299).
- Add support for Node16 execution handler.
Task 4.3.0
- Fix recursion cycle detection (#308) (contributed by Chad Smith).
- Add support for YAML multiple document files in variable files and inline variables (#287).
- Add support for JSON with comments variable files (#299).
Task 3.11.0
- Fix recursion cycle detection (#308) (contributed by Chad Smith).
- Add support for YAML multiple document files in variable files and inline variables (#287).
- Add support for JSON with comments variable files (#299).

New in 4.4.1

Task 4.2.1
- Fix compatibility with node 5.10.1 (#277).
Task 3.10.1
- Fix compatibility with node 5.10.1 (#277).

New in 4.4.0

Task 5.1.0
- Add support for inline variables (#252).
- Add support for recursive token replacement in values (#201).
- Add optional reworked feature to simplify empty and default values (this is a breaking change if enabled as the old Empty value is not used anymore but replaced by an empty variable declaration).
Task 4.2.0
- Add support for inline variables (#252).
- Add support for recursive token replacement in values (#201).
- Add optional reworked feature to simplify empty and default values (this is a breaking change if enabled as the old Empty value is not used anymore but replaced by an empty variable declaration).
Task 3.10.0
- Add support for inline variables (#252).
- Add support for recursive token replacement in values (#201).
- Add optional reworked feature to simplify empty and default values (this is a breaking change if enabled as the old Empty value is not used anymore but replaced by an empty variable declaration).

New in 4.3.0

Add task 5.0.0
- Breaking change: Migrate task to Node10 execution handler needing agent 2.144.0 minimum (#228, #230).

New in 4.2.1

Task 4.1.1
- Revert migrate tasks to Node10 execution handler (#233).
Task 3.9.1
- Revert migrate tasks to Node10 execution handler (#233).

New in 4.2.0

Task 4.1.0
- Migrate tasks to Node10 execution handler (#228, #230).
Task 3.9.0
- Migrate tasks to Node10 execution handler (#228, #230).

New in 4.1.0

Task 4.0.1
- Promoted to release.
- Add base64 transform (#163).
- Add action on no file processed (#210).
Task 3.8.0
- Add base64 transform (#163).
- Add action on no file processed (#210).

New in 4.0.0

Add support for multiple task versions.
Add task 4.x (preview)
- Breaking change: Add output variables (#160). (some older version of TFS/Azure Pipelines doesn't support output variables when used in release pipelines)
- Breaking change: Add dropdown parameter Token pattern to select token pattern (#131). (users with customized token pattern will need to manually select one or custom)

New in 3.7.1

Fix issue on binary files (#193).
Rollback output variables (#196).

New in 3.7.0

Add output variables tokenReplacedCount, tokenFoundCount and fileProcessedCount (#160).
Add support for variable transformations with Enable tranformations (#96).
Add default value for tokens not found with Default value (contribution from ClemensSutor).
Group log outputs in Azure Pipelines output.
Add support for variables in external YAML files with .yml or .yaml extension (#177).

New in 3.6.0

Add parameter Use legacy pattern with default value to false.

New in 3.5.2

Fix issue when token prefix present but not as a token prefix (#149).

New in 3.5.1

Fix issue when variable System.ServerType is not defined (#147).

New in 3.5.0

Add anonymous usage telemetry.
Ignore spaces between token prefix/suffix and variable name (#143).

New in 3.4.1

Fix JSON escaping of slash / (#138)

New in 3.4.0

Add summary in logs with number of tokens found and replaced (#126).
Add support for variables in external JSON files (#113).

New in 3.3.1

Breaking change: If you were using negative pattern you need to use the semi colon ; as a separator instead of new-line in Target files.
Fix negative pattern support (#127).

New in 3.3.0

Add support for custom output file and wildcard support (#114).

New in 3.2.2

Fix matching issue with directory (#122).

New in 3.2.1

Fix log issue with escaped secret values.

New in 3.2.0

Switch to jschardet for encoding detection when selecting auto in File encoding (#99).
Switch to azure-pipelines-task-lib v2.8.0.
Add auto to Escape type and set it as default value.
Move Escape type, Escape character and Characters to escape to the main paramters section for easier discoverability.

New in 3.1.0

Add Verbosity parameter to allow detail logs without using system.debug.

New in 3.0.0

Breaking change: If you were using the character escaping feature you need to select custom in Escape values type parameter.
Add support to escape JSON in variable values (contributed by Justin Gould)
Add support to escape XML in variable values (contributed by Justin Gould)
Add Windows 1252 and ISO 8859-1 encoding to File encoding (#67)

New in 2.3.0

Add support to escape characters in variable values (#52)

New in 2.2.1

Fix issue with backslash in default target files value on mac (#50)

New in 2.2.0

Fix issue on file not found when using network paths (#40, #41).

New in 2.1.0

Add support for comma-separator in Target files (#35).
Add Empty value parameter to allow token replacement with an empty string (#32).

New in 2.0.2

Fix invalid file permissions after saving files.

New in 2.0.0

Breaking change: Migrated code to typescript to support cross-platform agent. This change requires the use of an agent at least in version 2.105.0 which is not compatible with TFS 2015. If you need to install the task on TFS 2015, download the vsix from the repository: https://github.com/qetza/vsts-replacetokens-task/releases/download/v1.4.1/qetza.replacetokens-1.4.1.vsix
Breaking change: File encoding parameter is now used when reading and writing files. Previously it was only used when writing.
Breaking change: File encoding doesn't support 'utf-32' and 'utf-32 (big endian)' anymore.
Breaking change: Target files parameter now only uses the new line as a separator for multi-values (previously it used new-line and semi-colon).
Removed required Root directory, an empty value is equivalent to $(System.DefaultWorkingDirectory).

New in 1.4.1

Fix missing method issue with new xplat agent (2.104.1)

New in 1.4.0

Add variables expansion in variable values.
Escape token prefix and suffix in regex pattern.

New in 1.3.1

Fix wrong encoding constructors parameters.

New in 1.3.0

Replaced parameter Fail on missing with Action in Missing variables group.
Add Keep token parameter in 'Missing variables' group.
Fix issue on empty file.

New in 1.2.0

Add Root directory task parameter to configure file search root directory (contributed by Jesse Houwing).
Update Target files task parameter to support newline-separator (contributed by Jesse Houwing).