"definitions/boolean" in schema does not allow for boolean values, resulting in failing validation for a step with `continueOnError: true`

Originally reported here: python-jsonschema/check-jsonschema#42
A great deal of credit to the user in this case, he tells us exactly what the issue is.

This pipeline fails validation against the schema:

steps: 
  - script: echo "Hello world"
    continueOnError: true

The problem is here:

azure-pipelines-vscode/service-schema.json

Line 3995 in 2e9781b

"boolean": {

{"type": "boolean"} is not listed as one of the possible values.

I checked the language-server code to see if it was coercing bools to strings, and it is not:
https://github.com/microsoft/azure-pipelines-language-server/blob/71b20f92874c02dfe82ad2cc2dcc7fa64996be91/language-service/src/parser/yamlParser.ts#L144

(There is a nearby coercion of literal on, off, etc. to bools -- the other direction.)

There is a theme in this schema of treating everything as a string. But it's not applied consistently. For example:

azure-pipelines-vscode/service-schema.json

Line 5985 in 2e9781b

"type": "boolean",

So the options I see for accurate validation are:

definitions.boolean starts listing {"type": "boolean"} as an option
the schema becomes consistent one way or another about how booleans are treated

Thanks for reporting this @sirosen. I raised the original issue in the downstream repo, so will add a bit of context.
The Azure Pipelines VSCode extension actually does handle this error, but I was seeing errors when validating with a Python-based validator -- in Python, booleans are True and False, which fails against the boolean regex in the schema.

It looks like the language server casts bools to strings inside the BooleanASTNode object: https://github.com/microsoft/azure-pipelines-language-server/blob/71b20f92874c02dfe82ad2cc2dcc7fa64996be91/language-service/src/parser/jsonParser.ts#L387-L393

So for this specific case, there's no problem. But I agree with the sentiment that this extension should be consistent with its treatment of booleans.

It looks like the language server casts bools to strings inside the BooleanASTNode object: https://github.com/microsoft/azure-pipelines-language-server/blob/71b20f92874c02dfe82ad2cc2dcc7fa64996be91/language-service/src/parser/jsonParser.ts#L387-L393

Oy vey! I didn't catch that, thanks for pointing it out.

I see similar hacks in the handling of null and numeric types.

This is not a feature which can be emulated -- at least not trivially -- outside of the language-server implementation. The fact that the nodes in the parsed AST are responsible for their own validation is unusual. In most implementations of JSON Schema, the schema validation and data are fully decoupled.

It might be possible, in the particular case of python's jsonschema library, to customize the validators to get similar behavior. It's easy to turn all checks for {"type": "string"}, but getting non-string items coerced to strings at validation-time for pattern and other checks is not trivial.

Hi @sirosen, as far as I see this issue also related to ADO-defined schema, as I said in other ticket - the schema in this repository is generated on the Azure DevOps server, the VSCode extension only is just a schema consumer, our team not support schema on Azure DevOps side and to fix this issue it would be better to create a ticket on the Microsoft developer community.

Hello @sirosen, I'm closing this ticket, since we can't help you with issues that are related to the schema, our team doesn't support Azure DevOps YAML schema. Extension in fact is the same consumer of ADO schema as you and all the questions related to schema outside our ownership.