Wrong unevaluatedProperties validation errors
Closed this issue · 6 comments
Consider the following schema which uses unevaluatedProperties
in order to extend a closed schema following the official JSON schema documentation:
schema.json
{
"allOf": [
{
"type": "object",
"properties": {
"street_address": { "type": "string" },
"city": { "type": "string" },
"state": { "type": "string" }
},
"required": ["street_address", "city", "state"]
}
],
"properties": {
"type": { "enum": ["residential", "business"] }
},
"required": ["type"],
"unevaluatedProperties": false
}
When running jv
against the following valid document:
doc.json
{
"street_address": "123 Willow Road",
"city": "Middletown",
"state": "KY",
"type": "residential"
}
... jv
validates the document as expected:
$ jv schema.json doc.json
$
Now, if we make the document invalid by modifying one of the subschema properties - for example, by setting the street_address
to a number:
{
"street_address": 12,
"city": "Middletown",
"state": "KY",
"type": "residential"
}
... jv
correctly detects the street_address
type mismatch, however it also yields an unevaluatedProperties not allowed
error for every property that's declared by the subschema, although these properties are correct according to the schema:
$ jv schema.json doc.json
[I#] [S#] doesn't validate with file:///Users/vassilk/projects/test/json/schema.json#
[I#] [S#/allOf/0] allOf failed
[I#/street_address] [S#/allOf/0/properties/street_address/type] expected string, but got number
[I#/state] [S#/unevaluatedProperties] not allowed
[I#/street_address] [S#/unevaluatedProperties] not allowed
[I#/city] [S#/unevaluatedProperties] not allowed
the second document you have specified is invalid;
here allOf
has failed. so any properties evaluated by allOf
are still considered as unevaluated;
only if the validation succeeds, the properties are considered evaluated;
[edit]
the documentation you mentioned clearly says:
unevaluatedProperties works by collecting any properties that are successfully validated when processing the schemas and using those as the allowed list of properties.
from spec:
https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-01#name-unevaluatedproperties
The behavior of this keyword depends on the annotation results of adjacent keywords that apply to the instance location being validated.
but annotation results
are generated only when an instance is valid against schema. see https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-01#name-annotations
JSON Schema can annotate an instance with information, whenever the instance validates against the schema object containing the annotation, and all of its parent schema objects.
to better understand consider following example:
consider following files:
schema.json: ( allOf
is wrapped in not
)
{
"allOf": [ { "not":
{
"type": "object",
"properties": {
"street_address": { "type": "string" },
"city": { "type": "string" },
"state": { "type": "string" }
},
"required": ["street_address", "city", "state"]
}
}],
"properties": {
"type": { "enum": ["residential", "business"] }
},
"required": ["type"],
"unevaluatedProperties": false
}
{
"street_address": 123,
"city": "Middletown",
"state": "KY",
"type": "residential"
}
here street_address
is number so allOf
fails which implies not
succeeds. still validator considers that none of the properties are evaluated
see jv
output below:
jv schema.json doc.json
[I#] [S#] doesn't validate with file:///Users/santhosh/gh/santhosh-tekuri/jsonschema/cmd/jv/schema.json#
[I#/street_address] [S#/unevaluatedProperties] not allowed
[I#/city] [S#/unevaluatedProperties] not allowed
[I#/state] [S#/unevaluatedProperties] not allowed
if you want you can also check results at https://www.jsonschemavalidator.net
@vassilvk
the second document you have specified is invalid;
Yes, this was the main point.
The second document has one invalid property, and I expect to see a validation error only for that one property.
Seeing validation errors for all the other properties (which are in fact valid) makes no sense and is potentially confusing.
From customer's perspective, the experience is the following:
- They enter a number for street address (which they shouldn't, but they do).
- They get an error correctly indicating that address is not supposed to be a number.
- Then they get messages that address, city, and state are not allowed.
This last bit is technically wrong.
I understand that your goal is to stick to the specification, and that's great.
I am just surprised at how impractical unevaluatedProperties
becomes when it comes to subschemas.
in the beginning, this library used to show only single error and stop validation on first error.
later ability to show multiple errors is added.
It is always not trivial to show errors from customer's perspective.
for example when missing a semicolon in a program, compiler shows tons of errors. it is similar situation here.
we can do stop validating keywords that have dependent on other keywords. but it is little complicated to implement and maintain.
I understand. Thank you for the quick response and thank you for a great library!