JSON Schema validating a invalid JSON document does not produces error messages

[svenskmand]

Describe the bug
When I want to validate a JSON document against a schema and the document does not follow the schema I would expect an error message.

To Reproduce

[Fact]
public void TestFoo() {
	var schema = new Manatee.Json.Schema.JsonSchema();
	var serializer = new Manatee.Json.Serialization.JsonSerializer();
	schema.FromJson(new Manatee.Json.JsonObject {
		["properties"] = new Manatee.Json.JsonObject {
			["foo"] = new Manatee.Json.JsonObject {
				["type"] = "string",
			},
		},
	}, serializer);

	var doc1 = Manatee.Json.JsonValue.Parse(@"{""foo"":""bar""}");
	var result1 = schema.Validate(doc1);

	var doc2 = Manatee.Json.JsonValue.Parse(@"{""foo"":42}");
	var result2 = schema.Validate(doc2);
}

Expected behavior
I would expect the result of the validation to contain specific errors instead of having an empty validation tree with the IsValid flag saying the Json is invalid, but not saying why.

Desktop (please complete the following information):

• OS: Windows 10
• .Net Target Dotnet Core
• Version 3.0

[evman182]

The default verbosity is just the flag. If you want detailed output, you need to set the JsonSchemaOptions.OutputFormat property to Basic, Verbose, or Detailed

[svenskmand]

The default verbosity is just the flag. If you want detailed output, you need to set the JsonSchemaOptions.OutputFormat property to Basic, Verbose, or Detailed

Ok, thanks, but where do I change that? I cannot find any property on the Schema to change that :)

[evman182]

OutputFormat is a static property on JsonSchemaOptions

[svenskmand]

Is it not possible to configure this per instance of the JsonSchema type?

Also is there some kind of convinience method to turn the SchemaValidationResults into a list of strings that contains the error messages? The SchemaValidationResults is a tree structure and it is not clear which order the ErrorMessage and the ErrorMessage of the children in the field NestedResults have relative to each other.

[gregsdennis]

@svenskmand Is it not possible to configure this per instance of the JsonSchema type?

Can you explain your use case for wanting different output per instance of schema? I considered this an application-level setting, hence the static options.

I like the idea of a convenience method to pull the error messages into a list. It'll actually have to combine the error message with the location. For now, try setting the output format to Basic, which will output into a flat list. The docs have more information on validation results.

[svenskmand]

@svenskmand Is it not possible to configure this per instance of the JsonSchema type?

Can you explain your use case for wanting different output per instance of schema? I considered this an application-level setting, hence the static options.

I would just expect it to be configurable per instance, since having a global singleton configuration would mean that I could not verify documents in parallel using different JsonSchemaOptions configurations. Also since it is a global configuration there is no really good place to put it when you have production code and test code. So now I have simply put it into a wrapper validation method on a wrapper class that encapsulates the JsonSchema class in Manatee.Json.

I like the idea of a convenience method to pull the error messages into a list. It'll actually have to combine the error message with the location. For now, try setting the output format to Basic, which will output into a flat list. The docs have more information on validation results.

Yes, I looked into that, but as you mention I have to combine the values of many different fields which are not always set. Currently I just collect any of the ErrorMessage properties that are not null, I will also have to collect the InstanceLocation property to know where in the document the error is. That is why I thought a simple convenience method would be nice :)

[gregsdennis]

I could not verify documents in parallel using different JsonSchemaOptions configurations

This is my point. Can you give an example where you'd want different processing options per schema? These options are fairly generic, and I'd expect an application would want to process all schemas consistently. There's nothing in a schema that should prompt a change in how it's processed.

The only case I can conceive of is where you're building a web front-end for the validator where the options are user-selectable. (This would be awesome, and I'd love to see it.)

---

The Basic output will have all of the error messages and locations. The way it works is by building the Verbose structure then reducing it once validation completes.

[svenskmand]

I do not have an example where I want different validation error formatting currently, but one example could be to format errors that you emit in logs differently than errors you pass to your users.

But I think a much more important use-case is as I mentioned above that now I have to remember to set JsonSchemaOptions.OutputFormat everywhere I use Schema validation, i.e. it is not enough to set it during program startup, because unit-tests does not use the same initialization code. Another good reason for not having a global singleton static config is to be consistent with the rest of your library, i.e. all other classes I have used, e.g. serializers, etc. carry their own configuration with them and support having different configurations.

These are just my two cents :) Cool library :)

[gregsdennis]

I do have the serialization options as an instance class, and I copy the default options unless one is given to me during that. I've been meaning to move to the same model for schema, but it's likely a breaking change, so I haven't gotten to it yet.

In the meantime, I would suggest getting Verbose results back, then you can reduce the object how you see fit with the .Condense() and .Flatten() methods. These return copies that are reduced so that they don't change the original.

[svenskmand]

I do have the serialization options as an instance class, and I copy the default options unless one is given to me during that. I've been meaning to move to the same model for schema, but it's likely a breaking change, so I haven't gotten to it yet.

I think you should make that change as it will make the library better :)

Maybe you could also add a simple convenience method that can take the validation results and turn them into a human readable list of strings? I.e. contain a pointer/location to where the error occurs and a message saying what the error is? 😃

In the meantime, I would suggest getting Verbose results back, then you can reduce the object how you see fit with the .Condense() and .Flatten() methods. These return copies that are reduced so that they don't change the original.

Thanks that is a good way to support both use-cases in the mean time :)

[gregsdennis]

Yeah, to make instance options happen, I need to move all of the static properties to instance. That's a breaking change. It'd gonna have to wait until the next major version. If you would, please open a new issue for that as it doesn't really pertain to this one.

[gregsdennis]

Maybe you could also add a simple convenience method that can take the validation results and turn them into a human readable list of strings?

Making human-readable strings is not a concern of this library (or most libraries). This library provides results in a well-structured manner from which human-readable strings can be easily derived.

Human readability is an application concern. What works well for your application may not work well for others, and this library can't cater to everyone.

The simplest solution is the one that already exists. I provide the data; you translate/mutate that data for the users.