tv4.validate not finding problems

Question

tv4.validate not finding problems

Closed this issue 7 years ago · 23 comments

App Details:

Postman for Windows
Version 5.0.0
win32 10.0.10586 / x64

Hi - over the last few days, I've begun experimenting with Tiny Validator snippet functionality in the Postman test facility. This doesn't SEEM to be working, in that it doesn't find problems/bugs that I introduce into the schema

Here is a JSON response from one of our REST APIs (which, right now, doesn't show much information, and that is intentional in this public setting):

{
"_embedded": {
"clients": []
},
"_links": {
"self": {
"href": "https://10.7.104.144/elg/rest-api/clients"
},
"profile": {
"href": "https://10.7.104.144/elg/rest-api/profile/clients"
}
},
"page": {
"size": 20,
"totalElements": 0,
"totalPages": 0,
"number": 0
}
}

I copied this JSON data to https://jsonschema.net/#/editor, and generated the following JSON Schema:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"definitions": {},
"id": "http://example.com/example.json",
"properties": {
"_embedded": {
"id": "/properties/_embedded",
"properties": {
"clients": {
"id": "/properties/_embedded/properties/clients",
"items": {},
"type": "array"
}
},
"type": "object"
},
"_links": {
"id": "/properties/_links",
"properties": {
"profile": {
"id": "/properties/_links/properties/profile",
"properties": {
"href": {
"id": "/properties/_links/properties/profile/properties/href",
"type": "string"
}
},
"type": "object"
},
"self": {
"id": "/properties/_links/properties/self",
"properties": {
"href": {
"id": "/properties/_links/properties/self/properties/href",
"type": "string"
}
},
"type": "object"
}
},
"type": "object"
},
"page": {
"id": "/properties/page",
"properties": {
"number": {
"id": "/properties/page/properties/number",
"type": "integer"
},
"size": {
"id": "/properties/page/properties/size",
"type": "integer"
},
"totalElements": {
"id": "/properties/page/properties/totalElements",
"type": "integer"
},
"totalPages": {
"id": "/properties/page/properties/totalPages",
"type": "integer"
}
},
"type": "object"
}
},
"type": "object"
}

I used this schema to prepare this Postman test:
var jsonData = JSON.parse(responseBody);
var schema = {
"$schema": "http://json-schema.org/draft-04/schema#",
"definitions": {},
"id": "http://example.com/example.json",
"properties": {
"_embedded": {
"id": "/properties/_embedded",
"properties": {
"clients": {
"id": "/properties/_embedded/properties/clients",
"items": {},
"type": "array"
}
},
"type": "object"
},
"_links": {
"id": "/properties/_links",
"properties": {
"profile": {
"id": "/properties/_links/properties/profile",
"properties": {
"href": {
"id": "/properties/_links/properties/profile/properties/href",
"type": "string"
}
},
"type": "object"
},
"self": {
"id": "/properties/_links/properties/self",
"properties": {
"href": {
"id": "/properties/_links/properties/self/properties/href",
"type": "string"
}
},
"type": "object"
}
},
"type": "object"
},
"page": {
"id": "/properties/page",
"properties": {
"number": {
"id": "/properties/page/properties/number",
"type": "integer"
},
"size": {
"id": "/properties/page/properties/size",
"type": "integer"
},
"totalElements": {
"id": "/properties/page/properties/totalElements",
"type": "integer"
},
"totalPages": {
"id": "/properties/page/properties/totalPages",
"type": "integer"
}
},
"type": "object"
}
},
"type": "object"
};

tests["Validate JSON return"] = tv4.validate(jsonData, schema);
console.log("JSON validation failed: ", tv4.error);

The problem that I see is, if I modify the schema to induce a problem, tv4 doesn't detect this. For instance, in the schema, if I change "page" to "XXpage" and execute the test, tv4.validate still returns null (=success).

I'm doing this testing to ensure that, if my JSON output ever changes, the differences between it and the schema will be detected.

Is the tv4 code wrong, or is my schema wrong, or something else?

Answer 1 · 2017-06-19T12:29:04.000Z

Hi @tlemons, we use the tv4 library as-is. Can you try testing your code over at https://npm.runkit.com/tv4 and see if you're still seeing this problem?

Answer 2 · 2017-06-19T16:54:21.000Z

Hi - thanks very much for the reply. And thanks for the pointer to runkit. I tried it there, and see the same problem; if I purposely cause a mismatch between the schema and the json data, tv4 does not detect it. I'm using the JSON schema that was generated from my JSON data at https://jsonschema.net/#/editor; have there been any reports of problems with using JSON schema generated by that site?

Thanks!
tl

Answer 3 · 2017-06-20T08:40:11.000Z

Can you also create an issue here: https://github.com/geraintluff/tv4 since this looks like a TV4 issue?
I'm closing this, but please let us know here when this gets fixed there and we'll bump up our version to include the fix :)

Answer 4 · 2017-06-27T19:57:46.000Z

As you suggested, I opened a TV4 bug at geraintluff/tv4#257.

Answer 5 · 2017-06-30T17:10:32.000Z

I have not received an response to the TV4 bug that I posted, and I see dozens of issues at https://github.com/geraintluff/tv4/issues, some of which have been open for years.

I really want to do JSON return body checking in Postman, and like to idea of using the JSON Schema for this. If TV4 doesn't work and isn't supported, have you considered supporting a different JSON Schema-compliant checker in Postman?

Thanks
tl

Answer 6 · 2017-11-13T04:44:16.000Z

Hi,
I am facing a similar issue:
my response schema is

const resSchema =
{
"$schema": "http://json-schema.org/draft-04/schema#",
"required": ["T1","T2"] ,
"properties":
{
"topicId": { "type": "integer" },
"topicNamesssss": { "type": "string" },
"topicNote": { "type": "string" },
"grade": { "type": "integer", },
}

};

My response is :
{
"topicId": "1",
"topicName": "2",
"topicNote": "Area: Concept & informal units",

}

pm.expect(tv4.validate(response1, resSchema)).to.be.true; is always returning true.

Even if i am passing my response as "", tv4.validate is returning true.

I validated the same in a JSONvalidator and that correctly shows me the discrepancies between the schema and response. PFA screenshot for it.

Any help would be appreciated.

Thanks,
Sanam

Answer 7 · 2017-11-28T19:22:13.000Z

I have tried with the banUnknownProperties (https://github.com/geraintluff/tv4#the-banunknownproperties-flag) and it works.

Just do:

tv4.validate(JsonData, schema, false, true);

Answer 8 · 2017-12-20T05:12:14.000Z

is this issue fixed?

Answer 9 · 2017-12-25T20:53:51.000Z

Yes, with the additional parameters it works properly.

Answer 10 · 2018-03-15T19:01:49.000Z

It works with additional parameter but only if the variable is defined in testscript and not when it is get from environment variables.
This WORKS:
pm.test('User schema is valid', function() { pm.expect(tv4.validate(allData, user_schema, false, true)).to.be.true});
This Does NOT WORKS:
pm.test('User schema is valid', function() { pm.expect(tv4.validate(allData, pm.environment.get("user_schema"), false, true)).to.be.true});
Any suggestions to fix it???

Answer 11 · 2018-03-17T21:02:24.000Z

I did a simple test and it worked. In prerequest scripts:

const schema = {
    type:"object",
    properties:{
        "ip": {
            "type":"string"
        }
    }
};

pm.environment.set("SCHEMA", schema);

then in tests:

pm.test("schema", function() {
    pm.expect(tv4.validate(pm.response.json(), pm.environment.get("SCHEMA"), false, true), tv4.error).to.be.true;
})

in a Get request to https://postman-echo.com/ip

Try this way, if not work can you show your scripts?

Answer 12 · 2018-03-19T14:23:06.000Z

@rmlira There is still one difference (they way you did works for me too), but I did not set the variable using pm.environment.set, I defined the variable itself in "Manage Environment" and then it does not work, can you try that.
Just remove your set statement and set the variable first manually in your "Manage env" section.

Also noticed one more thing that, even if I set it in one request and try to test this in another request using "get" it does not work (always pass and not fail on valid failure)

Tried the same scenario with "Get request https://postman-echo.com/ip", same issue still

Answer 13 · 2018-03-19T23:39:30.000Z

Hi @mverma-va . I see your point, sorry, I had not understood before. I made this test:

On Postman > Manage Environments > Add a new environment > create in this new environment a variable 'schema' with the content:

schema:{type:"object",properties:{"ip": {"type":"string"}}}

(bulk edit)

Then in request, on Tests I remove the prerequest scripts and keep the schema test as follows:

pm.test("schema", function() {
    pm.expect(tv4.validate(pm.response.json(), pm.environment.get("schema"), false, true), tv4.error).to.be.true;
})

And it doesn't work properly, I mean it pass the test but if I change the 'ip' type to 'number' in environment it still passing the test when it should not.

I noticed that when you store schema in environment using 'pm.environment.set' it stores as an object ([Object object]), when you store it manually it will be stored as a string. So to make it works properly I change the schema stored in environment to:

schema:{"type":"object","properties":{"ip":{"type":"string"}}}
(bulk edit)
just adding a few quotation marks...

Then in test script i made a JSON parse from this stored schema as follows:

pm.test("schema", function() {
    pm.expect(tv4.validate(pm.response.json(), JSON.parse(pm.environment.get('schema')), false, true), tv4.error).to.be.true;
})

Converting from string to proper object.

Now it works properly. Can you check if this solves your problem? If not can you post your schema too?

Answer 14 · 2018-03-20T18:59:34.000Z

Hi @rmlira That worked for me properly. My variable already had quotes, it is basically JSON.parse() which solved the problem, thanks a lot!

Answer 15 · 2018-06-26T00:17:30.000Z

@mverma-va - could you please help , i get below error while running my scripts after last changes you have mentioned.
schema | JSONError: Unexpected token u in JSON at position 0

pm.test("schema", function()
{
pm.expect(tv4.validate(pm.response.json(), JSON.parse(pm.environment.get('schema')), false, true),tv4.error).to.be.true;
})

Answer 16 · 2018-06-26T12:24:11.000Z

@rmlira - could you please help , i get below error while running my scripts after last changes you have mentioned.
schema | JSONError: Unexpected token u in JSON at position 0

pm.test("schema", function()
{
pm.expect(tv4.validate(pm.response.json(), JSON.parse(pm.environment.get('schema')), false, true),tv4.error).to.be.true;
})

Answer 17 · 2018-06-26T14:43:41.000Z

@radauto not really sure, maybe you are missing some quotes or braces like "{" in your schema json object

Answer 18 · 2018-06-26T14:51:54.000Z

@mverma-va Thanks for your immediate response , i did some changes and now i get
schema | AssertionError: ValidationError: Invalid type: array (expected object): expected false to be true

Defined schema in pre request script
2.my test has
pm.test("schema", function() {
pm.expect(tv4.validate(pm.response.json(), pm.environment.get("SCHEMA"), false, true), tv4.error).to.be.true;
})

if i send you sample schema will you be able to help ?

Answer 19 · 2018-06-26T15:17:56.000Z

@radauto you can use these two tools to 1.create and 2.validate your schema.

Answer 20 · 2018-06-26T20:47:18.000Z

@mverma-va Thank you. i have corrected and its working fine.
but , my console shows as
Schema Validation | AssertionError: expected false to be true , when my schema is not matching response which is correct.

But is there a way , we can get actual Variable which is wrong as per schema ?

Answer 21 · 2019-05-14T17:50:58.000Z

@mverma-va @radauto My test is working fine and getting proper message.
Schema Validation | AssertionError: expected false to be true ,
Is there a way to capture exact object or property which is missing or incorrect. could you please share if you found the solution.

Answer 22 · 2019-05-14T18:03:34.000Z

@nsharma0101 tv4 is deprecated in favor of Ajv.
Refer #3300 (comment) for debugging schema validation errors using Ajv.

Answer 23 · 2019-05-14T18:39:25.000Z

@nsharma0101
you can check tv4 documentation here https://geraintluff.github.io/tv4/.

The method tv4.validate returns false or true if validation is succeeded or not and stores and object with error details which you can access calling tv4.error. Or you can use tv4.validateMultiple instead of tv4.validate. The result of this call will be an object with 'errors' property containing detailed causes of failure. The advantage is that you will have all schema errors reported in one call, not only the first error as returned by tv4.validate.

example:

let schemaValidation = tv4.validateMultiple(pm.response.json(), JSON_SCHEMA, false, true);

Now schemaValidation variable grabs result of validation with all errors that could be found.

To validate postman test you can do:

pm.expect(schemaValidation.valid, schemaValidation.errors).to.be.true;

in case of error all messages in schemaValidation.errors will be listed as result of pm.expect call.

Anyway now will be better if you migrate to Ajv as suggested by @codenirvana