0.4.2 test failures: tests/integration/test_validators.py::TestOAS{30,31}Validator::test_format_checkers
mgorny opened this issue · 1 comments
mgorny commented
When running the test suite in 0.4.2, I get the following failures:
========================================================= test session starts =========================================================
platform linux -- Python 3.10.9, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /tmp/openapi-schema-validator/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/openapi-schema-validator, configfile: pyproject.toml
plugins: flake8-1.1.0, cov-3.0.0
collected 58 items
tests/integration/test_validators.py::TestOAS30Validator::test_format_checkers FAILED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_null[boolean] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_null[array] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_null[integer] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_null[number] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_null[string] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_untyped[True] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_untyped[False] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable[boolean] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable[array] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable[integer] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable[number] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable[string] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_enum_without_none PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_enum_with_none PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_string_format_byte_valid[c3RyaW5n0] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_string_format_byte_valid[c3RyaW5n1] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_string_format_byte_invalid[string0] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_string_format_byte_invalid[string1] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_allof_required PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_required PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_read_only PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_write_only PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_required_read_only PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_required_write_only PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_oneof_required PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_oneof_discriminator[oneOf] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_oneof_discriminator[anyOf] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_oneof_discriminator[allOf] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_ref[True] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_ref[False] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[True-oneOf-None is not valid under any of the given schemas] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[True-anyOf-None is not valid under any of the given schemas] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[True-allOf-None for not nullable] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[False-oneOf-None is not valid under any of the given schemas] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[False-anyOf-None is not valid under any of the given schemas] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[False-allOf-None for not nullable] PASSED
tests/integration/test_validators.py::TestOAS31Validator::test_format_checkers FAILED
tests/integration/test_validators.py::TestOAS30ReadWriteValidatorValidate::test_read_only PASSED
tests/integration/test_validators.py::TestOAS30ReadWriteValidatorValidate::test_write_only PASSED
tests/integration/test_validators.py::TestOAS30ReadWriteValidatorValidate::test_required_read_only PASSED
tests/integration/test_validators.py::TestOAS30ReadWriteValidatorValidate::test_required_write_only PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_null[boolean] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_null[array] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_null[integer] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_null[number] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_null[string] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_nullable[boolean] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_nullable[array] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_nullable[integer] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_nullable[number] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_nullable[string] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_schema_validation PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_schema_ref PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_array_prefixitems[value0] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_array_prefixitems[value1] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_array_prefixitems_invalid[value0] PASSED
tests/unit/test_shortcut.py::ValidateTest::test_validate_does_not_mutate_schema_adding_nullable_key PASSED
============================================================== FAILURES ===============================================================
_______________________________________________ TestOAS30Validator.test_format_checkers _______________________________________________
self = <test_validators.TestOAS30Validator object at 0x7f0996aea470>
def test_format_checkers(self):
> assert set(OAS30Validator.FORMAT_CHECKER.checkers.keys()) == set([
# standard formats
"int32", "int64", "float", "double", "byte", "binary",
"date", "date-time", "password",
# extra formats
"uuid", "regex",
"ipv4", "ipv6", "email", "idn-email", "time"
])
E AssertionError: assert {'binary',\n 'byte',\n 'date',\n 'double',\n 'email',\n 'float',\n 'idn-email',\n 'idn-hostname',\n 'int32',\n 'int64',\n 'ipv4',\n 'ipv6',\n 'password',\n 'regex',\n 'time',\n 'uuid'} == {'binary',\n 'byte',\n 'date',\n 'date-time',\n 'double',\n 'email',\n 'float',\n 'idn-email',\n 'int32',\n 'int64',\n 'ipv4',\n 'ipv6',\n 'password',\n 'regex',\n 'time',\n 'uuid'}
E Extra items in the left set:
E 'idn-hostname'
E Extra items in the right set:
E 'date-time'
E Full diff:
E {
E 'binary',
E 'byte',
E 'date',
E - 'date-time',
E 'double',
E 'email',
E 'float',
E 'idn-email',
E + 'idn-hostname',
E 'int32',
E 'int64',
E 'ipv4',
E 'ipv6',
E 'password',
E 'regex',
E 'time',
E 'uuid',
E }
self = <test_validators.TestOAS30Validator object at 0x7f0996aea470>
tests/integration/test_validators.py:17: AssertionError
_______________________________________________ TestOAS31Validator.test_format_checkers _______________________________________________
self = <test_validators.TestOAS31Validator object at 0x7f09969c2920>
def test_format_checkers(self):
> assert set(OAS31Validator.FORMAT_CHECKER.checkers.keys()) == set([
# standard formats
"int32", "int64", "float", "double", "password",
# extra formats
"date", "date-time", "uuid", "regex",
"ipv4", "ipv6", "email", "idn-email", "time"
])
E AssertionError: assert {'date',\n 'double',\n 'email',\n 'float',\n 'idn-email',\n 'idn-hostname',\n 'int32',\n 'int64',\n 'ipv4',\n 'ipv6',\n 'password',\n 'regex',\n 'time',\n 'uuid'} == {'date',\n 'date-time',\n 'double',\n 'email',\n 'float',\n 'idn-email',\n 'int32',\n 'int64',\n 'ipv4',\n 'ipv6',\n 'password',\n 'regex',\n 'time',\n 'uuid'}
E Extra items in the left set:
E 'idn-hostname'
E Extra items in the right set:
E 'date-time'
E Full diff:
E {
E 'date',
E - 'date-time',
E 'double',
E 'email',
E 'float',
E 'idn-email',
E + 'idn-hostname',
E 'int32',
E 'int64',
E 'ipv4',
E 'ipv6',
E 'password',
E 'regex',
E 'time',
E 'uuid',
E }
self = <test_validators.TestOAS31Validator object at 0x7f09969c2920>
tests/integration/test_validators.py:474: AssertionError
--------------------------------- generated xml file: /tmp/openapi-schema-validator/reports/junit.xml ---------------------------------
---------- coverage: platform linux, python 3.10.9-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
-------------------------------------------------------------------------------------
openapi_schema_validator/__init__.py 13 0 0 0 100%
openapi_schema_validator/_format.py 41 7 6 1 79% 19, 25, 29, 36, 43-46
openapi_schema_validator/_types.py 12 0 0 0 100%
openapi_schema_validator/_validators.py 125 25 72 10 73% 37, 47-53, 62, 131, 133->exit, 146-150, 160, 184, 186->185, 188->195, 205, 207->206, 209->213, 222-237
openapi_schema_validator/shortcuts.py 13 1 2 1 87% 23
openapi_schema_validator/validators.py 37 0 6 0 100%
-------------------------------------------------------------------------------------
TOTAL 241 33 86 12 80%
Coverage XML written to file reports/coverage.xml
======================================================= short test summary info =======================================================
FAILED tests/integration/test_validators.py::TestOAS30Validator::test_format_checkers - AssertionError: assert {'binary',\n 'byte',\...
FAILED tests/integration/test_validators.py::TestOAS31Validator::test_format_checkers - AssertionError: assert {'date',\n 'double',\...
==================================================== 2 failed, 56 passed in 0.55s =====================================================I can reproduce it reliably in a venv via:
git clone https://github.com/p1c2u/openapi-schema-validator/
cd openapi-schema-validator/
git checkout 0.4.2
poetry install
pytest -vvOn a production system, I get even more mismatches:
_______________________________________________ TestOAS30Validator.test_format_checkers _______________________________________________
self = <test_validators.TestOAS30Validator object at 0x7f6ee63cceb0>
def test_format_checkers(self):
> assert set(OAS30Validator.FORMAT_CHECKER.checkers.keys()) == set([
# standard formats
"int32", "int64", "float", "double", "byte", "binary",
"date", "date-time", "password",
# extra formats
"uuid", "regex",
"ipv4", "ipv6", "email", "idn-email", "time"
])
E AssertionError: assert {'color', 'byte', 'idn-hostname', 'email', 'hostname', 'float', 'ipv6', 'idn-email', 'time', 'double', 'uri-template', 'password', 'uri-reference', 'duration', 'regex', 'uri', 'date', 'ipv4', 'int32', 'iri-reference', 'date-time', 'relative-json-pointer', 'iri', 'binary', 'uuid', 'json-pointer', 'int64'} == {'byte', 'time', 'ipv6', 'date', 'regex', 'double', 'email', 'float', 'int32', 'binary', 'password', 'uuid', 'ipv4', 'date-time', 'in
t64', 'idn-email'}
E Extra items in the left set:
E 'color'
E 'uri'
E 'relative-json-pointer'
E 'idn-hostname'
E 'hostname'
E 'iri'
E 'uri-template'
E 'uri-reference'
E 'iri-reference'
E 'duration'
E 'json-pointer'
E Full diff:
E {
E 'binary',
E 'byte',
E + 'color',
E 'date',
E 'date-time',
E 'double',
E + 'duration',
E 'email',
E 'float',
E + 'hostname',
E 'idn-email',
E + 'idn-hostname',
E 'int32',
E 'int64',
E 'ipv4',
E 'ipv6',
E + 'iri',
E + 'iri-reference',
E + 'json-pointer',
E 'password',
E 'regex',
E + 'relative-json-pointer',
E 'time',
E + 'uri',
E + 'uri-reference',
E + 'uri-template',
E 'uuid',
E }
self = <test_validators.TestOAS30Validator object at 0x7f6ee63cceb0>
tests/integration/test_validators.py:17: AssertionError
_______________________________________________ TestOAS31Validator.test_format_checkers _______________________________________________
self = <test_validators.TestOAS31Validator object at 0x7f6ee642d6f0>
def test_format_checkers(self):
> assert set(OAS31Validator.FORMAT_CHECKER.checkers.keys()) == set([
# standard formats
"int32", "int64", "float", "double", "password",
# extra formats
"date", "date-time", "uuid", "regex",
"ipv4", "ipv6", "email", "idn-email", "time"
])
E AssertionError: assert {'color', 'idn-hostname', 'email', 'hostname', 'float', 'ipv6', 'idn-email', 'time', 'double', 'uri-template', 'password', 'uri-reference', 'duration', 'regex', 'uri', 'date', 'ipv4', 'int32', 'iri-reference', 'date-time', 'relative-json-pointer', 'iri', 'uuid', 'json-pointer', 'int64'} == {'time', 'ipv6', 'date', 'regex', 'double', 'email', 'float', 'password', 'int32', 'uuid', 'ipv4', 'date-time', 'int64', 'idn-email'}
E Extra items in the left set:
E 'color'
E 'uri'
E 'relative-json-pointer'
E 'idn-hostname'
E 'hostname'
E 'iri'
E 'uri-template'
E 'uri-reference'
E 'iri-reference'
E 'duration'
E 'json-pointer'
E Full diff:
E {
E + 'color',
E 'date',
E 'date-time',
E 'double',
E + 'duration',
E 'email',
E 'float',
E + 'hostname',
E 'idn-email',
E + 'idn-hostname',
E 'int32',
E 'int64',
E 'ipv4',
E 'ipv6',
E + 'iri',
E + 'iri-reference',
E + 'json-pointer',
E 'password',
E 'regex',
E + 'relative-json-pointer',
E 'time',
E + 'uri',
E + 'uri-reference',
E + 'uri-template',
E 'uuid',
E }
self = <test_validators.TestOAS31Validator object at 0x7f6ee642d6f0>
tests/integration/test_validators.py:474: AssertionError
-- generated xml file: /tmp/portage/dev-python/openapi-schema-validator-0.4.2/work/openapi-schema-validator-0.4.2/reports/junit.xml ---
======================================================= short test summary info =======================================================p1c2u commented
Looks like test environment has more packages installed than just required ones. I will make a fix for that.