indeterminate test in test suite

[Goldziher]

We currently have some indeterminacy in the mocking of anyOf values in our test suite. As a result some of our checks occasionally fail.

[Goldziher]

fixed

[Goldziher]

Not fixed...

[Goldziher]

Stack trace:

tests/test_schema_tester.py:137: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
openapi_tester/schema_tester.py:366: in validate_response
    self.test_schema_section(
openapi_tester/schema_tester.py:247: in test_schema_section
    self.handle_any_of(schema_section=schema_section, data=data, reference=reference, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <openapi_tester.schema_tester.SchemaTester object at 0x7fe07633ed00>
schema_section = {'anyOf': [{'properties': {'brand': {'type': 'string'}, 'leaves': {'minimum': 1, 'type': 'integer'}, 'name': {'type': ...'type': 'string'}, 'num_of_friends': {'maximum': 3, 'type': 'integer'}, ...}, 'required': ['name'], 'type': 'object'}]}
data = {'brand': 'qLUJyfwFVYySnPCaLuQI', 'color': 'JRBBmlJQvozHKZsqfErj', 'id': 8857, 'leaves': 1, ...}
reference = 'init', kwargs = {'case_tester': None, 'ignore_case': None}
combined_sub_schemas = <map object at 0x7fe076211880>
schema = {'properties': {'color': {'type': 'string'}, 'id': {'type': 'integer'}, 'name': {'type': 'string'}, 'num_of_friends': {'maximum': 3, 'type': 'integer'}, ...}, 'required': ['name'], 'type': 'object'}

    def handle_any_of(self, schema_section: dict, data: Any, reference: str, **kwargs: Any):
        any_of: List[Dict[str, Any]] = schema_section.get("anyOf", [])
        combined_sub_schemas = map(
            lambda index: reduce(lambda x, y: combine_sub_schemas([x, y]), any_of[index:]),
            range(len(any_of)),
        )
    
        for schema in [*any_of, *combined_sub_schemas]:
            try:
                self.test_schema_section(schema_section=schema, data=data, reference=f"{reference}.anyOf", **kwargs)
                return
            except DocumentationError:
                continue
>       raise DocumentationError(f"{VALIDATE_ANY_OF_ERROR}\n\nReference: {reference}.anyOf")
E       openapi_tester.exceptions.DocumentationError: Expected data to match one or more of the documented anyOf schema types, but found no matches
E       
E       Reference: init.anyOf

openapi_tester/schema_tester.py:188: DocumentationError

[Goldziher]

{
  'anyOf': [
    {
      'properties': {
        'brand': {
          'type': 'string'
        },
        'leaves': {
          'minimum': 1,
          'type': 'integer'
        },
        'name': {
          'type': ...
          'type': 'string'
        },
        'num_of_friends': {
          'maximum': 3,
          'type': 'integer'
        },
        ...
      },
      'required': [
        'name'
      ],
      'type': 'object'
    }
  ]

data = {'brand': 'qLUJyfwFVYySnPCaLuQI', 'color': 'JRBBmlJQvozHKZsqfErj', 'id': 8857, 'leaves': 1, ...}

Look at the above, it appears to me missing the required field name.

[sondrelg]

Added a quick print here

    def convert_schema(self, schema: Dict[str, Any]) -> Any:
        print("Convert schema:", schema)   #  <-- here
        ...
        if "anyOf" in schema:
            while not sample:
                sample = random.sample(schema["anyOf"], random.randint(1, len(schema["anyOf"])))
            return self.convert_schema(combine_sub_schemas(sample))   # <-- here
        ...

So I guess we would expect it to print the whole schema, then the combined schema.

If you look at https://github.com/snok/drf-openapi-tester/pull/226/checks?check_run_id=1982854602 it actually only prints this on the second print

Convert schema: {'type': 'object', 'required': [], 'properties': {}}

I'll keep looking on this tomorrow if you don't find a solution before then. Seems like there's only a few places we could be going wrong though 🙂