Spintest task with wrong format does not seems to return to correct error
TisolaNox opened this issue · 0 comments
I have encounter an error will doing some simple test with the library.
I'm using
python==3.9.7
spintest==0.4.2
I'm testing a dummy API having it running or not does not affect the behavior that I'm encountering.
Here is my python file
from spintest import spintest
from spintest.types import Int
urls = ["http://localhost:8080/"]
task = [{
"method": "POST",
"route": "rollback/",
"output": "rollback_doc",
"body": {"task": "string"},
"expected": {
"code": 201,
"body": {
"task": "string",
"id": None
},
"expected_match": "partial"
},
"ignore": True
}, {
"method": "GET",
"route": "rollback/5",
"expected": {
"code": 200,
"body": {
"task": 'string',
"id": Int("{{ rollback_doc['id']}}"),
"status": "RUNNING"
},
"expected_match": "partial"
},
"fail_on": {
"expected_match": "partial",
"body": {"status": "ERROR"}
},
"rollback": ["delete_rollback"]
}, {
"name": "delete_rollback",
"method": "DELETE",
"route": "rollback/5",
"expected": {"code": 204}
}]
result = spintest(urls, task)
assert True is result
My tasks have 3 steps
- Creating a resource with a POST -> no issue expected (ignore: True so that you don't need to have a running API)
- Retrieving the resource with a wrong status -> Should trigger the rollback but will fail due to parsing issue
- Deleting the resource -> Should always delete
When i'm executing the task i'm getting the following output
ERROR - {
"name": null,
"status": "FAILED",
"timestamp": "Thu Apr 21 10:14:59 2022",
"duration_sec": 0.03,
"url": "http://localhost:8080/",
"route": "rollback/",
"message": "Request failed.",
"code": null,
"body": null,
"task": {
"method": "POST",
"route": "rollback/",
"output": "rollback_doc",
"ignore": true,
"body": {
"task": "string"
},
"expected": {
"code": 201,
"expected_match": "partial",
"body": {
"task": "string",
"id": null
}
},
"retry": 0,
"delay": 1,
"headers": {
"Accept": "application/json",
"Content-Type": "application/json"
},
"duration_sec": 0.03
},
"ignore": true
}
Traceback (most recent call last):
File "/lib/python3.9/site-packages/spintest/manager.py", line 193, in run
results.append(await self._next())
File "/lib/python3.9/site-packages/spintest/manager.py", line 178, in _next
return await self.stack.__anext__()
File "/lib/python3.9/site-packages/spintest/manager.py", line 116, in _executor
result = await Task(
File "/lib/python3.9/site-packages/spintest/task.py", line 174, in run
return self._response(
File "/lib/python3.9/site-packages/spintest/task.py", line 48, in _response
log_level.get(status, logger.critical)(json.dumps(result, indent=4))
File "python/lib/python3.9/json/__init__.py", line 234, in dumps
return cls(
File "python/lib/python3.9/json/encoder.py", line 201, in encode
chunks = list(chunks)
File "python/lib/python3.9/json/encoder.py", line 431, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "python/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "python/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "python/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
[Previous line repeated 1 more time]
File "python/lib/python3.9/json/encoder.py", line 438, in _iterencode
o = _default(o)
File "python/lib/python3.9/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Int is not JSON serializable
On this python file i have noticed that some the 'fail_on' value under task[1]['fail_on'] has the incorrect type.
It should be a list of dict and not directly a dict.
The issue here is that the error raised does not match the issue at end.
By moving the faulty key ('fail_on') to the first task. I'm getting a more coherent error
2022-04-21 10:10:45,564 - ERROR - {
"name": null,
"status": "FAILED",
"timestamp": "Thu Apr 21 10:10:45 2022",
"duration_sec": null,
"url": "http://localhost:8080/",
"route": "rollback/",
"message": "Task must follow this schema : Schema({'method': <class 'str'>, Optional('route'): <class 'str'>, Optional('name'): <class 'str'>, Optional('body'): <class 'dict'>, Optional('headers'): <class 'dict'>, Optional('output'): <class 'str'>, Optional('expected'): {Optional('code'): <class 'int'>, Optional('body'): Or(<class 'dict'>, <class 'str'>), Optional('expected_match'): Or('partial', 'strict')}, Optional('fail_on'): [{Optional('code'): <class 'int'>, Optional('body'): Or(<class 'dict'>, <class 'str'>), Optional('expected_match'): Or('partial', 'strict')}], Optional('retry'): <class 'int'>, Optional('delay'): <class 'int'>, Optional('ignore'): <class 'bool'>, Optional('rollback'): [Or(<class 'str'>, <class 'dict'>)]}).",
"code": null,
"body": null,
"task": {
"method": "POST",
"route": "rollback/",
"output": "rollback_doc",
"body": {
"task": "string"
},
"expected": {
"code": 201,
"body": {
"task": "string",
"id": null
},
"expected_match": "partial"
},
"fail_on": {
"expected_match": "partial",
"body": {
"status": "ERROR"
}
}
},
"ignore": false
}
Traceback (most recent call last):
File "pydevd.py", line 1415, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "test_spin.py", line 158, in <module>
assert True is result
AssertionError
From what i can understand.
the parsing of the second task fails here
.smoke/lib/python3.9/site-packages/spintest/task.py:172
validated_task = input_validator(self.task, TASK_SCHEMA)
Which lead to not parsing the task and skipping the casting the pytest.Int.
The task then failed here
.smoke/lib/python3.9/site-packages/spintest/task.py:48
log_level.get(status, logger.critical)(json.dumps(result, indent=4))
because result still have the "id": Int("{{ rollback_doc['id']}}"), unparsed causing the json.dumps to fail
I don't know the full depth of the parsing order but have noticed that in
.smoke/lib/python3.9/site-packages/spintest/validator.py:41
modifiing the
try:
return input_schema.validate(input)
except SchemaError:
return None
to
try:
return input_schema.validate(input)
except SchemaError:
raise
Raises the correct error schema.SchemaError: Key 'fail_on' error:{'expected_match': 'partial', 'body': {'status': 'ERROR'}} should be instance of 'list'
I don't know the full side effect of doing this change but doing something like this
or
Parsing the task (to remove cast) might be better to avoid dumping incorrect json.