`pysnc/client.py in _validate_response` fails when not json returned
Closed this issue · 2 comments
bwalchen commented
Good afternoon! 👋
I ran into an interesting error today when querying a table with pysnc: JSONDecodeError: Expecting value: line 1 column 1 (char 0).
Looking at the full error trace (attached at the bottom of this issue) I see that the line that is failing is in pysnc/client.py in _validate_response. It appears as though it is attempting to parse the server response as json but unfortunately the data that is being returned looks like HTML (at bottom of this issue).
Using my dapper debugger, I see that the pertinent lines in this HTML response are http.400 and java.lang.IllegalArgumentException: Request header is too large. It doesn't like the size of my request. If I make my request smaller and send it again it does not error out.
I would recommend modifying the below piece of code to first check whether the returned data is json or not before attempting to parse it. Let me know if you have questions!
pysnc/client.py in _validate_response(self, response)
116 raise AuthenticationException(rjson)
117 if code == 400:
--> 118 rjson = response.json()
119 raise RequestException(rjson)
Repro steps
sys_ids = [<SYS_IDS>] # place 300+ sys ids here
company_sys_ids_joined = ",".join(sys_ids)
gr = client_cpt.GlideRecord('<TABLE_NAME>')
query = 'companyIN' + company_sys_ids_joined # this is a string that contains ~300 sys ids
gr.add_encoded_query(query)
gr.fields = ['company', 'name']
gr.query()HTML server response
<html><head><title>ServiceNow - Error report</title><style><!--h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}--></style> </head><body><h1>HTTP Status 400 – </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception Report</p><p><b>Message</b> <u></u></p><p><b>Description</b> <u>http.400</u></p><p><b>Exception</b> <pre>java.lang.IllegalArgumentException: Request header is too large
org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:726)
org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:464)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:682)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1777)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1082)
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$2.completed(Nio2Endpoint.java:569)
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$2.completed(Nio2Endpoint.java:547)
java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219)
java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.base/java.lang.Thread.run(Thread.java:834)
</pre></p><p><b>Note</b> <u>The full stack trace of the root cause is available in the server logs.</u></p><HR size="1" noshade="noshade"><h3>ServiceNow</h3></body></html>Error trace
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
<ipython-input-25-cdc177364535> in <module>
----> 1 get_company_instances(0)
<ipython-input-24-98142c971225> in get_company_instances(index_start, index_end)
13 gr.add_encoded_query(query)
14 gr.fields = ['company', 'name']
---> 15 gr.query()
16
17 return pd.DataFrame(gr.to_pandas())
~/Documents/projects/sandbox-python/.venv/lib/python3.9/site-packages/pysnc/record.py in query(self)
237 :RequestException: If the transaction is canceled due to execution time
238 """
--> 239 response = self._client._list(self)
240 code = response.status_code
241 if code == 200:
~/Documents/projects/sandbox-python/.venv/lib/python3.9/site-packages/pysnc/client.py in _list(self, record)
125 params=params,
126 headers=dict(Accept="application/json"))
--> 127 self._validate_response(r)
128 return r
129
~/Documents/projects/sandbox-python/.venv/lib/python3.9/site-packages/pysnc/client.py in _validate_response(self, response)
116 raise AuthenticationException(rjson)
117 if code == 400:
--> 118 rjson = response.json()
119 raise RequestException(rjson)
120
~/Documents/projects/sandbox-python/.venv/lib/python3.9/site-packages/requests/models.py in json(self, **kwargs)
898 # used.
899 pass
--> 900 return complexjson.loads(self.text, **kwargs)
901
902 @property
~/.pyenv/versions/3.9.1/lib/python3.9/json/__init__.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
344 parse_int is None and parse_float is None and
345 parse_constant is None and object_pairs_hook is None and not kw):
--> 346 return _default_decoder.decode(s)
347 if cls is None:
348 cls = JSONDecoder
~/.pyenv/versions/3.9.1/lib/python3.9/json/decoder.py in decode(self, s, _w)
335
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
339 if end != len(s):
~/.pyenv/versions/3.9.1/lib/python3.9/json/decoder.py in raw_decode(self, s, idx)
353 obj, end = self.scan_once(s, idx)
354 except StopIteration as err:
--> 355 raise JSONDecodeError("Expecting value", s, err.value) from None
356 return obj, end
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
vetsin commented
Likely forcing tomcat to throw since the URI exceeds RFC max length. Question is if I handle the error more gracefully or transparently assist the user in executing said query.
vetsin commented
resolved this particular issue as part of the 52/batch-api branch -- now extra long urls are supported by POST'ing them via batch api