Getting Unpacking Error
nikita1610 opened this issue · 16 comments
Hi,
I am getting this error while intialising LIFO Queue.
error('unpack requires a buffer of 4 bytes')
Any ideas or solutions would be appreciated.
Thanks.
Could you provide a minimal, reproducible example?
Attaching the complete stack trace of the error for your reference.
File "multi_process_threading.py", line 102, in on_message_callback
parser_queue = LifoDiskQueue(LIFO_QUEUE_NAME)
File "/root/projects/parser1/parser/venv/lib/python3.7/site-packages/queuelib/queue.py", line 208, in __init__
(self.size,) = struct.unpack(self.SIZE_FORMAT, qsize)
struct.error: unpack requires a buffer of 4 bytes
Thank you, but a minimal, reproducible example would be more useful than a traceback.
Hi,
I am pushing data to queue from one file .Here is code for that.
data = json.loads(body)
parser_queue = LifoDiskQueue(LIFO_QUEUE_NAME)
parser_queue.push(json.dumps(data).encode('utf-8'))
parser_queue.close()
And ,
I am reading data from another file. Here is code for that
parser_queue = LifoDiskQueue(LIFO_QUEUE_NAME)
data = parser_queue.pop()
parser_queue.close()
I cannot reproduce your issue with the steps provided.
>>> import json
>>> from queuelib import LifoDiskQueue
>>> data = {}
>>> LIFO_QUEUE_NAME = "a"
>>> parser_queue = LifoDiskQueue(LIFO_QUEUE_NAME)
>>> parser_queue.push(json.dumps(data).encode('utf-8'))
>>> parser_queue.close()
>>> parser_queue = LifoDiskQueue(LIFO_QUEUE_NAME)I imagine the content of body matters. Please, provide the shortest version possible of body that can be used to reproduce this issue.
HI,
body is a json comprising of three fields:
body = {
url:"https://www.newsbreak.com/news/2817273523981/incumbent-luke-set-to-secure-third-term-as-denton-county-clerk",
website:"newsbreak.com",
html:"
<!DOCTYPE html><html lang="en"><head><meta name="ahrefs-site-verification" content="248e723ef2aed6ca374d0cb8a432ec8988410edcf8f308a5a9228e55fe8fc831"/><meta name="facebook-domain-verification" content="d8l40to31s3ca0n665syyaz01umz3b"/><meta name="ir-site-verification-token" content="1982472"/><meta name="google-site-verification" content="1FmhFigh6Hni9lhQJJChw-yphZjVZe_lKipzRtB2uA8"/><meta name="apple-mobile-web-app-capable" content="yes"/><meta name="format-detection" content="telephone=no"/><meta name="apple-mobile-web-app-status-bar-style" content="black"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><link rel="preconnect" href="https://static.newsbreak.com" crossorigin=""/><link rel="dns-prefetch" href="https://static.newsbreak.com"/><link rel="preconnect" href="https://img.particlenews.com" crossorigin=""/><link rel="dns-prefetch" href="https://img.particlenews.com"/><link rel="shortcut icon" href="#"/><link rel="icon" type="image/x-icon" href="https://static.newsbreak.com/static/favicon.ico"/><link rel="icon" type="image/png" sizes="32x32" href="https://static.newsbreak.com/static/favicon-32x32.png"/><link rel="icon" type="image/png" sizes="16x16" href="https://static.newsbreak.com/static/favicon-16x16.png"/><link rel="manifest" href="https://www.newsbreak.com/site.webmanifest"/><script>
--
| window.dataLayer = window.dataLayer \|\| [];
| // function gtag(){dataLayer.push(arguments);}
| </script><script>
| (function(w,d,s,l,i){w[l]=w[l]\|\|[];w[l].push({'gtm.start':
| new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
| j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
| 'h
<!DOCTYPE html><html lang="en"><head><meta name="ahrefs-site-verification" content="248e723ef2aed6ca374d0cb8a432ec8988410edcf8f308a5a9228e55fe8fc831"/><meta name="facebook-domain-verification" content="d8l40to31s3ca0n665syyaz01umz3b"/><meta name="ir-site-verification-token" content="1982472"/><meta name="google-site-verification" content="1FmhFigh6Hni9lhQJJChw-yphZjVZe_lKipzRtB2uA8"/><meta name="apple-mobile-web-app-capable" content="yes"/><meta name="format-detection" content="telephone=no"/><meta name="apple-mobile-web-app-status-bar-style" content="black"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><link rel="preconnect" href="https://static.newsbreak.com/" crossorigin=""/><link rel="dns-prefetch" href="https://static.newsbreak.com/"/><link rel="preconnect" href="https://img.particlenews.com/" crossorigin=""/><link rel="dns-prefetch" href="https://img.particlenews.com/"/><link rel="shortcut icon" href="[#](https://www.newsbreak.com/news/2817273523981/incumbent-luke-set-to-secure-third-term-as-denton-county-clerk#)"/><link rel="icon" type="image/x-icon" href="https://static.newsbreak.com/static/favicon.ico"/><link rel="icon" type="image/png" sizes="32x32" href="https://static.newsbreak.com/static/favicon-32x32.png"/><link rel="icon" type="image/png" sizes="16x16" href="https://static.newsbreak.com/static/favicon-16x16.png"/><link rel="manifest" href="https://www.newsbreak.com/site.webmanifest"/><script>
window.dataLayer = window.dataLayer || [];
// function gtag(){dataLayer.push(arguments);}
</script><script>
(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
'h"
}
That is neither a valid JSON nor valid Python.
This a valid JSON I am using ,
JSON contains three fields
{
url:"url",
website:"website",
html:"html of url"
}
Please, post the shortest-possible Python code that I can run to reproduce the issue.
I have to files push_to_queue.py and read_from_queue.py . Here is sample code for both of them.
push_to_queue.py
import json
from queuelib import LifoDiskQueue
data = {
"url":"https://lumtest.com/echo.json",
"website":"lumtest.com",
"html":"{"ip":"14.98.95.158","country":"IN","asn":{"asnum":45820,"org_name":"Tata Teleservices ISP AS"},"geo":{"city":"Faridabad","region":"HR","region_name":"Haryana","postal_code":"121003","latitude":28.4098,"longitude":77.31,"tz":"Asia/Kolkata","lum_city":"faridabad","lum_region":"hr"},"method":"GET","httpVersion":"1.1","url":"/echo.json","headers":{"Host":"lumtest.com","X-Real-IP":"14.98.95.158","X-Forwarded-For":"14.98.95.158","X-Forwarded-Proto":"https","X-Forwarded-Host":"lumtest.com","sec-ch-ua":"\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"}}"
}
LIFO_QUEUE_NAME = "a"
parser_queue = LifoDiskQueue(LIFO_QUEUE_NAME)
parser_queue.push(json.dumps(data).encode('utf-8'))
parser_queue.close()
read_from_queue.py
import json
from queuelib import LifoDiskQueue
LIFO_QUEUE_NAME ="a"
while(1):
try:
parser_queue = LifoDiskQueue(LIFO_QUEUE_NAME)
data = parser_queue.pop()
get_data(data)
parser_queue.close()
except Exception as e:
print(e)
The provided code is not valid Python. Please, test the example code you share before you share it.
How come the provided code is not valid python? The code I have shared is well tested.
There is a SyntaxError in push_to_queue.py (unescaped quotes) and in read_from_queue.py (try without except).
Please, test the code that you share here. It is not enough to test your own code, then change it, and then share here the modified version, because they modified version may not work (like in this case), and it may definitely not reproduce the issue.
oh Sorry my bad :").
Missed the exception of that case.
I have edited my above comment only.
The first file still has unescaped quotes. a = {"html": "{"ip": "b"}"} is not valid Python.
Closing until a minimal reproducible example is provided.