scrapy/queuelib

Question: can I load a queue from previous disk file

peter-wangxu opened this issue · 3 comments

Thanks for this easy to use library,

I can easily store data to file, If process quits, can i load stored data from existing file?

Thanks
Peter

Yes,

from queuelib import FifoDiskQueue

q = FifoDiskQueue("queuefile")

q.push(b'a')
q.push(b'b')
q.push(b'c')

q.close()

q = FifoDiskQueue("queuefile")
q.pop()
'a'
q.pop()
'b'
q.pop()
'c'

If you think about it, the whole point of writing a persistent file to disk is so that it can be retrieved back from disk. Hope that helps.

@peter-wangxu I just saw your persist-queue library and I really like it. Nice work!

@SpicyBits Thanks for your answer.

queuelib needs a explicit close and reopen of the queue in order to remember last position, what's more it's not thread-safe as it mentioned.

That's why I started persist-queue project.

Thanks
Peter