barrust/mediawiki

Support pagination (continue)

tbm opened this issue · 3 comments

tbm commented

.search("a", results=1000) will lead to 500 results since 500 is the maximum. However, a continue parameter is supported ("When more results are available, use this to continue"), which I suppose allows for paging through more results.

It would be nice if this was somehow supported (maybe as an iterator that does another query when needed).

barrust commented

This is definitely something that could be included. The library does something similar to this for the categorymembers() function so it could be added to this and, I would assume, other functions.

Another place we do this is in mediawikipage.py in the _continued_query() function. So it would likely be combining the wiki_request() and _continued_query() functions in some way.

I don't know how much time I have at the moment to make this change, but any support is welcome.

shay314 commented

Hey, is there a plan to support this feature?
Can be very useful.

Thanks,

barrust commented

I would love to add it but I haven't had time to implement anything yet. Not sure when time will become available. Perhaps over the holidays but no guarantees.

Any help would be appreciated.