Import of entire Twitter stream
muesli opened this issue · 3 comments
Not sure if this is due to some Twitter API limitation, but it'd be nice to import the entire Twitter stream of a user.
yes. that would in fact be nice for all modules. allmost all services allow API queries that would allow such an import. twitter even offers a downloadable archive.
bulk importing raises some questions or rather tasks:
- dealing with query limits like number of returned objects, pagination (ie. next page tokens)
- handling the load, if some thousand tweets are imported they have to be processed. since for example the twitter mod tries to download attached images, there has to be some kind of queueing mechanism
- filtering: optionally getting rid of retweets, mentions, muting hashtags
there are a couple of wordpress plugin doing just that: bulk importing tweets, instagrams, google plus statuses. may be even flickr etc.
A bunch of your points are still valid obviously, but I think we need to split them into a couple of separate issues. There's an argument to be had about building batch-processing / pagination into a generic API instead of re-writing the same pattern for each individual service e.g.
seems to work pretty well. thanks.