A simple way to access the webhose.io API from your Python code
import webhose
webhose.config(token=YOUR_API_KEY)
for post in webhose.search("github"):
print(post.title)
To make use of the webhose.io API, you need to obtain a token that would be used on every request. To obtain an API key, create an account at https://webhose.io/auth/signup, and then go into https://webhose.io/dashboard to see your token.
You can install from source:
$ git clone https://github.com/Buzzilla/webhose-python
$ cd webhose-python
$ python setup.py install
To get started, you need to import the library, and set your access token. (Replace YOUR_API_KEY with your actual API key).
>>> import webhose
>>> webhose.config(token=YOUR_API_KEY)
Now you can make a request and inspect the results:
>>> r = webhose.search("foobar")
>>> r.total
62
>>> len(r.posts)
62
>>> r.posts[0].language
'english'
>>> r.posts[0].title
'Putting quotes around dictionary keys in JS'
For your convenience, the Response object is iterable, so you can loop over it and get all the results. The iterator will create additional API requests to fetch additional pages.
>>> total_words = 0
>>> for post in r:
... total_words += len(post.text.split(" "))
...
>>> print(total_words)
56006
Warning: This method can use up your credits if your search has lots of results.
- config(token)
- token - your API key
- search(query, token=None)
- query - the search query, either as a search string, or as a Query object
- token - you can provide the API key directly to the search function if you want
Query object correspond to the advanced search options that appear on https://webhose.io/use
Query object have the following members:
all_terms
- a list of strings, all of which must appear in the resultssome_terms
- a list of strings, some of which must appear in the resultsphrase
- a phrase that must appear verbatim in the resultsexclude
- terms that should not appear in the resultssite_type
- one or more ofdiscussions
,news
,blogs
language
- one or more of language names, in lowercase englishsite
- one or more of site names, top level only (i.e., yahoo.com and not news.yahoo.com)title
- terms that must appear in the titlebody_text
- term that must appear in the body text
Query objects implement the __str__()
method, which shows the resulting search string.
Response objects have the following members:
total
- the total number of posts which match this searchmore
- the number of posts not included in this responseposts
- a list os Post objectsnext
- a URL for the next results page for this searchresponse
- the originalrequests
responseget_next()
- a method to fetch the next page of results. Returns a new Response object
Response objects implement the __iter__()
method, which can be used to loop
over all posts matching the query. (Automatic page fetching)
Post and Thread object contain the actual data returned from the API. Consult https://webhose.io/documentation to find out about their structure.
If you want to make repeated searches, performing an action whenever there are new results, use code like this:
r = webhose.search("skyrim")
while True:
for post in r:
perform_action(post)
time.sleep(300)
r = r.get_next()