Interface to call ScraperAPI.com easily from Node. All current API endpoints and features are implemented in this module. Requires Node 8+.
$ npm install --save scraper-api
const scraperAPI = require('scraper-api')({
// ... options
});
scraperAPI.get('http://httpbin.org/ip')
.then(result => {
// result => '<!doctype html> ...'
})
.catch(error => {
console.error(error);
//=> 'Internal server error'
});
scraperAPI(options) ⇒ ScraperAPI
⏏
Creates new instance of ScraperAPI with the provided options.
Kind: Exported function
Returns: ScraperAPI
- New instance of ScraperAPI.
Param | Type | Default | Description |
---|---|---|---|
options | Object |
Optional configuration options to pass into ScraperAPI. | |
[options.apiKey] | String |
process.env.SCRAPER_API_KEY |
API key for Scraper API. Defaults to pulling API Key from environment variable. |
[options.renderJs] | Boolean |
false |
Render JavaScript on the page before scraping the HTML for the page. |
[options.keepHeaders] | Boolean |
false |
Keep headers sent in the request to Scraper API in subsequent request(s) when scraping the provided url. You must set your headers in options.gotOptions.headers . |
[options.geoCode] | String |
'us' |
Geo code in which to use proxies for when scraping. See documentation for more information. |
[options.premium] | Boolean |
false |
Whether to use premium proxies. Caution: This will cost 10-25 times more than standard proxies. |
[options.sessionId] | Number |
A numeric session id to use to maintain the same proxy. See ScraperAPI.session() for more information. | |
[options.gotOptions] | Object |
{} |
Additional options to pass into got for requests to ScraperAPI. |
Example
const scraperAPI = require('scraper-api')({
// options...
});
Access to the uninstantiated ScraperAPI class.
Kind: static property of scraperAPI
Example
const ScraperAPI = require('scraper-api').ScraperAPI;
const scraperAPI = new ScraperAPI({
// options...
});
Kind: global class
- ScraperAPI
- new ScraperAPI([options])
- scraperAPI.session(id, [options]) ⇒
ScraperAPI
- scraperAPI.get(url, [options]) ⇒
Promise.<String>
- scraperAPI.post(url, data, [options]) ⇒
Promise.<Object>
Constructor for the ScraperAPI class.
Param | Type | Default | Description |
---|---|---|---|
[options] | Object |
{} |
Options for the ScraperAPI class. Options may be overridden on all methods for a single request. See options above for more information. |
scraperAPI.session(id, [options]) ⇒ ScraperAPI
Creates a new instance of ScraperAPI with the specified session id. Sessions allow subsequent requests with the same session id to go through the same proxy. See documentation for more information.
Kind: instance method of ScraperAPI
Returns: ScraperAPI
- A new instance of ScraperAPI with sessionId
set to the provided id.
Param | Type | Default | Description |
---|---|---|---|
id | Number |
Session ID in which to use for the given session. Must only contain numbers. | |
[options] | Object |
{} |
Options to override for all subsequent requests to Scraper API. Same as the global options. |
Example
const session = scraperAPI.session(1234);
let result = await scraperAPI.get('https://google.com');
// result -> '<!doctype html> ...'
Calls Scraper API with a GET request to the provided url.
Kind: instance method of ScraperAPI
Returns: Promise.<String>
- Promise that resolves with HTML source from requested URL.
Param | Type | Default | Description |
---|---|---|---|
url | String |
The URL in which to scrape. | |
[options] | Object |
{} |
Options to override for this specific request. |
Example
let result = await scraperAPI.get('https://google.com');
// result -> '<!doctype html> ...'
Calls Scraper API with a POST request to the provided url with the provided data.
Kind: instance method of ScraperAPI
Returns: Promise.<Object>
- Promise that resolves with object response from Scraper API. See documentation.
Param | Type | Default | Description |
---|---|---|---|
url | String |
The URL in which to scrape. | |
data | Object | form-data |
Data in which to post to the provided URL. Must be either a plain object or instance of form-data. | |
[options] | Object |
{} |
Options to override for this specific request. May be any of the global options and any additional options below. |
[options.form] | Boolean |
false |
Set true if provided data is form data and should be sent as such. By default, data will be sent as JSON. |
Example
let result = await scraperAPI.post('https://example.com/endpoint', {
hello: 'world',
some: 'data'
});
If you would like to pass custom headers through Scraper API to the destination, you may do so by setting your custom headers in options.gotOptions.headers
and enabling this option.
let result = await scraperAPI.get('https://google.com', {
keepHeaders: true,
gotOptions: {
headers: {
'My-Custom-Header': 'some value'
}
}
});
Tests are written and provided as part of the module. You may run the tests by calling:
$ npm run test
MIT License. See License in the repository.