A lightweight node.js module for retrieving and scraping ads from Kijiji.
- Retrieve single ads as JavaScript objects given their URL
- Retrieve the 20 latest ads matching given search criteria
- node.js - evented I/O for the backend
- cheerio - jQuery-like API for the server
- request - Simple HTTP request client
npm install kijiji-scraper
Will call callback
with an object representing the ad at url
.
url
- A Kijiji ad URL.callback(err, ad)
- A callback called after the ad has been scraped. If there is an error,err
will not be null. If everything was successful,ad
will contain the ad's properties. Thead
object is of the form
{
"title": "ad title",
"image": "ad image URL",
"images": ["ad image URL 1", "ad image URL 2", ..., "ad image URL n"],
"desc": "ad description"
"info": [<category-specific keys and values>]
}
The image URL given in image
is the featured image for the ad and will be up to 300x300. The image URLs given in images
are all of the images associated with the ad and each may be up to 1024x1024. If info
contains the key "Date Listed", its corresponding value will be converted to a JavaScript Date
object.
var kijiji = require("kijiji-scraper");
kijiji.scrape("<Kijiji ad URL>", function(err, ad) {
//Use the ad object
});
Will call callback
with an array of detailed ad objects.
prefs
- Contains Kijiji ad search category and location:
{
"locationId": <Kijiji location id>,
"categoryId": <Kijiji ad category id>,
"scrapeInnerAd": true/false (default true)
}
Values for locationId
and categoryId
can be found by performing a search and looking at the POST request parameters or the URL Kijiji redirects to. For example, after setting the location to Ottawa and selecting the "cars & vehicles" category, Kijiji redirects to http://www.kijiji.ca/b-cars-vehicles/ottawa/c27l1700185. The last part of the URL (c27l1700185) is formatted as c[categoryId]l[locationId]. So in this case, categoryId
is 27 and locationId
is 1700185.
By default, the details of each query result are scraped in separate, subsequent requests. To suppress this behavior and return only the data retrieved by the initial query, set the scrapeInnerAd
preference to false
.
params
- Contains Kijiji ad search criteria:
{
"minPrice": 0,
"maxPrice": 100,
"keywords": "keyword string with words separated by a '+'",
"adType": "OFFER"
}
There are many different search parameters, most of which vary by category type. They can be found by using your browser's developer tools and performing a custom search on Kijiji.
callback(err, ads)
- A callback called after Kijiji has been searched. If there is an error,err
will not be null. If everything was successful,ads
will contain detailed ad objects. These are different from the ad objects returned byscrape()
, since this function uses Kijiji's RSS functionality. They contain a key/value mapping for every field inside an ad's<item>
tag in the RSS feed plus aninnerAd
property. This property will contain an object identical to the ad object returned byscrape()
unlessscrapeInnerAd
was given asfalse
, in which case the property will contain an empty object. These more detailed ads are of the form
{
"title": "ad title",
"link": "ad URL",
"description": "ad description",
"pubDate": "date ad was published",
"guid": "ad URL",
"dc:date": "date ad was published",
"innerAd": [regular ad object]
}
var kijiji = require("kijiji-scraper");
kijiji.query(prefs, params, function(err, ads) {
//Use the ads array
});
Will return a string representation of an ad object.
ad
- Either an ad object returned byscrape()
or a detailed ad object from the array returned byquery()
.
var kijiji = require("kijiji-scraper");
kijiji.scrape("<Kijiji ad URL>", function(err, ad) {
console.log(kijiji.parse(ad)); //Converts ad to string
});