A web page article parser which returns an object containing the article's formatted text and other attributes including sentiment, keyphrases, people, places, organisations, spelling suggestions, in-article links, meta data & lighthouse audit results.
Node.js & NPM
npm install horseman-article-parser --save
| Param | Type | Description |
|---|---|---|
| options | Object |
the options object |
| socket | Object |
the optional socket |
Returns: Object - article parser results object
var parser = require('horseman-article-parser');
var options = {
url: "https://www.theguardian.com/politics/2018/sep/24/theresa-may-calls-for-immigration-based-on-skills-and-wealth",
enabled: ['lighthouse', 'screenshot', 'links', 'sentiment', 'entities', 'spelling', 'keywords']
}
parser.parseArticle(options)
.then(function (article) {
var response = {
title: article.title.text,
excerpt: article.excerpt,
metadescription: article.meta.description.text,
url: article.url,
sentiment: { score: article.sentiment.score, comparative: article.sentiment.comparative },
keyphrases: article.processed.keyphrases,
keywords: article.processed.keywords,
people: article.people,
orgs: article.orgs,
places: article.places,
text: {
raw: article.processed.text.raw,
formatted: article.processed.text.formatted,
html: article.processed.text.html
},
spelling: article.spelling,
meta: article.meta,
links: article.links,
lighthouse: article.lighthouse
}
console.log(response);
})
.catch(function (error) {
console.log(error.message)
console.log(error.stack);
})
parseArticle(options, <socket>) accepts an optional socket for pipeing the response object, status messages and errors to a front end UI.
See horseman-article-parser-ui as an example.
The options below are set by default
var options = {
// puppeteer options (https://github.com/GoogleChrome/puppeteer)
puppeteer: {
// puppeteer launch options (https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions)
launch: {
headless: true,
defaultViewport: null
},
// puppeteer goto options (https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagegotourl-options)
goto: {
waitUntil: 'domcontentloaded'
},
// Ignore content security policy
setBypassCSP: true
},
// clean-html options (https://ghub.io/clean-html)
cleanhtml: {
'add-remove-tags': ['blockquote', 'span'],
'remove-empty-tags': ['span'],
'replace-nbsp': true
},
// html-to-text options (https://ghub.io/html-to-text)
htmltotext: {
wordwrap: 100,
noLinkBrackets: true,
ignoreHref: true,
tables: true,
uppercaseHeadings: true
},
// retext-keywords options (https://ghub.io/retext-keywords)
retextkeywords: { maximum: 10 }
}
At a minimum you should pass a url
var options = {
url: "https://www.theguardian.com/politics/2018/sep/24/theresa-may-calls-for-immigration-based-on-skills-and-wealth"
}
If you want to enable the advanced features you should pass the following
var options = {
url: "https://www.theguardian.com/politics/2018/sep/24/theresa-may-calls-for-immigration-based-on-skills-and-wealth",
enabled: ['lighthouse', 'screenshot', 'links', 'sentiment', 'entities', 'spelling', 'keywords']
}
You may pass rules for returning an articles title & contents. This is useful in a case where the parser is unable to return the desired title or content e.g.
rules: [
{
host: 'www.bbc.co.uk',
content: () => {
var j = window.$
j('article section, article figure, article header').remove()
return j('article').html()
}
},
{
host: 'www.youtube.com',
title: () => {
return window.ytInitialData.contents.twoColumnWatchNextResults.results.results.contents[0].videoPrimaryInfoRenderer.title.runs[0].text
},
content: () => {
return window.ytInitialData.contents.twoColumnWatchNextResults.results.results.contents[1].videoSecondaryInfoRenderer.description.runs[0].text
}
}
]
If you want to pass cookies to puppeteer use the following
var options = {
puppeteer: {
cookies: [{ name: 'cookie1', value: 'val1', domain: '.domain1' },{ name: 'cookie2', value: 'val2', domain: '.domain2' }]
}
}
To strip tags before processing use the following
var options = {
striptags: ['.something', '#somethingelse']
}
If you need to dismiss any popups e.g. a privacy popup use the following
var options = {
clickelements: ['#button1', '#button2']
}
there are some additional "complex" options available
var options = {
// array of html elements to stip before analysis
striptags: [],
// array of resource types to block e.g. ['image' ]
blockedResourceTypes: [],
// array of resource source names (all resources from
// these sources are skipped) e.g. [ 'google', 'facebook' ]
skippedResources: [],
// readability options (https://ghub.io/node-readability)
readability: {},
// retext spell options (https://ghub.io/retext-spell)
retextspell: {}
// compromise nlp options
nlp: { plugins: [ myPlugin, anotherPlugin ] }
}
Compromise is the natural language processor that allows horseman-article-parser to return
topics e.g. people, places & organisations. You can now pass custom plugins to compromise to modify or add to the word lists like so:
/** add some names
let testPlugin = function(Doc, world) {
world.addWords({
'rishi': 'FirstName',
'sunak': 'LastName',
})
}
const options = {
url: 'https://www.theguardian.com/commentisfree/2020/jul/08/the-guardian-view-on-rishi-sunak-right-words-right-focus-wrong-policies',
enabled: ['lighthouse', 'screenshot', 'links', 'sentiment', 'entities', 'spelling', 'keywords'],
nlp: {
plugins: [testPlugin]
}
}
This allows us to match - for example - names which are not in the base compromise word lists.
Check out the compromise plugin docs for more info.
Please feel free to fork the repo or open pull requests to the development branch. I've used eslint for linting.
Build the dependencies with:
npm install
Lint the project files with:
npm run lint
Test the package with:
npm run test
Update API docs with:
npm run docs
- Puppeteer: High-level API to control Chrome or Chromium over the DevTools Protocol
- compromise: natural language processing in the browser
- retext: Natural language processor powered by plugins
- retext-pos: Plugin to add part-of-speech (POS) tags
- retext-keywords: Keyword extraction with Retext
- retext-spell: Spelling checker for retext
- sentiment: AFINN-based sentiment analysis for Node.js
- jquery: JavaScript library for DOM operations
- jsdom: A JavaScript implementation of many web standards
- lodash: Lodash modular utilities.
- absolutify: Relative to Absolute URL Replacer
- clean-html: HTML cleaner and beautifier
- dictionary-en-gb: English (United Kingdom) spelling dictionary in UTF-8
- html-to-text: Advanced html to plain text converter
- nlcst-to-string: Stringify NLCST
- vfile-reporter-json: JSON reporter for virtual files
- eslint: An AST-based pattern checker for JavaScript.
- eslint-config-standard: JavaScript Standard Style - ESLint Shareable Config
- eslint-plugin-import: Import with sanity.
- eslint-plugin-json: Lint JSON files
- eslint-plugin-node: Additional ESLint's rules for Node.js
- eslint-plugin-promise: Enforce best practices for JavaScript promises
- eslint-plugin-standard: ESlint Plugin for the Standard Linter
- jsdoc-to-markdown: Generates markdown API documentation from jsdoc annotated source code.
This project is licensed under the GNU GENERAL PUBLIC LICENSE Version 3 - see the LICENSE file for details
Due to node-readability being stale I have imported the relevent functions into this project and refactored it so it doesn't use request and therfor has no vulnrabilities.