Readiz/keyword-crawler

This is keyword crawler for node.js with phantomjs-prebuilt.

keyword crawler

This is keyword crawler for node.js with phantomjs-prebuilt.
This crawler fit to infinite scroll page, not pagination page.
grpah for crawled and morpheme parsing contents

installation

npm i

morpheme parser(korean)

• mecab-ko
• mecab-ko-dic

develop

first, run task runner(gulp)

1. eslint
2. babel

npm run dev

If you want to run another task,
Let's edit npm script in package.json or
run below command. (npm v5.2.0 or higher)

npx gulp TASK_NAME

second, write your crawler code

1. crawler code (/src/crawlContents.js)
2. crawled content's morpheme parsing code(/src/morphemeParser.js)
3. count morpheme word code(/src/countKeyword.js)

If you don't know how to write upper codes, Let's see code in demo directory. (It is korean recruitments site sample.)

third, run crawler(phantomjs-prebuilt)

npm start

If you want to run another crawler,
Let's edit npm script in package.json or
run below command. (npm v5.2.0 or higher)

npx phantomjs ANOTHER_CRAWLER.js

etc. run crawler demo

npm test