Clipper is a Node.js command line tool that allows you to easily clip content from web pages and convert it to Markdown. It uses Mozilla's Readability library and Turndown under the hood to parse web page content and convert it to Markdown.
Clipper provides a quick and simple way to save bits of content from the web for personal archival or note taking purposes, similar to browser extensions like Evernote Web Clipper or Notion Web Clipper. However, Clipper runs entirely in the terminal so you don't need to install any extensions or sign up for accounts.
HTML | Markdown |
---|---|
npm install -g @philschmid/clipper
note: for crawling you need playwright
and the browser dependencies.
Options:
-i, --input <file>
- Input file (html) to clip content from-u, --url <url>
- URL to clip content from-f, --format <format>
- Output format (markdown, json) (default: markdown)-o, --output <file>
- Output file for clipped content (default: output.md)
Examples:
- Clip content from a URL:
clipper clip -u <url>
- Clip content from a file:
clipper clip -i <file>
Warning
Only use this command if you know what you're doing. Crawling websites can be resource intensive and may cause issues for the website owner. Use at your own risk.
Options:
-u, --url <url>
- URL to crawl-g, --glob <glob>
- Glob pattern to match URLs against-o, --output <file>
- Output file for crawled content (default: dataset.jsonl)
Examples:
Crawl a site and clip all pages:
clipper crawl -u <url>
Results will be saved in dataset.jsonl
file in the current directory.
- Clone the repo
- Run
npm install
- Run
npm run test -- clip -u https://huggingface.co/docs/transformers/index
to test the CLI - Run
npm run test -- crawl -u https://awsdocs-neuron.readthedocs-hosted.com/en/v2.14.1/index.html -g https://awsdocs-neuron.readthedocs-hosted.com/en/v2.14.1/\*\*/\*
to crawl the AWS Neuron docs - Run
npm run build
to build for production - Run
npm install -g .
to symlink the CLI for local testing - Run
clipper clip -u https://huggingface.co/docs/transformers/index
to build for development - Remove the symlink with
npm r clipper -g
Clipper uses the following open source libraries:
- Mozilla Readability - For parsing article content
- Turndown - For converting HTML to Markdown
- Crawlee - For crawling websites
- Apache 2.0
- Remove old build files with
rm -rf dist
- Update the version in
package.json
if minor or major version - Run
npm run build
- Run
npm publish --access public
- Create a new release on GitHub
- Update the version in
package.json
to the next patch version