/fero

Sebuah side-project untuk tugas di per-kuli-ah-an

Primary LanguagePythonMIT LicenseMIT

fero

fero (fēro), Sebuah side-project untuk tugas di per-kuliah-an. Fero -> Steel -> Steal -> Scraping got it?.

Untuk apa?

  • Mengambil pranala (URL) dari sebuah portal atau mesin pencari
  • banyak.

Running

  • Dependencies.
flask
flask-bootstrap
beautifulsoup4
  • Quick Run
git clone https://github.com/zeroload/fero
cd fero
FLASK_APP=fero flask run

API

  • Scrapper
    • POST, /api/scrapper/<plugin_name>/fetch, keyword=<keyword>, should be GET
    • GET, ???
  • Auth / ETA SOON (TM)

Plugin system

It's a bit hacky and sketchy, but it works. How? long story short,

  • create a new directory plugins/ and a shiny new __init__.py.
  • load(app) this function will be called upon App initialization. could be useful for
  • fetch(keyword) returns a dict array contains title and url, accept a single string as keyword.
  • multiple keywords handled by calling fetch(keyword) multiple times.
[
	{"title": "string", "url": "valid-url"},
	{"title": "string", "url": "valid-url"}
]

The plugin directory structure should be like this,

.
├── fero
│   ├── ...
│   ├── plugins
│   │   ├── googlesearch
│   │   │   ├── assets
│   │   │   │   └── ...
│   │   │   └── __init__.py
...

TODO

  • Unit Test or something (YEP WE NEED THIS ;D)
  • Walk around for google search bot kicker :/
  • A pagination handler or something in plugins
    • Google, start parameter
    • Yahoo, b parameter
    • Liputan6, lazy-load or something
  • A working UI for the greater good

Contributors

  • Me
  • Myself
  • I