package should expose simple site generator configuration options
vinaymavi opened this issue · 2 comments
Feature request
It would be good if we expose all most all simple site generator configuration https://github.com/simplecrawler/simplecrawler#configuration that would help to control Crawler more.
The CLI does currently not provide all configuration options the core package has to offer. Since now no one missed it and I don't want to bloat the CLI with features no one uses.
The core package on the other hand supports all options the simplecrawler has to offer. If you have a more complicated use case you should use this instead of relying on the CLI.
Closing this, feel free to comment if you disagree.
Totally agree @lgraubner with you. all feature exposing is not really a good idea. I am thinking exposing only two options
- maxConcurrency
- respectRobotsTxt
can provide more flexibility to users as we have an existing issue #25 .