versioneye/crawl_r

[refactor] GosearchCrawler should save subpackages into module field and not new Product.

Closed this issue · 1 comments

Lessons learned from CpanCrawler:
If product has many submodules, then it seriously slows down saving data and the same data is duplicated.

TODO:

Save each subpackage as module in the Product model and also in version model as users may remove submodules between each release.

Actually, it doesnt work for Golang, because each submodule may include own set of dependencies;

For example, the etcd includes 2 subpackages client and auth. Gosearch api exposes very different set of dependencies for each of them:

So, i will close this tasks;