Design and implementation of the vulnerability-lookup
adulau opened this issue · 9 comments
vulnerability-lookup project
vulnerability-lookup is a cve-search rewrite to support the following functionalities. This project will be a new software project under the cve-search organisation.
Functionalities
Core
- Use a key-value store (kvrocks) instead of the old MongoDB back-end to handle the fast lookup of cve.circl.lu (peak ~7000 req per second);
- Import the actual NVD json file with some meta-data but keep the raw JSON import in a key;
- Expose an API compatible with the original cve-search with minimal API such as the two following entry points:
/api/cve/and/browse/. - Allow the import of other vulnerabilities without CVE-ID assigned;
- Import CPE from NVD or from additional sources;
Import
- Allow the import of CVE-ID state (not in the JSON) from https://cve.mitre.org/cve/search_cve_list.html;
- Allow the creation of vulnerability outside CVE-ID allocation;
- Allow the import of GSD source https://github.com/cloudsecurityalliance/gsd-database
- Add CISA known exploited vulnerabilities https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
- Add PyPi advisory https://github.com/pypa/advisory-database/tree/main/vulns
Work in progress on that (branch framework):
- Imports all CVEs individually straight into kvrocks (it takes 15 min and uses ~3G of ram to load the complete dataset)
- Supports updates for individual CVEs by erasing the old one
- API endpoint:
/vulnerability/{vulnerability_id}- Reason: will support any ID, from NVD or not
Maybe starting with a new importer like the GSD source GSD-database would be a good example of a second ID and also how to map the existing CVE with the GSD source too.
New extensions
- Allow to have a web interface similar to the original version cve-search
- Allow to have an admin interface (also expose via the API) to easily create vulnerability with a minimal set of vulnerability fields
- Allow to have a basic public or embargo/non-public vulnerability publishing system
- Improving debug and monitoring on the import process in vulnerability-lookup (e.g. how to monitor if the NVD API is down or giving 500 errors)
Some of the UI is implemented (search/list recent entries). Now let's discuss the system to create a new vulnerability.
This is the form to report an advisory via github:
Should it be similar to that? What is the minimal viable set of settings we want in the form?
Then, how to we identify the advisory before it is assigned a CVE? And do we do that? If yes, one way is to do something similar to github with something like that: GHSA-c647-pxm2-c52w
As found by @adulau , we should use this interface for edit/submit:
https://github.com/Vulnogram/Vulnogram
And push it to vulnerability-lookup instead of CVE for the ones created by our constituants
Open question regarding CVEList: it is more or less a duplicate of the NVD database, and it is not really possible to treat it as a new source.
For now, it will be imported as a meta for the CVE entries.
Potential new source to add - https://www.variotdbs.pl/vulns/