Data for vulnerabilityhistory.org
Every push and pull request is run against our integrity checkers on GitHub actions. Click on the above tag to see the status of the build.
Would you like to run the tests locally before pushing to your pull request? Here's what you do:
- You'll need Ruby 2.4+
- Run
gem install bundler(if you don't already have bundler) cdto the root of this repo, runbundle install- Run
rspecfrom the root of the repo to run all the tests. You'll see similar output as you get on GitHub Actions. Thousands of green dots is good.
mkdir tmp
cd tmp
git clone https://chromium.googlesource.com/chromium/src
Note: this will take a while. The Chromium repo is BIG. It may be a good idea to use one of Meneely's research servers for this (ask him if you don't know what those are).
Be in the root of this repository, and run:
rake cve:fixes
cd tmp
git clone https://chromium.googlesource.com/chromium/src
(takes a very long time!)
Be in the root of this repository, and run:
python scripts/get_cves_from_url.py https://chromereleases.googleblog.com/some/blog/post
This will scrape the page source for lines beginning with [$bounty] [bug_id] priority CVE-2017-12345: description. and create a new CVE in cves/ for each match.
Be in the root of this repository, and run:
ruby scripts/add_commit.rb --sha commit_sha_to_add
See the source code for other options.
When you want to make sure that any commit that's mentioned in a YAML is also in the gitlog, you can run this script. It will NOT figure out commits between VCC and Fix, however.
Be in the root of this repository, and run:
ruby scripts/add_mentioned_commits.rb
This will overwrite any commit and take a LONG time (5-10 minutes). If you just want to go quickly and add what's not already there, use:
ruby scripts/add_mentioned_commits.rb --skip-existing
So if a commit is already in gitlog.json then we won't look it up in the GitLog. This is a much faster option.
By default, this script checks the tmp/src directory. If you need, say, v8, there's an option for that.
Use VHP shepherd tools
Release data is scraped from this Wikipedia article
$ cd scripts
$ ruby get_releases.rb
Here's how you merge in student data once the assignment is finished.
- Make sure the current
devbranch is updated and works with the build - Switch
vulnerability-historylocally to pull fromdevinstead ofmaster. - Squash and merge the student pull req into
dev - Run
rails data:chromiumlocally. When it says "Loading data version " and then a git hash, make sure that matches up with the latest merge you just made (so you know you are pulling the latest chromium-vulnerabilities data). Alternatively, you can dorails data:clear data:chromium:load_onlywhich goes quicker. - If all is well, then do any spot-checks of their data to make sure everything got tagged just fine.
- If all is not well:
- You may need to merge their changes with any of your changes. This might be on GitHub itself, or locally.
- You may need to correct their YML structure to make it compatible with the loader. Make the change locally and push back to
devto fix it and re-run. - If things fail on an exception, you can always put in this snippet somewhere to figure out what file failed and use byebug to figure out the problem:
begin
# code where things when wrong
rescue
byebug
end
- You can always do
rails data:chromium:nogitto reload things without hitting GitHub - helpful for quicker debugging.
After we merge in a bunch of PRs, here's a checklist of what needs updating, and in what order:
- Add mentioned commits (default options should be good enough)
- Generate weeklies (skip existing)