# Environment
SNAP_DEBUG=1 - show debug info
SNAP_ENFORCE_RESQUASHFS=0 - don't enforce resquashfs tests
SNAP_ENFORCE_RESQUASHFS_COMP=0 - don't enforce resquashfs compression algorithm
SNAP_FAKEROOT_RESQUASHFS=1 - use fakeroot with resquashfs tests
SNAP_DEBUG_RESQUASHFS=1 - show debug info with failed resquashfs tests
SNAP_DEBUG_RESQUASHFS=2 - drop to a shell with failed resquashfs
SNAP_FORCE_STATE_CHECK=1 - force state checks on disallowed snaps
For snap-updates-available:
RT_SEND_EMAIL=1 - enable sending emails
RT_EMAIL_FROM=<addr> - override configured From address
RT_EMAIL_TO=<addr> - override calculated To addresses
RT_EMAIL_SERVER=<server> - use <server> instead of localhost
RT_EMAIL_NOPROMPT=1 - force sending of email without prompting
For extra tests:
RT_EXTRAS_PATH=/path/to/extras
# Contributing
1. Clone locally (create origin with master branch)
$ git clone git+ssh://<lpid>@git.launchpad.net/review-tools
$ git branch
* master
2. Add personal repo/fork remote in LP
$ git remote add <lpid> git+ssh://<lpid>@git.launchpad.net/~<lpid>/review-tools
3. Branch and submit
$ git fetch
$ git checkout -b mything origin/master
$ git push <lpid> mything
4. Propose for merging via:
https://code.launchpad.net/~<lpid>/review-tools/+git/review-tools/+ref/mything
5. If a committer, once approved can commit to master with:
$ git checkout master
$ git fetch && git pull --ff-only
$ git merge mything
$ git push origin master
$ git branch -D mything
$ git push --delete <lpid> mything # optional
# Tests
Runnable snap v2 tests:
- bin/snap-check-declaration: declaration tests
- bin/snap-check-lint: lint tests
- bin/snap-check-security: security tests
- bin/snap-run-checks: all tests
This gives an alternate view on bin/snap-run-checks:
- bin/snap-review
Running checks locally:
$ make check-deps # install what it tells you
$ PYTHONPATH=$PWD ./bin/snap-review /path/to/package
Importable tests:
- reviewtools/sr_lint.py: lint tests
- reviewtools/sr_security.py: security tests
- ...
In general, add or modify tests and report by using:
self._add_result(<type>, <name>, <message>)
Where <type> is one of 'info', 'warn', 'error'. <name> is the name of the
test (prefixed by <review_type>_), which is set when creating a SnapReview
object. After all tests are run, if there are any errors, the exit status is
'2', if there are no errors but some warnings, the exit status is '1',
otherwise it is '0.
See snap-check-skeleton and sr_skeleton.py for how to create new tests. In
short:
* create a snap-check-<something> and a sr_<something>.py script based off of
the skeleton. IMPORTANT: the new script must be snap-check-<something> so
other tools that use review-tools (eg, ubuntu-sdk) can find them.
* modify snap-check-<something> to use sr_<something>.py
* add tests to sr_<something>.py. If you name the tests 'check_<sometest>'
SnapReview.do_checks() will enumerate and run them automatically
To run tests, just execute:
$ ./run-tests # all tests
$ ./run-tests test_sr_security.py # only security tests
Or to run a specific test:
$ python3 -m unittest reviewtools.tests.<filename>.<class>.<test>
Invocations for existing tests:
* available.py
$ python3 -m unittest reviewtools.tests.test_available.TestAvailable.foo
* common.py
$ python3 -m unittest reviewtools.tests.test_common.TestCommon.bar
* debversion.py
$ python3 -m unittest reviewtools.tests.test_debversion.TestDebVersion.baz
* email.py
$ python3 -m unittest reviewtools.tests.test_email.TestEmail.norf
* modules.py
$ python3 -m unittest reviewtools.tests.test_modules.TestModules.corge
* sr_common.py
$ python3 -m unittest reviewtools.tests.test_sr_common.TestSnapReviewCommon.qux
* sr_declaration.py:
$ python3 -m unittest reviewtools.tests.test_sr_declaration.TestSnapReviewDeclaration.test_all_checks_as_v2
* sr_functional.py:
$ python3 -m unittest reviewtools.tests.test_sr_functional.TestSnapReviewFunctional.test_all_checks_as_v2
$ python3 -m unittest reviewtools.tests.test_sr_functional.TestSnapReviewFunctionalNoMock.quxx
* sr_lint.py
$ python3 -m unittest reviewtools.tests.test_sr_lint.TestSnapReviewLint.test_all_checks_as_v2
$ python3 -m unittest reviewtools.tests.test_sr_lint.TestSnapReviewLintNoMock.blah
* sr_security.py:
$ python3 -m unittest reviewtools.tests.test_sr_security.TestSnapReviewSecurity.test_all_checks_as_v2
$ python3 -m unittest reviewtools.tests.test_sr_security.TestSnapReviewSecurityNoMock.blah2
* store.py
$ python3 -m unittest reviewtools.tests.test_store.TestStore.blah3
* usn.py
$ python3 -m unittest reviewtools.tests.test_usn.TestUSN.blah4
Additional checks are:
$ ./run-flake8
$ ./run-black
$ ./tests/test.sh
$ ./tests/test.sh system # requies 'review-tools' snap to be installed
All tests (except './tests/test.sh system') can be run with:
$ make check
$ make coverage && make coverage-report # run unit tests with coverage
TODO: update for git hook
If you are going to develop the tools regularly, you might want to add a bzr
hook to run the testsuite before committing. Eg, add something like this to
~/.bazaar/plugins/hooks/__init__.py:
#!/usr/bin/python
from bzrlib.branch import Branch
def run_tests_rt(local, master, old_revno, old_revid, new_revno, new_revid,
seven, eight):
#print local, master, old_revno, old_revid, new_revno, new_revid, seven, eight
if 'click-reviewers-tools' in master.base:
import subprocess
print ''
rc = subprocess.call(['./run-tests'])
if rc != 0:
import sys
sys.exit(1)
Branch.hooks.install_named_hook('pre_commit',
run_tests_rt,
'click-reviewers-tools tests')
# Releases
The review-tools historically used tags for released versions and modified
debian/changelog to include what changed for that released version. The last
released version and tag that followed this was 0.48.
Today, the review-tools no longer releases versions and instead adds a git tag
to signify something that is usable for others (eg, a stable snap, the snap
store, etc). The debian/changelog file is still used and the version in it
loosely corresponds to a given tag. Eg, if there are tags for 20190930 and
20191018, then the changelog might look like:
review-tools (0.48+YYYYMMDD-1) UNRELEASED; urgency=medium
* new thing changed since git tag from 20191018...
...
review-tools (0.48+20191018-1) RELEASED; urgency=medium
* thing changed since previous git tag
* other thing changed since previous git tag
...
review-tools (0.48+20190930-1) RELEASED; urgency=medium
...
When it is time to issue a new git tag, the UNRELEASED entry should be updated
to have the date of the git tag and then changed to RELEASED. The general
procedure for creating a new tag is:
1. use 'dch -r' and update debian/changelog to use 0.48-YYYYMMDD-# with
RELEASED (substituting YYYYMMDD for today's date and '#' for '1'
(incrementing if multiple tags per day))
2. git commit debian/changelog && git push
3. Create a tag with: git tag $(date --utc "+%Y%m%d-%H%MUTC")
4. Push the tag with: git push origin $(git tag | tail -1)
These steps will result in a new version of the review-tools snap available in
the --edge channel. Before promoting to --beta, make sure further validations
are run on the new version:
1. Force the review-tools snap refresh with: 'snap refresh review-tools --edge'
2. Connect to $HOME interface if not connected already: 'sudo snap connect review-tools:home'
3. Run the extensive set of available automated tests with 'make check' and
'make functest-system' (the last one tests the installed snap)
4. Run any other manual test as needed
Whenever confident with validations, promote to --beta channel with:
'snapcraft promote --from-channel edge --to-channel beta review-tools'
# Updating reviewtools/data/snapd-base-declaration.yaml
Keeping reviewtools/data/snapd-base-declaration.yaml up to date is important
for certain tests. At a high level, run 'snap debug get-base-declaration' and
then pull those changes into reviewtools/data/snapd-base-declaration.yaml. The
format of snapd-base-declaration.yaml is generally the same as what 'snap debug
get-base-declaration', except for the following:
* there is a preamble (comments) in snapd-base-declaration.yaml
* the 'plugs' and 'slots' in snapd-base-declaration.yaml should be keys in the
"16" dictionary (for historical reasons)
* snapd-base-declaration.yaml should not contain the "dummy" test interface
* snapd-base-declaration.yaml should not contain the newline and '$builtin'
line from the end of 'snap debug get-base-declaration'
In general can do on a system with the current stable release of snapd:
* sudo snap debug base-declaration > /path/bd
* format /path/bd based on the above
* cp /path/bd reviewtools/data/snapd-base-declaration.yaml
* review the change with 'git diff'
(TODO: a tool could mostly automate this)
# Helpful commands
Install the required snaps for the below commands:
$ sudo snap install review-tools
$ sudo snap disconnect review-tools:home # optional, but good for security
To download a snap (and all of its assertions):
$ snap download emoj
Fetching snap "emoj"
Fetching assertions for "emoj"
Install the snap with:
snap ack emoj_53.assert
snap install emoj_53.snap
To perform a review via snap:
$ snap-review $HOME/snap/review-tools/common/path/to/snap
(need not be in SNAP_USER_COMMON if 'home' is connected)
To unpack the snap to 'squashfs-root' in the cwd (files are owned by invoking
user unless using sudo, which isn't recommended):
$ unsquashfs ./the.snap
Find the snap name for a given snap-id:
$ review-tools.store-query --snap-info HjdctDwq9fm9TnLPSvUXgyXeY9zmRsB5 | grep " name:"
name: emoj
Find the snap-id for a given snap name:
$ review-tools.store-query --snap-info emoj | grep 'snap-id'
snap-id: HjdctDwq9fm9TnLPSvUXgyXeY9zmRsB5
Download the snap.yaml for a given snap (and optionally channel and
architecture) from the store (defaults to latest/stable amd64):
$ review-tools.store-query --snap-yaml emoj
apps:
emoj:
command: command-emoj.wrapper
...
$ review-tools.store-query --snap-yaml --channel=17.03/stable --arch=amd64 docker
apps:
compose:
command: command-compose.wrapper
...
Download the snap-declaration for a given snap from the store:
$ review-tools.store-query --snap-decl ufw
...
plugs:
firewall-control:
allow-auto-connection: true
snap-id: Jb8klqgs5djfejP5egB9Za8KYVK686Pe
snap-name: ufw
type: snap-declaration
...
# References
* https://api.snapcraft.io/docs
* https://api.snapcraft.io/docs/info.html
* https://docs.ubuntu.com/core/en/reference/assertions
* https://assertions.ubuntu.com
* https://docs.ubuntu.com/core/en/reference/rest