Split out system tests into separate Kokoro job
tseaver opened this issue · 0 comments
tseaver commented
Working to reduce CI latency. Here are timings on my local machine (note the pre-run with --install-only to avoid measuring virtualenv creation time):
$ for job in $(nox --list | grep "^\*" | cut -d " " -f 2); do
echo $job;
nox -e $job --install-only;
time nox -re $job;
done
nox > Running session lint
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/lint
nox > python -m pip install flake8 black==19.10b0
nox > Skipping black run, as --install-only is set.
nox > Skipping flake8 run, as --install-only is set.
nox > Session lint was successful.
nox > Running session lint
nox > Re-using existing virtual environment at .nox/lint.
nox > python -m pip install flake8 black==19.10b0
nox > black --check docs google tests noxfile.py setup.py
All done! ✨ 🍰 ✨
64 files would be left unchanged.
nox > flake8 google tests
nox > Session lint was successful.
real 0m2.613s
user 0m6.705s
sys 0m0.242s
blacken
nox > Running session blacken
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/blacken
nox > python -m pip install black==19.10b0
nox > Skipping black run, as --install-only is set.
nox > Session blacken was successful.
nox > Running session blacken
nox > Re-using existing virtual environment at .nox/blacken.
nox > python -m pip install black==19.10b0
nox > black docs google tests noxfile.py setup.py
All done! ✨ 🍰 ✨
64 files left unchanged.
nox > Session blacken was successful.
real 0m0.912s
user 0m0.834s
sys 0m0.081s
lint_setup_py
nox > Running session lint_setup_py
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/lint_setup_py
nox > python -m pip install docutils pygments
nox > Skipping python run, as --install-only is set.
nox > Session lint_setup_py was successful.
nox > Running session lint_setup_py
nox > Re-using existing virtual environment at .nox/lint_setup_py.
nox > python -m pip install docutils pygments
nox > python setup.py check --restructuredtext --strict
running check
nox > Session lint_setup_py was successful.
real 0m1.064s
user 0m0.922s
sys 0m0.142s
unit-3.6
nox > Running session unit-3.6
nox > Creating virtual environment (virtualenv) using python3.6 in .nox/unit-3-6
nox > python -m pip install asyncmock pytest-asyncio -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.6.txt
nox > python -m pip install mock pytest pytest-cov -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.6.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.6.txt
nox > Skipping py.test run, as --install-only is set.
nox > Session unit-3.6 was successful.
nox > Running session unit-3.6
nox > Re-using existing virtual environment at .nox/unit-3-6.
nox > python -m pip install asyncmock pytest-asyncio -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.6.txt
nox > python -m pip install mock pytest pytest-cov -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.6.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.6.txt
nox > py.test --quiet --junitxml=unit_3.6_sponge_log.xml --cov=google/cloud --cov=tests/unit --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=0 tests/unit
........................................................................ [ 12%]
........................................................................ [ 24%]
........................................................................ [ 37%]
........................................................................ [ 49%]
........................................................................ [ 62%]
........................................................................ [ 74%]
........................................................................ [ 87%]
........................................................................ [ 99%]
.. [100%]
- generated xml file: /home/tseaver/projects/agendaless/Google/src/python-datastore/unit_3.6_sponge_log.xml -
578 passed in 5.67s
nox > Session unit-3.6 was successful.
real 0m9.988s
user 0m9.432s
sys 0m0.538s
unit-3.7
nox > Running session unit-3.7
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/unit-3-7
nox > python -m pip install asyncmock pytest-asyncio -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.7.txt
nox > python -m pip install mock pytest pytest-cov -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.7.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.7.txt
nox > Skipping py.test run, as --install-only is set.
nox > Session unit-3.7 was successful.
nox > Running session unit-3.7
nox > Re-using existing virtual environment at .nox/unit-3-7.
nox > python -m pip install asyncmock pytest-asyncio -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.7.txt
nox > python -m pip install mock pytest pytest-cov -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.7.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.7.txt
nox > py.test --quiet --junitxml=unit_3.7_sponge_log.xml --cov=google/cloud --cov=tests/unit --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=0 tests/unit
........................................................................ [ 12%]
........................................................................ [ 24%]
........................................................................ [ 37%]
........................................................................ [ 49%]
........................................................................ [ 62%]
........................................................................ [ 74%]
........................................................................ [ 87%]
........................................................................ [ 99%]
.. [100%]
- generated xml file: /home/tseaver/projects/agendaless/Google/src/python-datastore/unit_3.7_sponge_log.xml -
578 passed in 5.54s
nox > Session unit-3.7 was successful.
real 0m9.483s
user 0m8.993s
sys 0m0.482s
unit-3.8
nox > Running session unit-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/unit-3-8
nox > python -m pip install asyncmock pytest-asyncio -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > python -m pip install mock pytest pytest-cov -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > Skipping py.test run, as --install-only is set.
nox > Session unit-3.8 was successful.
nox > Running session unit-3.8
nox > Re-using existing virtual environment at .nox/unit-3-8.
nox > python -m pip install asyncmock pytest-asyncio -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > python -m pip install mock pytest pytest-cov -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > py.test --quiet --junitxml=unit_3.8_sponge_log.xml --cov=google/cloud --cov=tests/unit --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=0 tests/unit
........................................................................ [ 12%]
........................................................................ [ 24%]
........................................................................ [ 37%]
........................................................................ [ 49%]
........................................................................ [ 62%]
........................................................................ [ 74%]
........................................................................ [ 87%]
........................................................................ [ 99%]
.. [100%]
- generated xml file: /home/tseaver/projects/agendaless/Google/src/python-datastore/unit_3.8_sponge_log.xml -
578 passed in 5.29s
nox > Session unit-3.8 was successful.
real 0m8.927s
user 0m8.447s
sys 0m0.473s
unit-3.9
nox > Running session unit-3.9
nox > Creating virtual environment (virtualenv) using python3.9 in .nox/unit-3-9
nox > python -m pip install asyncmock pytest-asyncio -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.9.txt
nox > python -m pip install mock pytest pytest-cov -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.9.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.9.txt
nox > Skipping py.test run, as --install-only is set.
nox > Session unit-3.9 was successful.
nox > Running session unit-3.9
nox > Re-using existing virtual environment at .nox/unit-3-9.
nox > python -m pip install asyncmock pytest-asyncio -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.9.txt
nox > python -m pip install mock pytest pytest-cov -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.9.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.9.txt
nox > py.test --quiet --junitxml=unit_3.9_sponge_log.xml --cov=google/cloud --cov=tests/unit --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=0 tests/unit
........................................................................ [ 12%]
........................................................................ [ 24%]
........................................................................ [ 37%]
........................................................................ [ 49%]
........................................................................ [ 62%]
........................................................................ [ 74%]
........................................................................ [ 87%]
........................................................................ [ 99%]
.. [100%]
- generated xml file: /home/tseaver/projects/agendaless/Google/src/python-datastore/unit_3.9_sponge_log.xml -
578 passed in 5.72s
nox > Session unit-3.9 was successful.
real 0m9.258s
user 0m8.727s
sys 0m0.521s
system-3.8(disable_grpc=False)
nox > Running session system-3.8(disable_grpc=False)
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/system-3-8-disable_grpc-false
nox > python -m pip install --pre grpcio
nox > python -m pip install mock pytest google-cloud-testutils -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > Skipping py.test run, as --install-only is set.
nox > Session system-3.8(disable_grpc=False) was successful.
nox > Running session system-3.8(disable_grpc=False)
nox > Re-using existing virtual environment at .nox/system-3-8-disable_grpc-false.
nox > python -m pip install --pre grpcio
nox > python -m pip install mock pytest google-cloud-testutils -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > py.test --quiet --junitxml=system_3.8_sponge_log.xml tests/system
............................... [100%]
- generated xml file: /home/tseaver/projects/agendaless/Google/src/python-datastore/system_3.8_sponge_log.xml -
31 passed in 80.59s (0:01:20)
nox > Session system-3.8(disable_grpc=False) was successful.
real 1m23.987s
user 0m11.917s
sys 0m2.127s
system-3.8(disable_grpc=True)
nox > Running session system-3.8(disable_grpc=True)
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/system-3-8-disable_grpc-true
nox > python -m pip install --pre grpcio
nox > python -m pip install mock pytest google-cloud-testutils -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > Skipping py.test run, as --install-only is set.
nox > Session system-3.8(disable_grpc=True) was successful.
nox > Running session system-3.8(disable_grpc=True)
nox > Re-using existing virtual environment at .nox/system-3-8-disable_grpc-true.
nox > python -m pip install --pre grpcio
nox > python -m pip install mock pytest google-cloud-testutils -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > python -m pip install -e . -c /home/tseaver/projects/agendaless/Google/src/python-datastore/testing/constraints-3.8.txt
nox > py.test --quiet --junitxml=system_3.8_sponge_log.xml tests/system
............................... [100%]
- generated xml file: /home/tseaver/projects/agendaless/Google/src/python-datastore/system_3.8_sponge_log.xml -
31 passed in 63.67s (0:01:03)
nox > Session system-3.8(disable_grpc=True) was successful.
real 1m7.158s
user 0m11.968s
sys 0m0.528s
cover
nox > Running session cover
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/cover
nox > python -m pip install coverage pytest-cov
nox > Skipping coverage run, as --install-only is set.
nox > Skipping coverage run, as --install-only is set.
nox > Session cover was successful.
nox > Running session cover
nox > Re-using existing virtual environment at .nox/cover.
nox > python -m pip install coverage pytest-cov
nox > coverage report --show-missing --fail-under=100
Name Stmts Miss Branch BrPart Cover Missing
---------------------------------------------------------------------------------------------------------------------------------
google/cloud/datastore/__init__.py 9 0 0 0 100%
google/cloud/datastore/_app_engine_key_pb2.py 23 0 2 0 100%
google/cloud/datastore/_gapic.py 15 0 2 0 100%
google/cloud/datastore/_http.py 72 0 12 0 100%
google/cloud/datastore/batch.py 106 0 32 0 100%
google/cloud/datastore/client.py 230 0 118 0 100%
google/cloud/datastore/entity.py 23 0 6 0 100%
google/cloud/datastore/helpers.py 196 0 130 0 100%
google/cloud/datastore/key.py 202 0 76 0 100%
google/cloud/datastore/query.py 205 0 64 0 100%
google/cloud/datastore/transaction.py 51 0 10 0 100%
google/cloud/datastore/version.py 1 0 0 0 100%
google/cloud/datastore_admin_v1/services/__init__.py 0 0 0 0 100%
google/cloud/datastore_admin_v1/services/datastore_admin/__init__.py 3 0 0 0 100%
google/cloud/datastore_admin_v1/services/datastore_admin/async_client.py 90 0 20 0 100%
google/cloud/datastore_admin_v1/services/datastore_admin/client.py 178 0 58 0 100%
google/cloud/datastore_admin_v1/services/datastore_admin/pagers.py 42 0 10 0 100%
google/cloud/datastore_admin_v1/services/datastore_admin/transports/__init__.py 9 0 0 0 100%
google/cloud/datastore_admin_v1/services/datastore_admin/transports/base.py 46 0 8 0 100%
google/cloud/datastore_admin_v1/services/datastore_admin/transports/grpc.py 71 0 20 0 100%
google/cloud/datastore_admin_v1/services/datastore_admin/transports/grpc_asyncio.py 74 0 20 0 100%
google/cloud/datastore_admin_v1/types/__init__.py 3 0 0 0 100%
google/cloud/datastore_admin_v1/types/datastore_admin.py 74 0 0 0 100%
google/cloud/datastore_admin_v1/types/index.py 27 0 0 0 100%
google/cloud/datastore_v1/services/__init__.py 0 0 0 0 100%
google/cloud/datastore_v1/services/datastore/__init__.py 3 0 0 0 100%
google/cloud/datastore_v1/services/datastore/async_client.py 123 0 40 0 100%
google/cloud/datastore_v1/services/datastore/client.py 214 0 84 0 100%
google/cloud/datastore_v1/services/datastore/transports/__init__.py 9 0 0 0 100%
google/cloud/datastore_v1/services/datastore/transports/base.py 49 0 8 0 100%
google/cloud/datastore_v1/services/datastore/transports/grpc.py 78 0 24 0 100%
google/cloud/datastore_v1/services/datastore/transports/grpc_asyncio.py 81 0 24 0 100%
google/cloud/datastore_v1/types/__init__.py 4 0 0 0 100%
google/cloud/datastore_v1/types/datastore.py 76 0 0 0 100%
google/cloud/datastore_v1/types/entity.py 35 0 0 0 100%
google/cloud/datastore_v1/types/query.py 80 0 0 0 100%
tests/unit/__init__.py 0 0 0 0 100%
tests/unit/gapic/datastore_admin_v1/__init__.py 0 0 0 0 100%
tests/unit/gapic/datastore_admin_v1/test_datastore_admin.py 560 0 20 0 100%
tests/unit/gapic/datastore_v1/__init__.py 0 0 0 0 100%
tests/unit/gapic/datastore_v1/test_datastore.py 691 0 6 0 100%
tests/unit/test__gapic.py 32 0 2 0 100%
tests/unit/test__http.py 522 0 58 0 100%
tests/unit/test_batch.py 377 0 12 0 100%
tests/unit/test_client.py 1025 0 54 0 100%
tests/unit/test_entity.py 167 0 0 0 100%
tests/unit/test_helpers.py 741 0 14 0 100%
tests/unit/test_key.py 535 0 4 0 100%
tests/unit/test_query.py 583 0 24 0 100%
tests/unit/test_transaction.py 276 0 6 0 100%
---------------------------------------------------------------------------------------------------------------------------------
TOTAL 8011 0 968 0 100%
nox > coverage erase
nox > Session cover was successful.
real 0m2.396s
user 0m2.273s
sys 0m0.125s
docs
nox > Running session docs
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/docs
nox > python -m pip install -e .
nox > python -m pip install sphinx==4.0.1 alabaster recommonmark
nox > Skipping sphinx-build run, as --install-only is set.
nox > Session docs was successful.
nox > Running session docs
nox > Re-using existing virtual environment at .nox/docs.
nox > python -m pip install -e .
nox > python -m pip install sphinx==4.0.1 alabaster recommonmark
nox > sphinx-build -W -T -N -b html -d docs/_build/doctrees/ docs/ docs/_build/html/
Running Sphinx v4.0.1
making output directory... done
[autosummary] generating autosummary for: README.rst, UPGRADING.md, admin_client.rst, batches.rst, changelog.md, client.rst, entities.rst, helpers.rst, index.rst, keys.rst, queries.rst, transactions.rst
loading intersphinx inventory from https://python.readthedocs.org/en/latest/objects.inv...
loading intersphinx inventory from https://googleapis.dev/python/google-auth/latest/objects.inv...
loading intersphinx inventory from https://googleapis.dev/python/google-api-core/latest/objects.inv...
loading intersphinx inventory from https://grpc.github.io/grpc/python/objects.inv...
loading intersphinx inventory from https://proto-plus-python.readthedocs.io/en/latest/objects.inv...
loading intersphinx inventory from https://googleapis.dev/python/protobuf/latest/objects.inv...
intersphinx inventory has moved: https://python.readthedocs.org/en/latest/objects.inv -> https://python.readthedocs.io/en/latest/objects.inv
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 12 source files that are out of date
updating environment: [new config] 12 added, 0 changed, 0 removed
reading sources... [ 8%] README
reading sources... [ 16%] UPGRADING
/home/tseaver/projects/agendaless/Google/src/python-datastore/.nox/docs/lib/python3.8/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
reading sources... [ 25%] admin_client
reading sources... [ 33%] batches
reading sources... [ 41%] changelog
/home/tseaver/projects/agendaless/Google/src/python-datastore/.nox/docs/lib/python3.8/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
reading sources... [ 50%] client
reading sources... [ 58%] entities
reading sources... [ 66%] helpers
reading sources... [ 75%] index
reading sources... [ 83%] keys
reading sources... [ 91%] queries
reading sources... [100%] transactions
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 8%] README
writing output... [ 16%] UPGRADING
writing output... [ 25%] admin_client
writing output... [ 33%] batches
writing output... [ 41%] changelog
writing output... [ 50%] client
writing output... [ 58%] entities
writing output... [ 66%] helpers
writing output... [ 75%] index
writing output... [ 83%] keys
writing output... [ 91%] queries
writing output... [100%] transactions
generating indices... genindex py-modindex done
highlighting module code... [ 12%] google.cloud.datastore.batch
highlighting module code... [ 25%] google.cloud.datastore.client
highlighting module code... [ 37%] google.cloud.datastore.entity
highlighting module code... [ 50%] google.cloud.datastore.helpers
highlighting module code... [ 62%] google.cloud.datastore.key
highlighting module code... [ 75%] google.cloud.datastore.query
highlighting module code... [ 87%] google.cloud.datastore.transaction
highlighting module code... [100%] google.cloud.datastore_admin_v1.services.datastore_admin.client
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.
The HTML pages are in docs/_build/html.
nox > Session docs was successful.
real 0m7.417s
user 0m6.868s
sys 0m0.355sGiven that the system tests run twice, in ~ 1.5 minutes each, ISTM it would be good to break them out into a separate Kokoro job, running in parallel with the other test.
This change will require updates to the google3 internal configuration for Kokoro, similar to those @tswast made to enable them for googleapis/python-bigtable#390.