Failed to push cache: timeout waiting for push
geekflyer opened this issue · 6 comments
geekflyer commented
I'm trying to run makisu on our own Kubernetes cluster via via https://cirrus-ci.com.
While I somewhat got it working, every 5-th build or so get's stuck for a long a time (i.e. about 10 minutes) at a step like this:
1.5643073208686275e+09 info * Pushing with cache ID eb427ee8
1.5643073208691173e+09 info * Moving directories [] to /makisu-storage/sandbox/sandbox746334368/stages/MA==
After waiting for 10 minutes the build then usually prints this:
1.5643079235246627e+09 error Failed to push cache: timeout waiting for push
and then continues and successfully completes.
For context:
- We're using GCR
- We're using the GCR credential helper
- We're using HTTP cache
My suspicion is that this a problem that occurs in conjunction with the GCR credential helper.
Makisu version is: v0.1.11.
Registry config looks like this:
{
"gcr.io": {
"myproject-*/*": {
"push_chunk": -1,
"security": {
"credsStore": "gcr"
}
}
}
}
Makisu is invoked with this command:
- /makisu-internal/makisu build
--modifyfs
--log-fmt=console
--http-cache-addr http://$CIRRUS_HTTP_CACHE_HOST/
--registry-config="${REGISTRY_CONFIG}"
--commit=explicit
--push=gcr.io
--tag=myproject/$APP:$CIRRUS_CHANGE_IN_REPO
--replica=gcr.io/myproject/$APP:$CIRRUS_CHANGE_IN_REPO
--file=$CIRRUS_WORKING_DIR/Dockerfile
$CIRRUS_WORKING_DIR
yiranwang52 commented
Do you know how many layers your images have?
geekflyer commented
24 layers (includes the base layers). With commit=explicit it got down to about 16, but the error still occassionally occurs. The image is based off FROM node:10 but I also got this error a few times with another image which is based FROM python:3.6.6 and has a similar amount of layers.
evelynl94 commented
Do you know how big are the layers? Have you tried increasing the concurrency in registry configuration?
geekflyer commented
The layers are quite big but not huge. Altogether image A is compressed about 400Mb. The other one is even bigger, probably around 1Gig compressed.
How would increasing the concurrency help?
I'm pretty sure it's not a network / bandwidth issue. We're running the builds inside GKE and push it to GCR. Also, the builds in which this issue doesn't occur usually complete in less than 2 minutes, incl. pushing the layers (as mentioned this issue only occurs on every 5-th build or so).
My guess is honestly that somehow makisu swallows an error (maybe from the credential helper or maybe because of some invalidated token?) and that's why the timeout occurs. I also could imagine that it's somehow related to http cache. Unfortunately the error logs are not very useful with regards to what went wrong. I'll see if I can create a repro of this with a public repo once I got some time.
evelynl94 commented
The registry client is kinda verbose and would fail if credHelper returns an error but default timeout for pushing a layer is also 10 mins (which is too long). Could you provide us the entire build log? Maybe we missed something...
Also (probably unlikely), is it possible that you got rate limited by GCR? https://cloud.google.com/container-registry/quotas
geekflyer commented
Sure,
there you go:
cd resolve
/makisu-internal/makisu build --modifyfs --log-fmt=console --http-cache-addr http://$CIRRUS_HTTP_CACHE_HOST --registry-config="${REGISTRY_CONFIG}" --push=gcr.io --tag=my-project/$APP-makisu:$CIRRUS_CHANGE_IN_REPO --file=./Dockerfile .
1.564305915197019e+09 info Starting Makisu build (version=v0.1.11-10-g1b1102a)
1.5643059152263603e+09 info Using build context: /tmp/cirrus-ci-build/resolve
1.564305915227681e+09 info Using http server at http://127.0.0.1:12321 for cacheID storage
1.5643059154017467e+09 error Failed to fetch intermediate layer with cache ID ec8aa8fa: find layer ec8aa8fa: layer not found in cache
1.5643059154018242e+09 info * Stage 1/1 : (alias=0,latestfetched=-1)
1.56430591540184e+09 info * Step 1/14 (commit,modifyfs) : FROM python:3.6.6 (446839ca)
1.5643059154020095e+09 info * Started pulling image index.docker.io/library/python:3.6.6
1.5643059164474647e+09 info * Started pulling layer index.docker.io/library/python:sha256:bc9ab73e5b14b9fbd3687a4d8c1f1360533d6ee9ffc3f5ecc6630794b40257b7
1.564305916644417e+09 info * Started pulling layer index.docker.io/library/python:sha256:e5c3f8c317dc30af45021092a3d76f16ba7aa1ee5f18fec742c84d4960818580
1.5643059166470797e+09 info * Started pulling layer index.docker.io/library/python:sha256:193a6306c92af328dbd41bbbd3200a2c90802624cccfe5725223324428110d7f
1.5643059170155766e+09 info * Finished pulling layer library/python:e5c3f8c317dc30af45021092a3d76f16ba7aa1ee5f18fec742c84d4960818580
1.564305917079696e+09 info * Started pulling layer index.docker.io/library/python:sha256:a587a86c9dcb9df6584180042becf21e36ecd8b460a761711227b4b06889a005
1.5643059171277025e+09 info * Finished pulling layer library/python:193a6306c92af328dbd41bbbd3200a2c90802624cccfe5725223324428110d7f
1.5643059171895814e+09 info * Finished pulling layer library/python:bc9ab73e5b14b9fbd3687a4d8c1f1360533d6ee9ffc3f5ecc6630794b40257b7
1.5643059171962411e+09 info * Started pulling layer index.docker.io/library/python:sha256:72744d0a318b0788001cc4f5f83c6847ba4b753307fadd046b508bbc41eb9e29
1.5643059174475782e+09 info * Started pulling layer index.docker.io/library/python:sha256:6598fc9d11d10365ac9281071a87930a2382ee31d026f1b6d432717b31db387c
1.5643059177758055e+09 info * Finished pulling layer library/python:6598fc9d11d10365ac9281071a87930a2382ee31d026f1b6d432717b31db387c
1.5643059178413568e+09 info * Started pulling layer index.docker.io/library/python:sha256:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f
1.5643059178802233e+09 info * Finished pulling layer library/python:a587a86c9dcb9df6584180042becf21e36ecd8b460a761711227b4b06889a005
1.564305917945844e+09 info * Started pulling layer index.docker.io/library/python:sha256:93612f47cdc374d0b33057b9e71eac173ac469da3e1a631dc8a32ba6986a408a
1.5643059183229275e+09 info * Finished pulling layer library/python:93612f47cdc374d0b33057b9e71eac173ac469da3e1a631dc8a32ba6986a408a
1.5643059183920033e+09 info * Started pulling layer index.docker.io/library/python:sha256:1bc4b4b508703799ef67a807dacce4736045e642e87bcd49871cd0f23e7f5b8b
1.564305918423843e+09 info * Finished pulling layer library/python:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f
1.5643059184957218e+09 info * Started pulling layer index.docker.io/library/python:sha256:8256ec07b2ad5dbbf324da48b498658f2fa47ea4ce859cb35d926c0234b486cb
1.564305918810201e+09 info * Finished pulling layer library/python:1bc4b4b508703799ef67a807dacce4736045e642e87bcd49871cd0f23e7f5b8b
1.5643059188727944e+09 info * Finished pulling layer library/python:8256ec07b2ad5dbbf324da48b498658f2fa47ea4ce859cb35d926c0234b486cb
1.5643059205010023e+09 info * Finished pulling layer library/python:72744d0a318b0788001cc4f5f83c6847ba4b753307fadd046b508bbc41eb9e29
1.5643059205020914e+09 info * Finished pulling image index.docker.io/library/python:3.6.6 in 5.100004363s
1.564305920506579e+09 info * Processing FROM layer bc9ab73e5b14b9fbd3687a4d8c1f1360533d6ee9ffc3f5ecc6630794b40257b7
1.5643059219145367e+09 info * Untarred 7001 files to / in 1.408s
1.564305921918238e+09 info * Merged 7001 headers from tar to memfs
1.5643059219233994e+09 info * Processing FROM layer 193a6306c92af328dbd41bbbd3200a2c90802624cccfe5725223324428110d7f
1.5643059222076356e+09 info * Untarred 1304 files to / in 284ms
1.5643059222085664e+09 info * Merged 1303 headers from tar to memfs
1.5643059222108777e+09 info * Processing FROM layer e5c3f8c317dc30af45021092a3d76f16ba7aa1ee5f18fec742c84d4960818580
1.5643059223093393e+09 info * Untarred 267 files to / in 98ms
1.564305922309646e+09 info * Merged 262 headers from tar to memfs
1.564305922310478e+09 info * Processing FROM layer a587a86c9dcb9df6584180042becf21e36ecd8b460a761711227b4b06889a005
1.5643059239247441e+09 info * Untarred 6292 files to / in 1.614s
1.564305923928906e+09 info * Merged 6288 headers from tar to memfs
1.5643059239316592e+09 info * Processing FROM layer 72744d0a318b0788001cc4f5f83c6847ba4b753307fadd046b508bbc41eb9e29
1.5643059304517498e+09 info * Untarred 20547 files to / in 6.52s
1.5643059304594743e+09 info * Merged 20545 headers from tar to memfs
1.5643059304603536e+09 info * Processing FROM layer 6598fc9d11d10365ac9281071a87930a2382ee31d026f1b6d432717b31db387c
1.5643059306866426e+09 info * Untarred 672 files to / in 226ms
1.5643059306872816e+09 info * Merged 667 headers from tar to memfs
1.5643059306881106e+09 info * Processing FROM layer 4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f
1.5643059312738402e+09 info * Untarred 1104 files to / in 586ms
1.5643059312742963e+09 info * Merged 1104 headers from tar to memfs
1.5643059312751646e+09 info * Processing FROM layer 93612f47cdc374d0b33057b9e71eac173ac469da3e1a631dc8a32ba6986a408a
1.5643059312756255e+09 info * Untarred 7 files to / in 0s
1.5643059312756627e+09 info * Merged 7 headers from tar to memfs
1.5643059312765462e+09 info * Processing FROM layer 1bc4b4b508703799ef67a807dacce4736045e642e87bcd49871cd0f23e7f5b8b
1.564305931374282e+09 info * Untarred 596 files to / in 98ms
1.5643059313747387e+09 info * Merged 596 headers from tar to memfs
1.5643059313747761e+09 info * Execute FROM python:3.6.6 (446839ca) took 15.972880244s
1.5643059313752885e+09 info * Step 2/14 (commit,modifyfs) : MAINTAINER John Doe <john@sdoe.com> (ec8aa8fa)
1.564305931375339e+09 info * Execute MAINTAINER John Doe <john@sdoe.com> (ec8aa8fa) took 1.041µs
1.564305931375345e+09 info * Pushing with cache ID ec8aa8fa
1.564305931375978e+09 info * Step 3/14 (commit,modifyfs) : RUN mkdir -p /usr/src/resolve && mkdir -p /usr/src/resolve/models && pip install --upgrade pip (944aa9ef)
1.564305932638471e+09 info Collecting pip
1.5643059327380922e+09 info Downloading https://files.pythonhosted.org/packages/62/ca/94d32a6516ed197a491d17d46595ce58a83cbb2fca280414e57cd86b84dc/pip-19.2.1-py2.py3-none-any.whl (1.4MB)
1.564305932872864e+09 info Installing collected packages: pip
1.5643059328729243e+09 info Found existing installation: pip 18.1
1.5643059329885137e+09 info Uninstalling pip-18.1:
1.5643059331684113e+09 info Successfully uninstalled pip-18.1
1.564305933981709e+09 info Successfully installed pip-19.2.1
1.5643059341407273e+09 info * Execute RUN mkdir -p /usr/src/resolve && mkdir -p /usr/src/resolve/models && pip install --upgrade pip (944aa9ef) took 2.76471766s
1.5643059390795004e+09 info * Collecting filesystem diff
1.564305939913849e+09 info * Finished collecting diff in 834ms: 1025 files found
1.5643059401578593e+09 info * Created layer by scanning filesystem; 1025 files found
1.5643059401671553e+09 info * Committed gzipped layer sha256:d3fc069215c7ba6059cf21b0b463c00c996710b66fe4fee5339b20b83b6778d0 (8472575 bytes)
1.564305940167195e+09 info * Pushing with cache ID 944aa9ef
1.5643059401678967e+09 info * Step 4/14 (commit,modifyfs) : RUN pip install gunicorn==19.9.0 (1d7c7f3f)
1.5643059405633643e+09 info Collecting gunicorn==19.9.0
1.5643059407111268e+09 info Downloading https://files.pythonhosted.org/packages/8c/da/b8dd8deb741bff556db53902d4706774c8e1e67265f69528c14c003644e6/gunicorn-19.9.0-py2.py3-none-any.whl (112kB)
1.5643059407638924e+09 info Installing collected packages: gunicorn
1.564305940856079e+09 info Successfully installed gunicorn-19.9.0
1.5643059409177616e+09 info * Execute RUN pip install gunicorn==19.9.0 (1d7c7f3f) took 749.802494ms
1.564305941918179e+09 info * Collecting filesystem diff
1.5643059426625617e+09 info * Finished collecting diff in 744ms: 126 files found
1.5643059426728718e+09 info * Created layer by scanning filesystem; 126 files found
1.5643059426792467e+09 info * Committed gzipped layer sha256:34d73d444672f815723843b77fbb692893dc1e67fa779eb692b029994157ae74 (410075 bytes)
1.564305942679564e+09 info * Pushing with cache ID 1d7c7f3f
1.5643059426803443e+09 info * Step 5/14 (commit,modifyfs) : RUN pip install poetry (d3273579)
1.564305943067599e+09 info Collecting poetry
1.5643059432930908e+09 info Downloading https://files.pythonhosted.org/packages/c9/76/214acc400055aa0045ed21ce25378225b1887c5c091f99476955bf9f6984/poetry-0.12.17-py2.py3-none-any.whl (195kB)
1.5643059433920708e+09 info Collecting pyrsistent<0.15.0,>=0.14.2 (from poetry)
1.564305943430971e+09 info Downloading https://files.pythonhosted.org/packages/8c/46/4e93ab8a379d7efe93f20a0fb8a27bdfe88942cc954ab0210c3164e783e0/pyrsistent-0.14.11.tar.gz (104kB)
1.5643059438448305e+09 info Collecting pyparsing<3.0,>=2.2 (from poetry)
1.5643059439204583e+09 info Downloading https://files.pythonhosted.org/packages/ff/2c/afc36631a9e0dc9f2b8bd4c19fa2d330cbe07d6679a1c0910418a200acad/pyparsing-2.4.1.1-py2.py3-none-any.whl (64kB)
1.5643059439336166e+09 info Collecting pkginfo<2.0,>=1.4 (from poetry)
1.5643059440364084e+09 info Downloading https://files.pythonhosted.org/packages/e6/d5/451b913307b478c49eb29084916639dc53a88489b993530fed0a66bab8b9/pkginfo-1.5.0.1-py2.py3-none-any.whl
1.5643059440526583e+09 info Collecting requests<3.0,>=2.18 (from poetry)
1.5643059441294332e+09 info Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)
1.5643059441554132e+09 info Collecting cachecontrol[filecache]<0.13.0,>=0.12.4 (from poetry)
1.5643059441905816e+09 info Downloading https://files.pythonhosted.org/packages/5e/f0/2c193ed1f17c97ae539da7e1c2d48b80d8cccb1917163b26a91ca4355aa6/CacheControl-0.12.5.tar.gz
1.56430594443543e+09 info Collecting shellingham<2.0,>=1.1 (from poetry)
1.5643059446227052e+09 info Downloading https://files.pythonhosted.org/packages/2d/ac/e8a34d4b3d24bf554f40651b2aac549a3fc7223725bf10fbdfe2083b6372/shellingham-1.3.1-py2.py3-none-any.whl
1.5643059446344335e+09 info Collecting tomlkit<0.6.0,>=0.5.1 (from poetry)
1.564305944745968e+09 info Downloading https://files.pythonhosted.org/packages/29/22/716eb55bb154d2519752a2d91cf7e91d58dd24e8150c47aaaa67aae75aa6/tomlkit-0.5.5-py2.py3-none-any.whl
1.5643059447651706e+09 info Collecting cachy<0.3,>=0.2 (from poetry)
1.5643059447942119e+09 info Downloading https://files.pythonhosted.org/packages/26/37/8ce3e7b330078b6797a34e79a80a8ad6935e404a3b903765417182c9ce19/cachy-0.2.0-py2.py3-none-any.whl (59kB)
1.564305944810941e+09 info Collecting cleo<0.7.0,>=0.6.7 (from poetry)
1.564305944854549e+09 info Downloading https://files.pythonhosted.org/packages/a7/b9/270301a3a87587f09bc3985973f2e362ffa45fa5fcd5128501516b2f5e31/cleo-0.6.8-py2.py3-none-any.whl (264kB)
1.5643059448843765e+09 info Collecting jsonschema<4.0,>=3.0a3 (from poetry)
1.5643059449256701e+09 info Downloading https://files.pythonhosted.org/packages/aa/69/df679dfbdd051568b53c38ec8152a3ab6bc533434fc7ed11ab034bf5e82f/jsonschema-3.0.1-py2.py3-none-any.whl (54kB)
1.5643059449496784e+09 info Collecting html5lib<2.0,>=1.0 (from poetry)
1.5643059449796767e+09 info Downloading https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl (117kB)
1.564305945010845e+09 info Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry)
1.564305945043501e+09 info Downloading https://files.pythonhosted.org/packages/97/8a/d710f792d6f6ecc089c5e55b66e66c3f2f35516a1ede5a8f54c13350ffb0/requests_toolbelt-0.8.0-py2.py3-none-any.whl (54kB)
1.5643059450601454e+09 info Collecting six (from pyrsistent<0.15.0,>=0.14.2->poetry)
1.564305945094142e+09 info Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
1.5643059451044638e+09 info Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests<3.0,>=2.18->poetry)
1.564305945157288e+09 info Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
1.564305945190462e+09 info Collecting chardet<3.1.0,>=3.0.2 (from requests<3.0,>=2.18->poetry)
1.564305945220159e+09 info Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
1.564305945239445e+09 info Collecting idna<2.9,>=2.5 (from requests<3.0,>=2.18->poetry)
1.5643059452718954e+09 info Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
1.5643059452844913e+09 info Collecting certifi>=2017.4.17 (from requests<3.0,>=2.18->poetry)
1.5643059453219824e+09 info Downloading https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl (157kB)
1.564305945335415e+09 info Collecting msgpack (from cachecontrol[filecache]<0.13.0,>=0.12.4->poetry)
1.5643059453841977e+09 info Downloading https://files.pythonhosted.org/packages/92/7e/ae9e91c1bb8d846efafd1f353476e3fd7309778b582d2fb4cea4cc15b9a2/msgpack-0.6.1-cp36-cp36m-manylinux1_x86_64.whl (248kB)
1.5643059454019456e+09 info Collecting lockfile>=0.9 (from cachecontrol[filecache]<0.13.0,>=0.12.4->poetry)
1.5643059454295585e+09 info Downloading https://files.pythonhosted.org/packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl
1.5643059454398437e+09 info Collecting pylev<2.0,>=1.3 (from cleo<0.7.0,>=0.6.7->poetry)
1.5643059454654098e+09 info Downloading https://files.pythonhosted.org/packages/40/1c/7dff1d242bf1e19f9c6202f0ba4e6fd18cc7ecb8bc85b17b2d16c806e228/pylev-1.3.0-py2.py3-none-any.whl
1.564305945475453e+09 info Collecting pastel<0.2.0,>=0.1.0 (from cleo<0.7.0,>=0.6.7->poetry)
1.564305945500086e+09 info Downloading https://files.pythonhosted.org/packages/9b/7e/7d701686013c0d7dae62e0977467232a6adc2e562c23878eb3cd4f97d02e/pastel-0.1.0-py3-none-any.whl
1.564305945508103e+09 info Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from jsonschema<4.0,>=3.0a3->poetry) (40.4.3)
1.5643059455151021e+09 info Collecting attrs>=17.4.0 (from jsonschema<4.0,>=3.0a3->poetry)
1.5643059455465586e+09 info Downloading https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl
1.5643059455785832e+09 info Collecting webencodings (from html5lib<2.0,>=1.0->poetry)
1.5643059456049016e+09 info Downloading https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
1.5643059456131427e+09 info Building wheels for collected packages: pyrsistent, cachecontrol
1.564305945613847e+09 info Building wheel for pyrsistent (setup.py): started
1.5643059467257838e+09 info Building wheel for pyrsistent (setup.py): finished with status 'done'
1.5643059467273457e+09 info Created wheel for pyrsistent: filename=pyrsistent-0.14.11-cp36-cp36m-linux_x86_64.whl size=111967 sha256=da8666f6712c632306b3c20850594257afb26a7efb1c5469c8946e28229c9b5c
1.56430594672744e+09 info Stored in directory: /root/.cache/pip/wheels/83/59/9a/a037b9b3c3e93d9275ea0aff9d6064400f372879dfdab01afe
1.5643059467409644e+09 info Building wheel for cachecontrol (setup.py): started
1.564305947032157e+09 info Building wheel for cachecontrol (setup.py): finished with status 'done'
1.5643059470333097e+09 info Created wheel for cachecontrol: filename=CacheControl-0.12.5-cp36-none-any.whl size=18489 sha256=e90903deaf4e4baaa5895e02472a735e58449736e0fdb8a95da10bb5ebb8e371
1.564305947033484e+09 info Stored in directory: /root/.cache/pip/wheels/36/bd/5f/dbbee4f2d51f97ecd12a363f870361179cb1fd4bc1174ea08a
1.564305947039347e+09 info Successfully built pyrsistent cachecontrol
1.5643059471697376e+09 info Installing collected packages: six, pyrsistent, pyparsing, pkginfo, urllib3, chardet, idna, certifi, requests, msgpack, lockfile, cachecontrol, shellingham, tomlkit, cachy, pylev, pastel, cleo, attrs, jsonschema, webencodings, html5lib, requests-toolbelt, poetry
1.5643059482160053e+09 info Successfully installed attrs-19.1.0 cachecontrol-0.12.5 cachy-0.2.0 certifi-2019.6.16 chardet-3.0.4 cleo-0.6.8 html5lib-1.0.1 idna-2.8 jsonschema-3.0.1 lockfile-0.12.2 msgpack-0.6.1 pastel-0.1.0 pkginfo-1.5.0.1 poetry-0.12.17 pylev-1.3.0 pyparsing-2.4.1.1 pyrsistent-0.14.11 requests-2.22.0 requests-toolbelt-0.8.0 shellingham-1.3.1 six-1.12.0 tomlkit-0.5.5 urllib3-1.25.3 webencodings-0.5.1
1.5643059483234994e+09 info * Execute RUN pip install poetry (d3273579) took 5.643102222s
1.5643059493240607e+09 info * Collecting filesystem diff
1.564305950197831e+09 info * Finished collecting diff in 874ms: 1910 files found
1.5643059503661907e+09 info * Created layer by scanning filesystem; 1910 files found
1.5643059503718877e+09 info * Committed gzipped layer sha256:6f7445cbad659141bfb874ad19a9704bd25b43417e469e7ba697fda138bc8009 (5248391 bytes)
1.5643059503719156e+09 info * Pushing with cache ID d3273579
1.5643059503725297e+09 info * Step 6/14 (commit,modifyfs) : WORKDIR /usr/src/resolve (d566191)
1.5643059503725784e+09 info * Execute WORKDIR /usr/src/resolve (d566191) took 1.042µs
1.5643059503725832e+09 info * Pushing with cache ID d566191
1.5643059503730712e+09 info * Step 7/14 (commit,modifyfs) : COPY secrets secrets (42e75608)
1.564305950373489e+09 info * Execute COPY secrets secrets (42e75608) took 387.989µs
1.564305951374436e+09 info * Created copy layer with 6 files
1.5643059513761637e+09 info * Committed gzipped layer sha256:8a041c71e7ccc1418342afce952b8115ea1351db58f128c06b7beece6ebe561f (2124 bytes)
1.56430595137652e+09 info * Pushing with cache ID 42e75608
1.5643059513776245e+09 info * Step 8/14 (commit,modifyfs) : COPY pyproject.toml poetry.lock install_dependencies.sh /usr/src/resolve/ (c891b855)
1.5643059513787923e+09 info * Execute COPY pyproject.toml poetry.lock install_dependencies.sh /usr/src/resolve/ (c891b855) took 825.145µs
1.5643059523801625e+09 info * Created copy layer with 6 files
1.5643059523830843e+09 info * Committed gzipped layer sha256:b8a07cfb875b90ae3fd5d667dd651b0feba0d5956836b30e9ed9fb262fd68832 (22301 bytes)
1.564305952383427e+09 info * Pushing with cache ID c891b855
1.5643059523842015e+09 info * Step 9/14 (commit,modifyfs) : ARG GEMFURY_TOKEN (5d030392)
1.5643059523844893e+09 info * Execute ARG GEMFURY_TOKEN (5d030392) took 5.684µs
1.5643059523847103e+09 info * Pushing with cache ID 5d030392
1.56430595238546e+09 info * Step 10/14 (commit,modifyfs) : RUN GEMFURY_TOKEN=${GEMFURY_TOKEN} ./install_dependencies.sh (c1bf0f49)
1.564305953377296e+09 info Skipping virtualenv creation, as specified in config file.
1.564305954158073e+09 info Installing dependencies from lock file
1.564305954832293e+09 info
1.5643059548327742e+09 info
Package operations: 68 installs, 5 updates, 0 removals
- Installing pyasn1 (0.4.5)
1.5643059555791357e+09 info - Updating six (1.12.0 -> 1.11.0)
1.5643059561935637e+09 info - Installing cachetools (3.1.1)
1.5643059568062196e+09 info - Updating idna (2.8 -> 2.7)
1.564305957485989e+09 info - Installing protobuf (3.9.0)
1.5643059587350888e+09 info - Installing pyasn1-modules (0.2.5)
1.5643059595340626e+09 info - Installing rsa (4.0)
1.5643059602112775e+09 info - Updating urllib3 (1.25.3 -> 1.23)
1.5643059609435349e+09 info - Installing docutils (0.14)
1.5643059619535859e+09 info - Installing google-auth (1.6.3)
1.564305962628653e+09 info - Installing googleapis-common-protos (1.6.0)
1.5643059639617796e+09 info - Installing jmespath (0.9.4)
1.5643059645882602e+09 info - Installing python-dateutil (2.8.0)
1.5643059652957408e+09 info - Installing pytz (2019.1)
1.5643059663173704e+09 info - Updating requests (2.22.0 -> 2.19.1)
1.564305967041387e+09 info - Installing botocore (1.12.191)
1.564305969096692e+09 info - Installing google-api-core (1.14.0)
1.5643059697952476e+09 info - Installing appdirs (1.4.3)
1.5643059704192889e+09 info - Installing cymem (2.0.2)
1.5643059710335398e+09 info - Installing google-cloud-core (0.29.1)
1.5643059716883433e+09 info - Installing google-resumable-media (0.3.2)
1.5643059723538845e+09 info - Updating msgpack (0.6.1 -> 0.5.6)
1.5643059730519335e+09 info - Installing numpy (1.15.0)
1.5643059760313275e+09 info - Installing s3transfer (0.2.1)
1.5643059767053251e+09 info - Installing toolz (0.10.0)
1.5643059781025777e+09 info - Installing boto (2.49.0)
1.5643059795698204e+09 info - Installing boto3 (1.9.191)
1.5643059805252273e+09 info - Installing cytoolz (0.9.0.1)
1.5643060000662808e+09 info - Installing dill (0.2.8.2)
1.5643060013874466e+09 info - Installing fs (2.4.8)
1.564306002206525e+09 info - Installing google-cloud-storage (1.13.2)
1.564306002910627e+09 info - Installing marisa-trie (0.7.5)
1.564306019017573e+09 info - Installing markupsafe (1.1.1)
1.564306019688849e+09 info - Installing msgpack-numpy (0.4.3.2)
1.5643060203552153e+09 info - Installing murmurhash (1.0.2)
1.564306021031228e+09 info - Installing plac (0.9.6)
1.5643060217113945e+09 info - Installing preshed (2.0.1)
1.5643060224288566e+09 info - Installing tqdm (4.24.0)
1.5643060231660595e+09 info - Installing wrapt (1.10.11)
1.5643060250027034e+09 info - Installing click (7.0)
1.5643060257711194e+09 info - Installing enum34 (1.1.6)
1.5643060264251077e+09 info - Installing fs-gcsfs (0.4.1)
1.564306027680222e+09 info - Installing futures (3.1.1)
1.564306028963687e+09 info - Installing itsdangerous (1.1.0)
1.5643060296585975e+09 info - Installing jinja2 (2.10.1)
1.5643060304462152e+09 info - Installing langcodes (1.4.1)
1.5643060327761793e+09 info - Installing psutil (5.6.3)
1.564306035432153e+09 info - Installing regex (2018.01.10)
1.5643060463252249e+09 info - Installing scipy (1.3.0)
1.56430605047923e+09 info - Installing smart-open (1.8.4)
1.5643060518452435e+09 info - Installing thinc (6.12.1)
1.5643060528789866e+09 info - Installing ujson (1.35)
1.5643060551091783e+09 info - Installing websocket-client (0.56.0)
1.5643060558496518e+09 info - Installing werkzeug (0.15.5)
1.5643060567324436e+09 info - Installing beautifulsoup4 (4.6.3)
1.56430605749315e+09 info - Installing ddtrace (0.27.0)
1.5643060601991947e+09 info - Installing flask (1.0.2)
1.5643060609589949e+09 info - Installing gensim (3.6.0)
1.564306062983306e+09 info - Installing graphqlclient (0.2.4)
1.564306064265957e+09 info - Installing grpcio (1.18.0)
1.56430606597738e+09 info - Installing janome (0.3.6)
1.5643060971549213e+09 info - Installing jieba (0.39)
1.5643061017100897e+09 info - Installing joblib (0.11)
1.5643061025740614e+09 info - Installing more-itertools (4.1.0)
1.5643061033437889e+09 info - Installing nltk (3.3)
1.5643061062801018e+09 info - Installing pandas (0.21.1)
1.5643061110482614e+09 info - Installing pika (0.10.0)
1.564306111845532e+09 info - Installing pysolvvy (2.2.1)
1.5643061140786307e+09 info - Installing python-json-logger (0.1.10)
1.5643061147993839e+09 info - Installing scikit-learn (0.20.2)
1.5643061171768298e+09 info - Installing slackclient (1.1.0)
1.5643061185465374e+09 info - Installing spacy (2.0.18)
1.564306158019171e+09 info - Installing wordfreq (2.2.1)
1.5643061595072303e+09 error /usr/local/lib/python3.6/site-packages/poetry/version/requirements.py:170: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'specifier' on And expression collides with '_original_end' on contained expression
VERSION_SPEC = originalTextFor(_VERSION_SPEC)("specifier")
/usr/local/lib/python3.6/site-packages/poetry/version/requirements.py:173: UserWarning: warn_ungrouped_named_tokens_in_collection: setting results name 'marker' on And expression collides with '_original_end' on contained expression
MARKER_EXPR = originalTextFor(MARKER_EXPR())("marker")
1.5643061608907151e+09 info [nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data] Error downloading 'stopwords' from
[nltk_data] <https://raw.githubusercontent.com/nltk/nltk_data/gh-
[nltk_data] pages/packages/corpora/stopwords.zip>: <urlopen
[nltk_data] error [SSL: CERTIFICATE_VERIFY_FAILED] certificate
[nltk_data] verify failed (_ssl.c:841)>
[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data] Error downloading 'wordnet' from
[nltk_data] <https://raw.githubusercontent.com/nltk/nltk_data/gh-
[nltk_data] pages/packages/corpora/wordnet.zip>: <urlopen error
[nltk_data] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
[nltk_data] failed (_ssl.c:841)>
1.564306162135779e+09 info Collecting en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en_core_web_sm==2.0.0
1.5643061626807895e+09 info Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4MB)
1.564306164976116e+09 info Building wheels for collected packages: en-core-web-sm
1.5643061649767957e+09 info Building wheel for en-core-web-sm (setup.py): started
1.5643061674775896e+09 info Building wheel for en-core-web-sm (setup.py): finished with status 'done'
1.5643061675932443e+09 info Created wheel for en-core-web-sm: filename=en_core_web_sm-2.0.0-cp36-none-any.whl size=37405977 sha256=fc6520997c2d51144ce8d565b6bbe502aef2781568d460b3ef8b36928eedd787
Stored in directory: /tmp/pip-ephem-wheel-cache-jvhe75oa/wheels/54/7c/d8/f86364af8fbba7258e14adae115f18dd2c91552406edc3fdaa
1.5643061679790123e+09 info Successfully built en-core-web-sm
1.564306167979127e+09 info Installing collected packages: en-core-web-sm
1.5643061680692036e+09 info Successfully installed en-core-web-sm-2.0.0
1.5643061682762437e+09 info
Linking successful
/usr/local/lib/python3.6/site-packages/en_core_web_sm -->
/usr/local/lib/python3.6/site-packages/spacy/data/en
You can now load the model via spacy.load('en')
1.5643061693969872e+09 info Collecting de_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.0.0/de_core_news_sm-2.0.0.tar.gz#egg=de_core_news_sm==2.0.0
1.56430616987158e+09 info Downloading https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.0.0/de_core_news_sm-2.0.0.tar.gz (38.2MB)
1.5643061722450833e+09 info Building wheels for collected packages: de-core-news-sm
1.5643061722457318e+09 info Building wheel for de-core-news-sm (setup.py): started
1.5643061749243102e+09 info Building wheel for de-core-news-sm (setup.py): finished with status 'done'
1.5643061750428846e+09 info Created wheel for de-core-news-sm: filename=de_core_news_sm-2.0.0-cp36-none-any.whl size=38276259 sha256=e0690a31f47667c99a814fa47248ea4bdc3e37bd4402bdf99ff1ff5bd38cb5eb
1.5643061750429406e+09 info Stored in directory: /tmp/pip-ephem-wheel-cache-reyyky7j/wheels/a5/9e/96/785f498e5cea276b337aab8a79259d32a93f03bd744d71eef5
1.564306175453096e+09 info Successfully built de-core-news-sm
1.564306175453699e+09 info Installing collected packages: de-core-news-sm
1.564306175546654e+09 info Successfully installed de-core-news-sm-2.0.0
1.5643061757399292e+09 info
Linking successful
/usr/local/lib/python3.6/site-packages/de_core_news_sm -->
/usr/local/lib/python3.6/site-packages/spacy/data/de
You can now load the model via spacy.load('de')
1.5643061768918226e+09 info Collecting es_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/es_core_news_sm-2.0.0/es_core_news_sm-2.0.0.tar.gz#egg=es_core_news_sm==2.0.0
1.5643061774281178e+09 info Downloading https://github.com/explosion/spacy-models/releases/download/es_core_news_sm-2.0.0/es_core_news_sm-2.0.0.tar.gz (36.7MB)
1.5643061796637475e+09 info Building wheels for collected packages: es-core-news-sm
1.5643061796643817e+09 info Building wheel for es-core-news-sm (setup.py): started
1.5643061820971627e+09 info Building wheel for es-core-news-sm (setup.py): finished with status 'done'
1.5643061822095928e+09 info Created wheel for es-core-news-sm: filename=es_core_news_sm-2.0.0-cp36-none-any.whl size=36749073 sha256=11c8bf4a62ab78328e3ad43d2103d97e96658fa72e6005ac07784d7e2ed928fa
1.5643061822096481e+09 info Stored in directory: /tmp/pip-ephem-wheel-cache-bdvi887k/wheels/9e/28/c4/df4980946eb229379ed26d349566e427fa029dbf03546ccb94
1.564306182585842e+09 info Successfully built es-core-news-sm
1.5643061825859742e+09 info Installing collected packages: es-core-news-sm
1.564306182672747e+09 info Successfully installed es-core-news-sm-2.0.0
1.5643061828758357e+09 info
Linking successful
/usr/local/lib/python3.6/site-packages/es_core_news_sm -->
/usr/local/lib/python3.6/site-packages/spacy/data/es
You can now load the model via spacy.load('es')
1.5643061840248098e+09 info Collecting pt_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/pt_core_news_sm-2.0.0/pt_core_news_sm-2.0.0.tar.gz#egg=pt_core_news_sm==2.0.0
1.5643061845828686e+09 info Downloading https://github.com/explosion/spacy-models/releases/download/pt_core_news_sm-2.0.0/pt_core_news_sm-2.0.0.tar.gz (38.7MB)
1.5643061870466504e+09 info Building wheels for collected packages: pt-core-news-sm
1.5643061870473793e+09 info Building wheel for pt-core-news-sm (setup.py): started
1.564306189780281e+09 info Building wheel for pt-core-news-sm (setup.py): finished with status 'done'
1.564306189899478e+09 info Created wheel for pt-core-news-sm: filename=pt_core_news_sm-2.0.0-cp36-none-any.whl size=38749345 sha256=52a1e61705a91ade7b5e579dffbfb5d9ee6b78f6913d3af460c585bf08481d30
1.5643061898995314e+09 info Stored in directory: /tmp/pip-ephem-wheel-cache-k4_3ntbs/wheels/0b/f1/54/a8f759b41cf39d2a33d6da84cd90b8e2fefea549963396bbd2
1.5643061902959418e+09 info Successfully built pt-core-news-sm
1.5643061902959847e+09 info Installing collected packages: pt-core-news-sm
1.5643061903886478e+09 info Successfully installed pt-core-news-sm-2.0.0
1.5643061905912204e+09 info
Linking successful
/usr/local/lib/python3.6/site-packages/pt_core_news_sm -->
/usr/local/lib/python3.6/site-packages/spacy/data/pt
You can now load the model via spacy.load('pt')
1.5643061917029674e+09 info Collecting fr_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-2.0.0/fr_core_news_sm-2.0.0.tar.gz#egg=fr_core_news_sm==2.0.0
1.56430619242016e+09 info Downloading https://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-2.0.0/fr_core_news_sm-2.0.0.tar.gz (39.8MB)
1.5643061949027922e+09 info Building wheels for collected packages: fr-core-news-sm
1.564306194903339e+09 info Building wheel for fr-core-news-sm (setup.py): started
1.5643061977933373e+09 info Building wheel for fr-core-news-sm (setup.py): finished with status 'done'
1.56430619791312e+09 info Created wheel for fr-core-news-sm: filename=fr_core_news_sm-2.0.0-cp36-none-any.whl size=39841461 sha256=29a168848439ab9c70e303b113f7bc6cf2c749a0d5502d0e4a220086e1ccdd69
Stored in directory: /tmp/pip-ephem-wheel-cache-1ffxtfgb/wheels/50/cb/17/4b4c5a59786e77f375913b22b395922d768b069b7d7aa7bd24
1.5643061983669324e+09 info Successfully built fr-core-news-sm
1.564306198366972e+09 info Installing collected packages: fr-core-news-sm
1.5643061984732761e+09 info Successfully installed fr-core-news-sm-2.0.0
1.5643061986964142e+09 info
Linking successful
/usr/local/lib/python3.6/site-packages/fr_core_news_sm -->
/usr/local/lib/python3.6/site-packages/spacy/data/fr
You can now load the model via spacy.load('fr')
1.5643061998371525e+09 info Collecting it_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/it_core_news_sm-2.0.0/it_core_news_sm-2.0.0.tar.gz#egg=it_core_news_sm==2.0.0
1.5643062004175065e+09 info Downloading https://github.com/explosion/spacy-models/releases/download/it_core_news_sm-2.0.0/it_core_news_sm-2.0.0.tar.gz (36.5MB)
1.5643062027487109e+09 info Building wheels for collected packages: it-core-news-sm
1.5643062027492805e+09 info Building wheel for it-core-news-sm (setup.py): started
1.5643062051313343e+09 info Building wheel for it-core-news-sm (setup.py): finished with status 'done'
1.564306205242413e+09 info Created wheel for it-core-news-sm: filename=it_core_news_sm-2.0.0-cp36-none-any.whl size=36476091 sha256=c0ffa25968f429287832156371507b72791c4c37d88d7005ae56c056f2440359
1.5643062052424557e+09 info Stored in directory: /tmp/pip-ephem-wheel-cache-zt774awq/wheels/c4/cb/1c/c452364dfe51ffb6ab2727df879e833565d27e255c76ae2954
1.5643062056123633e+09 info Successfully built it-core-news-sm
1.564306205612407e+09 info Installing collected packages: it-core-news-sm
1.56430620570114e+09 info Successfully installed it-core-news-sm-2.0.0
1.5643062059075215e+09 info
Linking successful
/usr/local/lib/python3.6/site-packages/it_core_news_sm -->
/usr/local/lib/python3.6/site-packages/spacy/data/it
You can now load the model via spacy.load('it')
1.5643062070487492e+09 info Collecting nl_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/nl_core_news_sm-2.0.0/nl_core_news_sm-2.0.0.tar.gz#egg=nl_core_news_sm==2.0.0
1.5643062078050048e+09 info Downloading https://github.com/explosion/spacy-models/releases/download/nl_core_news_sm-2.0.0/nl_core_news_sm-2.0.0.tar.gz (36.7MB)
1.5643062101088996e+09 info Building wheels for collected packages: nl-core-news-sm
1.56430621010946e+09 info Building wheel for nl-core-news-sm (setup.py): started
1.564306212561634e+09 info Building wheel for nl-core-news-sm (setup.py): finished with status 'done'
1.5643062126733632e+09 info Created wheel for nl-core-news-sm: filename=nl_core_news_sm-2.0.0-cp36-none-any.whl size=36694098 sha256=169d6a1c470c7e4c8ebea3e949a81baff3d08c6d9931a2a2f0c12acb1484c498
1.5643062126734076e+09 info Stored in directory: /tmp/pip-ephem-wheel-cache-ux6ocaaf/wheels/20/e1/45/57eaecebfacb4fd1dcb8cc4e1ef0f71e4b8d72043b0ec8f1e5
1.564306213042146e+09 info Successfully built nl-core-news-sm
1.564306213042793e+09 info Installing collected packages: nl-core-news-sm
1.564306213130649e+09 info Successfully installed nl-core-news-sm-2.0.0
1.5643062133329954e+09 info
Linking successful
/usr/local/lib/python3.6/site-packages/nl_core_news_sm -->
/usr/local/lib/python3.6/site-packages/spacy/data/nl
You can now load the model via spacy.load('nl')
1.5643062139693282e+09 info
Linking successful
/usr/local/lib/python3.6/site-packages/jieba -->
/usr/local/lib/python3.6/site-packages/spacy/data/zh
You can now load the model via spacy.load('zh')
1.5643062145512326e+09 info
Linking successful
/usr/local/lib/python3.6/site-packages/janome -->
/usr/local/lib/python3.6/site-packages/spacy/data/ja
You can now load the model via spacy.load('ja')
1.5643062145969667e+09 info * Execute RUN GEMFURY_TOKEN=${GEMFURY_TOKEN} ./install_dependencies.sh (c1bf0f49) took 4m22.211239423s
1.5643062155974643e+09 info * Collecting filesystem diff
1.5643062184818888e+09 info * Finished collecting diff in 2.884s: 16726 files found
1.564306231832618e+09 info * Created layer by scanning filesystem; 16726 files found
1.5643062318390176e+09 info * Committed gzipped layer sha256:e7d8cecf15ca1d353859cb01d3dc981d83c2688f67c0c28e63a4a5059b6e7d7b (822886598 bytes)
1.564306231839078e+09 info * Pushing with cache ID c1bf0f49
1.5643062318397386e+09 info * Step 11/14 (commit,modifyfs) : COPY . /usr/src/resolve (bb7e73c6)
1.564306231853637e+09 info * Execute COPY . /usr/src/resolve (bb7e73c6) took 13.845266ms
1.5643062350982347e+09 info * Created copy layer with 115 files
1.5643062351020508e+09 info * Committed gzipped layer sha256:5a48e99eaaff29fe6775cd72fb0457da90977cbdb3a66acaafa5a9dcd4723d2a (141533 bytes)
1.5643062351020858e+09 info * Pushing with cache ID bb7e73c6
1.5643062351027906e+09 info * Step 12/14 (commit,modifyfs) : ENV GOOGLE_APPLICATION_CREDENTIALSS=/usr/src/resolve/secrets/my-project-key.json PYTHONUNBUFFERED=TRUE PYTHONDONTWRITEBYTECODE=TRUE PYTHONHASHSEED=0 PYTHONPATH="/usr/src/resolve/src" (20ed4a5c)
1.564306235102849e+09 info * Execute ENV GOOGLE_APPLICATION_CREDENTIALSS=/usr/src/resolve/secrets/my-project-key.json PYTHONUNBUFFERED=TRUE PYTHONDONTWRITEBYTECODE=TRUE PYTHONHASHSEED=0 PYTHONPATH="/usr/src/resolve/src" (20ed4a5c) took 1.619µs
1.564306235102856e+09 info * Pushing with cache ID 20ed4a5c
1.5643062351034133e+09 info * Step 13/14 (commit,modifyfs) : EXPOSE 6062 (b35c2a2a)
1.5643062351034636e+09 info * Execute EXPOSE 6062 (b35c2a2a) took 1.35µs
1.5643062351034722e+09 info * Pushing with cache ID b35c2a2a
1.5643062351040144e+09 info * Step 14/14 (commit,modifyfs) : ENTRYPOINT ddtrace-run gunicorn --name=resolve_api --log-config=src/config/gunicorn_logging.conf --config=src/config/gunicorn_config.py --access-logformat=\"{\'remote_ip\':\'%\(h\)s\'\,\'code\':\'%\(s\)s\'\,\'method\':\'%\(m\)s\'\,\'path\':\'%\(U\)s\'\,\'querystring\':\'%\(q\)s\'\,\'timetaken\':\'%\(D\)s\'\,\'response_length\':\'%\(B\)s\'}\" resolve_api:configure_app\(remote_model_dir=\"$REMOTE_MODEL_DIR\"\,initial_model_version=\"$INITIAL_MODEL_VERSION\"\,local_model_dir=\"/data\"\,org_id=$ORG_ID\) (eb427ee8)
1.564306235104085e+09 info * Execute ENTRYPOINT ddtrace-run gunicorn --name=resolve_api --log-config=src/config/gunicorn_logging.conf --config=src/config/gunicorn_config.py --access-logformat=\"{\'remote_ip\':\'%\(h\)s\'\,\'code\':\'%\(s\)s\'\,\'method\':\'%\(m\)s\'\,\'path\':\'%\(U\)s\'\,\'querystring\':\'%\(q\)s\'\,\'timetaken\':\'%\(D\)s\'\,\'response_length\':\'%\(B\)s\'}\" resolve_api:configure_app\(remote_model_dir=\"$REMOTE_MODEL_DIR\"\,initial_model_version=\"$INITIAL_MODEL_VERSION\"\,local_model_dir=\"/data\"\,org_id=$ORG_ID\) (eb427ee8) took 1.439µs
1.5643062351040921e+09 info * Pushing with cache ID eb427ee8
1.5643062351046875e+09 info * Moving directories [] to /makisu-storage/sandbox/sandbox590207551/stages/MA==
1.5643068379290779e+09 error Failed to push cache: timeout waiting for push
1.564306837930396e+09 info Computed total image size 1189314536 {"total_image_size": 1189314536}
1.5643068379304373e+09 info Successfully built image my-project/resolve-makisu-makisu:f597821dfabe48685ce3e1c0cd27071ebe833cd0
1.5643068379305375e+09 info * Started pushing image gcr.io/my-project/resolve-makisu-makisu:f597821dfabe48685ce3e1c0cd27071ebe833cd0
1.564306838519872e+09 info * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:bc9ab73e5b14b9fbd3687a4d8c1f1360533d6ee9ffc3f5ecc6630794b40257b7
1.5643068385219464e+09 info * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:e5c3f8c317dc30af45021092a3d76f16ba7aa1ee5f18fec742c84d4960818580
1.564306838522682e+09 info * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:193a6306c92af328dbd41bbbd3200a2c90802624cccfe5725223324428110d7f
1.5643068387359588e+09 info * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:6598fc9d11d10365ac9281071a87930a2382ee31d026f1b6d432717b31db387c
1.5643068387402036e+09 info * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:72744d0a318b0788001cc4f5f83c6847ba4b753307fadd046b508bbc41eb9e29
1.564306838744728e+09 info * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:a587a86c9dcb9df6584180042becf21e36ecd8b460a761711227b4b06889a005
1.564306838940623e+09 info * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f
1.5643068389598777e+09 info * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:93612f47cdc374d0b33057b9e71eac173ac469da3e1a631dc8a32ba6986a408a
1.5643068389639986e+09 info * Skipped pushing existing layer my-project/resolve-makisu-makisu:sha256:1bc4b4b508703799ef67a807dacce4736045e642e87bcd49871cd0f23e7f5b8b
1.5643068393929317e+09 info * Started pushing layer sha256:d3fc069215c7ba6059cf21b0b463c00c996710b66fe4fee5339b20b83b6778d0
1.5643068393939884e+09 info * Started pushing layer sha256:34d73d444672f815723843b77fbb692893dc1e67fa779eb692b029994157ae74
1.5643068394151993e+09 info * Started pushing layer sha256:6f7445cbad659141bfb874ad19a9704bd25b43417e469e7ba697fda138bc8009
1.5643068411155717e+09 info * Finished pushing layer sha256:34d73d444672f815723843b77fbb692893dc1e67fa779eb692b029994157ae74
1.5643068413501308e+09 info * Finished pushing layer sha256:6f7445cbad659141bfb874ad19a9704bd25b43417e469e7ba697fda138bc8009
1.564306841523365e+09 info * Finished pushing layer sha256:d3fc069215c7ba6059cf21b0b463c00c996710b66fe4fee5339b20b83b6778d0
1.564306841618725e+09 info * Started pushing layer sha256:8a041c71e7ccc1418342afce952b8115ea1351db58f128c06b7beece6ebe561f
1.5643068418106127e+09 info * Started pushing layer sha256:b8a07cfb875b90ae3fd5d667dd651b0feba0d5956836b30e9ed9fb262fd68832
1.564306841900909e+09 info * Started pushing layer sha256:e7d8cecf15ca1d353859cb01d3dc981d83c2688f67c0c28e63a4a5059b6e7d7b
1.5643068433530157e+09 info * Finished pushing layer sha256:8a041c71e7ccc1418342afce952b8115ea1351db58f128c06b7beece6ebe561f
1.5643068436314905e+09 info * Finished pushing layer sha256:b8a07cfb875b90ae3fd5d667dd651b0feba0d5956836b30e9ed9fb262fd68832
1.564306843823328e+09 info * Started pushing layer sha256:5a48e99eaaff29fe6775cd72fb0457da90977cbdb3a66acaafa5a9dcd4723d2a
1.5643068440943959e+09 info * Started pushing image config sha256:c9e8fec1c953d0b0e8b763953761d7a62ceb486c04099b42615e340769e0525b
1.5643068457586443e+09 info * Finished pushing layer sha256:5a48e99eaaff29fe6775cd72fb0457da90977cbdb3a66acaafa5a9dcd4723d2a
1.5643068459129927e+09 info * Finished pushing image config sha256:c9e8fec1c953d0b0e8b763953761d7a62ceb486c04099b42615e340769e0525b
1.564306875721196e+09 info * Finished pushing layer sha256:e7d8cecf15ca1d353859cb01d3dc981d83c2688f67c0c28e63a4a5059b6e7d7b
1.564306876771674e+09 info * Finished pushing image gcr.io/my-project/resolve-makisu-makisu:f597821dfabe48685ce3e1c0cd27071ebe833cd0 in 38.841108748s
1.564306876772124e+09 info Successfully pushed gcr.io/my-project/resolve-makisu-makisu:f597821dfabe48685ce3e1c0cd27071ebe833cd0 to gcr.io
1.5643068767724044e+09 info Finished building my-project/resolve-makisu-makisu:f597821dfabe48685ce3e1c0cd27071ebe833cd0
As for the rate limits: I think that is highly unlikely. I encountered this issue mostly when experimenting with makisu off-hours where we don't have much activity anyways, but even during the day I doubt we'd even get close to the rate limits.
Could it have to do somethign with the http cache though?