Cannot find a primary and inelligble to become primary
Closed this issue · 4 comments
ayuhito commented
Thank you for the awesome new release! I moved over to the new version and really wanted to try the new exec feature for database migrations. Unfortunately, I seem to have run on an infinite loop of the following:
Logs
2023-04-26T09:20:11.758 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:12.912 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:14.066 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:15.220 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:16.425 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:17.579 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:18.761 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:19.922 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:21.071 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:22.315 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:23.465 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:24.775 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
2023-04-26T09:20:26.048 app[a9311843] sjc [info] BF2ABFEB3AB37220: cannot find primary & ineligible to become primary, retrying: no primary
LiteFS configuration for reference in case I missed something there.
This occurred with initially fly scale count 1. I tried to do fly scale count 2 too but since there seems to be no primary detected, both fail. Maybe something for handover failed there?
ayuhito commented
I reset everything by running fly scale count 0 and then tried spinning up two new machines.
This time we successfully launched one machine. However, while launching the second machine, things started breaking:
Logs
2023-04-26T11:04:43.318 app[3d8d317a296158] fra [warn] Virtual machine exited abruptly
2023-04-26T11:06:08.303 runner[4d89705c6d5ee8] lhr [info] Pulling container image registry.fly.io/fontsource:deployment-01GYYKVBVMYDS54V3YF22W74T9
2023-04-26T11:06:14.535 runner[4d89705c6d5ee8] lhr [info] Unpacking image
2023-04-26T11:06:27.981 runner[4d89705c6d5ee8] lhr [info] Successfully prepared image registry.fly.io/fontsource:deployment-01GYYKVBVMYDS54V3YF22W74T9
2023-04-26T11:06:29.937 runner[4d89705c6d5ee8] lhr [info] Configuring firecracker
2023-04-26T11:06:30.336 app[4d89705c6d5ee8] lhr [info] Starting init (commit: 1578345)...
2023-04-26T11:06:30.352 app[4d89705c6d5ee8] lhr [info] Preparing to run: `docker-entrypoint.sh litefs mount` as root
2023-04-26T11:06:30.373 app[4d89705c6d5ee8] lhr [info] 2023/04/26 11:06:30 listening on [fdaa:0:e7e0:a7b:8e:2adb:129b:2]:22 (DNS: [fdaa::3]:53)
2023-04-26T11:06:30.418 app[4d89705c6d5ee8] lhr [info] config file read from /app/litefs.yml
2023-04-26T11:06:30.418 app[4d89705c6d5ee8] lhr [info] LiteFS v0.4.0, commit=6421a2236f96b1d7bd88d18a1cc46aa1fd5c2ae7
2023-04-26T11:06:30.418 app[4d89705c6d5ee8] lhr [info] Using Consul to determine primary
2023-04-26T11:06:30.898 health[4d89705c6d5ee8] lhr [warn] Health check on port 8080 is in a 'warning' state. Your app may not be responding properly. Services exposed on ports [80, 443] may have intermittent failures until the health check passes.
2023-04-26T11:06:32.037 app[4d89705c6d5ee8] lhr [info] initializing consul: key=fontsource/primary url=https://:e56d6a0c-6c59-e30e-fc67-3e412f38afb9@consul-fra-3.fly-shared.net/fontsource-6r85yqled4r92pvl/ hostname=4d89705c6d5ee8 advertise-url=http://4d89705c6d5ee8.vm.fontsource.internal:20202
2023-04-26T11:06:32.042 app[4d89705c6d5ee8] lhr [info] LiteFS mounted to: /litefs
2023-04-26T11:06:32.042 app[4d89705c6d5ee8] lhr [info] http server listening on: http://localhost:20202
2023-04-26T11:06:32.042 app[4d89705c6d5ee8] lhr [info] waiting to connect to cluster
2023-04-26T11:06:32.585 app[4d89705c6d5ee8] lhr [info] CB81CA6E2B6B7B86: primary lease acquired, advertising as http://4d89705c6d5ee8.vm.fontsource.internal:20202
2023-04-26T11:06:32.585 app[4d89705c6d5ee8] lhr [info] connected to cluster, ready
2023-04-26T11:06:32.585 app[4d89705c6d5ee8] lhr [info] node is a candidate, automatically promoting to primary
2023-04-26T11:06:32.585 app[4d89705c6d5ee8] lhr [info] node is already primary, skipping promotion
2023-04-26T11:06:32.585 app[4d89705c6d5ee8] lhr [info] proxy server listening on: http://localhost:8080
2023-04-26T11:06:32.585 app[4d89705c6d5ee8] lhr [info] executing command: npm [run migrations]
2023-04-26T11:06:33.362 app[4d89705c6d5ee8] lhr [info] > migrations
2023-04-26T11:06:33.362 app[4d89705c6d5ee8] lhr [info] > knex migrate:latest
2023-04-26T11:06:33.557 app[4d89705c6d5ee8] lhr [info] Using environment: production
2023-04-26T11:06:33.748 app[4d89705c6d5ee8] lhr [info] database file is zero length on initialization: /data/litefs/dbs/sqlite.db/database
2023-04-26T11:06:33.810 app[4d89705c6d5ee8] lhr [info] Batch 1 run: 5 migrations
2023-04-26T11:06:33.823 app[4d89705c6d5ee8] lhr [info] starting background subprocess: npm [run start]
2023-04-26T11:06:33.823 app[4d89705c6d5ee8] lhr [info] waiting for signal or subprocess to exit
2023-04-26T11:06:34.208 app[4d89705c6d5ee8] lhr [info] > start
2023-04-26T11:06:34.208 app[4d89705c6d5ee8] lhr [info] > remix-serve build
2023-04-26T11:06:35.475 app[4d89705c6d5ee8] lhr [info] Remix App Server started at http://localhost:8081 (http://172.19.133.250:8081)
2023-04-26T11:06:36.901 health[4d89705c6d5ee8] lhr [info] Health check on port 8080 is now passing.
2023-04-26T11:06:39.924 runner[5683d01a1043e8] lhr [info] Pulling container image registry.fly.io/fontsource:deployment-01GYYKVBVMYDS54V3YF22W74T9
2023-04-26T11:06:43.567 runner[5683d01a1043e8] lhr [info] Unpacking image
2023-04-26T11:06:52.966 runner[5683d01a1043e8] lhr [info] Successfully prepared image registry.fly.io/fontsource:deployment-01GYYKVBVMYDS54V3YF22W74T9
2023-04-26T11:07:02.465 app[4d89705c6d5ee8] lhr [info] GET /robots.txt 404 - - 32.081 ms
2023-04-26T11:07:02.577 app[4d89705c6d5ee8] lhr [info] GET / 200 - - 340.140 ms
2023-04-26T11:07:07.531 app[4d89705c6d5ee8] lhr [info] Fetched metadata for fira-mono
2023-04-26T11:07:08.387 runner[5683d01a1043e8] lhr [info] Configuring firecracker
2023-04-26T11:07:08.640 app[4d89705c6d5ee8] lhr [info] Fetched metadata for roboto-condensed
2023-04-26T11:07:08.796 app[5683d01a1043e8] lhr [info] Starting init (commit: 1578345)...
2023-04-26T11:07:08.812 app[5683d01a1043e8] lhr [info] Preparing to run: `docker-entrypoint.sh litefs mount` as root
2023-04-26T11:07:08.833 app[5683d01a1043e8] lhr [info] 2023/04/26 11:07:08 listening on [fdaa:0:e7e0:a7b:a98:eeff:954f:2]:22 (DNS: [fdaa::3]:53)
2023-04-26T11:07:08.867 app[5683d01a1043e8] lhr [info] config file read from /app/litefs.yml
2023-04-26T11:07:08.867 app[5683d01a1043e8] lhr [info] LiteFS v0.4.0, commit=6421a2236f96b1d7bd88d18a1cc46aa1fd5c2ae7
2023-04-26T11:07:08.868 app[5683d01a1043e8] lhr [info] Using Consul to determine primary
2023-04-26T11:07:08.982 app[4d89705c6d5ee8] lhr [info] Fetched metadata for dm-sans
2023-04-26T11:07:09.284 health[5683d01a1043e8] lhr [warn] Health check on port 8080 is in a 'warning' state. Your app may not be responding properly. Services exposed on ports [80, 443] may have intermittent failures until the health check passes.
2023-04-26T11:07:09.519 app[4d89705c6d5ee8] lhr [info] Fetched metadata for dm-mono
2023-04-26T11:07:09.715 app[4d89705c6d5ee8] lhr [info] Fetched metadata for noto-sans-tc
2023-04-26T11:07:09.876 app[4d89705c6d5ee8] lhr [info] Fetched metadata for noto-sans-jp
2023-04-26T11:07:10.173 app[4d89705c6d5ee8] lhr [info] Fetched metadata for manrope
2023-04-26T11:07:10.870 app[4d89705c6d5ee8] lhr [info] Fetched metadata for inter
2023-04-26T11:07:10.936 app[4d89705c6d5ee8] lhr [info] Fetched metadata for lato
2023-04-26T11:07:11.199 app[5683d01a1043e8] lhr [info] initializing consul: key=fontsource/primary url=https://:e56d6a0c-6c59-e30e-fc67-3e412f38afb9@consul-fra-3.fly-shared.net/fontsource-6r85yqled4r92pvl/ hostname=5683d01a1043e8 advertise-url=http://5683d01a1043e8.vm.fontsource.internal:20202
2023-04-26T11:07:11.207 app[5683d01a1043e8] lhr [info] LiteFS mounted to: /litefs
2023-04-26T11:07:11.207 app[5683d01a1043e8] lhr [info] http server listening on: http://localhost:20202
2023-04-26T11:07:11.207 app[5683d01a1043e8] lhr [info] waiting to connect to cluster
2023-04-26T11:07:11.239 app[5683d01a1043e8] lhr [info] 0834326F3C26742B: existing primary found (4d89705c6d5ee8), connecting as replica
2023-04-26T11:07:11.242 app[4d89705c6d5ee8] lhr [info] CB81CA6E2B6B7B86: stream connected
2023-04-26T11:07:11.247 app[5683d01a1043e8] lhr [info] database file is zero length on initialization: /data/litefs/dbs/sqlite.db/database
2023-04-26T11:07:11.242 app[4d89705c6d5ee8] lhr [info] starting from txid 0000000000000001, writing snapshot
2023-04-26T11:07:11.244 app[4d89705c6d5ee8] lhr [info] writing snapshot "sqlite.db" @ 0000000000000131
2023-04-26T11:07:11.278 app[5683d01a1043e8] lhr [info] snapshot received for "sqlite.db", removing other ltx files: 0000000000000001-0000000000000131.ltx
2023-04-26T11:07:11.291 app[5683d01a1043e8] lhr [info] connected to cluster, ready
2023-04-26T11:07:11.291 app[5683d01a1043e8] lhr [info] node is a candidate, automatically promoting to primary
2023-04-26T11:07:11.357 app[5683d01a1043e8] lhr [info] 0834326F3C26742B: disconnected from primary, retrying
2023-04-26T11:07:11.356 app[4d89705c6d5ee8] lhr [info] CB81CA6E2B6B7B86: exiting primary, preserving lease for handoff
2023-04-26T11:07:11.357 app[5683d01a1043e8] lhr [info] 0834326F3C26742B: acquiring existing lease from handoff
2023-04-26T11:07:11.356 app[4d89705c6d5ee8] lhr [info] CB81CA6E2B6B7B86: stream disconnected
2023-04-26T11:07:11.357 app[4d89705c6d5ee8] lhr [info] fuse: write(): journal error: read only replica
2023-04-26T11:07:11.358 app[4d89705c6d5ee8] lhr [info] fuse: commit error: read only replica
2023-04-26T11:07:11.368 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.369 app[4d89705c6d5ee8] lhr [info] Fetched metadata for source-sans-pro
2023-04-26T11:07:11.372 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.397 app[4d89705c6d5ee8] lhr [info] CB81CA6E2B6B7B86: existing primary found (4d89705c6d5ee8), connecting as replica
2023-04-26T11:07:11.397 app[4d89705c6d5ee8] lhr [info] http: POST /stream: error: cannot connect to self
2023-04-26T11:07:11.397 app[4d89705c6d5ee8] lhr [info] CB81CA6E2B6B7B86: disconnected from primary with error, retrying: connect to primary: invalid response: code=400 ('http://4d89705c6d5ee8.vm.fontsource.internal:20202')
2023-04-26T11:07:11.405 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.439 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.492 app[5683d01a1043e8] lhr [info] 0834326F3C26742B: primary lease acquired, advertising as http://5683d01a1043e8.vm.fontsource.internal:20202
2023-04-26T11:07:11.493 app[5683d01a1043e8] lhr [info] proxy server listening on: http://localhost:8080
2023-04-26T11:07:11.493 app[5683d01a1043e8] lhr [info] executing command: npm [run migrations]
2023-04-26T11:07:11.498 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.508 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.509 app[4d89705c6d5ee8] lhr [info] Fetched metadata for roboto
2023-04-26T11:07:11.602 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.608 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.638 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.648 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.653 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.659 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.666 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.677 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.683 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.687 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.717 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.744 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.794 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.881 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.892 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.945 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.956 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.967 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.973 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.982 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:11.987 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.009 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.010 app[4d89705c6d5ee8] lhr [info] Fetched metadata for ibm-plex-mono
2023-04-26T11:07:12.013 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.022 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.048 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.099 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.164 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.182 app[5683d01a1043e8] lhr [info] > migrations
2023-04-26T11:07:12.182 app[5683d01a1043e8] lhr [info] > knex migrate:latest
2023-04-26T11:07:12.212 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.217 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.242 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.269 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.279 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.298 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.318 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.335 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.355 app[5683d01a1043e8] lhr [info] Using environment: production
2023-04-26T11:07:12.380 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.385 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.396 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.417 app[4d89705c6d5ee8] lhr [info] CB81CA6E2B6B7B86: existing primary found (5683d01a1043e8), connecting as replica
2023-04-26T11:07:12.420 app[5683d01a1043e8] lhr [info] 0834326F3C26742B: stream connected
2023-04-26T11:07:12.528 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.541 app[5683d01a1043e8] lhr [info] Already up to date
2023-04-26T11:07:12.546 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.551 app[5683d01a1043e8] lhr [info] starting background subprocess: npm [run start]
2023-04-26T11:07:12.551 app[5683d01a1043e8] lhr [info] waiting for signal or subprocess to exit
2023-04-26T11:07:12.555 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.568 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.584 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.588 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.612 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.652 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.660 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.677 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.720 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.832 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.864 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.874 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.879 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.880 app[4d89705c6d5ee8] lhr [info] Fetched metadata for poppins
2023-04-26T11:07:12.906 app[5683d01a1043e8] lhr [info] > start
2023-04-26T11:07:12.906 app[5683d01a1043e8] lhr [info] > remix-serve build
2023-04-26T11:07:12.933 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.947 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:12.997 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.029 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.122 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.179 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.213 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.230 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.277 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.326 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.456 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.511 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.517 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.577 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.583 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.584 app[4d89705c6d5ee8] lhr [info] Fetched metadata for barlow
2023-04-26T11:07:13.655 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.843 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.858 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:13.859 app[4d89705c6d5ee8] lhr [info] Fetched metadata for noto-sans
2023-04-26T11:07:13.892 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:14.072 app[5683d01a1043e8] lhr [info] Remix App Server started at http://localhost:8081 (http://172.19.136.242:8081)
2023-04-26T11:07:14.151 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:14.456 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:18.283 health[5683d01a1043e8] lhr [info] Health check on port 8080 is now passing.
2023-04-26T11:07:34.124 app[4d89705c6d5ee8] lhr [info] GET / 200 - - 143.277 ms
2023-04-26T11:07:34.838 app[4d89705c6d5ee8] lhr [info] fuse: create(): cannot create journal: read only replica
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] SqliteError: insert into `fonts` (`category`, `defSubset`, `family`, `id`, `lastModified`, `license`, `npmVersion`, `source`, `styles`, `subsets`, `type`, `variable`, `version`, `weights`) values ('monospace', 'latin', 'Source Code Pro', 'source-code-pro', '2022-09-22', '{"type":"OFL-1.1","url":"http://scripts.sil.org/OFL","attribution":"© 2010 - 2020 Adobe Systems Incorporated (http://www.adobe.com/), with Reserved Font Name ‘Source’."}', '4.5.0', 'https://github.com/google/fonts', 'italic,normal', 'cyrillic,cyrillic-ext,greek,greek-ext,latin,latin-ext,vietnamese', 'google', true, 'v22', '200,300,400,500,600,700,800,900') on conflict (`id`) do update set `category` = excluded.`category`, `defSubset` = excluded.`defSubset`, `family` = excluded.`family`, `id` = excluded.`id`, `lastModified` = excluded.`lastModified`, `license` = excluded.`license`, `npmVersion` = excluded.`npmVersion`, `source` = excluded.`source`, `styles` = excluded.`styles`, `subsets` = excluded.`subsets`, `type` = excluded.`type`, `variable` = excluded.`variable`, `version` = excluded.`version`, `weights` = excluded.`weights` - attempt to write a readonly database
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] at Client_BetterSQLite3._query (/app/node_modules/.pnpm/knex@2.4.2_better-sqlite3@8.3.0/node_modules/knex/lib/dialects/better-sqlite3/index.js:39:38)
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] at executeQuery (/app/node_modules/.pnpm/knex@2.4.2_better-sqlite3@8.3.0/node_modules/knex/lib/execution/internal/query-executioner.js:37:17)
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] at Client_BetterSQLite3.query (/app/node_modules/.pnpm/knex@2.4.2_better-sqlite3@8.3.0/node_modules/knex/lib/client.js:146:12)
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] at Runner.query (/app/node_modules/.pnpm/knex@2.4.2_better-sqlite3@8.3.0/node_modules/knex/lib/execution/runner.js:123:36)
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] at ensureConnectionCallback (/app/node_modules/.pnpm/knex@2.4.2_better-sqlite3@8.3.0/node_modules/knex/lib/execution/internal/ensure-connection-callback.js:13:17)
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] at Runner.ensureConnection (/app/node_modules/.pnpm/knex@2.4.2_better-sqlite3@8.3.0/node_modules/knex/lib/execution/runner.js:300:20)
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] at Runner.run (/app/node_modules/.pnpm/knex@2.4.2_better-sqlite3@8.3.0/node_modules/knex/lib/execution/runner.js:30:19)
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] at fetchMetadata (/app/build/index.js:1981:3)
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] at getCss (/app/build/index.js:1956:7) {
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] code: 'UNKNOWN_SQLITE_ERROR_1544'
2023-04-26T11:07:34.844 app[4d89705c6d5ee8] lhr [info] }
For reference, when you see a log line such as Fetched metadata for inter, assume the app is attempting to write to the DB. Another thing to note, we're spinning this up without using volumes, so these databases are initialised empty.
I also attempted redeploying with fly scale count 0 and starting again. This time I made sure no writes occurred until both VMs spun up and I'm still seeing the same fuse errors. Scaling down to 1 machine seems to have no issues.
Edit: Eventually I scaled up again and now the build is stable. But there are definitely some rough edges to consider.
benbjohnson commented
Thank you for the awesome new release! I moved over to the new version and really wanted to try the new exec feature for database migrations. Unfortunately, I seem to have run on an infinite loop of the following...
cannot find primary & ineligible to become primary, retrying: no primary
Thanks for the feedback! Since your lease.candidate is set to ${FLY_REGION == PRIMARY_REGION}, this will occur if one of your nodes is not in the PRIMARY_REGION. It looks like the first set of logs were running in sjc but the second set was able get a primary in lhr.
Another thing to note, we're spinning this up without using volumes, so these databases are initialised empty.
I definitely don't recommend this approach. If all your nodes go down at the same time then you'll lose your data since it's all on ephemeral disks. We have plans for making ephemeral disks work in the future but it's not advisable right now.
fuse: create(): cannot create journal: read only replica
If you're seeing these errors and you're using the LiteFS built-in proxy then you're probably issuing write transactions on GET requests. The proxy expects GET requests to be read-only and it'll proxy non-GET requests to the current primary.
ayuhito commented
Since your lease.candidate is set to ${FLY_REGION == PRIMARY_REGION}, this will occur if one of your nodes is not in the PRIMARY_REGION. It looks like the first set of logs were running in sjc but the second set was able get a primary in lhr.
Oh, this seems to be new for Apps V2! This makes sense to me now, I was wondering where the primary region env variable was coming from.
I definitely don't recommend this approach. If all your nodes go down at the same time then you'll lose your data since it's all on ephemeral disks. We have plans for making ephemeral disks work in the future but it's not advisable right now.
I'm using the DB as a cache so persistence isn't something I needed (especially for development). Will use volumes once I deploy for production!
If you're seeing these errors and you're using the LiteFS built-in proxy then you're probably issuing write transactions on GET requests. The proxy expects GET requests to be read-only and it'll proxy non-GET requests to the current primary.
I see! That makes a lot of sense. I'll have to rearchitect the code to handle that. Prior to upgrading to 0.4 and Fly Apps V2 I was only using one machine so I suppose that's how I missed this until now. Thanks a lot for your time!
cayblood commented
If you're seeing these errors and you're using the LiteFS built-in proxy then you're probably issuing write transactions on
GETrequests. The proxy expectsGETrequests to be read-only and it'll proxy non-GETrequests to the current primary.
@benbjohnson I have a remix app that seems to be experiencing intermittent failures ("fuse: create(): cannot create journal: read only replica") and I'm thinking it might be due to this. I'm assuming it works sometimes because I'm already on the primary.
In the particular case I'm dealing with, a user has clicked on an email link inviting them to use our app. I normally would avoid doing modifications in a GET request, but in this case it seemed like the most obvious approach. Any suggestions on how to refactor this? I was thinking of maybe initiating a fetch post inside the page's loader.
UPDATE: In case it helps anyone else, I have succeeded in getting this working by doing a fetch with method: 'POST' inside the loader to the same URL, passing on the same cookies from my loader's request. I then implemented an action inside that route that does the modifying calls to litefs.