canonical/microcloud

rpc error: code = Unknown desc = exec (try: 500): database is locked

kcrumpler10 opened this issue · 3 comments

kcrumpler10 commented

This error happens constantly and quite frankly I'm overwhelmed with where to begin looking into this issue, what are some of the things that I should try to look into to fix this? I have a MicroCloud configured with MicroCeph remote storage. On the LXD Cluster Hosts, I've deployed Charmed MicroK8s. So the DQLite database appears to be locking, and that is also indirectly dependent on the Ceph Remote Storage.

What things should I consider while attempting to debug this issue?

MggMuggins commented

Hi! What command(s) did you run to produce the error? There are a few dqlite clusters in a typical MicroCloud deployment (LXD, MicroCeph, MicroOVN, MicroCloud), and MicroK8s also includes one. I'd like to determine which tool is producing the error and we can continue troubleshooting from there.

It would also be helpful if you could indicate the versions of each tool you're using (snap list should be sufficient). Thanks!

kcrumpler10 commented

Hey!

On the MicroCloud Host Machine,

Name        Version                 Rev    Tracking       Publisher   Notes
core        16-2.61.4-20240607      17200  latest/stable  canonical✓  core
core20      20240416                2318   latest/stable  canonical✓  base
core22      20240408                1380   latest/stable  canonical✓  base
jq          1.5+dfsg-1              6      latest/stable  mvo✪        -
lxd         6.1-c14927a             29551  latest/stable  canonical✓  -
microceph   18.2.0+snapcba31e8c75   999    reef/stable    canonical✓  -
microcloud  1.1-04a1c49             734    latest/stable  canonical✓  -
microovn    22.03.3+snap0e23a0e4f5  395    22.03/stable   canonical✓  -
snapd       2.63                    21759  latest/stable  canonical✓  snapd

Then I just ran,

Bootstrap a Juju controller onto MicroCloud

juju bootstrap microcloud microcloud-controller --bootstrap-base=ubuntu@22.04

Deploy Charmed MicroK8s

juju deploy microk8s -n 4 --channel="1.28/stable"

I had to restart the LXD VM's with,

lxc stop --all
lxc start --all

And restart the MicroK8s installation,

juju exec --all -- sudo microk8s stop
juju exec --all -- sudo microk8s start

This finally got the MicroK8s model to display as operational.

Then when I go to add-on anything like microk8s enable istio or metallb it does the database is locked error.

This is strange because it has worked before and was able to enable load istio and metallb, but then when I misconfigured istio and just deleted the juju model and redeployed the Charmed MicroK8s and now it's having issues where it won't install.

I know this is convoluted, am I doing some kind of architecture pattern than just shouldn't be used?

I.e. Installing Charmed Microk8s into the MicroCloud as a Juju private cloud target?

I feel like if I don't use a standard architecture pattern that is widely adopted then I'll just continue to get overwritten 😅

This is the snap list for the developer machine that is connected to the microcloud,

Name        Version             Rev    Tracking          Publisher       Notes
bare        1.0                 5      latest/stable     canonical✓      base
charmcraft  2.7.1               4162   latest/stable     canonical✓      classic
core        16-2.61.4-20240607  17200  latest/stable     canonical✓      core
core20      20240416            2318   latest/stable     canonical✓      base
core22      20240408            1380   latest/stable     canonical✓      base
jhack       0.4.2.1             365    latest/candidate  ppasotti        -
jq          1.5+dfsg-1          6      latest/stable     mvo✪            -
juju        3.5.3               28060  3/stable          canonical✓      -
kubectl     1.30.3              3315   latest/stable     canonical✓      classic
lxd         6.1-c14927a         29551  latest/stable     canonical✓      -
pebble      v1.14.1             784    latest/stable     canonical✓      classic
rclone      1.67.0              504    latest/stable     aoilinux        -
rockcraft   1.5.3               2218   latest/stable     canonical✓      classic
skaffold    v0.25.0             1      latest/stable     terraform-snap  -
snapcraft   8.3.1               12136  latest/stable     canonical✓      classic
snapd       2.63                21759  latest/stable     canonical✓      snapd
tree        1.8.0+pkg-3fd6      18     latest/stable     brlin           -
vault       1.15.6              2226   1.15/stable       canonical✓      -
MggMuggins commented

Please file a bug in MicroK8s or Charmed MicroK8s with the following information:

  • juju status
  • snap list from inside one of the Microk8s units (juju ssh 0 or similar)
  • Any relevant log information from /var/log/juju in the Microk8s units
  • The command you ran to get database is locked and its output
  • Any additional information requested in the bug report template

Thanks!

  • 👍1