attic_server::error: Database error: Query Error: error returned from database: (code: 6410) disk I/O error
Opened this issue · 1 comments
voidus commented
Hey folks,
I'm getting the error from the when running in github actions. My flake.lock
lists attic with revision fbe252a5c21febbe920c025560cbd63b20e24f3b
The github actions log shows:
error: unable to download 'https://nix-cache.<my-server>.tld/my-cache/nar/<some-nar-hash>.nar': HTTP error 500
response body:
{"code":500,"error":"InternalServerError","message":"The server encountered an internal error or misconfiguration."}
The systemd journal shows:
Feb 16 21:13:44 electra atticd[1751]: 2024-02-16T21:13:44.884923Z ERROR attic_server::error: Database error: Query Error: error returned from database: (code: 6410) disk I/O error
Feb 16 21:13:44 electra atticd[1751]: 0: attic_server::api::binary_cache::get_nar
Feb 16 21:13:44 electra atticd[1751]: at server/src/api/binary_cache.rs:167
Feb 16 21:13:44 electra atticd[1751]: 1: tower_http::trace::make_span::request
Feb 16 21:13:44 electra atticd[1751]: with method=GET uri=/cannahaus/nar/qs1mrabfkk50p5alxf9cml1qvfgrxc7x.nar version=HTTP/1.1
Feb 16 21:13:44 electra atticd[1751]: at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tower-http-0.4.4/src/trace/make_span.rs:109
Feb 16 21:13:44 electra atticd[1751]: 2: tokio::task::runtime.spawn
Feb 16 21:13:44 electra atticd[1751]: with kind=task task.name= task.id=326290 loc.file="/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/hyper-0.14.27>
Feb 16 21:13:44 electra atticd[1751]: at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.34.0/src/util/trace.rs:17
Feb 16 21:13:44 electra atticd[1751]: 2024-02-16T21:13:44.887729Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=21 ms
This repeats a bunch of times. There is nothing related, journalctl -k
is not showing anything either.
The relevant nixos config:
{
config,
pkgs,
...
}: let
host = "nix-cache.<my-server>.tld";
local-port = "8080";
inherit (pkgs.lib) concatStringsSep;
in {
sops.secrets = {
atticd_ssh_private_key = {owner = config.services.atticd.user;};
atticd_credentials_file = {owner = config.services.atticd.user;};
atticd_cifs_environmentfile = {};
};
environment.systemPackages = [
pkgs.cifs-utils
];
systemd.services.atticd = {
serviceConfig = {
DynamicUser = pkgs.lib.mkForce false;
};
unitConfig = {
RequiresMountsFor = "/var/lib/atticd/storage";
};
};
users = let
u = config.services.atticd.user;
in {
groups.${u} = {};
users.${u} = {
isSystemUser = true;
group = config.users.groups.${u}.name;
extraGroups = [config.users.groups.keys.name];
};
};
services = {
atticd = {
enable = true;
credentialsFile = config.sops.secrets.atticd_credentials_file.path;
settings = {
listen = "[::]:${local-port}";
allowed-hosts = [host];
api-endpoint = "https://${host}/";
garbage-collection.default-retention-period = "6 months";
# Data chunking
#
# Warning: If you change any of the values here, it will be
# difficult to reuse existing chunks for newly-uploaded NARs
# since the cutpoints will be different. As a result, the
# deduplication ratio will suffer for a while after the change.
chunking = {
# The minimum NAR size to trigger chunking
#
# If 0, chunking is disabled entirely for newly-uploaded NARs.
# If 1, all NARs are chunked.
nar-size-threshold = 64 * 1024; # 64 KiB
# The preferred minimum size of a chunk, in bytes
min-size = 16 * 1024; # 16 KiB
# The preferred average size of a chunk, in bytes
avg-size = 64 * 1024; # 64 KiB
# The preferred maximum size of a chunk, in bytes
max-size = 256 * 1024; # 256 KiB
};
};
};
nginx = {
enable = true;
recommendedProxySettings = true;
recommendedTlsSettings = true;
clientMaxBodySize = "1g";
virtualHosts.${host} = {
enableACME = true;
forceSSL = true;
locations = {
"/" = {
proxyPass = "http://[::]:${local-port}";
};
};
};
};
};
}
I'm somewhat at a loss how to debug this. I've tried adding RUST_LOG=debug
to the credentials file, but I don't think it changed anything.
There's a few other services running on the machine, so let me know if anything else might be relevant.
voidus commented
Just set everything up on a new VM to make sure, but this still happens. I think I'll have to set up a bucket instead :/