Agola CPU spikes to 120% and goes into tight futex/nanosleep loop after some time

Question

Agola CPU spikes to 120% and goes into tight futex/nanosleep loop after some time

milosivanovic opened this issue 10 months ago · 7 comments

What happened:
I'm not sure what triggers it, but after some time, agola starts to use 120% CPU and never stops, even though it is functionally doing nothing:

Once it gets into this state, restarting the agola server does not help - it immediately uses 120%+ CPU right after starting back up. Agola still seems to function normally during this time, but the high CPU usage does not drop until I delete all the agola databases and start over with a fresh installation, but then I lose all the build data and history.

What you expected to happen:
Agola should be able to maintain 0% CPU when it's not doing anything.

How to reproduce it (as minimally and precisely as possible):
Unfortunately, I am not sure. It seems to start happening after I push several commits, but I haven't been able to pinpoint an exact cause. Let me know if I can provide any additional debugging details.

This is the second time that this has happened. The first time, I deleted all the databases which cleared all the configuration and history, and it fixed the issue. It worked for a couple weeks until the issue happened again, so I figured I would open a bug to document the findings and hopefully identify a cause.

Environment:

Agola version: git master branch (b179cb2)
Agola running environment (if useful to understand the bug): Linux

I ran strace -f -p <pid> and I can see it is stuck in a tight futex/nanosleep loop:

comet ~ # strace -f -p 16530                                                                                                                                                                                            18:11:21 [39810/39815]
strace: Process 16530 attached with 23 threads
[pid 16555] futex(0xc000b81148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16554] futex(0xc000d86148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16551] futex(0xc000c00148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16550] futex(0xc000b80148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16548] futex(0xc00055e148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16546] futex(0xc000a80148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16545] futex(0xc000a00148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16544] futex(0xc0000ff948, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16543] futex(0xc000600948, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16535] futex(0xc000500148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16530] futex(0x41d9ea0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16547] futex(0xc000c00948, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 16533] futex(0xc0000d3948, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16547] <... futex resumed>)        = 0
[pid 16537] futex(0xc000600148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16532] futex(0xc0000d3148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16531] restart_syscall(<... resuming interrupted futex ...> <unfinished ...>
[pid 16547] fcntl(14, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=120, l_len=1} <unfinished ...>
[pid 16553] futex(0xc000b80948, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16552] futex(0xc000c00948, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16549] futex(0xc000b00148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16539] futex(0xc0006a0148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16547] <... fcntl resumed>)        = 0
[pid 16538] futex(0xc0004be148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16547] fcntl(14, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=124, l_len=1} <unfinished ...>
[pid 16536] epoll_pwait(4,  <unfinished ...>
[pid 16534] futex(0x423bb38, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 16547] <... fcntl resumed>)        = 0
[pid 16547] getrandom("\x00\x80\x8b\x22\x01\xf8\xda\x86\xdc\x7c\x4f\xff\x7e\x7c\x4c\xc8", 16, 0) = 16
[pid 16547] fcntl(14, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=124, l_len=1}) = 0
[pid 16547] fcntl(14, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=120, l_len=1}) = 0
[pid 16552] <... futex resumed>)        = -1 EAGAIN (Resource temporarily unavailable)
[pid 16547] fcntl(14, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=120, l_len=1} <unfinished ...>
[pid 16536] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 16547] <... fcntl resumed>)        = 0
[pid 16547] fcntl(14, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=124, l_len=1}) = 0
[pid 16547] getrandom("\x68\xff\x7b\xb2\x9f\x12\xa2\xc2\x9f\xe8\x1f\x90\x6c\xa6\xc0\xc3", 16, 0) = 16
[pid 16547] fcntl(14, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=124, l_len=1} <unfinished ...>
[pid 16552] epoll_pwait(4,  <unfinished ...>
[pid 16547] <... fcntl resumed>)        = 0
[pid 16552] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 16547] fcntl(14, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=120, l_len=1} <unfinished ...>
[pid 16536] epoll_pwait(4,  <unfinished ...>
[pid 16547] <... fcntl resumed>)        = 0
[pid 16536] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 16552] futex(0xc000d86148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 16536] epoll_pwait(4,  <unfinished ...>
[pid 16552] <... futex resumed>)        = 1
[pid 16536] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 16554] <... futex resumed>)        = 0
<...snip...>

Here is the system call summary after 10s:

comet ~ # timeout 10 strace -c -f -p 16530
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 84.17   19.429929          41    463870      6614 futex
  8.62    1.990271          17    111794       130 nanosleep
  6.50    1.500132         615      2436         1 epoll_pwait
  0.38    0.087532           0    261162           fcntl
  0.11    0.026024           5      5107           tgkill
  0.10    0.022536           4      5107           getpid
  0.06    0.014966           0     65283           getrandom
  0.04    0.009026           1      5107       174 rt_sigreturn
  0.01    0.002656           1      2306           sched_yield
  0.00    0.000914           4       215           write
  0.00    0.000695           1       409       119 read
  0.00    0.000498           1       375           madvise
  0.00    0.000052          52         1           restart_syscall
  0.00    0.000013           2         5           openat
  0.00    0.000007           0        10           getdents64
  0.00    0.000005           1         5           close
  0.00    0.000000           0        20           pwrite64
  0.00    0.000000           0         5         5 epoll_ctl
  0.00    0.000000           0        15           newfstatat
------ ----------- ----------- --------- --------- ----------------
100.00   23.085256          25    923232      7043 total

Answer 1 · 2024-02-16T06:24:25.000Z

@milosivanovic since it's just probably something looping forever to see what's happening you should provide a stack trace of the processes (SIGQUIT) and perhaps the logs of when this is happening. Syscalls aren't useful in such case.

Answer 2 · 2024-02-16T06:37:04.000Z

Sure - here is the raw output after sending SIGQUIT to the main process which was running for just a few seconds and had the 120% CPU issue:

agola_stack_traces.txt

Answer 3 · 2024-02-16T06:47:39.000Z

I re-ran agola for 8 seconds with debug: true set in each service config. The CPU immediately spiked to 120% (as expected) and agola produced a small chunk of log output to stdout every 2 seconds:

comet ~/d/agola # ./bin/agola serve --config /root/swarm/agola/config_nondocker.yml --components all-base,executor
2024-02-15T22:43:57.134413215-08:00 DBG internal/services/notification/webhooks.go:132 > webhookCleanerLoop
2024-02-15T22:43:57.134659182-08:00 DBG internal/services/executor/executor.go:1089 > executorStatusSenderLoop
2024-02-15T22:43:57.134613827-08:00 DBG internal/services/notification/commitstatus.go:138 > commitStatusesCleanerLoop
2024-02-15T22:43:57.134726757-08:00 DBG internal/services/executor/executor.go:1106 > executorTasksStatusSenderLoop
2024-02-15T22:43:57.134820864-08:00 DBG internal/services/executor/executor.go:1142 > tasksUpdater
2024-02-15T22:43:57.134728233-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:43:57.134824973-08:00 DBG internal/services/executor/executor.go:1270 > tasksDataCleaner
2024-02-15T22:43:57.134825894-08:00 DBG internal/services/executor/executor.go:1576 > tasksTimeoutCleaner
2024-02-15T22:43:57.135324643-08:00 DBG internal/services/runservice/scheduler.go:1345 > fetcher
2024-02-15T22:43:57.135325424-08:00 DBG internal/services/runservice/scheduler.go:1495 > finished run archiver loop
2024-02-15T22:43:57.135344569-08:00 DBG internal/services/runservice/scheduler.go:1361 > fetcher
2024-02-15T22:43:57.135351647-08:00 DBG internal/services/runservice/scheduler.go:1511 > finished run archiver
2024-02-15T22:43:57.135353153-08:00 DBG internal/services/runservice/scheduler.go:1658 > objectsCleaner
2024-02-15T22:43:57.135360984-08:00 DBG internal/services/runservice/scheduler.go:1450 > runsSchedulerLoop
2024-02-15T22:43:57.135397426-08:00 DBG internal/services/runservice/scheduler.go:1466 > runsScheduler
2024-02-15T22:43:57.135363169-08:00 DBG internal/services/runservice/scheduler.go:1599 > cacheCleaner
2024-02-15T22:43:57.135359596-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:43:57.13536822-08:00 DBG internal/services/runservice/scheduler.go:970 > runTasksUpdater
2024-02-15T22:43:57.135413945-08:00 DBG internal/services/runservice/scheduler.go:986 > runTasksUpdater
2024-02-15T22:43:57.135353614-08:00 DBG internal/services/runservice/scheduler.go:1641 > logCleanerLoop
2024-02-15T22:43:57.135452566-08:00 DBG internal/services/runservice/scheduler.go:1658 > objectsCleaner
2024-02-15T22:43:57.135505688-08:00 INF internal/services/runservice/runservice.go:43 > maintenance mode watcher: maintenance mode enabled: false
2024-02-15T22:43:57.135547184-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:43:57.136755848-08:00 DBG internal/services/executor/executor.go:1169 > ets: []*types.ExecutorTask{}
2024-02-15T22:43:57.136923357-08:00 INF internal/services/configstore/configstore.go:44 > maintenance mode watcher: maintenance mode enabled: false
2024-02-15T22:43:57.136951925-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:43:57.137265074-08:00 DBG internal/services/executor/executor.go:734 > send executor status: &types.ExecutorStatus{
  ListenURL: "http://192.168.88.3:4001",
  Archs: []types.Arch{
    "amd64",
  },
  Labels: map[string]string{},
  AllowPrivilegedContainers: false,
  ActiveTasksLimit: 2,
  ActiveTasks: 0,
  Dynamic: false,
  ExecutorGroup: "515ba71a-a58a-4d58-b699-f77d060acc1a",
  SiblingsExecutors: []string{
    "515ba71a-a58a-4d58-b699-f77d060acc1a",
  },
}
2024-02-15T22:43:58.136955321-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:43:58.136976599-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:43:58.137037368-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:43:58.13734305-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:43:59.13493095-08:00 DBG internal/services/executor/executor.go:1270 > tasksDataCleaner
2024-02-15T22:43:59.134930006-08:00 DBG internal/services/executor/executor.go:1576 > tasksTimeoutCleaner
2024-02-15T22:43:59.13492976-08:00 DBG internal/services/executor/executor.go:1106 > executorTasksStatusSenderLoop
2024-02-15T22:43:59.135540026-08:00 DBG internal/services/runservice/scheduler.go:1345 > fetcher
2024-02-15T22:43:59.135573187-08:00 DBG internal/services/runservice/scheduler.go:1361 > fetcher
2024-02-15T22:43:59.1361108-08:00 DBG internal/services/runservice/scheduler.go:1495 > finished run archiver loop
2024-02-15T22:43:59.136138183-08:00 DBG internal/services/runservice/scheduler.go:1511 > finished run archiver
2024-02-15T22:43:59.136899799-08:00 DBG internal/services/executor/executor.go:1142 > tasksUpdater
2024-02-15T22:43:59.136898747-08:00 DBG internal/services/runservice/scheduler.go:1450 > runsSchedulerLoop
2024-02-15T22:43:59.136961853-08:00 DBG internal/services/runservice/scheduler.go:1466 > runsScheduler
2024-02-15T22:43:59.137112258-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:43:59.137111236-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:43:59.137170212-08:00 DBG internal/services/executor/executor.go:1169 > ets: []*types.ExecutorTask{}
2024-02-15T22:43:59.137140897-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:43:59.137784528-08:00 DBG internal/services/executor/executor.go:1089 > executorStatusSenderLoop
2024-02-15T22:43:59.139432447-08:00 DBG internal/services/executor/executor.go:734 > send executor status: &types.ExecutorStatus{
  ListenURL: "http://192.168.88.3:4001",
  Archs: []types.Arch{
    "amd64",
  },
  Labels: map[string]string{},
  AllowPrivilegedContainers: false,
  ActiveTasksLimit: 2,
  ActiveTasks: 0,
  Dynamic: false,
  ExecutorGroup: "515ba71a-a58a-4d58-b699-f77d060acc1a",
  SiblingsExecutors: []string{
    "515ba71a-a58a-4d58-b699-f77d060acc1a",
  },
}
2024-02-15T22:43:59.139706655-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:00.137285332-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:00.137378263-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:00.137470329-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:00.141127454-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:01.134987177-08:00 DBG internal/services/executor/executor.go:1106 > executorTasksStatusSenderLoop
2024-02-15T22:44:01.1349885-08:00 DBG internal/services/executor/executor.go:1576 > tasksTimeoutCleaner
2024-02-15T22:44:01.134989389-08:00 DBG internal/services/executor/executor.go:1270 > tasksDataCleaner
2024-02-15T22:44:01.135794413-08:00 DBG internal/services/runservice/scheduler.go:1345 > fetcher
2024-02-15T22:44:01.135806459-08:00 DBG internal/services/runservice/scheduler.go:1361 > fetcher
2024-02-15T22:44:01.136311023-08:00 DBG internal/services/runservice/scheduler.go:1495 > finished run archiver loop
2024-02-15T22:44:01.136324857-08:00 DBG internal/services/runservice/scheduler.go:1511 > finished run archiver
2024-02-15T22:44:01.137071986-08:00 DBG internal/services/runservice/scheduler.go:1450 > runsSchedulerLoop
2024-02-15T22:44:01.137086227-08:00 DBG internal/services/runservice/scheduler.go:1466 > runsScheduler
2024-02-15T22:44:01.13723224-08:00 DBG internal/services/executor/executor.go:1142 > tasksUpdater
2024-02-15T22:44:01.137388638-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:01.137477751-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:01.13748769-08:00 DBG internal/services/executor/executor.go:1169 > ets: []*types.ExecutorTask{}
2024-02-15T22:44:01.137704348-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:01.139839714-08:00 DBG internal/services/executor/executor.go:1089 > executorStatusSenderLoop
2024-02-15T22:44:01.141516077-08:00 DBG internal/services/executor/executor.go:734 > send executor status: &types.ExecutorStatus{
  ListenURL: "http://192.168.88.3:4001",
  Archs: []types.Arch{
    "amd64",
  },
  Labels: map[string]string{},
  AllowPrivilegedContainers: false,
  ActiveTasksLimit: 2,
  ActiveTasks: 0,
  Dynamic: false,
  ExecutorGroup: "515ba71a-a58a-4d58-b699-f77d060acc1a",
  SiblingsExecutors: []string{
    "515ba71a-a58a-4d58-b699-f77d060acc1a",
  },
}
2024-02-15T22:44:01.142895198-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:02.137454546-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:02.137522325-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:02.13775333-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:02.144428202-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:03.135157315-08:00 DBG internal/services/executor/executor.go:1270 > tasksDataCleaner
2024-02-15T22:44:03.13515986-08:00 DBG internal/services/executor/executor.go:1576 > tasksTimeoutCleaner
2024-02-15T22:44:03.135160018-08:00 DBG internal/services/executor/executor.go:1106 > executorTasksStatusSenderLoop
2024-02-15T22:44:03.135953122-08:00 DBG internal/services/runservice/scheduler.go:1345 > fetcher
2024-02-15T22:44:03.135979465-08:00 DBG internal/services/runservice/scheduler.go:1361 > fetcher
2024-02-15T22:44:03.136412754-08:00 DBG internal/services/runservice/scheduler.go:1495 > finished run archiver loop
2024-02-15T22:44:03.136424991-08:00 DBG internal/services/runservice/scheduler.go:1511 > finished run archiver
2024-02-15T22:44:03.137201763-08:00 DBG internal/services/runservice/scheduler.go:1450 > runsSchedulerLoop
2024-02-15T22:44:03.137211015-08:00 DBG internal/services/runservice/scheduler.go:1466 > runsScheduler
2024-02-15T22:44:03.13750215-08:00 DBG internal/services/executor/executor.go:1142 > tasksUpdater
2024-02-15T22:44:03.137595098-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:03.13762987-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:03.137645347-08:00 DBG internal/services/executor/executor.go:1169 > ets: []*types.ExecutorTask{}
2024-02-15T22:44:03.13782317-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:03.141838273-08:00 DBG internal/services/executor/executor.go:1089 > executorStatusSenderLoop
2024-02-15T22:44:03.143436024-08:00 DBG internal/services/executor/executor.go:734 > send executor status: &types.ExecutorStatus{
  ListenURL: "http://192.168.88.3:4001",
  Archs: []types.Arch{
    "amd64",
  },
  Labels: map[string]string{},
  AllowPrivilegedContainers: false,
  ActiveTasksLimit: 2,
  ActiveTasks: 0,
  Dynamic: false,
  ExecutorGroup: "515ba71a-a58a-4d58-b699-f77d060acc1a",
  SiblingsExecutors: []string{
    "515ba71a-a58a-4d58-b699-f77d060acc1a",
  },
}
2024-02-15T22:44:03.146057005-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:04.137649644-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:04.137690427-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:04.137936275-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:04.147450947-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:05.135229251-08:00 DBG internal/services/executor/executor.go:1576 > tasksTimeoutCleaner
2024-02-15T22:44:05.135229394-08:00 DBG internal/services/executor/executor.go:1270 > tasksDataCleaner
2024-02-15T22:44:05.135228478-08:00 DBG internal/services/executor/executor.go:1106 > executorTasksStatusSenderLoop
2024-02-15T22:44:05.136136144-08:00 DBG internal/services/runservice/scheduler.go:1345 > fetcher
2024-02-15T22:44:05.136160683-08:00 DBG internal/services/runservice/scheduler.go:1361 > fetcher
2024-02-15T22:44:05.136520172-08:00 DBG internal/services/runservice/scheduler.go:1495 > finished run archiver loop
2024-02-15T22:44:05.136545153-08:00 DBG internal/services/runservice/scheduler.go:1511 > finished run archiver
2024-02-15T22:44:05.137328518-08:00 DBG internal/services/runservice/scheduler.go:1450 > runsSchedulerLoop
2024-02-15T22:44:05.137352317-08:00 DBG internal/services/runservice/scheduler.go:1466 > runsScheduler
2024-02-15T22:44:05.137705609-08:00 DBG internal/services/executor/executor.go:1142 > tasksUpdater
2024-02-15T22:44:05.137743451-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:05.137770757-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:05.137868515-08:00 DBG internal/services/executor/executor.go:1169 > ets: []*types.ExecutorTask{}
2024-02-15T22:44:05.138083771-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:05.143809241-08:00 DBG internal/services/executor/executor.go:1089 > executorStatusSenderLoop
2024-02-15T22:44:05.144960861-08:00 DBG internal/services/executor/executor.go:734 > send executor status: &types.ExecutorStatus{
  ListenURL: "http://192.168.88.3:4001",
  Archs: []types.Arch{
    "amd64",
  },
  Labels: map[string]string{},
  AllowPrivilegedContainers: false,
  ActiveTasksLimit: 2,
  ActiveTasks: 0,
  Dynamic: false,
  ExecutorGroup: "515ba71a-a58a-4d58-b699-f77d060acc1a",
  SiblingsExecutors: []string{
    "515ba71a-a58a-4d58-b699-f77d060acc1a",
  },
}
2024-02-15T22:44:05.148948977-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
^C
comet ~/d/agola #

Answer 4 · 2024-02-16T07:36:57.000Z

@milosivanovic Thanks for the stack trace. I think I know what's going on and will open a PR soon.

Answer 5 · 2024-02-16T08:00:17.000Z

Should be fixed by #487

Answer 6 · 2024-02-16T08:21:27.000Z

@sgotti wow, thank you so much - I checked out the #487 PR locally and confirmed that it fixed the bug!

Answer 7 · 2024-02-16T08:48:21.000Z

@milosivanovic Thanks for reporting and testing. Merged #487.