Azure: Hitting API limit - maybe add throttling?
cheekoo opened this issue · 8 comments
Running Komiser on mac, against Azure cloud. Hitting 429, RESPONSE 429: 429 Too Many Requests. Can we add either throttling or re-starting where previous run failed?
Logs below -
_```
komiser start
INIT!!!
INFO[2024-04-13T19:55:15-05:00] Debug logging is enabled
INFO[2024-04-13T19:55:15-05:00] Data will be stored in SQLite
WARN[2024-04-13T19:55:15-05:00] Azure account cannot be inserted to database
constraint failed: UNIQUE constraint failed: accounts.credentials (2067)
INFO[2024-04-13T19:55:15-05:00] there are no new migrations to run (database is up to date)
INFO[2024-04-13T19:55:15-05:00] Komiser version: 3.1.15, commit: 783b8a4, buildt: 1712400646
INFO[2024-04-13T19:55:15-05:00] Fetching resources workflow has started
INFO[2024-04-13T19:55:15-05:00] Workflow triggered for client: Sample Subscription
INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=0 service="Application Gateway"
INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=0 service=Image
INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=0 service=Firewall
INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=0 service=Snapshot
INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=0 service=Databox
INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=3 service="PostgreSQL Database Servers"
INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=63 service=Disk
INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=6 service="Load Balancer"
INFO[2024-04-13T19:55:18-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=370 service=Table
INFO[2024-04-13T19:55:18-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=1 service=Queue
WARN[2024-04-13T19:55:34-05:00] failed to query usage: POST https://management.azure.com/subscriptions/xyz/providers/Microsoft.CostManagement/query
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
{
"error": {
"code": "429",
"message": "Too many requests. Please retry."
}
}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x28 pc=0x102135b28]
goroutine 50 [running]:
github.com/tailwarden/komiser/providers/azure/compute.VirtualMachines({0x103b11d40, 0x105d4d6e0}, {0x0, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, ...})
/home/runner/work/komiser/komiser/providers/azure/compute/virtual_machines.go:58 +0x768
github.com/tailwarden/komiser/providers/azure.FetchResources.func1()
/home/runner/work/komiser/komiser/providers/azure/azure.go:37 +0x7c
github.com/tailwarden/komiser/providers.(*WorkerPool).worker(0x14001100040)
/home/runner/work/komiser/komiser/providers/providers.go:85 +0x3c
created by github.com/tailwarden/komiser/providers.(*WorkerPool).Start in goroutine 7
/home/runner/work/komiser/komiser/providers/providers.go:69 +0x2c
Hey, thank you for opening your first Issue! 🙂 While a Tailwarden team member takes a look at your issue we would like to invite you to join our official Discord server, where you can interact directly with other contributors and Tailwarden team members. Link here: https://discord.tailwarden.com