go-graphite/buckytools

bucky modify panic : truncate : invalid argument

Thorsieger opened this issue · 8 comments

Thorsieger commented

Hello everyone,

I use buckytools for managing a graphite infrastructure (carbon-c-relay -> go-carbon -> carbonapi).
I'm trying to use the modify command to resize whisper files but the program failed.

General informations :

  • Buckytools version : 0.4.2-gg

  • Go : go version go1.17.2 linux/amd64

Output of the command :

/go $ bucky modify -index 1 -retention 60s:10d -f /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp 
panic: truncate /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp: invalid argument

goroutine 1 [running]:
main.modifyCommand({{0x6cc5ca, 0x6}, 0x6e8878, {0x6cd25c, 0x9}, {0x6dafe1, 0x2c}, {0x6e1e1c, 0x36e}, 0xc000100720})
	/go/src/github.com/go-graphite/buckytools/cmd/bucky/modify.go:84 +0x44e
main.main()
	/go/src/github.com/go-graphite/buckytools/cmd/bucky/main.go:101 +0x198
deniszh commented

@Thorsieger : is file /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp really exist and it's writable for bucky daemon? invalid argument could mean that file has wrong permission and bucky can't operate on it with current permissions - golang/go#9640

  • 👍1
Thorsieger commented

Yep. I'm running buckytools as carbon user and he can r/w the file

/go $ whoami
carbon
/go $ ls -l /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp
-rw-r--r--    1 carbon   carbon       25960 Nov  2 15:44 /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp
deniszh commented

Hi @Thorsieger,

Well, the I have no ideas. Error is very simple - golang tried to truncate file and got "invalid argument" error back, which looks like permission error. At least I can't see how that can be explained differently or fixed from bucky side....

bom-d-van commented

yep, it seems like an issue with the whisper file or the file system. you can try to use whisper-resizepy to see check if it can do the job.

sudo -u carbon whisper-resize.py xxx
Thorsieger commented

whisper-resize.py is working fine :

whisper-resize.py /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp 60s:10d
Retrieving all data from the archives
Creating new whisper database: /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp.tmp
Created: /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp.tmp (172828 bytes)
Migrating data without aggregation...
Renaming old database to: /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp.bak
Renaming new database to: /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp

I find here that (File).Truncate get an error on linux because it use syscall.Ftruncate.

using os.Truncate(resizeFilename,0) instead work.

But then a get the following error :

panic: write /opt/graphite/storage/whisper/haggar/agent/0/metrics/1.wsp: bad file descriptor

goroutine 1 [running]:
main.modifyCommand({{0x6cb10a, 0x6}, 0x6e7500, {0x6cbd9c, 0x9}, {0x6d9bc9, 0x2c}, {0x6e0a61, 0x36e}, 0xc00007c720})
	/go/src/github.com/go-graphite/buckytools/cmd/bucky/modify.go:118 +0x411
main.main()
	/go/src/github.com/go-graphite/buckytools/cmd/bucky/main.go:101 +0x198
  • 👍1
Thorsieger commented

I'm a golang beginner but I think that I found something interesting :

the modify command use os.Open() : https://github.com/go-graphite/buckytools/blob/master/cmd/bucky/modify.go#L63

but this fonction open the file as read only : https://cs.opensource.google/go/go/+/refs/tags/go1.17.3:src/os/file.go;l=317
So then the whisper.Create2 fail to write the data to the file

it's maybe the same reason that make truncate function not work. Using OpenFile() with os.O_RDWR option could solve this. I will try next week.

  • 👍1
bom-d-van commented

Yes, it's the root cause. Not sure why the issue didn't show up when I was working on it.

  • 👍1
Thorsieger commented

#49 fix this I think