rackspace/rack

fatal error: runtime: out of memory

Closed this issue · 1 comments

mxxcon commented

I have a 11,410,658,517 bytes file which I'm trying to upload to a Cloud Files bucket using 15GB Performance server.

   rack version 1.0.1
   commit: 71a8d7c80b3652b4dd39e683bc423b8a542b0167

root@prod-ci-mongodb1:~# ./rack files object upload --container MongoDump --file /data/mongodb/backup/dump/full_09_21_2015_dump.zip --name full_09_21_2015_dump.zip
fatal error: runtime: out of memory

runtime stack:
runtime.SysMap(0xc408300000, 0x200000000, 0xbb7600, 0xbc23b8)
        /usr/local/go/src/runtime/mem_linux.c:149 +0x98
runtime.MHeap_SysAlloc(0xbc7aa0, 0x200000000, 0x434d52)
        /usr/local/go/src/runtime/malloc.c:284 +0x124
runtime.MHeap_Alloc(0xbc7aa0, 0x100000, 0x10100000000, 0x0)
        /usr/local/go/src/runtime/mheap.c:240 +0x66

goroutine 12 [running]:
runtime.switchtoM()
        /usr/local/go/src/runtime/asm_amd64.s:198 fp=0xc20826b538 sp=0xc20826b530
runtime.mallocgc(0x1fffffe00, 0x7922c0, 0x1, 0x1000)
        /usr/local/go/src/runtime/malloc.go:199 +0x9f3 fp=0xc20826b5e8 sp=0xc20826b538
runtime.newarray(0x7922c0, 0x1fffffe00, 0xc208262380)
        /usr/local/go/src/runtime/malloc.go:365 +0xc1 fp=0xc20826b620 sp=0xc20826b5e8
runtime.makeslice(0x784500, 0x1fffffe00, 0x1fffffe00, 0x0, 0x0, 0x0)
        /usr/local/go/src/runtime/slice.go:32 +0x15c fp=0xc20826b668 sp=0xc20826b620
bytes.makeSlice(0x1fffffe00, 0x0, 0x0, 0x0)
        /usr/local/go/src/bytes/buffer.go:191 +0x6a fp=0xc20826b6b8 sp=0xc20826b668
bytes.(*Buffer).ReadFrom(0xc20800f1f0, 0x7f57cccb65c0, 0xc208243f60, 0xfffffe00, 0x0, 0x0)
        /usr/local/go/src/bytes/buffer.go:163 +0xda fp=0xc20826b750 sp=0xc20826b6b8
io.Copy(0x7f57cccb65e8, 0xc20800f1f0, 0x7f57cccb65c0, 0xc208243f60, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/io.go:358 +0x13d fp=0xc20826b808 sp=0xc20826b750
github.com/rackspace/rack/internal/github.com/rackspace/gophercloud/openstack/objectstorage/v1/objects.Create(0xc2080955f0, 0xc208029350, 0x9, 0xc208243f00, 0x18, 0x7f57cccb64f8, 0xc208036068, 0x7f57cccb6528, 0xc2081c5b20, 0x0, ...)
        /home/travis/gopath/src/github.com/rackspace/rack/internal/github.com/rackspace/gophercloud/openstack/objectstorage/v1/objects/requests.go:219 +0x5c8 fp=0xc20826bb98 sp=0xc20826b808
github.com/rackspace/rack/internal/github.com/rackspace/gophercloud/rackspace/objectstorage/v1/objects.Create(0xc2080955f0, 0xc208029350, 0x9, 0xc208243f00, 0x18, 0x7f57cccb64f8, 0xc208036068, 0x7f57cccb6528, 0xc2081c5b20, 0x0, ...)
        /home/travis/gopath/src/github.com/rackspace/rack/internal/github.com/rackspace/gophercloud/rackspace/objectstorage/v1/objects/delegate.go:37 +0xaa fp=0xc20826bc38 sp=0xc20826bb98
github.com/rackspace/rack/commands/filescommands/objectcommands.(*commandUpload).Execute(0xc208036020, 0xc208038e60)
        /home/travis/gopath/src/github.com/rackspace/rack/commands/filescommands/objectcommands/upload.go:163 +0x17f fp=0xc20826bf68 sp=0xc20826bc38
github.com/rackspace/rack/handler.func·003()
        /home/travis/gopath/src/github.com/rackspace/rack/handler/handle.go:175 +0xf0 fp=0xc20826bfe0 sp=0xc20826bf68
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20826bfe8 sp=0xc20826bfe0
created by github.com/rackspace/rack/handler.handleExecute
        /home/travis/gopath/src/github.com/rackspace/rack/handler/handle.go:178 +0xc0e

goroutine 1 [chan receive]:
github.com/rackspace/rack/handler.Handle(0x7f57cccaf538, 0xc208036020)
        /home/travis/gopath/src/github.com/rackspace/rack/handler/handle.go:110 +0x4cc
github.com/rackspace/rack/commands/filescommands/objectcommands.actionUpload(0xc20807ab60)
        /home/travis/gopath/src/github.com/rackspace/rack/commands/filescommands/objectcommands/upload.go:82 +0xd4
github.com/rackspace/rack/internal/github.com/codegangsta/cli.Command.Run(0x90cd50, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc2080387d0, 0x50, 0x93acb0, ...)
        /home/travis/gopath/src/github.com/rackspace/rack/internal/github.com/codegangsta/cli/command.go:118 +0x1044
github.com/rackspace/rack/internal/github.com/codegangsta/cli.(*App).RunAsSubcommand(0xc208069700, 0xc20807a9a0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/rackspace/rack/internal/github.com/codegangsta/cli/app.go:257 +0xc2a
github.com/rackspace/rack/internal/github.com/codegangsta/cli.Command.startApp(0x909cb0, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x979430, 0x25, 0x0, ...)
        /home/travis/gopath/src/github.com/rackspace/rack/internal/github.com/codegangsta/cli/command.go:183 +0x505
github.com/rackspace/rack/internal/github.com/codegangsta/cli.Command.Run(0x909cb0, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x979430, 0x25, 0x0, ...)
        /home/travis/gopath/src/github.com/rackspace/rack/internal/github.com/codegangsta/cli/command.go:45 +0x14ee
github.com/rackspace/rack/internal/github.com/codegangsta/cli.(*App).RunAsSubcommand(0xc208069600, 0xc20807a8c0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/rackspace/rack/internal/github.com/codegangsta/cli/app.go:257 +0xc2a
github.com/rackspace/rack/internal/github.com/codegangsta/cli.Command.startApp(0x906890, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x979a70, 0x23, 0x0, ...)
        /home/travis/gopath/src/github.com/rackspace/rack/internal/github.com/codegangsta/cli/command.go:183 +0x505
github.com/rackspace/rack/internal/github.com/codegangsta/cli.Command.Run(0x906890, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x979a70, 0x23, 0x0, ...)
        /home/travis/gopath/src/github.com/rackspace/rack/internal/github.com/codegangsta/cli/command.go:45 +0x14ee
github.com/rackspace/rack/internal/github.com/codegangsta/cli.(*App).Run(0xc208069500, 0xc20800a000, 0xa, 0xa, 0x0, 0x0)
        /home/travis/gopath/src/github.com/rackspace/rack/internal/github.com/codegangsta/cli/app.go:160 +0xd0c
main.main()
        /home/travis/gopath/src/github.com/rackspace/rack/main.go:35 +0x3b7

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

When I run this command I can see that one of rack's threads starts using up all available mem until it crashes.

jrperritt commented

Hmm, I need to do some testing on this, but I suspect there are 2 issues here:

  1. Currently, rack doesn't support large object uploads (where size is greater than 5 GB), so unfortunately I'd expect this would've failed even if there had been enough memory. Can you file an issue for large object support?
  2. Though rack treats the file to be uploaded as just an io.ReadSeeker, Gophercloud does by default copy the contents from the io.ReadSeeker into a buffer using an io.TeeReader so that we can compute the object's hash and let you know if it was completely uploaded. But the io.TeeReader gives us an io.Reader and we need an io.ReadSeeker for the upload (hence the copy to the buffer, which implements io.Seek). The current implementation may be a bit naive and can likely be improved. I'll look at it today.