tanaikech/goodls

Out of memory when downloading very large files

code-chicken opened this issue · 6 comments

code-chicken commented

When downloading very large files from Google I get the following error:

Now downloading.
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x759761, 0x16)
        C:/Go/src/runtime/panic.go:605 +0x95
runtime.sysMap(0xc620600000, 0x200000000, 0x0, 0x92ded8)
        C:/Go/src/runtime/mem_linux.go:216 +0x1d0
runtime.(*mheap).sysAlloc(0x914e60, 0x200000000, 0x427812)
        C:/Go/src/runtime/malloc.go:470 +0xd7
runtime.(*mheap).grow(0x914e60, 0x100000, 0x0)
        C:/Go/src/runtime/mheap.go:887 +0x60
runtime.(*mheap).allocSpanLocked(0x914e60, 0x100000, 0x92dee8, 0xc420041eb0)
        C:/Go/src/runtime/mheap.go:800 +0x334
runtime.(*mheap).alloc_m(0x914e60, 0x100000, 0xc420320101, 0x1)
        C:/Go/src/runtime/mheap.go:666 +0x118
runtime.(*mheap).alloc.func1()
        C:/Go/src/runtime/mheap.go:733 +0x4d
runtime.systemstack(0xc420041f08)
        C:/Go/src/runtime/asm_amd64.s:360 +0xab
runtime.(*mheap).alloc(0x914e60, 0x100000, 0xc420010101, 0xc420000180)
        C:/Go/src/runtime/mheap.go:732 +0xa1
runtime.largeAlloc(0x1fffffe00, 0xc420000101, 0x100000000000004)
        C:/Go/src/runtime/malloc.go:827 +0x98
runtime.mallocgc.func1()
        C:/Go/src/runtime/malloc.go:722 +0x46
runtime.systemstack(0xc420029300)
        C:/Go/src/runtime/asm_amd64.s:344 +0x79
runtime.mstart()
        C:/Go/src/runtime/proc.go:1135

goroutine 1 [running]:
runtime.systemstack_switch()
        C:/Go/src/runtime/asm_amd64.s:298 fp=0xc420075650 sp=0xc420075648 pc=0x454b40
runtime.mallocgc(0x1fffffe00, 0x6e2be0, 0xc420075701, 0x0)
        C:/Go/src/runtime/malloc.go:721 +0x7ae fp=0xc4200756f8 sp=0xc420075650 pc=0x4108ee
runtime.makeslice(0x6e2be0, 0x1fffffe00, 0x1fffffe00, 0xc520600000, 0xc4200b2528, 0xc4200b25e8)
        C:/Go/src/runtime/slice.go:54 +0x77 fp=0xc420075728 sp=0xc4200756f8 pc=0x440a07
bytes.makeSlice(0x1fffffe00, 0x0, 0x0, 0x0)
        C:/Go/src/bytes/buffer.go:231 +0x77 fp=0xc420075768 sp=0xc420075728 pc=0x46b1a7
bytes.(*Buffer).ReadFrom(0xc420075810, 0x7f76cbe1d4a8, 0xc4200b2500, 0xc42039e200, 0x0, 0x200)
        C:/Go/src/bytes/buffer.go:203 +0x291 fp=0xc4200757e0 sp=0xc420075768 pc=0x46b0e1
io/ioutil.readAll(0x7f76cbe1d4a8, 0xc4200b2500, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
        C:/Go/src/io/ioutil/ioutil.go:33 +0x12c fp=0xc420075888 sp=0xc4200757e0 pc=0x4dfbec
io/ioutil.ReadAll(0x7f76cbe1d4a8, 0xc4200b2500, 0xc4200b2500, 0x7f76cbe1d4a8, 0xc4200b2500, 0x5d, 0x5d)
        C:/Go/src/io/ioutil/ioutil.go:42 +0x3e fp=0xc4200758d8 sp=0xc420075888 pc=0x4dfcce
main.(*para).saveFile(0xc420075c98, 0xc42043a000, 0x0, 0x0)
        D:/build/goodls/goodls.go:51 +0x149 fp=0xc4200759b0 sp=0xc4200758d8 pc=0x6aac79
main.(*para).downloadLargeFile(0xc420159c98, 0xc4200fe210, 0xa9)
        D:/build/goodls/goodls.go:87 +0x22c fp=0xc420075a40 sp=0xc4200759b0 pc=0x6ab45c
main.(*para).download(0xc420159c98, 0x7ffd2e3393c6, 0x52, 0x7ffd2e3393c6, 0x52)
        D:/build/goodls/goodls.go:166 +0x2b3 fp=0xc420075b00 sp=0xc420075a40 pc=0x6abef3
main.handler(0xc4200b06e0)
        D:/build/goodls/goodls.go:192 +0x725 fp=0xc420075d30 sp=0xc420075b00 pc=0x6ac9b5
github.com/urfave/cli.HandleAction(0x6e1620, 0x769090, 0xc4200b06e0, 0xc42005e1e0, 0x0)
        C:/Go/projects/src/github.com/urfave/cli/app.go:492 +0x7c fp=0xc420075d58 sp=0xc420075d30 pc=0x699cbc
github.com/urfave/cli.(*App).Run(0xc420010d00, 0xc42000e140, 0x5, 0x5, 0x0, 0x0)
        C:/Go/projects/src/github.com/urfave/cli/app.go:264 +0x635 fp=0xc420075f38 sp=0xc420075d58 pc=0x697db5
main.main()
        D:/build/goodls/goodls.go:255 +0x84 fp=0xc420075f80 sp=0xc420075f38 pc=0x6acf54
runtime.main()
        C:/Go/src/runtime/proc.go:195 +0x226 fp=0xc420075fe0 sp=0xc420075f80 pc=0x42b7f6
runtime.goexit()
        C:/Go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420075fe8 sp=0xc420075fe0 pc=0x4576c1

goroutine 51 [running]:
        goroutine running on other thread; stack unavailable
created by net/http.(*http2Transport).newClientConn
        C:/Go/src/net/http/h2_bundle.go:7053 +0x6b9

This happens after some minutes. No data has been saved then.
putty

tanaikech commented

I'm sorry for the inconvenience. And thank you for your report. Can I ask you about the size of the large files?

  • 👍1
code-chicken commented

Thank you for your quick reaction. The size is about 5 Gigabytes, I think. I could send you a private link, if you want.

tanaikech commented

Thank you for the information. I fixed this issue. So I report it. You can download it as v1.0.2. Could you please confirm it?

At the shared files on Google Drive, when the download a file, the file size cannot be retrieved without the access token before downloading. So I cannot implement the progression like the progress-bar. I'm sorry about this situation.

  • 👍1
  • ❤️1
code-chicken commented

Thank you so much for solving this issue so quickly!
Everything is working fine now!

  • 👍1
tanaikech commented

Welcome. Thank you for reporting it. I'm glad your issue was solved. It could also update this application. And, I could study from this issue. Thank you, too.

I reported this update here.

tanaikech commented

I'm really sorry for late response. I updated goodls. Unfortunately, when OAuth2 is not used to download, the file size cannot be retrieved. But I noticed that the progress of downloading can be displayed. So I added this function. From this version, the progress of downloading got to be able to be displayed. If this was useful for you, I'm glad.

  • 👍1