googleapis/google-api-ruby-client

storage: Out of memory when `upload_chunk_size` is set to 0

stanhu opened this issue · 1 comments

stanhu commented

Environment details

  • OS: Linux
  • Ruby version: 2.7.7
  • Gem name and version: google-apis-core 0.10.0

Steps to reproduce

  1. Set Google::Apis::RequestOptions.default.upload_chunk_size = 0.
  2. Attempt to upload a 10 GB file.

#13283 added support for disabling chunking. However, instead of opening and passing in an IO object, the upload attempts to read the entire file:

google-api-ruby-client/google-apis-core/lib/google/apis/core/storage_upload.rb

Line 149 in 4b61f59

chunk_body = StringIO.new(upload_io.read(current_chunk_size))

I think the right thing to do is pass along the unfettered upload_io if upload_chunk_size is 0 instead of reading the entire chunk.

cc: @bajajneha27, @dazuma

stanhu commented

#13341 seems to help here.