capacity overflow when send large Uint8List to rust(> 2gb)

Question

capacity overflow when send large Uint8List to rust(> 2gb)

Closed this issue a month ago · 8 comments

Describe the bug

In flutter capacity overflow when send large Uint8List to rust(> 2gb)

Steps to reproduce

Hint: A simple way to reproduce is to clone and modify the https://github.com/fzyzcjy/flutter_rust_bridge/tree/master/frb_example/flutter_via_create example package according to your needs.

add a new function to return vec len at rust/src/api/simple.rs

pub fn test_vec(data: Vec<u8>) -> usize {
    data.len()
}

run flutter_rust_bridge_codegen generate
add test code to dart

import 'dart:typed_data';

import 'package:flutter/material.dart';
import 'package:flutter_via_create/src/rust/api/simple.dart';
import 'package:flutter_via_create/src/rust/frb_generated.dart';

Future<void> main() async {
  await RustLib.init();
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(title: const Text('flutter_rust_bridge quickstart')),
        body: Center(
          child: TextButton(
            onPressed: () {
              final result = greet(name: "world");
              print(result);
              testVec(data: Uint8List(1024 * 1024 * 1024 * 2)).then((value) => print(value));
            },
            child: const Text("data"),
          ),
        ),
      ),
    );
  }
}

click button , then this will get a panic.

Logs

thread '<unnamed>' panicked at library/alloc/src/raw_vec.rs:25:5:
capacity overflow
stack backtrace:
   0:        0x102c107c4 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h243268f17d714c7f
   1:        0x102c536fc - core::fmt::write::hb3cfb8a30e72d7ff
   2:        0x102c067a8 - std::io::Write::write_fmt::hfb2314975de9ecf1
   3:        0x102c12cd4 - std::panicking::default_hook::{{closure}}::h14c7718ccf39d316
   4:        0x102c128f8 - std::panicking::default_hook::hc62e60da3be2f352
   5:        0x102a071fc - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::ha3771cb704b88a16
   6:        0x102a0e92c - flutter_rust_bridge::misc::panic_backtrace::PanicBacktrace::setup::{{closure}}::hf7eef2f7421ac658
   7:        0x102c138f0 - std::panicking::rust_panic_with_hook::h09e8a656f11e82b2
   8:        0x102c131d8 - std::panicking::begin_panic_handler::{{closure}}::h1230eb3cc91b241c
   9:        0x102c10c50 - std::sys::backtrace::__rust_end_short_backtrace::hc3491307aceda2c2
  10:        0x102c12ec8 - _rust_begin_unwind
  11:        0x102c76534 - core::panicking::panic_fmt::ha4b80a05b9fff47a
  12:        0x102c450e4 - alloc::raw_vec::capacity_overflow::h88d6b8d70532516f
  13:        0x102c760cc - alloc::raw_vec::handle_error::h3874354ba7b66dcc
  14:        0x102a05c68 - alloc::raw_vec::RawVec<T,A>::reserve_exact::h09bb17568e2b0d00
  15:        0x102a0fd68 - alloc::vec::Vec<T,A>::reserve_exact::h025a1c8279936d8b
  16:        0x102a08d2c - flutter_rust_bridge::ffi_binding::vec_resize::h02bf0f6657da6c5a
  17:        0x102a08c64 - _frb_rust_vec_u8_resize
thread '<unnamed>' panicked at library/core/src/panicking.rs:221:5:
panic in a function that cannot unwind
stack backtrace:
   0:        0x102c107c4 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h243268f17d714c7f
   1:        0x102c536fc - core::fmt::write::hb3cfb8a30e72d7ff
   2:        0x102c067a8 - std::io::Write::write_fmt::hfb2314975de9ecf1
   3:        0x102c12cd4 - std::panicking::default_hook::{{closure}}::h14c7718ccf39d316
   4:        0x102c128f8 - std::panicking::default_hook::hc62e60da3be2f352
   5:        0x102a071fc - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::ha3771cb704b88a16
   6:        0x102a0e92c - flutter_rust_bridge::misc::panic_backtrace::PanicBacktrace::setup::{{closure}}::hf7eef2f7421ac658
   7:        0x102c138f0 - std::panicking::rust_panic_with_hook::h09e8a656f11e82b2
   8:        0x102c131d8 - std::panicking::begin_panic_handler::{{closure}}::h1230eb3cc91b241c
   9:        0x102c10c50 - std::sys::backtrace::__rust_end_short_backtrace::hc3491307aceda2c2
  10:        0x102c12ec8 - _rust_begin_unwind
  11:        0x102c76564 - core::panicking::panic_nounwind_fmt::h91ee161184879b56
  12:        0x102c765dc - core::panicking::panic_nounwind::heab7ebe7a6cd845c
  13:        0x102c76754 - core::panicking::panic_cannot_unwind::hedc43d82620205bf
  14:        0x102a08c58 - _frb_rust_vec_u8_resize
thread caused non-unwinding panic. aborting.

Expected behavior

No response

Generated binding code

No response

OS

MacOS 15.1 Apple chip

Version of `flutter_rust_bridge_codegen`

2.6.0

Flutter info

[✓] Flutter (Channel stable, 3.24.4, on macOS 15.1 24B83 darwin-arm64, locale en-CN)
    • Flutter version 3.24.4 on channel stable at /Users/littleq/Code/Work/Library/flutter_stl
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 603104015d (2 weeks ago), 2024-10-24 08:01:25 -0700
    • Engine revision db49896cf2
    • Dart version 3.5.4
    • DevTools version 2.37.3

[!] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/littleq/Library/Android/sdk
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/to/macos-android-setup for more details.

[!] Xcode - develop for iOS and macOS (Xcode 16.0)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 16A242
    ! iOS 18.0 Simulator not installed; this may be necessary for iOS and macOS development.
      To download and install the platform, open Xcode, select Xcode > Settings > Platforms,
      and click the GET button for the required platform.

      For more information, please visit:
        https://developer.apple.com/documentation/xcode/installing-additional-simulator-runtimes
    • CocoaPods version 1.15.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] Android Studio (version 2024.2)
    • Android Studio at /Users/littleq/Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 21.0.3+-79915917-b509.11)

[✓] IntelliJ IDEA Ultimate Edition (version 2024.2.3)
    • IntelliJ at /Users/littleq/Applications/IntelliJ IDEA Ultimate.app
    • Flutter plugin version 80.0.2
    • Dart plugin version 242.20629

[✓] IntelliJ IDEA Community Edition (version 2024.2.3)
    • IntelliJ at /Users/littleq/Applications/IntelliJ IDEA Community Edition.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.95.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension can be installed from:
      🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (3 available)
    • macOS (desktop)                 • macos                 • darwin-arm64   • macOS 15.1 24B83 darwin-arm64
    • Mac Designed for iPad (desktop) • mac-designed-for-ipad • darwin         • macOS 15.1 24B83 darwin-arm64
    • Chrome (web)                    • chrome                • web-javascript • Google Chrome 130.0.6723.117

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 2 categories.

Version of `clang++`

Apple clang version 16.0.0 (clang-1600.0.26.3)

Additional context

No response

Answer 1 · 2024-11-09T03:22:43.000Z

Hi! Thanks for opening your first issue here! 😄

Answer 2 · 2024-11-09T03:39:39.000Z

Hmm interesting. Looks like vec_resize caused that.

Firstly, could you please install LLVM (https://cjycode.com/flutter_rust_bridge/manual/ffigen-troubleshooting#llvm-needs-to-be-installed) and enable full_dep: true in config yaml? Because for your case DCO codec may be much faster than the SSE codec without full_dep enabled.

Secondly, curious why do you need to transfer 2GB data? Maybe we can discuss and see whether it is really needed (or maybe e.g. things can live on Rust side without moving to Dart side)

Answer 3 · 2024-11-09T04:11:35.000Z

Firstly, thank you for your response. I enabled the configuration and reran flutter_rust_bridge_codegen generate, but the error still persists.

thread '<unnamed>' panicked at library/alloc/src/raw_vec.rs:25:5:
capacity overflow
stack backtrace:
   0:        0x100ca529c - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h243268f17d714c7f
   1:        0x100ce81d4 - core::fmt::write::hb3cfb8a30e72d7ff
   2:        0x100c9b280 - std::io::Write::write_fmt::hfb2314975de9ecf1
   3:        0x100ca77ac - std::panicking::default_hook::{{closure}}::h14c7718ccf39d316
   4:        0x100ca73d0 - std::panicking::default_hook::hc62e60da3be2f352
   5:        0x100a9bcd4 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::ha3771cb704b88a16
   6:        0x100aa3404 - flutter_rust_bridge::misc::panic_backtrace::PanicBacktrace::setup::{{closure}}::hf7eef2f7421ac658
   7:        0x100ca83c8 - std::panicking::rust_panic_with_hook::h09e8a656f11e82b2
   8:        0x100ca7cb0 - std::panicking::begin_panic_handler::{{closure}}::h1230eb3cc91b241c
   9:        0x100ca5728 - std::sys::backtrace::__rust_end_short_backtrace::hc3491307aceda2c2
  10:        0x100ca79a0 - _rust_begin_unwind
  11:        0x100d0b014 - core::panicking::panic_fmt::ha4b80a05b9fff47a
  12:        0x100cd9bbc - alloc::raw_vec::capacity_overflow::h88d6b8d70532516f
  13:        0x100d0abac - alloc::raw_vec::handle_error::h3874354ba7b66dcc
  14:        0x100a9ff2c - <u8 as alloc::vec::spec_from_elem::SpecFromElem>::from_elem::h2af9457ae79cd135
  15:        0x100aa4f18 - alloc::vec::from_elem::hf2fba866b1b568fd
  16:        0x100aa35f8 - flutter_rust_bridge::for_generated::pointer::new_leak_vec_ptr::h18db222fc1c5c161
  17:        0x100a8cf30 - _frbgen_flutter_via_create_cst_new_list_prim_u_8_loose
thread '<unnamed>' panicked at library/core/src/panicking.rs:221:5:
panic in a function that cannot unwind
stack backtrace:
   0:        0x100ca529c - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h243268f17d714c7f
   1:        0x100ce81d4 - core::fmt::write::hb3cfb8a30e72d7ff
   2:        0x100c9b280 - std::io::Write::write_fmt::hfb2314975de9ecf1
   3:        0x100ca77ac - std::panicking::default_hook::{{closure}}::h14c7718ccf39d316
   4:        0x100ca73d0 - std::panicking::default_hook::hc62e60da3be2f352
   5:        0x100a9bcd4 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::ha3771cb704b88a16
   6:        0x100aa3404 - flutter_rust_bridge::misc::panic_backtrace::PanicBacktrace::setup::{{closure}}::hf7eef2f7421ac658
   7:        0x100ca83c8 - std::panicking::rust_panic_with_hook::h09e8a656f11e82b2
   8:        0x100ca7cb0 - std::panicking::begin_panic_handler::{{closure}}::h1230eb3cc91b241c
   9:        0x100ca5728 - std::sys::backtrace::__rust_end_short_backtrace::hc3491307aceda2c2
  10:        0x100ca79a0 - _rust_begin_unwind
  11:        0x100d0b044 - core::panicking::panic_nounwind_fmt::h91ee161184879b56
  12:        0x100d0b0bc - core::panicking::panic_nounwind::heab7ebe7a6cd845c
  13:        0x100d0b234 - core::panicking::panic_cannot_unwind::hedc43d82620205bf
  14:        0x100a8cf24 - _frbgen_flutter_via_create_cst_new_list_prim_u_8_loose
thread caused non-unwinding panic. aborting.

Secondly, In my current situation where I need to transfer 2GB of data, I need to implement an encryption and decryption function. The content being processed is quite large. For the PC platform, I can use files for transmission, but for other platforms like the Web, it's not feasible to use files for transmission. Therefore, I'm passing a &[u8] array as the input parameter, and the return value is also the same type of array.

Answer 4 · 2024-11-09T04:53:35.000Z

I see. But I am worried about:

OOM error: On phones/websites, not sure whether can allocate that huge memory on everyone's device. Minimum is 2GB on Dart and 2GB on Rust, so it is 4GB.
UI freeze: When transfering between Dart and Rust, the UI will freeze. Usually this is not noticeable since it is very fast, but for 2GB I do not know whether that will be noticeable.

I need to implement an encryption and decryption function

For this specific scenario, I personally suggest to implement some chunking mechanism. For example, chunk 2GB into many 10MB blocks, and encrypt/decrypt each of them separately.

Answer 5 · 2024-11-09T05:12:31.000Z

Thank you very much for your suggestion. I should consider modifying my design to process the content in chunks.

Answer 6 · 2024-11-09T06:49:12.000Z

You are welcome!

Answer 7 · 2024-11-13T00:49:04.000Z

Close since this seems to be solved. Feel free to reopen if needed!

Answer 8 · 2024-11-27T01:33:09.000Z

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new issue.

Describe the bug

Steps to reproduce

Logs

Expected behavior

Generated binding code

OS

Version of flutter_rust_bridge_codegen

Flutter info

Version of clang++

Additional context

Version of `flutter_rust_bridge_codegen`

Version of `clang++`