google/fuzzing

Running libfuzzer with /MD on Windows & MSVC is using incontrollable memory consumption?

rajeemm opened this issue · 0 comments

I'm trying to fuzz test my library which compiled on Windows using /MD (C Runtime) with MSVC compiler.
I get incontrollable memory consumption while using libfuzzer

Repro steps:

  1. I cloned LLVM repo
  2. To allow MD compilation of LLVM : at https://github.com/llvm/llvm-project/blob/master/compiler-rt/CMakeLists.txt#L337, I replaced the block of if(COMPILER_RT_HAS_MT_FLAG) block with include(ChooseMSVCCRT) (I know it's not 100% supported, but I think it will be and I'm not sure if it's the cause?)
  3. Compile LLVM with LLVM_USE_CRT_RELEASE=MD
  4. Compile my own code using Windows 10, MSVC compiler (v14.28.29333) while linking libfuzzer libs: 'clang_rt.fuzzer-x86_64.lib','libsancov.lib', 'clang_rt.asan_dynamic-x86_64.lib', 'clang_rt.asan_dynamic_runtime_thunk-x86_64.lib' (some from MSVC libs dir, some from LLVM compilation) and using CXXFLAGS: /fsanitize=address', '/fsanitize-coverage=inline-8bit-counters', '/fsanitize-coverage=edge', '/fsanitize-coverage=trace-cmp', '/fsanitize-coverage=trace-div'.
  5. Libfuzzer crashes because of too much memory usage (even with -rss_limit_mb=4000), so I did a sanity check to make sure no memory leak on my code:
  • Run 1 (with libfuzzer libs & CXXFLAGS as explained above):
FUZZ_EXPORT int __cdecl LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
  const char* tt = "66";
  RunFuzzingTest((size_t)2,(const char*)tt);
  return 0;
}
  • Run 2 (without libfuzzer libs & CXXFLAGS as above):
int main() {
  while (true) {
  const char* tt = "66";
  RunFuzzingTest((size_t)2,(const char*)tt);
  }
  return 0;  
}
  • Results:
    Run of (1 - libfuzzer with same input) resulted in huge and increasing memory usage (2.6GB after 5 mins)
    Run of (2 - my own main with same input) resulted in pretty constant memory usage (10MB after more than 20 mins).

Thanks