tcp_metadata_exchange proxy test fails on big-endian platforms

Question

tcp_metadata_exchange proxy test fails on big-endian platforms

knm3000 opened this issue 2 years ago · 0 comments

After we had introduced big-endian support in proxy-wasm #198 and fixed
#197 the following new error started happening (regression of #198):
tcp_metadata_exchange test from proxy repo fails on big-endian platform (envoy binary crash) https://github.com/istio/proxy/blob/master/test/envoye2e/tcp_metadata_exchange/tcp_metadata_exchange_test.go

go test ./test/envoye2e/tcp_metadata_exchange -v
...

[2022-07-25 11:05:29.028][240769][critical][assert] [external/envoy/source/exe/main_common.cc:84] panic: out of memory
[2022-07-25 11:05:29.091][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x3ac8100000000
[2022-07-25 11:05:29.226][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-07-25 11:05:29.226][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: b977767d8be6ce88e1956cd1f6165242aab2a572/1.20.6/Modified/RELEASE/OpenSSL
[2022-07-25 11:05:29.256][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #0: [0x20025f33490]
[2022-07-25 11:05:29.257][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #1: __pthread_kill_implementation [0x2002679eb86]
[2022-07-25 11:05:29.258][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #2: gsignal [0x200267500e0]
[2022-07-25 11:05:29.258][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #3: abort [0x2002672b100]
[2022-07-25 11:05:29.362][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #4: Envoy::MainCommonBase::MainCommonBase()::$_1::__invoke() [0x2aa3cea39bc]
[2022-07-25 11:05:29.372][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #5: tcmalloc::cpp_throw_oom() [0x2aa4076b0a0]
[2022-07-25 11:05:29.381][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #6: tcmalloc::allocate_full_cpp_throw_oom() [0x2aa4076bd1c]
[2022-07-25 11:05:29.399][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #7: flatbuffers::FlatBufferBuilder::CreateString() [0x2aa3ce7140a]
[2022-07-25 11:05:29.408][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #8: Wasm::Common::extractLocalNodeFlatBuffer() [0x2aa3ce6aa66]
[2022-07-25 11:05:29.417][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #9: proxy_wasm::null_plugin::Stats::PluginRootContext::configure() [0x2aa3cdbbcc0]
[2022-07-25 11:05:29.426][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #10: proxy_wasm::null_plugin::Stats::PluginRootContext::onConfigure() [0x2aa3cdbbc18]
[2022-07-25 11:05:29.435][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #11: std::_Function_handler<>::_M_invoke() [0x2aa3d9a2898]
[2022-07-25 11:05:29.445][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #12: proxy_wasm::ContextBase::onConfigure() [0x2aa3d9a83c8]
[2022-07-25 11:05:29.463][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #13: proxy_wasm::WasmBase::configure() [0x2aa3d9c42b4]
[2022-07-25 11:05:29.474][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #14: proxy_wasm::createWasm() [0x2aa3d9c66da]
[2022-07-25 11:05:29.484][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #15: Envoy::Extensions::Common::Wasm::createWasm()::$_8::operator()() [0x2aa3d6d05f4]
[2022-07-25 11:05:29.512][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #16: Envoy::Extensions::Common::Wasm::createWasm() [0x2aa3d6cd924]
[2022-07-25 11:05:29.527][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #17: Envoy::Extensions::NetworkFilters::Wasm::FilterConfig::FilterConfig() [0x2aa3d5ecf9a]
[2022-07-25 11:05:29.537][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #18: Envoy::Extensions::NetworkFilters::Wasm::WasmFilterConfig::createFilterFactoryFromProtoTyped() [0x2aa3d5eab6c]
[2022-07-25 11:05:29.547][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #19: Envoy::Extensions::NetworkFilters::Common::FactoryBase<>::createFilterFactoryFromProto() [0x2aa3d5eb976]
[2022-07-25 11:05:29.557][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #20: Envoy::Server::ProdListenerComponentFactory::createNetworkFilterFactoryList_() [0x2aa3f96c45a]
[2022-07-25 11:05:29.575][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #21: Envoy::Server::ListenerFilterChainFactoryBuilder::buildFilterChainInternal() [0x2aa3f97985a]
[2022-07-25 11:05:29.586][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #22: Envoy::Server::ListenerFilterChainFactoryBuilder::buildFilterChain() [0x2aa3f979526]
[2022-07-25 11:05:29.599][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #23: Envoy::Server::FilterChainManagerImpl::addFilterChains() [0x2aa3f996c4c]
[2022-07-25 11:05:29.609][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #24: Envoy::Server::ListenerImpl::ListenerImpl() [0x2aa3f95f46a]
[2022-07-25 11:05:29.620][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #25: Envoy::Server::ListenerManagerImpl::addOrUpdateListenerInternal() [0x2aa3f974652]
[2022-07-25 11:05:29.630][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #26: Envoy::Server::ListenerManagerImpl::addOrUpdateListener() [0x2aa3f9735d8]
[2022-07-25 11:05:29.658][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #27: Envoy::Server::LdsApiImpl::onConfigUpdate() [0x2aa3f9e8e16]
[2022-07-25 11:05:29.691][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #28: Envoy::Server::LdsApiImpl::onConfigUpdate() [0x2aa3f9ea7f8]
[2022-07-25 11:05:29.718][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #29: Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate() [0x2aa3fb27e70]
[2022-07-25 11:05:29.739][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #30: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0x2aa3fb309a8]
[2022-07-25 11:05:29.748][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #31: Envoy::Grpc::AsyncStreamCallbacks<>::onReceiveMessageRaw() [0x2aa3fb34a3c]
[2022-07-25 11:05:29.763][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #32: Envoy::Grpc::AsyncStreamImpl::onData() [0x2aa3fb570c6]
[2022-07-25 11:05:29.780][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #33: Envoy::Http::AsyncStreamImpl::encodeData() [0x2aa3fb60b4c]
[2022-07-25 11:05:29.789][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #34: Envoy::Router::UpstreamRequest::decodeData() [0x2aa3fbcb9f0]
[2022-07-25 11:05:29.798][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #35: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x2aa3f81b63e]
[2022-07-25 11:05:29.807][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #36: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x2aa3faa37a0]
[2022-07-25 11:05:29.816][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #37: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_22::__invoke() [0x2aa3fab0824]
[2022-07-25 11:05:29.836][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #38: nghttp2_session_on_data_received [0x2aa3fd088e4]
[2022-07-25 11:05:29.846][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #39: nghttp2_session_mem_recv [0x2aa3fd0ab0a]
[2022-07-25 11:05:29.855][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #40: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x2aa3faa1a00]
[2022-07-25 11:05:29.864][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #41: Envoy::Http::CodecClient::onData() [0x2aa3f8cb478]
[2022-07-25 11:05:29.873][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #42: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x2aa3f8cd5e2]
[2022-07-25 11:05:29.882][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #43: Envoy::Network::FilterManagerImpl::onRead() [0x2aa3fb19de2]
[2022-07-25 11:05:29.891][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #44: Envoy::Network::ConnectionImpl::onReadReady() [0x2aa3fb0f666]
[2022-07-25 11:05:29.900][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #45: Envoy::Network::ConnectionImpl::onFileEvent() [0x2aa3fb0ccc8]
[2022-07-25 11:05:29.908][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #46: std::_Function_handler<>::_M_invoke() [0x2aa3fae79da]
[2022-07-25 11:05:29.917][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #47: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x2aa3fae8d84]
[2022-07-25 11:05:29.926][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #48: event_process_active_single_queue [0x2aa3fcf8b3e]
[2022-07-25 11:05:29.936][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #49: event_base_loop [0x2aa3fcf757e]
[2022-07-25 11:05:29.945][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #50: Envoy::Event::DispatcherImpl::run() [0x2aa3fae512e]
[2022-07-25 11:05:29.973][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #51: Envoy::Server::InstanceImpl::run() [0x2aa3f4cadf6]
[2022-07-25 11:05:29.983][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #52: Envoy::MainCommonBase::run() [0x2aa3cea1312]
[2022-07-25 11:05:29.992][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #53: Envoy::MainCommon::main() [0x2aa3cea1c76]
[2022-07-25 11:05:30.001][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #54: main [0x2aa3ce9d13e]
[2022-07-25 11:05:30.001][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #55: __libc_start_call_main [0x200267317f2]
[2022-07-25 11:05:30.002][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #56: __libc_start_main@GLIBC_2.2 [0x200267318d0]
[2022-07-25 11:05:30.010][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #57: [0x2aa3cc9ee90]
AsyncClient 0x2aa43ecb080, stream_id_: 8385127285791150716

The root cause is that tcp_metadata_exchange test uses null-vm, and in #198 we introduced byte swapping on big-endian for all wasm runtimes (including null-vm). Since null-vm is not using wasm binary, we should not reverse bytes for null-vm (for other wasm runtimes we still need to reverse bytes on big-endian).
Opened #282 to fix it.