Race conditions when processing packet in the egress pipeline.

After using the TSANs tool on our tests with a simple grpc service with processing the packet in the egress pipeline, I noticed that it resulted in some flaky tests with race conditions. In particular, it occurs in tests that inject packets that get multicasted.

This is where TSAN is complaining about it.

phv->get_field("standard_metadata.packet_length").set(
    packet->get_register(RegisterAccess::PACKET_LENGTH_REG_IDX));
egress_mau->apply(packet.get()); // <---------------- TSAN error

Here is the complete ThreadSanitizer report.

==================
WARNING: ThreadSanitizer: data race (pid=1283)
  Write of size 8 at 0xffffa89537e8 by thread T28:
    #0 set_source_info third_party/p4lang_behavioral_model/include/bm/bm_sim/actions.h:637:22 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x851ef0) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #1 execute third_party/p4lang_behavioral_model/src/bm_sim/actions.cpp:371:14 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x851ef0)
    #2 bm::ActionFnEntry::execute(bm::Packet*) const third_party/p4lang_behavioral_model/src/bm_sim/actions.cpp:411:15 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x851ef0)
    #3 bm::ActionFnEntry::operator()(bm::Packet*) const third_party/p4lang_behavioral_model/src/bm_sim/actions.cpp:429:5 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x8522e4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #4 bm::MatchTableAbstract::apply_action(bm::Packet*) third_party/p4lang_behavioral_model/src/bm_sim/match_tables.cpp:138:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x8a5fb8) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #5 bm::MatchActionTable::operator()(bm::Packet*) const third_party/p4lang_behavioral_model/src/bm_sim/tables.cpp:42:34 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x925e40) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #6 bm::Pipeline::apply(bm::Packet*) third_party/p4lang_behavioral_model/src/bm_sim/pipeline.cpp:44:12 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x911618) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #7 SimpleSwitch::egress_thread(unsigned long) third_party/p4lang_behavioral_model/targets/simple_switch/simple_switch.cpp:687:17 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x751b00) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #8 __invoke<void (SimpleSwitch::*)(unsigned long), SimpleSwitch *, unsigned long, void> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__type_traits/invoke.h:118:25 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x763ef4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #9 __thread_execute<std::__tsan::unique_ptr<std::__tsan::__thread_struct, std::__tsan::default_delete<std::__tsan::__thread_struct> >, void (SimpleSwitch::*)(unsigned long), SimpleSwitch *, unsigned long, 2UL, 3UL> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__thread/thread.h:193:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x763ef4)
    #10 void* std::__tsan::__thread_proxy<std::__tsan::tuple<std::__tsan::unique_ptr<std::__tsan::__thread_struct, std::__tsan::default_delete<std::__tsan::__thread_struct>>, void (SimpleSwitch::*)(unsigned long), SimpleSwitch*, unsigned long>>(void*) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__thread/thread.h:202:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x763ef4)

  Previous write of size 8 at 0xffffa89537e8 by thread T30:
    #0 set_source_info third_party/p4lang_behavioral_model/include/bm/bm_sim/actions.h:637:22 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x851ef0) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #1 execute third_party/p4lang_behavioral_model/src/bm_sim/actions.cpp:371:14 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x851ef0)
    #2 bm::ActionFnEntry::execute(bm::Packet*) const third_party/p4lang_behavioral_model/src/bm_sim/actions.cpp:411:15 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x851ef0)
    #3 bm::ActionFnEntry::operator()(bm::Packet*) const third_party/p4lang_behavioral_model/src/bm_sim/actions.cpp:429:5 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x8522e4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #4 bm::MatchTableAbstract::apply_action(bm::Packet*) [third_party/p4lang_behavioral_model/src/bm_sim/match_tables.cpp:138]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x8a5fb8) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #5 bm::MatchActionTable::operator()(bm::Packet*) const third_party/p4lang_behavioral_model/src/bm_sim/tables.cpp:4234 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x925e40) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #6 bm::Pipeline::apply(bm::Packet*) third_party/p4lang_behavioral_model/src/bm_sim/pipeline.cpp:44:12 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x911618) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #7 SimpleSwitch::egress_thread(unsigned long) third_party/p4lang_behavioral_model/targets/simple_switch/simple_switch.cpp:687:17 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x751b00) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #8 __invoke<void (SimpleSwitch::*)(unsigned long), SimpleSwitch *, unsigned long, void> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__type_traits/invoke.h:118]:25 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x763ef4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #9 __thread_execute<std::__tsan::unique_ptr<std::__tsan::__thread_struct, std::__tsan::default_delete<std::__tsan::__thread_struct> >, void (SimpleSwitch::*)(unsigned long), SimpleSwitch *, unsigned long, 2UL, 3UL> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__thread/thread.h:193]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x763ef4)
    #10 void* std::__tsan::__thread_proxy<std::__tsan::tuple<std::__tsan::unique_ptr<std::__tsan::__thread_struct, std::__tsan::default_delete<std::__tsan::__thread_struct>>, void (SimpleSwitch::*)(unsigned long), SimpleSwitch*, unsigned long>>(void*) [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__thread/thread.h:202]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x763ef4)

  Location is heap block of size 32 at 0xffffa89537e0 allocated by thread T33:
    #0 operator new(unsigned long) [third_party/llvm/llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x7350e4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #1 operator() [third_party/p4lang_behavioral_model/src/bm_sim/core/primitives.cpp:34]:1 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x86d3fc) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #2 __invoke<bm::core::(lambda at third_party/p4lang_behavioral_model/src/bm_sim/core/primitives.cpp:34:1) &> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__type_traits/invoke.h:150]:25 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x86d3fc)
    #3 __call<bm::core::(lambda at third_party/p4lang_behavioral_model/src/bm_sim/core/primitives.cpp:34:1) &> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__type_traits/invoke.h:217]:12 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x86d3fc)
    #4 operator() [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:211]:12 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x86d3fc)
    #5 std::__tsan::unique_ptr<bm::ActionPrimitive_, std::__tsan::default_delete<bm::ActionPrimitive_>> std::__tsan::__function::__policy_invoker<std::__tsan::unique_ptr<bm::ActionPrimitive_, std::__tsan::default_delete<bm::ActionPrimitive_>> ()>::__call_impl<std::__tsan::__function::__default_alloc_func<bm::core::$_0, std::__tsan::unique_ptr<bm::ActionPrimitive_, std::__tsan::default_delete<bm::ActionPrimitive_>> ()>>(std::__tsan::__function::__policy_storage const*) [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:611]:12 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x86d3fc)
    #6 operator() [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:717]:12 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x8516e4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #7 operator() [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:990]:10 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x8516e4)
    #8 bm::ActionOpcodesMap::get_primitive(std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>> const&) [third_party/p4lang_behavioral_model/src/bm_sim/actions.cpp:352]:10 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x8516e4)
    #9 bm::P4Objects::get_primitive(std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>> const&) [third_party/p4lang_behavioral_model/src/bm_sim/P4Objects.cpp:2462]:51 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x80c590) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #10 bm::P4Objects::add_primitive_to_action(Json::Value const&, bm::ActionFn*) [third_party/p4lang_behavioral_model/src/bm_sim/P4Objects.cpp:300]:20 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x80c1e8) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #11 bm::P4Objects::init_actions(Json::Value const&) [third_party/p4lang_behavioral_model/src/bm_sim/P4Objects.cpp:1450]:7 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x81dde4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #12 bm::P4Objects::init_objects(std::__tsan::basic_istream<char, std::__tsan::char_traits<char>>*, bm::LookupStructureFactory*, unsigned long, unsigned int, std::__tsan::shared_ptr<bm::TransportIface>, std::__tsan::set<std::__tsan::pair<std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>, std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>>, std::__tsan::less<std::__tsan::pair<std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>, std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>>>, std::__tsan::allocator<std::__tsan::pair<std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>, std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>>>> const&, bm::P4Objects::ForceArith const&) [third_party/p4lang_behavioral_model/src/bm_sim/P4Objects.cpp:2195]:5 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x827a68) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #13 init_objects [third_party/p4lang_behavioral_model/src/bm_sim/context.cpp:803]:30 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x868bf4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #14 bm::Context::load_new_config(std::__tsan::basic_istream<char, std::__tsan::char_traits<char>>*, bm::LookupStructureFactory*, std::__tsan::set<std::__tsan::pair<std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>, std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>>, std::__tsan::less<std::__tsan::pair<std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>, std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>>>, std::__tsan::allocator<std::__tsan::pair<std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>, std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>>>>> const&, bm::P4Objects::ForceArith const&) [third_party/p4lang_behavioral_model/src/bm_sim/context.cpp:822]3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x868bf4)
    #15 bm::SwitchWContexts::load_new_config(std::__tsan::basic_string<char, std::__tsan::char_traits<char>, std::__tsan::allocator<char>> const&) [third_party/p4lang_behavioral_model/src/bm_sim/switch.cpp:345]24 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x920fe4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #16 _pi_update_device_start [third_party/p4lang_behavioral_model/PI/src/pi_imp.cpp:101]:38 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x7fe400) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #17 pi_update_device_start [third_party/p4lang_PI/src/pi.c:210]:7 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x7f27f8) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #18 pi::fe::proto::DeviceMgrImp::pipeline_config_set(p4::v1::SetForwardingPipelineConfigRequest_Action, p4::v1::ForwardingPipelineConfig const&) [third_party/p4lang_PI/proto/frontend/src/device_mgr.cpp:587]:19 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x788928) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #19 pi::fe::proto::DeviceMgr::pipeline_config_set(p4::v1::SetForwardingPipelineConfigRequest_Action, p4::v1::ForwardingPipelineConfig const&) [third_party/p4lang_PI/proto/frontend/src/device_mgr.cpp:3120]:16 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x78826c) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #20 pi::server::(anonymous namespace)::P4RuntimeServiceImpl::SetForwardingPipelineConfig(grpc::ServerContext*, p4::v1::SetForwardingPipelineConfigRequest const*, p4::v1::SetForwardingPipelineConfigResponse*) [third_party/p4lang_PI/proto/server/pi_server.cpp:459]:31 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x782468) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #21 grpc::Status std::__tsan::__function::__policy_invoker<grpc::Status (p4::v1::grpc::P4Runtime::Service*, grpc::ServerContext*, p4::v1::SetForwardingPipelineConfigRequest const*, p4::v1::SetForwardingPipelineConfigResponse*)>::__call_impl<std::__tsan::__function::__default_alloc_func<p4::v1::grpc::P4Runtime::Service::Service()::$_2, grpc::Status (p4::v1::grpc::P4Runtime::Service*, grpc::ServerContext*, p4::v1::SetForwardingPipelineConfigRequest const*, p4::v1::SetForwardingPipelineConfigResponse*)>>(std::__tsan::__function::__policy_storage const*, p4::v1::grpc::P4Runtime::Service*, grpc::ServerContext*, p4::v1::SetForwardingPipelineConfigRequest const*, p4::v1::SetForwardingPipelineConfigResponse*) p4runtime.grpc.pb.cc (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x9a6c28) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #22 grpc::internal::RpcMethodHandler<p4::v1::grpc::P4Runtime::Service, p4::v1::SetForwardingPipelineConfigRequest, p4::v1::SetForwardingPipelineConfigResponse, proto2::MessageLite, proto2::MessageLite>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) <null> (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x9a6e90) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #23 grpc::Server::SyncRequest::ContinueRunAfterInterception() [third_party/grpc/src/cpp/server/server_cc.cc:471]:14 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x14193c4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #24 grpc::Server::SyncRequest::Run(std::__tsan::shared_ptr<grpc::Server::GlobalCallbacks> const&, bool) [third_party/grpc/src/cpp/server/server_cc.cc:459]:7 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x14190d4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #25 grpc::Server::SyncRequestThreadManager::DoWork(void*, bool, bool) [third_party/grpc/src/cpp/server/server_cc.cc:830]15 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x1418a8c) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #26 grpc::ThreadManager::MainWorkLoop() [third_party/grpc/src/cpp/thread_manager/thread_manager.cc:214]:9 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x142b1e8) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #27 Run [third_party/grpc/src/cpp/thread_manager/thread_manager.cc:48]:13 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x142be94) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #28 operator() [third_party/grpc/src/cpp/thread_manager/thread_manager.cc:40]:69 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x142be94)
    #29 grpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) [third_party/grpc/src/cpp/thread_manager/thread_manager.cc:40]:7 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x142be94)
    #30 (anonymous namespace)::ThreadBodyWithCleanup(void (*)(void*), void*, (anonymous namespace)::ThreadInternalsGoogleThread*) [third_party/grpc/google_specific/src/core/support/thd_google.cc:117]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62444) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #31 __invoke<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__type_traits/invoke.h:150]:25 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #32 invoke<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:28]10 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #33 operator()<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/bind_front.h:35]:12 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #34 operator()<void> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/perfect_forward.h:77]:12 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #35 __invoke<std::__tsan::__bind_front_t<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *> > [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__type_traits/invoke.h:150]:25 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #36 invoke<std::__tsan::__bind_front_t<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *> > [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:28]:10 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #37 InvokeR<void, std::__tsan::__bind_front_t<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *>, void> [third_party/absl/functional/internal/any_invocable.h:132]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #38 void absl::internal_any_invocable::RemoteInvoker<false, void, std::__tsan::__bind_front_t<void (*)(void (*)(void*), void*, (anonymous namespace)::ThreadInternalsGoogleThread*), void (*)(void*), void*, (anonymous namespace)::ThreadInternalsGoogleThread*>&&>(absl::internal_any_invocable::TypeErasedState*) [third_party/absl/functional/internal/any_invocable.h:368]:10 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #39 operator() [third_party/absl/functional/internal/any_invocable.h:876]:1 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0xae7624) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #40 ClosureThread::Run() [thread/thread.h:464]:25 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0xae7624)
    #41 Thread::ThreadBody(void*) [thread/thread.cc:1355]:16 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2c497f4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)

  Thread T28 (tid=1325, running) created by main thread at:
    #0 pthread_create [third_party/llvm/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1023]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x6a4898) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #1 __libcpp_thread_create [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__thread/support/pthread.h:182]:10 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x752f2c) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #2 std::__tsan::thread::thread<void (SimpleSwitch::*)(unsigned long), SimpleSwitch*, unsigned long&, 0>(void (SimpleSwitch::*&&)(unsigned long), SimpleSwitch*&&, unsigned long&) [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__thread/thread.h:212]:14 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x752f2c)
    #3 SimpleSwitch::start_and_return_() [third_party/p4lang_behavioral_model/targets/simple_switch/simple_switch.cpp:279]:24 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x74e21c) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #4 bm::SwitchWContexts::start_and_return() [third_party/p4lang_behavioral_model/src/bm_sim/switch.cpp:87]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x91f230) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #5 sswitch_grpc::SimpleSwitchGrpcRunner::init_and_start(bm::OptionsParser const&) [third_party/p4lang_behavioral_model/targets/simple_switch_grpc/switch_runner.cpp:670]:18 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x73af58) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #6 main [third_party/p4lang_behavioral_model/targets/simple_switch_grpc/main.cpp:238]:23 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x7382b8) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)

  Thread T30 (tid=1327, running) created by main thread at:
    #0 pthread_create third_party/llvm/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1023:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x6a4898) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #1 __libcpp_thread_create third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__thread/support/pthread.h:182:10 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x752f2c) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #2 std::__tsan::thread::thread<void (SimpleSwitch::*)(unsigned long), SimpleSwitch*, unsigned long&, 0>(void (SimpleSwitch::*&&)(unsigned long), SimpleSwitch*&&, unsigned long&) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__thread/thread.h:212:14 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x752f2c)
    #3 SimpleSwitch::start_and_return_() third_party/p4lang_behavioral_model/targets/simple_switch/simple_switch.cpp:279:24 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x74e21c) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #4 bm::SwitchWContexts::start_and_return() third_party/p4lang_behavioral_model/src/bm_sim/switch.cpp:87:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x91f230) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #5 sswitch_grpc::SimpleSwitchGrpcRunner::init_and_start(bm::OptionsParser const&) third_party/p4lang_behavioral_model/targets/simple_switch_grpc/switch_runner.cpp:670:18 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x73af58) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #6 main third_party/p4lang_behavioral_model/targets/simple_switch_grpc/main.cpp:238:23 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x7382b8) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)

  Thread T33 'grpcpp_sync_server' (tid=1333, running) created by thread T25 at:
    #0 pthread_create third_party/llvm/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1023:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x6a4898) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #1 Thread::CreatePthread(pthread_attr_t&) [thread/thread.cc:503]:13 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2c49218) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #2 Thread::Start() [thread/thread.cc:679]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2c49d90) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #3 (anonymous namespace)::ThreadInternalsGoogleThread::Start() [third_party/grpc/google_specific/src/core/support/thd_google.cc:106]:36 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62590) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #4 Start [third_party/grpc/src/core/lib/gprpp/thd.h:153]:14 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x142b320) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #5 Start [third_party/grpc/src/cpp/thread_manager/thread_manager.h:124]25 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x142b320)
    #6 grpc::ThreadManager::MainWorkLoop() [third_party/grpc/src/cpp/thread_manager/thread_manager.cc:186]:23 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x142b320)
    #7 Run [third_party/grpc/src/cpp/thread_manager/thread_manager.cc:48]:13 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x142be94) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #8 operator() [third_party/grpc/src/cpp/thread_manager/thread_manager.cc:40]:69 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x142be94)
    #9 grpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) [third_party/grpc/src/cpp/thread_manager/thread_manager.cc:40]:7 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x142be94)
    #10 (anonymous namespace)::ThreadBodyWithCleanup(void (*)(void*), void*, (anonymous namespace)::ThreadInternalsGoogleThread*) [third_party/grpc/google_specific/src/core/support/thd_google.cc:117]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62444) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #11 __invoke<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__type_traits/invoke.h:150]:25 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #12 invoke<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:28]:10 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #13 operator()<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/bind_front.h:35]:12 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #14 operator()<void> [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/perfect_forward.h:77]:12 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #15 __invoke<std::__tsan::__bind_front_t<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *> > [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__type_traits/invoke.h:150]:25 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #16 invoke<std::__tsan::__bind_front_t<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *> > [third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:28]:10 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #17 InvokeR<void, std::__tsan::__bind_front_t<void (*)(void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *), void (*)(void *), void *, (anonymous namespace)::ThreadInternalsGoogleThread *>, void> [third_party/absl/functional/internal/any_invocable.h:132]:3 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #18 void absl::internal_any_invocable::RemoteInvoker<false, void, std::__tsan::__bind_front_t<void (*)(void (*)(void*), void*, (anonymous namespace)::ThreadInternalsGoogleThread*), void (*)(void*), void*, (anonymous namespace)::ThreadInternalsGoogleThread*>&&>(absl::internal_any_invocable::TypeErasedState*) [third_party/absl/functional/internal/any_invocable.h:368]:10 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2a62644)
    #19 operator() [third_party/absl/functional/internal/any_invocable.h:876]:1 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0xae7624) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)
    #20 ClosureThread::Run() [thread/thread.h:464]:25 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0xae7624)
    #21 Thread::ThreadBody(void*) [thread/thread.cc:1355]:16 (7cbb561ee76890897533062d9b05b236b11b59e2da08549042867c7da37c3a8f_02000fd58b30+0x2c497f4) (BuildId: 9ded8881ecb23c96d5ab6691f705d26a)

SUMMARY: ThreadSanitizer: data race [third_party/p4lang_behavioral_model/include/bm/bm_sim/actions.h:637]:22 in set_source_info
==================

@smolkaj , @jonathan-dilorenzo

Interesting find.

I am not familiar enough with this code to be certain, but I would not be surprised if this set_source_info() call here: https://github.com/p4lang/behavioral-model/blob/main/src/bm_sim/actions.cpp#L371

would be best done once at P4 program load time, rather than once every time an action is invoked while processing a packet. That would be somewhat more efficient, as well as hopefully avoiding this data race.

would be best done once at P4 program load time, rather than once every time an action is invoked while processing a packet. That would be somewhat more efficient, as well as hopefully avoiding this data race.

This was my first thought, but it is not an option here. There is a single ActionPrimitive object instantiated for each primitive action, and it is shared by all pipelines whenever the primitive is executed, no matter which action it is executed from.

The source info we store here is related to the primitive action "call", and so it depends on the location of the actual call site. On primitive can be used many times in a given program.

Fortunately, there is a simple fix here. The field needs to be switched to a static thread_local one. It is no different from how the pkt and phv fields are handled.

behavioral-model/include/bm/bm_sim/actions.h

Lines 641 to 648 in 8e183a3

    
           // This used to be regular members in ActionPrimitive, but there could be a 
        
           // race condition. Making them thread_local solves the issue. I moved these 
        
           // from ActionPrimitive because g++4.8 has a bug when using thread_local class 
        
           // variables with templates 
        
           // (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60056). 
        
           static thread_local Packet *pkt; 
        
           static thread_local PHV *phv; 
        
           SourceInfo *call_source_info{nullptr};

I can take care of it a bit later, unless someone wants to beat me to it.

Sorry, I got a bit confused regarding which field you are referring to that needs to be switched to a static thread_local

I included a code snippet in my post. It's the only field shown there that is not static thread_local.

I tried to change the field to a static thread_local SourceInfo *call_source_info. I had issues with building and running my tests. Not entirely clear what's wrong.

error: undefined symbol: bm::ActionPrimitive_::call_source_info
>>> referenced by primitives.cpp
>>>               blaze-out/k8-fastbuild/bin/third_party/p4lang_behavioral_model/_objs/bm_sim/primitives.pic.o:(bm::core::assert_::operator()(bm::Data const&))
>>> referenced by primitives.cpp
>>>               blaze-out/k8-fastbuild/bin/third_party/p4lang_behavioral_model/_objs/bm_sim/primitives.pic.o:(bm::core::assume_::operator()(bm::Data const&))
>>> referenced by primitives.cpp
>>>               blaze-out/k8-fastbuild/bin/third_party/p4lang_behavioral_model/_objs/bm_sim/primitives.pic.o:(bm::core::log_msg::operator()(std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>> const&, std::__u::vector<bm::Data, std::__u::allocator<bm::Data>>))
>>> referenced 2 more times
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I see that there are references to call_source_info inside other files like primitives.cpp from a quick search, not particularly sure if these get affected or not.

Did you define the static field in actions.cpp as well?

behavioral-model/src/bm_sim/actions.cpp

Lines 481 to 482 in 8e183a3

    
           thread_local Packet *ActionPrimitive_::pkt = nullptr; 
        
           thread_local PHV *ActionPrimitive_::phv = nullptr;

	// This used to be regular members in ActionPrimitive, but there could be a
	// race condition. Making them thread_local solves the issue. I moved these
	// from ActionPrimitive because g++4.8 has a bug when using thread_local class
	// variables with templates
	// (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60056).
	static thread_local Packet *pkt;
	static thread_local PHV *phv;
	SourceInfo *call_source_info{nullptr};

	thread_local Packet *ActionPrimitive_::pkt = nullptr;
	thread_local PHV *ActionPrimitive_::phv = nullptr;