osrf/buildfarm-tools

Buildfarmer log 2024/05/13 - 2024/05/20

Closed this issue · 12 comments

Previous log #48

May 13

Build regressions

Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__benchmark_ubuntu_noble_amd64
- Failing 100.0% in 10 out of 10 builds
	projectroot.benchmark_string_conversions
	  (also in: "Jci__benchmark_ubuntu_noble_amd64" job)

Issue #2 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 8 out of 8 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)

Issue #3 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 8 out of 8 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 8 out of 8 builds
	projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)

Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 8 out of 8 builds
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 8 out of 8 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #7 in nightly_linux-aarch64_repeated
- Failing 40.0% in 6 out of 15 builds
	rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice
	projectroot.test_sequential_compression_writer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #8 in nightly_linux_iron_coverage
- Failing 35.71% in 5 out of 14 builds
	test_tracetools.test_tracetools.test.test_timer.TestTimer.test_all

Issue #9 in nightly_linux_iron_coverage
- Failing 35.71% in 5 out of 14 builds
	projectroot.test_timer
	  (also in: "nightly_win_rep" job)

Issue #10 in nightly_win_rep
- Failing 50.0% in 5 out of 10 builds
	rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock
	projectroot.test.rclcpp.test_clock
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #11 in nightly_linux-rhel_repeated
- Failing 33.33% in 5 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_debug" job)

Issue #12 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #13 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)

Issue #14 in nightly_linux-aarch64_repeated
- Failing 26.67% in 4 out of 15 builds
	rosbag2_transport.RecordIntegrationTestFixture.can_record_again_after_stop
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #15 in nightly_linux-aarch64_repeated
- Failing 26.67% in 4 out of 15 builds
	projectroot.test_record__rmw_fastrtps_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #16 in nightly_win_rep
- Failing 40.0% in 4 out of 10 builds
	rclpy.test_service.xunit.missing_result
	projectroot.test_service

Issue #17 in nightly_win_rep
- Failing 40.0% in 4 out of 10 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #18 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 37.5% in 3 out of 8 builds
	rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #19 in nightly_win_rep
- Failing 30.0% in 3 out of 10 builds
	rclcpp.test_executor.gtest.missing_result
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #20 in nightly_win_rep
- Failing 30.0% in 3 out of 10 builds
	rclcpp.TestInterfaceTraits.has_node_base_interface
	rclcpp.TestExecutors/SingleThreadedExecutor.detachOnDestruction
	projectroot.test.rclcpp.test_logger_service
	projectroot.test.rclcpp.test_interface_traits
	  (also in: "nightly_win_rel" job)

Issue #21 in nightly_win_rep
- Failing 30.0% in 3 out of 10 builds
	projectroot.test.rclcpp.test_events_executor
	  (also in: "nightly_win_rel" job)

Issue #22 in gz_transport-12-win
- Failing 40.0% in 2 out of 5 builds
	playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-13-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)

Issue #23 in nightly_linux-aarch64_repeated
- Failing 13.33% in 2 out of 15 builds
	rosbag2_py.test_transport_py.xunit.missing_result
	projectroot.test_transport_py

Issue #24 in nightly_linux-rhel_repeated
- Failing 13.33% in 2 out of 15 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[1-3]
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #25 in nightly_win_rep
- Failing 20.0% in 2 out of 10 builds
	rclcpp.TestLoggerService.test_set_and_get_multi_logging_level

Issue #26 in nightly_win_rep
- Failing 20.0% in 2 out of 10 builds
	projectroot.test.rclcpp.test_subscription_topic_statistics
	  (also in: "nightly_win_rel" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-connext_ubuntu_noble_amd64" job)

Issue #27 in nightly_win_rep
- Failing 20.0% in 2 out of 10 builds
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp
	projectroot.test.rclcpp.test_storage_policy_common
	projectroot.test.rclcpp.test_static_single_threaded_executor
	projectroot.test.rclcpp.test_graph_listener
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #28 in nightly_win_rep
- Failing 20.0% in 2 out of 10 builds
	rclcpp.TestWaitSet.add_remove_wait
	rclcpp.TestWaitSet.add_guard_condition_to_two_different_wait_set
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testTimer2CancelledWithExecutorSpin
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testTimer1CancelledWithExecutorSpin
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testHeadTimerCancelThenResetBehavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBothTimerCancelThenResetT2Behavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBothTimerCancelThenResetT1Behavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBackTimerCancelThenResetBehavior
	rclcpp.TestThreadSafeStorage.wait_waitable
	rclcpp.TestThreadSafeStorage.wait_timer
	rclcpp.TestThreadSafeStorage.wait_subscription
	rclcpp.TestThreadSafeStorage.wait_client_service
	rclcpp.TestThreadSafeStorage.iterables_construct_destruct
	rclcpp.TestThreadSafeStorage.default_construct_destruct
	rclcpp.TestThreadSafeStorage.add_remove_out_of_scope
	rclcpp.TestThreadSafeStorage.add_remove_nullptr
	rclcpp.TestThreadSafeStorage.add_remove_dynamically
	rclcpp.TestSubscriptionTopicStatisticsFixture.test_invalid_publish_period
	rclcpp.TestSubscriptionOptions.topic_statistics_options_node_default_mode
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_resize_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_fini_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_clear_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_timer_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_subscription_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_service_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_guard_condition_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_client_error
	rclcpp.TestStoragePolicyCommon.add_waitable_error
	rclcpp.TestStaticStorage.wait_waitable
	rclcpp.TestStaticStorage.wait_timer
	rclcpp.TestStaticStorage.wait_subscription
	rclcpp.TestStaticStorage.wait_guard_condition
	rclcpp.TestStaticStorage.wait_client_service
	rclcpp.TestStaticStorage.iterables_construct_destruct
	rclcpp.TestStaticStorage.fixed_storage_needs_pruning
	rclcpp.TestStaticSingleThreadedExecutor.add_callback_group_trigger_guard_failed
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_parent_log
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_node_rosout_disabled
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_getchild_hierarchy
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_getchild
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_child_log
	rclcpp.TestMultiThreadedExecutor.timer_over_take
	rclcpp.TestIntraprocessExecutors/SingleThreadedExecutor.testIntraprocessRetrigger
	rclcpp.TestIntraProcessWaitable.test_that_waitable_stays_ready_after_second_wait
	rclcpp.TestGraphListener.construction_and_destruction
	rclcpp.TestExecutorNotifyWaitable.add_remove_guard_conditions
	rclcpp.TestExecutorEntitiesCollector.add_remove_node
	rclcpp.TestEventsExecutor.run_pub_sub
	rclcpp.TestDynamicStorage.wait_waitable
	rclcpp.TestDynamicStorage.wait_timer
	rclcpp.TestDynamicStorage.wait_subscription
	rclcpp.TestDynamicStorage.wait_client_service
	rclcpp.TestDynamicStorage.iterables_construct_destruct
	rclcpp.TestDynamicStorage.default_construct_destruct
	rclcpp.TestDynamicStorage.add_remove_out_of_scope
	rclcpp.TestDynamicStorage.add_remove_nullptr
	rclcpp.TestDynamicStorage.add_remove_dynamically
	rclcpp.TestContentFilterSubscription.set_content_filter_error
	rclcpp.TestContentFilterSubscription.set_content_filter
	rclcpp.TestContentFilterSubscription.is_cft_enabled
	rclcpp.TestContentFilterSubscription.get_content_filter_error
	rclcpp.TestContentFilterSubscription.get_content_filter
	rclcpp.TestContentFilterSubscription.content_filter_reset
	rclcpp.TestContentFilterSubscription.content_filter_get_later
	rclcpp.TestContentFilterSubscription.content_filter_get_begin
	rclcpp.TestCallbackGroup.valid_callback_group
	rclcpp.TestCallbackGroup.invalid_callback_group
	projectroot.test.rclcpp.test_wait_set
	projectroot.test.rclcpp.test_thread_safe_synchronization
	projectroot.test.rclcpp.test_subscription_options
	projectroot.test.rclcpp.test_static_storage
	projectroot.test.rclcpp.test_rosout_subscription
	projectroot.test.rclcpp.test_multi_threaded_executor
	projectroot.test.rclcpp.test_intra_process_waitable
	projectroot.test.rclcpp.test_executors_timer_cancel_behavior
	projectroot.test.rclcpp.test_executors_intraprocess
	projectroot.test.rclcpp.test_executors_callback_group_behavior
	projectroot.test.rclcpp.test_executor_notify_waitable
	projectroot.test.rclcpp.test_entities_collector
	projectroot.test.rclcpp.test_dynamic_storage
	  (also in: "nightly_win_rel" job)

Issue #29 in Rci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 7.14% in 1 out of 14 builds
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin

Issue #30 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_minimal_scene.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_marker_manager.test_ran
	INTEGRATION_camera_tracking.test_ran

Issue #31 in gz_sim-main-win
- Failing 20.0% in 1 out of 5 builds
	INTEGRATION_mesh_inertia_calculation.test_ran
	  (also in: "gz_sim-8-win" job)

Issue #32 in nightly_linux-rhel_repeated
- Failing 6.67% in 1 out of 15 builds
	ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #33 in nightly_linux-rhel_repeated
- Failing 6.67% in 1 out of 15 builds
	rclcpp.TestAddCallbackGroupsToExecutorStable/EventsExecutor.subscriber_triggered_to_receive_message
	  (also in: "nightly_win_rel" job)

Issue #34 in nightly_win_rep
- Failing 10.0% in 1 out of 10 builds
	test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
	  (also in: "nightly_linux-aarch64_release" job)

Issue #35 in nightly_linux-aarch64_repeated
- Failing 6.67% in 1 out of 15 builds
	test_rclcpp.TestNNodes.test_10_nodes
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #36 in nightly_linux-aarch64_repeated
- Failing 6.67% in 1 out of 15 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #37 in nightly_win_rep
- Failing 10.0% in 1 out of 10 builds
	projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__non_secure_comm_3
	projectroot.image_transport_qos_override
	image_transport.TestQosOverride.qos_override_subscriber_without_options
	image_transport.TestQosOverride.qos_override_subscriber_with_options
	image_transport.TestQosOverride.qos_override_publisher_with_options

Garden and harmonic broken homebrew bottles | Build regression | Keeping track 👀

Referecence builds:

Steve handling bottles broekn by gdal in: osrf/homebrew-simulation#2651

Gz-rendering8 Windows | Build regression | Investigating 🕵️

Reference build: https://build.osrfoundation.org/job/gz_rendering-8-win/48/console

Log output:

CMake Error at C:/vcpkg/scripts/buildsystems/vcpkg.cmake:782 (_find_package):
  By not providing "Findgz-cmake3.cmake" in CMAKE_MODULE_PATH this project
  has asked CMake to find a package configuration file provided by
  "gz-cmake3", but CMake did not find one.

  Could not find a package configuration file provided by "gz-cmake3" with
  any of the following names:

    gz-cmake3Config.cmake
    gz-cmake3-config.cmake

  Add the installation prefix of "gz-cmake3" to CMAKE_PREFIX_PATH or set
  "gz-cmake3_DIR" to a directory containing one of the above files.  If
  "gz-cmake3" provides a separate development package or SDK, be sure it has
  been installed.
Call Stack (most recent call first):
  CMakeLists.txt:12 (find_package)


-- Configuring incomplete, errors occurred!
See also "C:/J/workspace/gz_rendering-8-win/ws/build/gz-rendering8/CMakeFiles/CMakeOutput.log".
---
--- stderr: gz-rendering8

Not sure if it's the right change, but I see gazebo-release/gz-rendering8-release#4. However, these are change to Noble, not windows.

May 14

Build regressions
nightly_linux_coverage|2385|2024-05-14 05:11:37.755000|Number of failed test exceeds the threshold|coverage analysis
nightly_linux_humble_coverage|799|2024-05-14 05:24:33.734000|Packages rosbag2_transport  Failed|build
Ici__nightly-performance_ubuntu_jammy_amd64|97|2024-05-14 02:33:22.137000|Packages performance_test Failed|build workspace in isolation and install
Rci__nightly-performance_ubuntu_noble_amd64|17|2024-05-13 20:44:29.454000|Packages performance_test Failed|build workspace in isolation and install
gz-msgs11-debbuilder|315|2024-05-14 10:29:49.664000|Failed to compile and install build|install build dependencies from debian/control
gz_launch-7-win|46|2024-05-13 21:06:37.300000|Packages gz-rendering8  Failed|colcon compilation without test for dependencies of gz-launch7
gz_physics-7-win|50|2024-05-14 05:32:50.193000|Jenkins channel closing down (Agent disconnected)|colcon compilation with tests for gz-physics7
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__benchmark_ubuntu_noble_amd64
- Failing 100.0% in 11 out of 11 builds
	projectroot.benchmark_string_conversions
	  (also in: "Jci__benchmark_ubuntu_noble_amd64" job)

Issue #2 in nightly_win_deb
- Failing 60.0% in 9 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_debug" job)

Issue #3 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 8 out of 8 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)

Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 8 out of 8 builds
	projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)

Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 8 out of 8 builds
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 8 out of 8 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #7 in nightly_linux-aarch64_repeated
- Failing 50.0% in 8 out of 16 builds
	rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice
	projectroot.test_sequential_compression_writer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #8 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 8 out of 8 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #9 in nightly_linux-rhel_repeated
- Failing 31.25% in 5 out of 16 builds
	projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp

Issue #10 in nightly_win_rep
- Failing 41.67% in 5 out of 12 builds
	projectroot.test_timer
	  (also in: "nightly_linux_iron_coverage" job)

Issue #11 in nightly_win_rep
- Failing 41.67% in 5 out of 12 builds
	projectroot.test_subscription
	  (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job)
	  (also in: "Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #12 in nightly_win_rep
- Failing 41.67% in 5 out of 12 builds
	rclpy.test_waitable.xunit.missing_result
	rclpy.test_wait_for_message.xunit.missing_result
	rclpy.test_timer.xunit.missing_result
	rclpy.test_time_source.xunit.missing_result
	rclpy.test_subscription.xunit.missing_result
	rclpy.test_service_introspection.xunit.missing_result
	rclpy.test_service.xunit.missing_result
	projectroot.test_waitable
	projectroot.test_wait_for_message
	projectroot.test_time_source
	projectroot.test_service_introspection

Issue #13 in nightly_win_rep
- Failing 41.67% in 5 out of 12 builds
	rclcpp.TestInterfaceTraits.has_node_base_interface
	projectroot.test.rclcpp.test_logger_service
	projectroot.test.rclcpp.test_interface_traits
	  (also in: "nightly_win_rel" job)

Issue #14 in nightly_win_rep
- Failing 41.67% in 5 out of 12 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #15 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 50.0% in 4 out of 8 builds
	rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #16 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #17 in nightly_win_rep
- Failing 33.33% in 4 out of 12 builds
	rclcpp.test_executor.gtest.missing_result
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #18 in nightly_win_rep
- Failing 33.33% in 4 out of 12 builds
	rclpy.test_node.xunit.missing_result
	projectroot.test_node

Issue #19 in nightly_win_rep
- Failing 33.33% in 4 out of 12 builds
	rclcpp.TestUtilities.wait_for_message_twice_one_sub
	rclcpp.TestUtilities.wait_for_message_indefinitely
	rclcpp.TestUtilities.wait_for_message
	rclcpp.TestTimeSource.reattach
	rclcpp.TestTimeSource.parameter_activation
	rclcpp.TestTimeSource.no_pre_jump_callback
	rclcpp.TestTimeSource.clock_sleep_until_with_ros_time_basic
	rclcpp.TestTimeSource.check_use_clock_thread_value
	rclcpp.TestTimeSource.check_clock_thread_status
	rclcpp.TestTimeSource.callbacks
	rclcpp.TestTimeSource.ROS_time_valid_wall_time
	rclcpp.TestTimeSource.ROS_time_valid_sim_time
	rclcpp.TestTimeSource.ROS_time_valid_attach_detach
	rclcpp.TestTimeSource.ROS_invalid_sim_time
	rclcpp.TestLoggerService.check_connect_get_logger_service
	rclcpp.TestExecutors/SingleThreadedExecutor.detachOnDestruction
	projectroot.test.rclcpp.test_wait_for_message
	  (also in: "nightly_win_rel" job)

Issue #20 in nightly_win_rep
- Failing 33.33% in 4 out of 12 builds
	rclcpp.TestTimeSource.clock
	rclcpp.TestTimeSource.callback_handler_erasure
	projectroot.test.rclcpp.test_time_source
	  (also in: "nightly_win_rel" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #21 in nightly_win_rep
- Failing 33.33% in 4 out of 12 builds
	projectroot.test.rclcpp.test_events_executor
	  (also in: "nightly_win_rel" job)

Issue #22 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 3 out of 3 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #23 in nightly_win_rep
- Failing 25.0% in 3 out of 12 builds
	projectroot.test_publisher (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	  (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	  (also in: "Jci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	  (also in: "Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)

Issue #24 in nightly_win_rep
- Failing 25.0% in 3 out of 12 builds
	rclpy.test_rosout_subscription.xunit.missing_result
	rclpy.test_rate.xunit.missing_result
	rclpy.test_qos_overriding_options.xunit.missing_result
	rclpy.test_qos_event.xunit.missing_result
	rclpy.test_publisher.xunit.missing_result
	rclpy.test_parameter_service.xunit.missing_result
	rclpy.test_parameter_event_handler.xunit.missing_result
	rclpy.test_parameter_client.xunit.missing_result
	projectroot.test_rosout_subscription
	projectroot.test_rate
	projectroot.test_qos_overriding_options
	projectroot.test_qos_event
	projectroot.test_parameter_service
	projectroot.test_parameter_event_handler
	projectroot.test_parameter_client

Issue #25 in nightly_win_rep
- Failing 25.0% in 3 out of 12 builds
	projectroot.test_default_state_machine

Issue #26 in nightly_win_rep
- Failing 25.0% in 3 out of 12 builds
	projectroot.test.rclcpp.test_subscription_topic_statistics
	  (also in: "nightly_win_rel" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-connext_ubuntu_noble_amd64" job)

Issue #27 in nightly_win_rep
- Failing 25.0% in 3 out of 12 builds
	projectroot.test.rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp
	projectroot.test.rclcpp.test_storage_policy_common
	projectroot.test.rclcpp.test_static_single_threaded_executor
	projectroot.test.rclcpp.test_graph_listener
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #28 in nightly_win_rep
- Failing 25.0% in 3 out of 12 builds
	rclcpp.TestWaitSet.add_remove_wait
	rclcpp.TestWaitSet.add_guard_condition_to_two_different_wait_set
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testTimer2CancelledWithExecutorSpin
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testTimer1CancelledWithExecutorSpin
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testHeadTimerCancelThenResetBehavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBothTimerCancelThenResetT2Behavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBothTimerCancelThenResetT1Behavior
	rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBackTimerCancelThenResetBehavior
	rclcpp.TestThreadSafeStorage.wait_waitable
	rclcpp.TestThreadSafeStorage.wait_timer
	rclcpp.TestThreadSafeStorage.wait_subscription
	rclcpp.TestThreadSafeStorage.wait_client_service
	rclcpp.TestThreadSafeStorage.iterables_construct_destruct
	rclcpp.TestThreadSafeStorage.default_construct_destruct
	rclcpp.TestThreadSafeStorage.add_remove_out_of_scope
	rclcpp.TestThreadSafeStorage.add_remove_nullptr
	rclcpp.TestThreadSafeStorage.add_remove_dynamically
	rclcpp.TestSubscriptionTopicStatisticsFixture.test_invalid_publish_period
	rclcpp.TestSubscriptionOptions.topic_statistics_options_node_default_mode
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_resize_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_fini_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_clear_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_timer_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_subscription_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_service_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_guard_condition_error
	rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_client_error
	rclcpp.TestStoragePolicyCommon.add_waitable_error
	rclcpp.TestStaticStorage.wait_waitable
	rclcpp.TestStaticStorage.wait_timer
	rclcpp.TestStaticStorage.wait_subscription
	rclcpp.TestStaticStorage.wait_guard_condition
	rclcpp.TestStaticStorage.wait_client_service
	rclcpp.TestStaticStorage.iterables_construct_destruct
	rclcpp.TestStaticStorage.fixed_storage_needs_pruning
	rclcpp.TestStaticSingleThreadedExecutor.add_callback_group_trigger_guard_failed
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_parent_log
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_node_rosout_disabled
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_getchild_hierarchy
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_getchild
	rclcpp.TestRosoutSubscription.test_rosoutsubscription_child_log
	rclcpp.TestMultiThreadedExecutor.timer_over_take
	rclcpp.TestIntraprocessExecutors/SingleThreadedExecutor.testIntraprocessRetrigger
	rclcpp.TestIntraProcessWaitable.test_that_waitable_stays_ready_after_second_wait
	rclcpp.TestGraphListener.construction_and_destruction
	rclcpp.TestExecutorNotifyWaitable.add_remove_guard_conditions
	rclcpp.TestExecutorEntitiesCollector.add_remove_node
	rclcpp.TestEventsExecutor.run_pub_sub
	rclcpp.TestDynamicStorage.wait_waitable
	rclcpp.TestDynamicStorage.wait_timer
	rclcpp.TestDynamicStorage.wait_subscription
	rclcpp.TestDynamicStorage.wait_client_service
	rclcpp.TestDynamicStorage.iterables_construct_destruct
	rclcpp.TestDynamicStorage.default_construct_destruct
	rclcpp.TestDynamicStorage.add_remove_out_of_scope
	rclcpp.TestDynamicStorage.add_remove_nullptr
	rclcpp.TestDynamicStorage.add_remove_dynamically
	rclcpp.TestContentFilterSubscription.set_content_filter_error
	rclcpp.TestContentFilterSubscription.set_content_filter
	rclcpp.TestContentFilterSubscription.is_cft_enabled
	rclcpp.TestContentFilterSubscription.get_content_filter_error
	rclcpp.TestContentFilterSubscription.get_content_filter
	rclcpp.TestContentFilterSubscription.content_filter_reset
	rclcpp.TestContentFilterSubscription.content_filter_get_later
	rclcpp.TestContentFilterSubscription.content_filter_get_begin
	rclcpp.TestCallbackGroup.valid_callback_group
	rclcpp.TestCallbackGroup.invalid_callback_group
	projectroot.test.rclcpp.test_wait_set
	projectroot.test.rclcpp.test_thread_safe_synchronization
	projectroot.test.rclcpp.test_subscription_options
	projectroot.test.rclcpp.test_static_storage
	projectroot.test.rclcpp.test_rosout_subscription
	projectroot.test.rclcpp.test_multi_threaded_executor
	projectroot.test.rclcpp.test_intra_process_waitable
	projectroot.test.rclcpp.test_executors_timer_cancel_behavior
	projectroot.test.rclcpp.test_executors_intraprocess
	projectroot.test.rclcpp.test_executors_callback_group_behavior
	projectroot.test.rclcpp.test_executor_notify_waitable
	projectroot.test.rclcpp.test_entities_collector
	projectroot.test.rclcpp.test_dynamic_storage
	  (also in: "nightly_win_rel" job)

Issue #29 in gz_transport-12-win
- Failing 50.0% in 2 out of 4 builds
	playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-13-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)

Issue #30 in nightly_linux_repeated
- Failing 13.33% in 2 out of 15 builds
	test_rclcpp.test_node_name__rmw_fastrtps_cpp.xunit.missing_result
	projectroot.test_node_name__rmw_fastrtps_cpp

Issue #31 in nightly_win_rel
- Failing 13.33% in 2 out of 15 builds
	ros2cli.ros2cli.test.test_daemon.test_get_name

Issue #32 in nightly_win_rep
- Failing 16.67% in 2 out of 12 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #33 in nightly_win_rep
- Failing 16.67% in 2 out of 12 builds
	projectroot.test_action_server

Issue #34 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 16.67% in 1 out of 6 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Hci__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #35 in gz_gui-ci-gz-gui7-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	UNIT_Screenshot_TEST.test_ran
	  (also in: "gz_gui-ci-ign-gui6-focal-amd64" job)

Issue #36 in gz_gui-ci-gz-gui7-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	INTEGRATION_minimal_scene.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_marker_manager.test_ran
	INTEGRATION_camera_tracking.test_ran
	  (also in: "gz_gui-ci-ign-gui6-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)

Issue #37 in gz_gui-ci-gz-gui7-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	UNIT_PointCloud_TEST.test_ran
	INTEGRATION_transport_scene_manager.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_deprecated_TEST.test_ran
	INTEGRATION_Examples_TEST.test_ran
	INTEGRATION_ExamplesBuild_TEST.test_ran

Issue #38 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_tpelib_World_TEST.test_ran
	UNIT_tpelib_Utils_TEST.test_ran
	UNIT_tpelib_Shape_TEST.test_ran
	UNIT_tpelib_Model_TEST.test_ran
	UNIT_tpelib_Link_TEST.test_ran
	UNIT_tpelib_Engine_TEST.test_ran
	UNIT_tpelib_Collision_TEST.test_ran
	UNIT_tpelib_CollisionDetector_TEST.test_ran
	UNIT_tpelib_AABBTree_TEST.test_ran
	UNIT_tpe_SimulationFeatures_TEST.test_ran
	UNIT_tpe_SDFFeatures_TEST.test_ran
	UNIT_tpe_EntityManagement_TEST.test_ran
	UNIT_tpe_Base_TEST.test_ran
	UNIT_WorldFeatures_TEST.test_ran
	UNIT_SDFFeatures_TEST.test_ran
	UNIT_EntityManagement_TEST.test_ran
	UNIT_Base_TEST.test_ran
	UNIT_AddedMassFeatures_TEST.test_ran
	SimulationFeaturesTestFreeGroup/0.FreeGroup
	SimulationFeaturesTestFeaturesContactPropertiesCallback/0.ContactPropertiesCallback
	SimulationFeaturesTestBasic/0.ShapeBoundingBox
	SimulationFeaturesTestBasic/0.RetrieveContacts
	SimulationFeaturesTestBasic/0.MultipleCollisions
	SimulationFeaturesTestBasic/0.CollideBitmasks
	SimulationFeaturesStepTest/0.StepWorld (known issue in: https://github.com/gazebosim/gz-physics/issues/483 - CLOSED)
	SimulationFeaturesShapeFeaturesTest/0.ShapeFeatures (known issue in: https://github.com/gazebosim/gz-physics/issues/483 - CLOSED)
	SimulationFeaturesFallingTest/0.Falling (known issue in: https://github.com/gazebosim/gz-physics/issues/483 - CLOSED)
	SimulationFeaturesContactsTest/0.Contacts
	SimulationFeaturesCollisionPairMaxContactsTest/0.CollisionPairMaxContacts
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job)

Issue #39 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Manager_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)
	  (also in: "gz_launch-main-win" job) (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)
	  (also in: "gz_launch-ign-launch5-win" job) (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)
	  (also in: "gz_launch-6-win" job) (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)

Issue #40 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_thermal_camera.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering3-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)

Issue #41 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Camera_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	INTEGRATION_depth_camera.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	INTEGRATION_camera.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering6-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering3-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)

Issue #42 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Util_TEST.test_ran
	UNIT_Sensor_TEST.test_ran
	UNIT_Noise_TEST.test_ran
	UNIT_Lidar_TEST.test_ran
	INTEGRATION_wide_angle_camera.test_ran
	INTEGRATION_triggered_camera.test_ran
	INTEGRATION_triggered_boundingbox_camera.test_ran
	INTEGRATION_segmentation_camera.test_ran
	INTEGRATION_rgbd_camera.test_ran
	INTEGRATION_gpu_lidar_sensor.test_ran
	INTEGRATION_distortion_camera.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2084 - CLOSED)
	INTEGRATION_boundingbox_camera.test_ran

Issue #43 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/ServerFixture.RunNonBlockingMultiple/0

Issue #44 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	ros2topic.ros2topic.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #45 in nightly_linux_repeated
- Failing 6.67% in 1 out of 15 builds
	ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #46 in nightly_win_deb
- Failing 6.67% in 1 out of 15 builds
	rclcpp_lifecycle.test_lifecycle_service_client.gtest.missing_result
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #47 in nightly_win_deb
- Failing 6.67% in 1 out of 15 builds
	rclcpp_lifecycle.TestDefaultStateMachine.wait_for_graph_change
	rclcpp_lifecycle.TestDefaultStateMachine.test_graph_topics
	rclcpp_lifecycle.TestDefaultStateMachine.test_graph_services_by_node
	rclcpp_lifecycle.TestDefaultStateMachine.test_graph_services
	rclcpp_lifecycle.TestDefaultStateMachine.test_getters
	rclcpp_lifecycle.TestDefaultStateMachine.test_callback_groups
	rclcpp_lifecycle.TestDefaultStateMachine.lifecycle_subscriber
	rclcpp_lifecycle.TestDefaultStateMachine.declare_parameters
	rclcpp_lifecycle.TestDefaultStateMachine.check_parameters
	  (also in: "nightly_win_rep" job)

Issue #48 in nightly_win_deb
- Failing 6.67% in 1 out of 15 builds
	projectroot.test_lifecycle_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_iron_coverage" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #49 in nightly_win_deb
- Failing 6.67% in 1 out of 15 builds
	projectroot.test_client
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-connext_ubuntu_noble_amd64" job)

Issue #50 in nightly_win_deb
- Failing 6.67% in 1 out of 15 builds
	rclcpp_lifecycle.TestStateMachineInfo.transition_graph
	rclcpp_lifecycle.TestStateMachineInfo.available_transitions
	rclcpp_lifecycle.TestStateMachineInfo.available_states
	rclcpp_lifecycle.TestService.construction_and_destruction
	rclcpp_lifecycle.TestRegisterCustomCallbacks.custom_callbacks
	rclcpp_lifecycle.TestDefaultStateMachine.trigger_transition_with_error_code
	rclcpp_lifecycle.TestDefaultStateMachine.trigger_transition_rcl_errors
	rclcpp_lifecycle.TestDefaultStateMachine.trigger_transition
	rclcpp_lifecycle.TestDefaultStateMachine.good_mood
	rclcpp_lifecycle.TestDefaultStateMachine.empty_initializer_rcl_errors
	rclcpp_lifecycle.TestDefaultStateMachine.empty_initializer
	rclcpp_lifecycle.TestDefaultStateMachine.check_logger_services_exist
	rclcpp_lifecycle.TestDefaultStateMachine.call_transitions_without_code
	rclcpp_lifecycle.TestDefaultStateMachine.call_transitions_with_error_code
	rclcpp_lifecycle.TestDefaultStateMachine.bad_mood
	rclcpp_lifecycle.TestClient.construction_with_free_function
	rclcpp_lifecycle.TestClient.construction_and_destruction
	rclcpp_lifecycle.TestCallbackExceptions.positive_on_error_with_code
	rclcpp_lifecycle.TestCallbackExceptions.positive_on_error
	rclcpp_lifecycle.TestCallbackExceptions.negative_on_error_with_code
	rclcpp_lifecycle.TestCallbackExceptions.negative_on_error
	projectroot.test_tutorial_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp
	projectroot.test_state_machine_info
	projectroot.test_register_custom_callbacks
	projectroot.test_callback_exceptions

Issue #51 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	rosbag2_transport.RecordIntegrationTestFixture.published_messages_from_multiple_services_are_recorded
	  (also in: "nightly_win_deb" job)

Issue #52 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	ros2cli.ros2cli.test.test_daemon.test_get_namespace

Issue #53 in nightly_win_deb
- Failing 6.67% in 1 out of 15 builds
	projectroot.test_service
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)

Issue #54 in nightly_win_rep
- Failing 8.33% in 1 out of 12 builds
	test_rclcpp.test_waitable.waitable_with_timer
	test_cli_remapping.TestCLIRemappingAfterShutdown.test_processes_finished_gracefully[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]
	test_cli_remapping.TestCLIRemapping.test_topic_replacement[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]
	rclcpp.TestAddCallbackGroupsToExecutor/MultiThreadedExecutor.remove_callback_group
	rclcpp.TestAddCallbackGroupsToExecutor/MultiThreadedExecutor.one_node_many_callback_groups_many_executors
	rcl_lifecycle.TestDefaultStateMachine.default_sequence
	rcl_action.TestActionServer.test_action_server_notify_goal_done
	projectroot.test.rclcpp.test_add_callback_groups_to_executor__rmw_fastrtps_cpp
	projectroot.gtest_waitable__rmw_fastrtps_cpp

Failing homebrew bottles | Build regression | Investigating 🕵️

Gz-garden and gz-harmonic install botlle jobs have been failing since March 1st:

Log outputs shows a problem with pkg_resources module:

==> Testing osrf/simulation/gz-garden
�[34m==>�[0m �[1m/usr/local/Cellar/gz-garden/1.0.0_10/libexec/bin/vcs validate --input /usr/l�[0m
Last 15 lines from /Users/jenkins/Library/Logs/Homebrew/gz-garden/test.01.vcs:
2024-05-14 03:55:55 -0700

/usr/local/Cellar/gz-garden/1.0.0_10/libexec/bin/vcs
validate
--input
/usr/local/Cellar/gz-garden/1.0.0_10/share/gz/gz-garden/gazebodistro/collection-garden.yaml

Traceback (most recent call last):
  File "/usr/local/Cellar/gz-garden/1.0.0_10/libexec/bin/vcs", line 5, in <module>
    from vcstool.commands.vcs import main
  File "/usr/local/Cellar/gz-garden/1.0.0_10/libexec/lib/python3.12/site-packages/vcstool/commands/vcs.py", line 3, in <module>
    from vcstool.commands.help import get_entrypoint
  File "/usr/local/Cellar/gz-garden/1.0.0_10/libexec/lib/python3.12/site-packages/vcstool/commands/help.py", line 4, in <module>
    from pkg_resources import load_entry_point
ModuleNotFoundError: No module named 'pkg_resources'
Error: osrf/simulation/gz-garden: failed
Warning: Removed Sorbet lines from backtrace!
Rerun with `--verbose` to see the original backtrace
An exception occurred within a child process:
  BuildError: Failed executing: /usr/local/Cellar/gz-garden/1.0.0_10/libexec/bin/vcs validate --input /usr/local/Cellar/gz-garden/1.0.0_10/share/gz/gz-garden/gazebodistro/collection-garden.yaml

May 15

Build regressions
nightly_linux_coverage|2386|2024-05-15 04:48:35.224000|Number of failed test exceeds the threshold|coverage analysis
nightly_linux_iron_coverage|431|2024-05-15 05:54:32.205000|Jenkins channel closing down (Agent disconnected)|build
Rci__nightly-performance_ubuntu_noble_amd64|18|2024-05-15 00:50:43.778000|Packages performance_test Failed|build workspace in isolation and install
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__benchmark_ubuntu_noble_amd64
- Failing 100.0% in 11 out of 11 builds
	projectroot.benchmark_string_conversions
	  (also in: "Jci__benchmark_ubuntu_noble_amd64" job)

Issue #2 in nightly_linux-aarch64_repeated
- Failing 56.25% in 9 out of 16 builds
	rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice
	projectroot.test_sequential_compression_writer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #3 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 7 out of 7 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_linux-aarch64_debug" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 7 out of 7 builds
	projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)

Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 7 out of 7 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 7 out of 7 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #7 in nightly_win_deb
- Failing 46.67% in 7 out of 15 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)

Issue #8 in nightly_linux_debug
- Failing 40.0% in 6 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_debug" job)

Issue #9 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 57.14% in 4 out of 7 builds
	rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #10 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #11 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #12 in nightly_linux_repeated
- Failing 26.67% in 4 out of 15 builds
	rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock
	projectroot.test.rclcpp.test_clock
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #13 in nightly_linux_repeated
- Failing 26.67% in 4 out of 15 builds
	launch_testing_examples.launch_testing_examples.check_node_launch_test.launch_testing_examples.check_node_launch_test
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #14 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 60.0% in 3 out of 5 builds
	INTEGRATION_gpu_lidar.test_ran

Issue #15 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[1-3]
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #16 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	projectroot.test_record__rmw_cyclonedds_cpp

Issue #17 in nightly_linux_repeated
- Failing 13.33% in 2 out of 15 builds
	ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #18 in nightly_win_deb
- Failing 13.33% in 2 out of 15 builds
	test_rclcpp.test_multithreaded.multi_access_publisher
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)

Issue #19 in nightly_win_rel
- Failing 13.33% in 2 out of 15 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy.test_generate_policy
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)

Issue #20 in nightly_win_rep
- Failing 16.67% in 2 out of 12 builds
	rosidl_typesupport_introspection_tests.MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage
	projectroot.test_multi_nested_message_introspection

Issue #21 in nightly_win_rep
- Failing 16.67% in 2 out of 12 builds
	rclcpp.TestTypeSupport.test_service_ts_set_params_atomically_srv
	rclcpp.TestTypeSupport.test_service_ts_list_params_srv
	rclcpp.TestTypeSupport.test_service_ts_describe_params_srv
	projectroot.test.rclcpp.test_type_support
	  (also in: "nightly_win_rel" job)

Issue #22 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 16.67% in 1 out of 6 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Hci__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #23 in gz_gui-ci-gz-gui7-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	UNIT_Screenshot_TEST.test_ran
	  (also in: "gz_gui-ci-ign-gui6-focal-amd64" job)

Issue #24 in gz_gui-ci-gz-gui7-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	INTEGRATION_minimal_scene.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_marker_manager.test_ran
	INTEGRATION_camera_tracking.test_ran
	  (also in: "gz_gui-ci-ign-gui6-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)

Issue #25 in gz_gui-ci-gz-gui7-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	UNIT_PointCloud_TEST.test_ran
	INTEGRATION_transport_scene_manager.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_deprecated_TEST.test_ran
	INTEGRATION_Examples_TEST.test_ran
	INTEGRATION_ExamplesBuild_TEST.test_ran

Issue #26 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_tpelib_World_TEST.test_ran
	UNIT_tpelib_Utils_TEST.test_ran
	UNIT_tpelib_Shape_TEST.test_ran
	UNIT_tpelib_Model_TEST.test_ran
	UNIT_tpelib_Link_TEST.test_ran
	UNIT_tpelib_Engine_TEST.test_ran
	UNIT_tpelib_Collision_TEST.test_ran
	UNIT_tpelib_CollisionDetector_TEST.test_ran
	UNIT_tpelib_AABBTree_TEST.test_ran
	UNIT_tpe_SimulationFeatures_TEST.test_ran
	UNIT_tpe_SDFFeatures_TEST.test_ran
	UNIT_tpe_EntityManagement_TEST.test_ran
	UNIT_tpe_Base_TEST.test_ran
	UNIT_WorldFeatures_TEST.test_ran
	UNIT_SDFFeatures_TEST.test_ran
	UNIT_EntityManagement_TEST.test_ran
	UNIT_Base_TEST.test_ran
	UNIT_AddedMassFeatures_TEST.test_ran
	SimulationFeaturesTestFreeGroup/0.FreeGroup
	SimulationFeaturesTestFeaturesContactPropertiesCallback/0.ContactPropertiesCallback
	SimulationFeaturesTestBasic/0.ShapeBoundingBox
	SimulationFeaturesTestBasic/0.RetrieveContacts
	SimulationFeaturesTestBasic/0.MultipleCollisions
	SimulationFeaturesTestBasic/0.CollideBitmasks
	SimulationFeaturesStepTest/0.StepWorld (known issue in: https://github.com/gazebosim/gz-physics/issues/483 - CLOSED)
	SimulationFeaturesShapeFeaturesTest/0.ShapeFeatures (known issue in: https://github.com/gazebosim/gz-physics/issues/483 - CLOSED)
	SimulationFeaturesFallingTest/0.Falling (known issue in: https://github.com/gazebosim/gz-physics/issues/483 - CLOSED)
	SimulationFeaturesContactsTest/0.Contacts
	SimulationFeaturesCollisionPairMaxContactsTest/0.CollisionPairMaxContacts
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job)

Issue #27 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Manager_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)
	  (also in: "gz_launch-main-win" job) (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)
	  (also in: "gz_launch-ign-launch5-win" job) (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)
	  (also in: "gz_launch-6-win" job) (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)

Issue #28 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_thermal_camera.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering3-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)

Issue #29 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Camera_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	INTEGRATION_depth_camera.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	INTEGRATION_camera.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering6-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering3-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)

Issue #30 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Util_TEST.test_ran
	UNIT_Sensor_TEST.test_ran
	UNIT_Noise_TEST.test_ran
	UNIT_Lidar_TEST.test_ran
	INTEGRATION_wide_angle_camera.test_ran
	INTEGRATION_triggered_camera.test_ran
	INTEGRATION_triggered_boundingbox_camera.test_ran
	INTEGRATION_segmentation_camera.test_ran
	INTEGRATION_rgbd_camera.test_ran
	INTEGRATION_gpu_lidar_sensor.test_ran
	INTEGRATION_distortion_camera.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2084 - CLOSED)
	INTEGRATION_boundingbox_camera.test_ran

Issue #31 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/ServerFixture.RunNonBlockingMultiple/0

Issue #32 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[/home/jenkins-agent/workspace/nightly_linux-rhel_repeated/ws/src/ros2/system_tests/test_cli_remapping/test/name_maker.py]
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[/home/jenkins-agent/workspace/nightly_linux-rhel_repeated/ws/build/test_cli_remapping/name_maker_rclcpp]

Issue #33 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #34 in nightly_linux_repeated
- Failing 6.67% in 1 out of 15 builds
	rcl.TestGraphFixture.test_graph_guard_condition_trigger_check

Issue #35 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	rclpy.test_rate.xunit.missing_result
	projectroot.test_rate
	  (also in: "nightly_win_rep" job)

Issue #36 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	rclpy.test_node.xunit.missing_result
	projectroot.test_node
	  (also in: "nightly_win_rep" job)

Issue #37 in nightly_win_rep
- Failing 8.33% in 1 out of 12 builds
	test_tf2.BufferClientTestAfterShutdown.test_exit_code
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #38 in nightly_win_rep
- Failing 8.33% in 1 out of 12 builds
	projectroot.test_static_publisher.launch.py (known issue in: https://github.com/ros2/geometry2/issues/583 - CLOSED)

May 16

Build regressions
nightly_linux_coverage|2387|2024-05-16 05:01:36.273000|Packages rosbag2_py  Failed|build
nightly_linux_humble_coverage|801|2024-05-16 05:30:54.415000|Packages interactive_markers  Failed|build
packaging_linux|3441|2024-05-16 06:56:34.417000|Early fail|No last section
packaging_linux-rhel|1933|2024-05-16 06:57:11.141000|Early fail|No last section
nightly_win_deb|3095|2024-05-16 04:00:01.558000|Number of failed test exceeds the threshold|test
nightly_win_rel|3047|2024-05-16 04:00:01.565000|Number of failed test exceeds the threshold|test
nightly_win_rep|3356|2024-05-16 04:00:01.571000|Number of failed test exceeds the threshold|test
Rci__nightly-performance_ubuntu_noble_amd64|19|2024-05-15 19:00:32.210000|Packages performance_test Failed|build workspace in isolation and install
gz_sensors-main-win|45|2024-05-16 01:06:41.878000|Jenkins channel closing down (Agent disconnected)|colcon compilation with tests for gz-sensors9
gz_fuel_tools-9-win|47|2024-05-16 01:30:44.975000|Jenkins channel closing down (Agent disconnected)|colcon compilation without test for dependencies of gz-fuel_tools9
gz_gui-8-win|47|2024-05-16 08:06:22.632000|Packages gz-rendering8  Failed|colcon compilation without test for dependencies of gz-gui8
gz_rendering-8-win|49|2024-05-15 14:25:58.118000|Packages gz-rendering8  Failed|colcon compilation with tests for gz-rendering8
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__benchmark_ubuntu_noble_amd64
- Failing 100.0% in 12 out of 12 builds
	projectroot.benchmark_string_conversions
	  (also in: "Jci__benchmark_ubuntu_noble_amd64" job)

Issue #2 in nightly_linux-aarch64_repeated
- Failing 62.5% in 10 out of 16 builds
	rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice
	projectroot.test_sequential_compression_writer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #3 in nightly_linux_debug
- Failing 46.67% in 7 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #4 in nightly_linux_iron_coverage
- Failing 46.67% in 7 out of 15 builds
	test_tracetools.test_tracetools.test.test_timer.TestTimer.test_all

Issue #5 in nightly_linux_iron_coverage
- Failing 46.67% in 7 out of 15 builds
	projectroot.test_timer
	  (also in: "nightly_win_rep" job)

Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 6 out of 6 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #7 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 6 out of 6 builds
	projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)

Issue #8 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 6 out of 6 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #9 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 6 out of 6 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #10 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 66.67% in 4 out of 6 builds
	rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #11 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #12 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #13 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 60.0% in 3 out of 5 builds
	INTEGRATION_gpu_lidar.test_ran

Issue #14 in nightly_linux_repeated
- Failing 20.0% in 3 out of 15 builds
	ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #15 in gz_gui-ci-gz-gui7-homebrew-amd64
- Failing 50.0% in 1 out of 2 builds
	UNIT_Screenshot_TEST.test_ran
	  (also in: "gz_gui-ci-ign-gui6-focal-amd64" job)

Issue #16 in gz_gui-ci-gz-gui7-homebrew-amd64
- Failing 50.0% in 1 out of 2 builds
	INTEGRATION_minimal_scene.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_marker_manager.test_ran
	INTEGRATION_camera_tracking.test_ran
	  (also in: "gz_gui-ci-ign-gui6-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)

Issue #17 in gz_gui-ci-gz-gui7-homebrew-amd64
- Failing 50.0% in 1 out of 2 builds
	UNIT_PointCloud_TEST.test_ran
	INTEGRATION_transport_scene_manager.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_deprecated_TEST.test_ran
	INTEGRATION_Examples_TEST.test_ran
	INTEGRATION_ExamplesBuild_TEST.test_ran

Issue #18 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	UNIT_Manager_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)
	  (also in: "gz_launch-main-win" job) (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)
	  (also in: "gz_launch-ign-launch5-win" job) (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)
	  (also in: "gz_launch-6-win" job) (known issue in: https://github.com/gazebosim/gz-launch/issues/203 - CLOSED | https://github.com/gazebosim/gz-launch/issues/203 - OPEN)

Issue #19 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	INTEGRATION_thermal_camera.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering3-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)

Issue #20 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	UNIT_Camera_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	INTEGRATION_depth_camera.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	INTEGRATION_camera.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering6-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering3-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)

Issue #21 in gz_sensors-ci-gz-sensors7-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	UNIT_Util_TEST.test_ran
	UNIT_Sensor_TEST.test_ran
	UNIT_Noise_TEST.test_ran
	UNIT_Lidar_TEST.test_ran
	INTEGRATION_wide_angle_camera.test_ran
	INTEGRATION_triggered_camera.test_ran
	INTEGRATION_triggered_boundingbox_camera.test_ran
	INTEGRATION_segmentation_camera.test_ran
	INTEGRATION_rgbd_camera.test_ran
	INTEGRATION_gpu_lidar_sensor.test_ran
	INTEGRATION_distortion_camera.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2084 - CLOSED)
	INTEGRATION_boundingbox_camera.test_ran

Issue #22 in gz_sim-ign-gazebo6-win
- Failing 33.33% in 1 out of 3 builds
	ServerRepeat/ServerFixture.RunNonBlockingMultiple/0

Gz-gazebo3 focal | Flaky test regression | Keeping tack 👀

Reference build: https://build.osrfoundation.org/job/gz_sim-ci-ign-gazebo3-focal-amd64/44/

Test started failing again since 2024-05-02 (last time it was seen was in 2024-02-20).

Flaky ratio for the last 15 days:

job_name last_fail first_fail build_count failure_count failure_percentage
gz_sim-ci-ign-gazebo3-focal-amd64 2024-05-14 2024-05-02 5 3 60.0
Test output diff:
❯ diff passing failing                                                                                                                (base) 
2,12c2,4
< 167: [Dbg] [Sensors.cc:307] SensorsPrivate::Stop
< 167: [Dbg] [Sensors.cc:293] SensorsPrivate::RenderThread stopped
< 167: [Dbg] [SimulationRunner.cc:533] Exiting postupdate worker thread (0)
< 167: [Dbg] [Sensors.cc:307] SensorsPrivate::Stop
< 167: [       OK ] GpuLidarTest.GpuLidarBox (2360 ms)
< 167: [----------] 1 test from GpuLidarTest (2360 ms total)
< 167: 
< 167: [----------] Global test environment tear-down
< 167: [==========] 1 test from 1 test suite ran. (2360 ms total)
< 167: [  PASSED  ] 1 test.
< 167/178 Test #167: INTEGRATION_gpu_lidar ................................   Passed    2.46 sec
---
> 167: /home/jenkins/workspace/gz_sim-ci-ign-gazebo3-focal-amd64/gz-sim/test/integration/gpu_lidar.cc:93: Failure
> 167: Expected: (laserMsgs.size()) > (0u), actual: 0 vs 0
> 167/178 Test #167: INTEGRATION_gpu_lidar ................................***Exception: SegFault  5.42 sec

This test has passed and failed on the same agent multiple times. I can't say the reason right now. I'll build it multiple times to see if anything changes and report again

Ros2 windows nightlies | Build regression | Keepin track 👀

Windows builds showed Windows fala exceptions last night:

Windows debug: sros2 test output:
218865 --- stderr: sros2^M
218866 Windows fatal exception: access violation^M
218867 ^M
218868 Current thread 0x00002c94 (most recent call first):^M
218869   File "C:\ci\ws\src\ros2\sros2\sros2\sros2\verb\generate_artifacts.py", line 44 in main^M
218870   File "C:\ci\ws\src\ros2\sros2\sros2\sros2\command\security.py", line 36 in main^M
218871   File "C:\ci\ws\install\Lib\site-packages\ros2cli\cli.py", line 91 in main^M
218872   File "C:\ci\ws\src\ros2\sros2\sros2\test\sros2\commands\security\verbs\test_generate_artifacts.py", line 111 in test_cli_policies_args^M
218873   File "C:\Python38\lib\site-packages\_pytest\python.py", line 183 in pytest_pyfunc_call^M
218874   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
218875   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
218876   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
218877   File "C:\Python38\lib\site-packages\_pytest\python.py", line 1641 in runtest^M
218878   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 162 in pytest_runtest_call^M
218879   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
218880   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
218881   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
218882   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 255 in <lambda>^M
218883   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 311 in from_call^M
218884   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 254 in call_runtest_hook^M
218885   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 215 in call_and_report^M
218886   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 126 in runtestprotocol^M
218887   File "C:\Python38\lib\site-packages\pytest_rerunfailures.py", line 608 in pytest_runtest_protocol^M
218888   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
218889   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
218890   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
218891   File "C:\Python38\lib\site-packages\_pytest\main.py", line 348 in pytest_runtestloop^M
218892   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
218893   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
218894   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
218895   File "C:\Python38\lib\site-packages\_pytest\main.py", line 323 in _main^M
218896   File "C:\Python38\lib\site-packages\_pytest\main.py", line 269 in wrap_session^M
218897   File "C:\Python38\lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main^M
218898   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
218899   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
218900   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
218901   File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 162 in main^M
218902   File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 185 in console_main^M
218903   File "C:\Python38\lib\site-packages\pytest\__main__.py", line 5 in <module>^M
218904   File "C:\Python38\lib\runpy.py", line 87 in _run_code^M
218905   File "C:\Python38\lib\runpy.py", line 194 in _run_module_as_main^M
218906 ---^M
218907 Failed   <<< sros2 [3min 8s, exited with code 3221225477]^M
Windows release ros2component output:
238589 --- stderr: ros2component^M
238590 Windows fatal exception: access violation^M
238592 Thread 0x00000be8 (most recent call first):^M
238593   File "C:\Python38\lib\threading.py", line 306 in wait^M
238594   File "C:\Python38\lib\threading.py", line 558 in wait^M
238595   File "C:\Python38\lib\threading.py", line 1252 in run^M
238596   File "C:\Python38\lib\threading.py", line 932 in _bootstrap_inner^M
238597   File "C:\Python38\lib\threading.py", line 890 in _bootstrap^M
238598 ^M
238599 Current thread 0x00000fdc (most recent call first):^M
238600   File "C:\ci\ws\install\Lib\site-packages\rclpy\node.py", line 189 in __init__^M
238601   File "C:\ci\ws\install\Lib\site-packages\rclpy\__init__.py", line 180 in create_node^M
238602   File "C:\ci\ws\install\Lib\site-packages\ros2cli\node\direct.py", line 48 in __init__^M
238603   File "C:\ci\ws\install\Lib\site-packages\ros2cli\node\strategy.py", line 36 in __init__^M
238604   File "C:\ci\ws\src\ros2\ros2cli\ros2component\test\test_api.py", line 26 in test_find_container_node_names^M
238605   File "C:\Python38\lib\site-packages\_pytest\python.py", line 183 in pytest_pyfunc_call^M
238606   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
238607   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
238608   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
238609   File "C:\Python38\lib\site-packages\_pytest\python.py", line 1641 in runtest^M
238610   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 162 in pytest_runtest_call^M
238611   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
238612   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
238613   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
238614   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 255 in <lambda>^M
238615   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 311 in from_call^M
238616   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 254 in call_runtest_hook^M
238617   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 215 in call_and_report^M
238618   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 126 in runtestprotocol^M
238619   File "C:\Python38\lib\site-packages\pytest_rerunfailures.py", line 608 in pytest_runtest_protocol^M
238620   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
238621   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
238622   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
238623   File "C:\Python38\lib\site-packages\_pytest\main.py", line 348 in pytest_runtestloop^M
238624   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
238625   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
238626   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
238627   File "C:\Python38\lib\site-packages\_pytest\main.py", line 323 in _main^M
238628   File "C:\Python38\lib\site-packages\_pytest\main.py", line 269 in wrap_session^M
238629   File "C:\Python38\lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main^M
238630   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
238631   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
238632   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
238633   File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 162 in main^M
238634   File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 185 in console_main^M
238635   File "C:\Python38\lib\site-packages\pytest\__main__.py", line 5 in <module>^M
238636   File "C:\Python38\lib\runpy.py", line 87 in _run_code^M
238637   File "C:\Python38\lib\runpy.py", line 194 in _run_module_as_main^M
238638 ---^M
238639 Failed   <<< ros2component [23.5s, exited with code 3221225477]^M
Windows repeated test_launch_ros output:
432026 --- stderr: test_launch_ros^M
432027 Windows fatal exception: access violation^M
432028 ^M
432029 Thread 0x0000206c (most recent call first):^M
432030   File "C:\Python38\lib\selectors.py", line 314 in _select^M
432031   File "C:\Python38\lib\selectors.py", line 323 in select^M
432032   File "C:\Python38\lib\asyncio\base_events.py", line 1823 in _run_once^M
432033   File "C:\Python38\lib\asyncio\base_events.py", line 570 in run_forever^M
432034   File "C:\ci\ws\install\Lib\site-packages\launch\utilities\signal_management.py", line 138 in run_background_loop^M
432035   File "C:\Python38\lib\threading.py", line 870 in run^M
432036   File "C:\Python38\lib\threading.py", line 932 in _bootstrap_inner^M
432037   File "C:\Python38\lib\threading.py", line 890 in _bootstrap^M
432038 ^M
432039 Thread 0x000002fc (most recent call first):^M
432040   File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 685 in _wait_for_ready_callbacks^M
432041   File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 781 in wait_for_ready_callbacks^M
432042   File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 811 in _spin_once_impl^M
432043   File "C:\ci\ws\install\Lib\site-packages\rclpy\executors.py", line 827 in spin_once^M
432044   File "C:\ci\ws\install\Lib\site-packages\rclpy\__init__.py", line 240 in spin^M
432045   File "C:\Python38\lib\threading.py", line 870 in run^M
432046   File "C:\Python38\lib\threading.py", line 932 in _bootstrap_inner^M
432047   File "C:\Python38\lib\threading.py", line 890 in _bootstrap^M
432048 ^M
432049 Thread 0x00000ab8 (most recent call first):^M
432050   File "C:\Python38\lib\concurrent\futures\thread.py", line 78 in _worker^M
432051   File "C:\Python38\lib\threading.py", line 870 in run^M
432052   File "C:\Python38\lib\threading.py", line 932 in _bootstrap_inner^M
432053   File "C:\Python38\lib\threading.py", line 890 in _bootstrap^M
432054 ^M
432055 Thread 0x000019a4 (most recent call first):^M
432056   File "C:\Python38\lib\concurrent\futures\thread.py", line 78 in _worker^M
432057   File "C:\Python38\lib\threading.py", line 870 in run^M
432058   File "C:\Python38\lib\threading.py", line 932 in _bootstrap_inner^M
432059   File "C:\Python38\lib\threading.py", line 890 in _bootstrap^M
432060 ^M
432061 Current thread 0x00001e84 (most recent call first):^M
432062   File "C:\ci\ws\install\Lib\site-packages\rclpy\node.py", line 189 in __init__^M
432063   File "C:\ci\ws\install\Lib\site-packages\rclpy\__init__.py", line 180 in create_node^M
432064   File "C:\ci\ws\install\Lib\site-packages\launch_ros\ros_adapters.py", line 60 in start^M
432065   File "C:\ci\ws\install\Lib\site-packages\launch_ros\ros_adapters.py", line 52 in __init__^M
432066   File "C:\ci\ws\install\Lib\site-packages\launch_ros\ros_adapters.py", line 119 in get_ros_adapter^M
432067   File "C:\ci\ws\install\Lib\site-packages\launch_ros\ros_adapters.py", line 136 in get_ros_node^M
432068   File "C:\ci\ws\install\Lib\site-packages\launch_ros\actions\load_composable_nodes.py", line 224 in execute^M
432069   File "C:\ci\ws\install\Lib\site-packages\launch\action.py", line 109 in visit^M
432070   File "C:\ci\ws\install\Lib\site-packages\launch\utilities\visit_all_entities_and_collect_futures_impl.py", line 38 in visit_all_entities_and_collect_futures^M
432071   File "C:\ci\ws\install\Lib\site-packages\launch\utilities\visit_all_entities_and_collect_futures_impl.py", line 45 in visit_all_entities_and_collect_futures^M
432072   File "C:\ci\ws\install\Lib\site-packages\launch\utilities\visit_all_entities_and_collect_futures_impl.py", line 45 in visit_all_entities_and_collect_futures^M
432073   File "C:\ci\ws\install\Lib\site-packages\launch\utilities\visit_all_entities_and_collect_futures_impl.py", line 45 in visit_all_entities_and_collect_futures^M
432074   File "C:\ci\ws\install\Lib\site-packages\launch\launch_service.py", line 250 in __process_event^M
432075   File "C:\ci\ws\install\Lib\site-packages\launch\launch_service.py", line 230 in _process_one_event^M
432076   File "C:\Python38\lib\asyncio\events.py", line 81 in _run^M
432077   File "C:\Python38\lib\asyncio\base_events.py", line 1851 in _run_once^M
432078   File "C:\Python38\lib\asyncio\base_events.py", line 570 in run_forever^M
432079   File "C:\Python38\lib\asyncio\windows_events.py", line 316 in run_forever^M
432080   File "C:\Python38\lib\asyncio\base_events.py", line 603 in run_until_complete^M
432081   File "C:\ci\ws\install\Lib\site-packages\launch\launch_service.py", line 383 in run^M
432082   File "C:\ci\ws\src\ros2\launch_ros\test_launch_ros\test\test_launch_ros\actions\test_load_composable_nodes.py", line 93 in _assert_launch_no_errors^M
432083   File "C:\ci\ws\src\ros2\launch_ros\test_launch_ros\test\test_launch_ros\actions\test_load_composable_nodes.py", line 541 in test_load_node_with_global_remaps_in_group^M
432084   File "C:\Python38\lib\site-packages\_pytest\python.py", line 183 in pytest_pyfunc_call^M
432085   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
432086   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
432087   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
432088   File "C:\Python38\lib\site-packages\_pytest\python.py", line 1641 in runtest^M
432089   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 162 in pytest_runtest_call^M
432090   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
432091   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
432092   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
432093   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 255 in <lambda>^M
432094   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 311 in from_call^M
432095   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 254 in call_runtest_hook^M
432096   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 215 in call_and_report^M
432097   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 126 in runtestprotocol^M
432098   File "C:\Python38\lib\site-packages\_pytest\runner.py", line 109 in pytest_runtest_protocol^M
432099   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
432100   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
432101   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
432102   File "C:\Python38\lib\site-packages\_pytest\main.py", line 348 in pytest_runtestloop^M
432103   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
432104   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
432105   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
432106   File "C:\Python38\lib\site-packages\_pytest\main.py", line 323 in _main^M
432107   File "C:\Python38\lib\site-packages\_pytest\main.py", line 269 in wrap_session^M
432108   File "C:\Python38\lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main^M
432109   File "C:\Python38\lib\site-packages\pluggy\_callers.py", line 103 in _multicall^M
432110   File "C:\Python38\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec^M
432111   File "C:\Python38\lib\site-packages\pluggy\_hooks.py", line 513 in __call__^M
432112   File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 162 in main^M
432113   File "C:\Python38\lib\site-packages\_pytest\config\__init__.py", line 185 in console_main^M
432114   File "C:\Python38\lib\site-packages\pytest\__main__.py", line 5 in <module>^M
432115   File "C:\Python38\lib\runpy.py", line 87 in _run_code^M
432116   File "C:\Python38\lib\runpy.py", line 194 in _run_module_as_main^M
432117 ---^M
432118 Failed   <<< test_launch_ros [2min 55s, exited with code 3221225477]^M

It's odd that it happened in 3 different machines in 3 different packages in the same night. I'll keep track this problem

May 17

Build regressions
nightly_linux_coverage|2388|2024-05-17 05:32:46.090000|Number of failed test exceeds the threshold|coverage analysis
nightly_linux-rhel_debug|1916|2024-05-17 04:00:01.079000|Jenkins channel closing down (Agent disconnected)|test
nightly_win_deb|3096|2024-05-17 04:00:01.572000|Number of failed test exceeds the threshold|test
gz_sensors-8-win|46|2024-05-16 12:06:34.299000|Packages gz-rendering8  Failed|colcon compilation without test for dependencies of gz-sensors8
gz_math-ign-math6-win|43|2024-05-16 22:06:42.284000|Jenkins channel closing down (Agent disconnected)|vcpkg: install all dependencies
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__benchmark_ubuntu_noble_amd64
- Failing 100.0% in 12 out of 12 builds
	projectroot.benchmark_string_conversions
	  (also in: "Jci__benchmark_ubuntu_noble_amd64" job)

Issue #2 in nightly_linux-aarch64_repeated
- Failing 68.75% in 11 out of 16 builds
	rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice
	projectroot.test_sequential_compression_writer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #3 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 6 out of 6 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 6 out of 6 builds
	projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)

Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 6 out of 6 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 6 out of 6 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #7 in nightly_win_rep
- Failing 50.0% in 6 out of 12 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #8 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 83.33% in 5 out of 6 builds
	rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #9 in nightly_linux-rhel_repeated
- Failing 31.25% in 5 out of 16 builds
	projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp

Issue #10 in nightly_win_rep
- Failing 41.67% in 5 out of 12 builds
	rclcpp.test_executor.gtest.missing_result
	  (also in: "nightly_win_rel" job)

Issue #11 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #12 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #13 in nightly_linux_repeated
- Failing 26.67% in 4 out of 15 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp

Issue #14 in nightly_linux_repeated
- Failing 26.67% in 4 out of 15 builds
	ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #15 in nightly_win_rep
- Failing 33.33% in 4 out of 12 builds
	rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock
	projectroot.test.rclcpp.test_clock
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #16 in nightly_linux-rhel_repeated
- Failing 25.0% in 4 out of 16 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #17 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	test_rclcpp.TestNNodes.test_10_nodes
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #18 in nightly_linux_repeated
- Failing 20.0% in 3 out of 15 builds
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp

Issue #19 in nightly_win_rel
- Failing 21.43% in 3 out of 14 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)

Issue #20 in nightly_win_rep
- Failing 25.0% in 3 out of 12 builds
	test_rclcpp.test_timeout_subscriber.timeout_subscriber

Issue #21 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #22 in nightly_linux_iron_coverage
- Failing 13.33% in 2 out of 15 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Ici__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Hci__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #23 in nightly_win_rep
- Failing 16.67% in 2 out of 12 builds
	rclcpp_action.TestServer.publish_status_succeeded
	rclcpp_action.TestServer.publish_status_canceling
	rclcpp_action.TestServer.publish_status_canceled
	rclcpp_action.TestServer.publish_status_accepted
	rclcpp_action.TestServer.publish_status_aborted
	rclcpp_action.TestServer.publish_feedback
	rclcpp_action.TestServer.handle_goal_called
	rclcpp_action.TestServer.handle_cancel_unknown_goal
	rclcpp_action.TestServer.handle_cancel_terminated_goal
	rclcpp_action.TestServer.handle_cancel_reject
	rclcpp_action.TestServer.handle_cancel_called
	rclcpp_action.TestServer.handle_accepted_called
	rclcpp_action.TestServer.get_result_deferred
	rclcpp_action.TestServer.get_result
	rclcpp_action.TestServer.deferred_execution
	rclcpp_action.TestServer.construction_and_destruction_wait_set_error
	rclcpp_action.TestServer.construction_and_destruction_sub_node
	rclcpp_action.TestServer.construction_and_destruction_server_init_error
	rclcpp_action.TestServer.construction_and_destruction_callback_group
	projectroot.test_server

Issue #24 in nightly_win_rep
- Failing 16.67% in 2 out of 12 builds
	projectroot.test_client
	  (also in: "nightly_win_deb" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-connext_ubuntu_noble_amd64" job)

Issue #25 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #26 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy.test_generate_policy
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)

Issue #27 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	ros2param.ros2param.test.test_verb_dump.test_verb_dump (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #28 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #29 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #30 in nightly_win_rep
- Failing 8.33% in 1 out of 12 builds
	rclcpp_action.TestServer.construction_and_destruction_after_node
	rclcpp_action.TestServer.construction_and_destruction
	rclcpp_action.TestClientAgainstServer.send_rcl_errors
	rclcpp_action.TestClientAgainstServer.execute_rcl_errors
	rclcpp_action.TestClientAgainstServer.deadlock_in_callbacks
	rclcpp_action.TestClientAgainstServer.async_cancel_some_goals_with_callback
	rclcpp_action.TestClientAgainstServer.async_cancel_some_goals
	projectroot.test_showimage_cam2image__rmw_fastrtps_cpp
	projectroot.gtest_timeout_subscriber__rmw_cyclonedds_cpp
	image_tools.TestExecutablesDemo.test_reliable_qos

May 18

Build regressions

Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__benchmark_ubuntu_noble_amd64
- Failing 100.0% in 12 out of 12 builds
	projectroot.benchmark_string_conversions
	  (also in: "Jci__benchmark_ubuntu_noble_amd64" job)

Issue #2 in nightly_linux-aarch64_repeated
- Failing 73.33% in 11 out of 15 builds
	rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice
	projectroot.test_sequential_compression_writer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #3 in nightly_win_rep
- Failing 54.55% in 6 out of 11 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 5 out of 5 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 5 out of 5 builds
	projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)

Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 5 out of 5 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #7 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 5 out of 5 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #8 in nightly_linux-rhel_repeated
- Failing 33.33% in 5 out of 15 builds
	projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp

Issue #9 in nightly_win_rep
- Failing 45.45% in 5 out of 11 builds
	rclcpp.test_executor.gtest.missing_result
	  (also in: "nightly_win_rel" job)

Issue #10 in gz_rendering-main-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #11 in nightly_linux_repeated
- Failing 28.57% in 4 out of 14 builds
	ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #12 in nightly_win_rep
- Failing 36.36% in 4 out of 11 builds
	rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock
	projectroot.test.rclcpp.test_clock
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #13 in nightly_linux-rhel_repeated
- Failing 26.67% in 4 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #14 in gz_physics-ci-ign-physics2-homebrew-amd64
- Failing 100.0% in 3 out of 3 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job)
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #15 in nightly_linux-rhel_repeated
- Failing 20.0% in 3 out of 15 builds
	test_rclcpp.TestNNodes.test_10_nodes
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #16 in nightly_linux_repeated
- Failing 21.43% in 3 out of 14 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp

Issue #17 in nightly_win_rel
- Failing 23.08% in 3 out of 13 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)

Issue #18 in nightly_win_rep
- Failing 27.27% in 3 out of 11 builds
	test_rclcpp.test_timeout_subscriber.timeout_subscriber

Issue #19 in nightly_linux-rhel_repeated
- Failing 20.0% in 3 out of 15 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #20 in nightly_linux_iron_coverage
- Failing 14.29% in 2 out of 14 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Ici__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Hci__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #21 in nightly_linux_repeated
- Failing 14.29% in 2 out of 14 builds
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp

Issue #22 in nightly_win_rep
- Failing 18.18% in 2 out of 11 builds
	rclcpp_action.TestServer.publish_status_succeeded
	rclcpp_action.TestServer.publish_status_canceling
	rclcpp_action.TestServer.publish_status_canceled
	rclcpp_action.TestServer.publish_status_accepted
	rclcpp_action.TestServer.publish_status_aborted
	rclcpp_action.TestServer.publish_feedback
	rclcpp_action.TestServer.handle_goal_called
	rclcpp_action.TestServer.handle_cancel_unknown_goal
	rclcpp_action.TestServer.handle_cancel_terminated_goal
	rclcpp_action.TestServer.handle_cancel_reject
	rclcpp_action.TestServer.handle_cancel_called
	rclcpp_action.TestServer.handle_accepted_called
	rclcpp_action.TestServer.get_result_deferred
	rclcpp_action.TestServer.get_result
	rclcpp_action.TestServer.deferred_execution
	rclcpp_action.TestServer.construction_and_destruction_wait_set_error
	rclcpp_action.TestServer.construction_and_destruction_sub_node
	rclcpp_action.TestServer.construction_and_destruction_server_init_error
	rclcpp_action.TestServer.construction_and_destruction_callback_group
	projectroot.test_server

Issue #23 in nightly_win_rep
- Failing 18.18% in 2 out of 11 builds
	projectroot.test_client
	  (also in: "nightly_win_deb" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-connext_ubuntu_noble_amd64" job)

Issue #24 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #25 in nightly_linux-rhel_repeated
- Failing 6.67% in 1 out of 15 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy.test_generate_policy
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)

Issue #26 in nightly_linux-rhel_repeated
- Failing 6.67% in 1 out of 15 builds
	ros2param.ros2param.test.test_verb_dump.test_verb_dump (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #27 in nightly_linux-rhel_repeated
- Failing 6.67% in 1 out of 15 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #28 in nightly_linux-rhel_repeated
- Failing 6.67% in 1 out of 15 builds
	projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #29 in nightly_win_rep
- Failing 9.09% in 1 out of 11 builds
	rclcpp_action.TestServer.construction_and_destruction_after_node
	rclcpp_action.TestServer.construction_and_destruction
	rclcpp_action.TestClientAgainstServer.send_rcl_errors
	rclcpp_action.TestClientAgainstServer.execute_rcl_errors
	rclcpp_action.TestClientAgainstServer.deadlock_in_callbacks
	rclcpp_action.TestClientAgainstServer.async_cancel_some_goals_with_callback
	rclcpp_action.TestClientAgainstServer.async_cancel_some_goals
	projectroot.test_showimage_cam2image__rmw_fastrtps_cpp
	projectroot.gtest_timeout_subscriber__rmw_cyclonedds_cpp
	image_tools.TestExecutablesDemo.test_reliable_qos

May 19

Build regressions

Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_linux-aarch64_repeated
- Failing 78.57% in 11 out of 14 builds
	rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice
	projectroot.test_sequential_compression_writer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #2 in Rci__benchmark_ubuntu_noble_amd64
- Failing 100.0% in 10 out of 10 builds
	projectroot.benchmark_string_conversions
	  (also in: "Jci__benchmark_ubuntu_noble_amd64" job)

Issue #3 in nightly_win_rep
- Failing 54.55% in 6 out of 11 builds
	projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 5 out of 5 builds
	projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)

Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 5 out of 5 builds
	projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)

Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 5 out of 5 builds
	projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)

Issue #7 in Jci__nightly-cyclonedds_ubuntu_noble_amd64
- Failing 100.0% in 5 out of 5 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN)
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN)
	rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some
	rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all
	  (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN)
	  (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job)

Issue #8 in nightly_linux-rhel_repeated
- Failing 35.71% in 5 out of 14 builds
	projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp

Issue #9 in nightly_win_rep
- Failing 45.45% in 5 out of 11 builds
	rclcpp.test_executor.gtest.missing_result
	  (also in: "nightly_win_rel" job)

Issue #10 in gz_physics-ci-ign-physics5-homebrew-amd64
- Failing 100.0% in 4 out of 4 builds
	JointFeaturesFixture.JointDetach
	  (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job)

Issue #11 in gz_rendering-main-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #12 in nightly_linux_repeated
- Failing 30.77% in 4 out of 13 builds
	ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #13 in nightly_win_rep
- Failing 36.36% in 4 out of 11 builds
	rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock
	projectroot.test.rclcpp.test_clock
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #14 in nightly_linux-rhel_repeated
- Failing 21.43% in 3 out of 14 builds
	test_rclcpp.TestNNodes.test_10_nodes
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #15 in nightly_linux_repeated
- Failing 23.08% in 3 out of 13 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp

Issue #16 in nightly_win_rel
- Failing 25.0% in 3 out of 12 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)

Issue #17 in nightly_win_rep
- Failing 27.27% in 3 out of 11 builds
	test_rclcpp.test_timeout_subscriber.timeout_subscriber

Issue #18 in nightly_linux-rhel_repeated
- Failing 21.43% in 3 out of 14 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #19 in nightly_linux-rhel_repeated
- Failing 21.43% in 3 out of 14 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #20 in nightly_linux_repeated
- Failing 15.38% in 2 out of 13 builds
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp

Issue #21 in nightly_win_rep
- Failing 18.18% in 2 out of 11 builds
	rclcpp_action.TestServer.publish_status_succeeded
	rclcpp_action.TestServer.publish_status_canceling
	rclcpp_action.TestServer.publish_status_canceled
	rclcpp_action.TestServer.publish_status_accepted
	rclcpp_action.TestServer.publish_status_aborted
	rclcpp_action.TestServer.publish_feedback
	rclcpp_action.TestServer.handle_goal_called
	rclcpp_action.TestServer.handle_cancel_unknown_goal
	rclcpp_action.TestServer.handle_cancel_terminated_goal
	rclcpp_action.TestServer.handle_cancel_reject
	rclcpp_action.TestServer.handle_cancel_called
	rclcpp_action.TestServer.handle_accepted_called
	rclcpp_action.TestServer.get_result_deferred
	rclcpp_action.TestServer.get_result
	rclcpp_action.TestServer.deferred_execution
	rclcpp_action.TestServer.construction_and_destruction_wait_set_error
	rclcpp_action.TestServer.construction_and_destruction_sub_node
	rclcpp_action.TestServer.construction_and_destruction_server_init_error
	rclcpp_action.TestServer.construction_and_destruction_callback_group
	projectroot.test_server

Issue #22 in nightly_win_rep
- Failing 18.18% in 2 out of 11 builds
	projectroot.test_client
	  (also in: "nightly_win_deb" job)
	  (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job)
	  (also in: "Jci__nightly-connext_ubuntu_noble_amd64" job)

Issue #23 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #24 in nightly_linux-rhel_repeated
- Failing 7.14% in 1 out of 14 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy.test_generate_policy
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)

Issue #25 in nightly_linux-rhel_repeated
- Failing 7.14% in 1 out of 14 builds
	ros2param.ros2param.test.test_verb_dump.test_verb_dump (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #26 in nightly_linux_iron_coverage
- Failing 7.69% in 1 out of 13 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Ici__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Hci__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #27 in nightly_linux-rhel_repeated
- Failing 7.14% in 1 out of 14 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #28 in nightly_linux-rhel_repeated
- Failing 7.14% in 1 out of 14 builds
	projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #29 in nightly_win_rep
- Failing 9.09% in 1 out of 11 builds
	rclcpp_action.TestServer.construction_and_destruction_after_node
	rclcpp_action.TestServer.construction_and_destruction
	rclcpp_action.TestClientAgainstServer.send_rcl_errors
	rclcpp_action.TestClientAgainstServer.execute_rcl_errors
	rclcpp_action.TestClientAgainstServer.deadlock_in_callbacks
	rclcpp_action.TestClientAgainstServer.async_cancel_some_goals_with_callback
	rclcpp_action.TestClientAgainstServer.async_cancel_some_goals
	projectroot.test_showimage_cam2image__rmw_fastrtps_cpp
	projectroot.gtest_timeout_subscriber__rmw_cyclonedds_cpp
	image_tools.TestExecutablesDemo.test_reliable_qos