osrf/buildfarm-tools

Buildfarmer log 2024/01/22 - 2024/01/29

Closed this issue · 9 comments

Previous log #23

Jan 22

Build regressions
nightly_win_deb|2980|2024-01-22 04:00:01.449000|Number of failed test exceeds the threshold|test
gz-msgs11-debbuilder|99|2024-01-22 10:44:38.438000|Error in configuration dependencies|create deb packages
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 28.57% in 4 out of 14 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in nightly_win_rep
- Failing 21.43% in 3 out of 14 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp

Issue #3 in nightly_linux_repeated
- Failing 13.33% in 2 out of 15 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #4 in nightly_linux_repeated
- Failing 13.33% in 2 out of 15 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp

Issue #5 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 33.33% in 1 out of 3 builds
	INTEGRATION_minimal_scene.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_marker_manager.test_ran

Issue #6 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Server_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2282 - OPEN)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2282 - OPEN)

Issue #7 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 20.0% in 1 out of 5 builds
	ServerRepeat/OdometryPublisherTest.GaussianNoiseTest/0

Issue #8 in gz_sim-ci-main-jammy-amd64
- Failing 20.0% in 1 out of 5 builds
	UNIT_Gui_TEST.test_ran

Issue #9 in gz_sim-main-win
- Failing 20.0% in 1 out of 5 builds
	INTEGRATION_mesh_inertia_calculation.test_ran

Jan 23

Build regressions
nightly_linux-aarch64_debug|2651|2024-01-23 04:00:00.919000|Jenkins channel closing down (Agent disconnected)|build
packaging_linux-aarch64|2667|2024-01-23 04:00:01.434000|Jenkins channel closing down (Agent disconnected)|build underlay packages
nightly_win_deb|2981|2024-01-23 04:00:01.419000|Number of failed test exceeds the threshold|test
gz-msgs11-debbuilder|100|2024-01-23 09:48:38.445000|Error in configuration dependencies|create deb packages
gz_rendering-ci-main-jammy-amd64|24|2024-01-22 15:02:16.922000|Unknown Reason|compiling
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 31.25% in 5 out of 16 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test_n_nodes__rmw_fastrtps_cpp
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in nightly_linux_repeated
- Failing 20.0% in 3 out of 15 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #3 in nightly_win_rep
- Failing 12.5% in 2 out of 16 builds
	test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully

Issue #4 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 20.0% in 1 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #5 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 33.33% in 1 out of 3 builds
	INTEGRATION_minimal_scene.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_marker_manager.test_ran

Issue #6 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_gpu_lidar.test_ran

Issue #7 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 20.0% in 1 out of 5 builds
	ServerRepeat/OdometryPublisherTest.GaussianNoiseTest/0

Issue #8 in nightly_linux_release
- Failing 6.25% in 1 out of 16 builds
	projectroot.xmllint
	  (also in: "nightly_linux_debug" job)

Issue #9 in nightly_linux_release
- Failing 6.25% in 1 out of 16 builds
	rosidl_adapter.xmllint.xunit.missing_result
	resource_retriever.resource_retriever_test.gtest.missing_result
	projectroot.resource_retriever_test

Issue #10 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	projectroot.test_tutorial_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp
	  (also in: "nightly_win_rep" job)

Issue #11 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	projectroot.test_tutorial_talker_listener__rmw_fastrtps_cpp
	projectroot.test_tutorial_content_filtering_publisher_content_filtering_subscriber__rmw_fastrtps_cpp
	projectroot.test_tutorial_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp
	  (also in: "nightly_win_rep" job)

Issue #12 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorialAfterShutdown.test_last_process_exit_code
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)

Issue #13 in nightly_win_rep
- Failing 6.25% in 1 out of 16 builds
	test_launch_ros.test.test_launch_ros.utilities.test_track_node_names.test_launch_nodes_with_different_names[3-3]
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__secure_comm_3

Rendering main | Build regression | Investigating 🕵️

Reference build: https://build.osrfoundation.org/job/gz_rendering-ci-main-jammy-amd64/24/

Log output:

[ 93%] �[32m�[1mLinking CXX shared library ../../lib/libgz-rendering9-ogre2.so�[0m
/usr/bin/ld: ../../lib/libterra.a(OgreHlmsTerra.cpp.o): warning: relocation against `_ZN4Ogre11PbsProperty14LegacyMathBrdfE' in read-only section `.text'
/usr/bin/ld: ../../lib/libterra.a(OgreHlmsTerra.cpp.o): relocation R_X86_64_PC32 against symbol `_ZTVN4Ogre9HlmsTerraE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[2]: *** [ogre2/src/CMakeFiles/gz-rendering9-ogre2.dir/build.make:987: lib/libgz-rendering9-ogre2.so.9.0.0] Error 1
make[1]: *** [CMakeFiles/Makefile2:3493: ogre2/src/CMakeFiles/gz-rendering9-ogre2.dir/all] Error 2

Jan 24

Build regressions
nightly_linux_humble_coverage|685|2024-01-24 05:42:20.594000|Jenkins channel closing down (Agent disconnected)|build
nightly_linux_release|2958|2024-01-24 06:39:33.517000|Jenkins channel closing down (Agent disconnected)|build
nightly_linux-aarch64_debug|2653|2024-01-24 04:00:01.028000|Jenkins channel closing down (Agent disconnected)|build
gz-msgs11-debbuilder|101|2024-01-24 09:49:44.360000|Error in configuration dependencies|create deb packages
gz_physics-ci-main-jammy-amd64|24|2024-01-24 09:06:28.358000|Early fail|No last section
gz_rendering-main-win|16|2024-01-23 22:45:08.369000|Jenkins channel closing down (Agent disconnected)|colcon compilation without test for dependencies of gz-rendering9
gz_sim-7-win|23|2024-01-23 16:06:13.333000|Jenkins channel closing down (Agent disconnected)|colcon compilation with tests for gz-sim7
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 31.25% in 5 out of 16 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in nightly_win_rep
- Failing 18.75% in 3 out of 16 builds
	rclcpp.TestTimersManager.timers_thread
	projectroot.test.rclcpp.test_timers_manager

Issue #3 in nightly_win_rep
- Failing 12.5% in 2 out of 16 builds
	test_communication.TestPublisherSubscriberAfterShutdown.test_processes_finished_gracefully[MultiNested] (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)

Issue #4 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 20.0% in 1 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #5 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 33.33% in 1 out of 3 builds
	INTEGRATION_minimal_scene.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_marker_manager.test_ran

Issue #6 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_gpu_lidar.test_ran

Issue #7 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 20.0% in 1 out of 5 builds
	ServerRepeat/OdometryPublisherTest.GaussianNoiseTest/0

Issue #8 in nightly_win_rep
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_publisher_subscriber__rclcpp__rmw_fastrtps_cpp
	  (also in: "nightly_linux_repeated" job)

Issue #9 in nightly_win_rep
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_showimage_cam2image__rmw_fastrtps_cpp
	image_tools.TestExecutablesDemo.test_reliable_qos

Rrel_sync-packages-to-testing_jammy_amd64 | Build regression | Keeping track 👀

Reference build: https://build.ros2.org/job/Rrel_sync-packages-to-testing_jammy_amd64/602/console

Desktop ros2 package is missing from building repository.

And Rrel_sync-packages-to-testing_jammy_amd64/602/ failed because of:

The following binary packages are missing to sync: 
    - desktop

rmw_cyclonedds_cpp and rmw_fastrtps_shared_cpp started to fail because of a problem in tracetools.

This change was made and released. However, cyclonedds nor fastrtps_shared were released to match tracetools changes.

Jan 25

Build regressions
nightly_linux_humble_coverage|687|2024-01-25 05:46:29.288000|Jenkins channel closing down (Agent disconnected)|test
nightly_linux_release|2960|2024-01-25 07:39:54.949000|Jenkins channel closing down (Agent disconnected)|build
nightly_win_deb|2983|2024-01-25 04:00:01.412000|Number of failed test exceeds the threshold|test
Ici__nightly-cyclonedds_ubuntu_jammy_amd64|89|2024-01-25 07:15:02.850000|Jenkins channel closing down (Agent disconnected)|run tests
gz_gui-ci-main-jammy-amd64|34|2024-01-25 07:06:23.763000|Unknown Reason|NVIDIA MISMATCH RECOVERY
gz_physics-ci-main-jammy-amd64|25|2024-01-24 13:09:44.628000|Early fail|No last section
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 31.25% in 5 out of 16 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 20.0% in 1 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #3 in gz_sensors-ci-main-homebrew-amd64
- Failing 20.0% in 1 out of 5 builds
	DopplerVelocityLogTests/DopplerVelocityLogTest.BottomTrackingWhileStatic/ogre

Issue #4 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_gpu_lidar.test_ran

Issue #5 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
	  (also in: "nightly_win_rep" job)

Issue #6 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	test_security.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_1.xunit.missing_result
	test_security.TestSecurePublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_not_connecting_1
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_not_connecting_0
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_3
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_2
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_1
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_0
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_not_connecting_1
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_not_connecting_0
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_3
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_2
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_1
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_0

Issue #7 in nightly_win_rep
- Failing 6.25% in 1 out of 16 builds
	test_launch_ros.test.test_launch_ros.utilities.test_track_node_names.test_launch_nodes_with_different_names[1-3]
	test_communication.TestActionClientServerAfterShutdown.test_processes_finished_gracefully[NestedMessage]
	projectroot.test_action_client_server__rclpy__rmw_fastrtps_cpp

Jan 26

Build regressions
nightly_linux_coverage|2260|2024-01-26 05:40:04.064000|Jenkins channel closing down (Agent disconnected)|test
nightly_linux_release|2962|2024-01-26 06:54:35.199000|Jenkins channel closing down (Agent disconnected)|test
packaging_linux|3310|2024-01-25 19:46:32.820000|Jenkins channel closing down (Agent disconnected)|build underlay packages
packaging_linux-rhel|1809|2024-01-25 19:46:57.757000|Packages ament_cmake_core  Failed|build underlay packages
gz-msgs11-debbuilder|103|2024-01-26 09:50:02.787000|Error in configuration dependencies|create deb packages
gz-msgs11-debbuilder|102|2024-01-25 12:53:17.567000|Error in configuration dependencies|create deb packages
gz_sensors-ci-ign-sensors6-focal-amd64|24|2024-01-25 15:46:57.401000|Early fail|No last section
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 31.25% in 5 out of 16 builds
	rosidl_typesupport_introspection_tests.MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage
	projectroot.test_multi_nested_message_introspection

Issue #2 in gz_sensors-ci-main-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	DopplerVelocityLogTests/DopplerVelocityLogTest.BottomTrackingWhileStatic/ogre

Issue #3 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 33.33% in 1 out of 3 builds
	INTEGRATION_gpu_lidar.test_ran

Jan 27

Build regressions
nightly_linux-aarch64_release|2619|2024-01-27 04:00:00.832000|Jenkins channel closing down (Agent disconnected)|build
packaging_linux-rhel|1811|2024-01-26 15:21:31.571000|Packages ament_cmake_core  Failed|build underlay packages
nightly_win_deb|2985|2024-01-27 04:00:00.929000|Number of failed test exceeds the threshold|test
gz-msgs11-debbuilder|104|2024-01-27 09:47:23.377000|Error in configuration dependencies|create deb packages
gz_cmake-main-win|20|2024-01-27 10:26:06.929000|Jenkins channel closing down (Agent disconnected)|colcon compilation with tests for gz-cmake4
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 37.5% in 6 out of 16 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in gz_sensors-ci-main-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	DopplerVelocityLogTests/DopplerVelocityLogTest.BottomTrackingWhileStatic/ogre

Jan 28

Build regressions
nightly_linux_coverage|2263|2024-01-28 04:46:02.517000|Number of failed test exceeds the threshold|ccache stats (after)
nightly_linux_debug|2948|2024-01-28 06:07:16.005000|Number of failed test exceeds the threshold|ccache stats (after)
nightly_linux_humble_coverage|691|2024-01-28 05:38:21.295000|Packages rosbag2_transport  Failed|build
nightly_linux_release|2965|2024-01-28 06:15:40.233000|Number of failed test exceeds the threshold|ccache stats (after)
nightly_linux-aarch64_debug|2658|2024-01-28 04:00:01.171000|Number of failed test exceeds the threshold|ccache stats (after)
nightly_linux-aarch64_release|2621|2024-01-28 04:00:01.175000|Number of failed test exceeds the threshold|ccache stats (after)
nightly_linux-aarch64_repeated|2649|2024-01-28 04:00:01.183000|Number of failed test exceeds the threshold|ccache stats (after)
nightly_linux-rhel_debug|1801|2024-01-28 04:00:01.196000|Packages rviz_default_plugins  Failed|build
nightly_linux-rhel_release|1783|2024-01-28 04:00:01.203000|Number of failed test exceeds the threshold|test
nightly_linux-rhel_repeated|1784|2024-01-28 04:00:01.208000|Number of failed test exceeds the threshold|test
nightly_win_deb|2986|2024-01-28 04:00:01.228000|Number of failed test exceeds the threshold|test
gz-msgs11-debbuilder|105|2024-01-28 09:49:46.670000|Error in configuration dependencies|create deb packages
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 43.75% in 7 out of 16 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in gz_sensors-ci-main-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	DopplerVelocityLogTests/DopplerVelocityLogTest.BottomTrackingWhileStatic/ogre

Issue #3 in nightly_win_rep
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_play_services__rmw_fastrtps_cpp (known issue in: https://github.com/ros2/rosbag2/issues/862 - CLOSED)
	  (also in: "nightly_linux_debug" job) (known issue in: https://github.com/ros2/rosbag2/issues/862 - CLOSED)

Issue #4 in nightly_win_rel
- Failing 6.25% in 1 out of 16 builds
	rosidl_generator_py.test
	projectroot.test_cli_extension
	launch_testing.test.launch_testing.test_io_handler_and_assertions.TestIoHandlerAndAssertions.test_regex_matching
	launch_testing.test.launch_testing.test_io_handler_and_assertions.TestIoHandlerAndAssertions.test_process_names
	launch_testing.test.launch_testing.test_io_handler_and_assertions.TestIoHandlerAndAssertions.test_assertInStdout_notices_too_many_matching_procs
	launch_testing.test.launch_testing.test_io_handler_and_assertions.TestIoHandlerAndAssertions.test_assertInStdout_notices_no_matching_proc
	launch_pytest.test.launch_pytest.test_plugin.test_generator_yields_twice_module_scope
	launch_pytest.test.launch_pytest.test_plugin.test_examples
	launch_pytest.test.launch_pytest.test_plugin.test_asyncgenerator_yields_twice_module_scope
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)

Issue #5 in nightly_win_rep
- Failing 6.25% in 1 out of 16 builds
	launch_testing.test.launch_testing.test_io_handler_and_assertions.TestIoHandlerAndAssertions.test_asserts_on_missing_text
	launch_pytest.test.launch_pytest.test_plugin.test_generator_yields_twice_module_scope[3-3]
	launch_pytest.test.launch_pytest.test_plugin.test_generator_yields_twice_module_scope[2-3]
	launch_pytest.test.launch_pytest.test_plugin.test_generator_yields_twice_module_scope[1-3]
	launch_pytest.test.launch_pytest.test_plugin.test_examples[3-3]
	launch_pytest.test.launch_pytest.test_plugin.test_examples[2-3]
	launch_pytest.test.launch_pytest.test_plugin.test_examples[1-3]
	launch_pytest.test.launch_pytest.test_plugin.test_asyncgenerator_yields_twice_module_scope[3-3]
	launch_pytest.test.launch_pytest.test_plugin.test_asyncgenerator_yields_twice_module_scope[2-3]
	launch_pytest.test.launch_pytest.test_plugin.test_asyncgenerator_yields_twice_module_scope[1-3]