Transition Windows/Mac CI farms to cloud virtualization

Question

Transition Windows/Mac CI farms to cloud virtualization

Closed this issue 4 years ago · 9 comments

Answer 1 · 2020-01-13T20:35:54.000Z

I've run into an issue attempting to build in a directory mounted with docker run -v. When building, cmake runs a compiler check on a simple program in debug mode and with the /Zi build flag. This creates a pdb file with debug symbols. Any references to the debug symbols are requested through mspdbsrv.exe. I suspect this has to do with file handle access across the mounted directory, between the containerized OS and the host OS. Building in a directory that's not in a mounted location, but copying build/install results back to the mounted directory seems to work.

Example of output failure.

cl /c /Zi /W3 /WX- /diagnostics:classic /Od /Ob0 /D WIN32 /D _WINDOWS /D "CMAKE_INTDIR=\"Debug\"" /D _MBCS /Gm- /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Z
c:forScope /Zc:inline /Fo"cmTC_c31aa.dir\Debug\\" /Fd"cmTC_c31aa.dir\Debug\vc141.pdb" /Gd /TC /errorReport:queue C:\TEMP\workdir\ws\build\poco_vendor\CMakeFiles\CMakeTmp
\testCCompiler.c
[19.406s]
[19.406s]       testCCompiler.c
[19.406s]     LINK : fatal error LNK1318: Unexpected PDB error; RPC (23) '(0x000006E7)' [C:\TEMP\workdir\ws\build\poco_vendor\CMakeFiles\CMakeTmp\cmTC_c31aa.vcxproj]

Answer 2 · 2020-01-13T20:40:38.000Z

To run a containerized Windows OS on Windows, the containerized OS has compatibility requirements with the host OS. See https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility?tabs=windows-server-1909%2Cwindows-10-1909

This means that without Hyper-V enabled, the containerized OS must match the host OS because they both use the same kernel. With Hyper-V, it generally means the Release Id of the OS of the container must the same or older than the host OS. I've added logic in my PR for the Release ID to be passed into the docker image when building on a jenkins job.

However, this also means that if someone downloads the image from the cloud instance to run on their own machine, they will have to run a matching Release Id of the cloud instance, or one compatible through Hyper-V.

To find the Release Id on a windows machine, run:

powershell $(Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion').ReleaseId

Answer 3 · 2020-01-13T20:55:01.000Z

Currently, it is not possible to install RTI Connext through the command line on Windows. The installer provides a headless mode and a text-based mode. Unfortunately, the headless mode is not available for the evaluation installer, and the text-base mode is not available on Windows.

https://community.rti.com/static/documentation/connext-dds/5.2.0/doc/manuals/connext_dds/html_files/RTI_ConnextDDS_CoreLibraries_GettingStarted/Content/GettingStarted/Installing_ConnextDDS.htm

Answer 4 · 2020-01-23T13:32:16.000Z

@brawner I assigned you in order to avoid the issue appearing again in our triagging process.
Feel free to unassing yourself if you think that the assignment is wrong.

Answer 5 · 2020-01-24T23:29:28.000Z

Thanks to @cottsay (brawner#1), PR #361 now has Connext functionality. He added a git submodule pointing to a private OSRF repo to download a professional Connext installer. The professional installer can be installed headless in a docker container, which was not possible with the evaluation installer.

Answer 6 · 2020-02-05T01:53:27.000Z

`ci_windows` and `ci_windows-container` build differences

`ci_windows-container` catches build errors with more up-to-date installs

Turtlesim incompatible with Qt 5.12.7. (Addressed by moving to 5.14.1 in #383)

ci_windows 9203 ci_windows-container 88
ci_windows 9157 ci_windows-container 39
ci_windows 9133 ci_windows-container 14

Caught MSBuild warnings in newer versions of Visual Studio. This was already fixed in ros2/rclcpp#963

ci_windows 9172 ci_windows-container 50
ci_windows 9156 ci_windows-container 38

`cppcheck` 1.90 issues: Addressed by:

#387
ros2/rclcpp#1000
ros2/system_tests#400
https://gitlab.com/micro-ROS/ros_tracing/ros2_tracing/-/merge_requests/140
Tracetools may still fail though. (https://gitlab.com/micro-ROS/ros_tracing/ros2_tracing/issues/69)

ci_windows 9400 ci_windows-container 238
ci_windows 9397 ci_windows-container 235
ci_windows 9395 ci_windows-container 234
ci_windows 9394 ci_windows-container 233
ci_windows 9391 ci_windows-container 231
ci_windows 9390 ci_windows-container 230
ci_windows 9366 ci_windows-container 211
ci_windows 9331 ci_windows-container 188
ci_windows 9330 ci_windows-container 187
ci_windows 9316 ci_windows-container 177
ci_windows 9300 ci_windows-container 160
ci_windows 9305 ci_windows-container 169
ci_windows 9269 ci_windows-container 135
ci_windows 9255 ci_windows-container 127

Open issues for `ci_windows-container`

Timing related flakiness

ci_windows 9397 ci_windows-container 235
ci_windows 9394 ci_windows-container 233
ci_windows 9391 ci_windows-container 231
ci_windows 9390 ci_windows-container 230
ci_windows 9346 ci_windows-container 200
ci_windows 9337 ci_windows-container 194
ci_windows 9331 ci_windows-container 188
ci_windows 9328 ci_windows-container 185
ci_windows 9316 ci_windows-container 177
ci_windows 9261 ci_windows-container 130
ci_windows 9255 ci_windows-container 127
ci_windows 9272 ci_windows-container 138
ci_windows 9275 ci_windows-container 142
ci_windows 9277 ci_windows-container 144
ci_windows 9280 ci_windows-container 147
ci_windows 9283 ci_windows-container 150
ci_windows 9282 ci_windows-container 149
ci_windows 9205 ci_windows-container 88 (This one failed for a java related reason)
ci_windows 9184 ci_windows-container 58 (This one failed for a java related reason)
ci_windows 9174 ci_windows-container 51
ci_windows 9176 ci_windows-container 52
ci_windows 9166 ci_windows-container 45
ci_windows 9165 ci_windows-container 44
ci_windows 9150 ci_windows-container 32
ci_windows 9146 ci_windows-container 26
ci_windows 9132 ci_windows-container 13