actions/setup-go

Shared objects built by Github Actions are very slow to use

mpenkov opened this issue · 8 comments

mpenkov commented

Description:

I've identified a problem with building shared libraries from Go source using Github Actions.

The library source contains a single function, called cNoop, which does nothing. I then build this library into a shared object using Github Actions. The workflow is here. Nothing fancy, just a go build -buildmode=c-shared.

The problem is that the resulting shared object is extremely slow to use. Calling the cNoop function 1 million times from e.g. Python takes nearly 30s.

In contrast, if I build the shared object locally on my machine, the same task (1 million calls to cNoop) takes a fraction of a second.

$ RUNPY_LIB_PATH=builds/lib.so.gha2 python run.py  # github actions, build 2
made 1000000 calls in 33.91s
$ RUNPY_LIB_PATH=builds/lib.so.local python run.py # local build
made 1000000 calls in 0.26s

Minimum reproducible example is here.

Action version:

5

Platform:

  • Ubuntu
  • macOS
  • Windows

Runner type:

  • Hosted
  • Self-hosted

Tools version:

1.23.0

Repro steps:

https://github.com/mpenkov/go-shared

Expected behavior:

The shared object should be reasonably fast to use.
Here's an example of a shared object that I built locally:

$ RUNPY_LIB_PATH=builds/lib.so.local python run.py # local build
made 1000000 calls in 0.26s

Actual behavior:

The shared object built using Github Actions is extremely slow (up to 100 times slower).

$ RUNPY_LIB_PATH=builds/lib.so.gha2 python run.py  # github actions, build 2
made 1000000 calls in 33.91s
gowridurgad commented

Hello @mpenkov ,
Thank you for creating this issue. We will investigate it and provide feedback as soon as we have some updates.

  • 👍1
priyagupta108 commented

Hello @mpenkov 👋,
The performance differences you've observed between your local environment and GitHub Actions (GHA) can be influenced by several factors:

  • Hardware Differences: Variations in CPU, memory, and disk speed between the local environment and GHA runners can significantly impact performance.

  • Operating System and Configuration: The underlying operating system and its configuration, including system overheads and background processes, can affect application performance.

  • Execution Environment: Differences in runtime versions (e.g., Python, Go), environment variables, and the presence of different system libraries can lead to performance variations.

To provide a clearer example, I've attached screenshots of my macOS local environment setup and the macOS-13 GHA configuration:

  • Local Environment:
Screenshot 2024-09-10 at 4 44 14 PM
  • GHA:
Screenshot 2024-09-12 at 12 51 36 PM

I hope this clarifies the situation.

mpenkov commented

Hi @priyagupta108 , thank you for looking into this.

I think you may be misunderstanding the problem because your explanation focuses on the execution environment. The point isn't to run the shared library on GHA. The point is to build the shared library on GHA, and then to use the shared library in the production environment.

The issue demonstrates that there is something wrong with the way that GHA is building the shared library. If I try to use the shared library built by GHA, it is very slow. If I replace the shared library built by GHA with one that I built somewhere else, then it is no longer slow. Therefore, the problem is with GHA.

EDIT:

I've updated my example code to also use the macos-latest runner. I can confirm that the problem does not appear with the macos runner, which is consistent with the results that you posted.

Therefore, we can narrow the problem down further: the problem is with the build environment on the GHA Ubuntu runner.

priyagupta108 commented

Hi @mpenkov 👋 ,
Could you please provide the Go version installed in your local environment (use go version to obtain this).
This information will help us investigate the performance difference between the shared libraries built on GitHub Actions and local machine.
Thank you!

mpenkov commented 
$ go version
go version go1.21.4 linux/amd64
priyagupta108 commented

Hi @mpenkov,
After further investigation, I've identified a performance issue related to Go versions released from 2024-07-02 onwards. On Ubuntu environments, the execution time is notably longer with these versions compared to those released before this date.
Please find the test results below:
Versions from July 2, 2024, onwards:

  • go1.23.1:

    Screenshot 2024-09-19 at 3 44 33 PM

  • go1.22.5:

    Screenshot 2024-09-19 at 3 45 25 PM

Versions released before July 2, 2024:

  • go1.22.4:

    Screenshot 2024-09-19 at 3 00 50 PM

  • go1.21.4:

    Screenshot 2024-09-19 at 2 58 40 PM

I recommend setting a Go version released before July 2, 2024, such as 1.21.4, in the GitHub Actions workflow to observe the performance difference.

mpenkov commented

Good catch! So this is a problem with the go utility itself, and not Github Actions?

This seems relevant: golang/go#68587

priyagupta108 commented

Thank you! Yes, this is an issue with the Go utility itself. The golang/go#68587 issue you referenced is relevant and might be worth following for any updates or potential fixes from the Go team.
Closing this issue for now. Please feel free to reach out to us in case of any concerns.

  • 👍1