Expecting error exit code, but getting success

[aghuddleston]

Bug report

I am converting our pipeline from CircleCI to Github Actions. In our Circle pipeline, we call the Chromatic cli directly npx --no-install chromatic (with a bunch of options).

In the Github actions workflow, I am trying to use the action provided by Chromatic, but now instead of an error code when there is a visual change detected, I am getting a success code.

I think I should still be getting an error exit code, but maybe my understanding is wrong?

On CircleCI (Chromatic CLI v11.3.0) I see this output:

Started build 706
    → View build details at https://www.chromatic.com/build?appId=XXX&number=706
Running 395 tests
    → This may take a few minutes
ℹ Speed up Continuous Integration
Your project is linked to GitHub so Chromatic will report results there.
This means you can pass the --exit-once-uploaded flag to skip waiting for build results.
Read more here: https://www.chromatic.com/docs/cli#chromatic-options
    → [                    ] 1/395  
    ....
    → [====================] 395/395  
✖ Found 1 visual change: Review the changes at https://www.chromatic.com/build?appId=XXX&number=706

ℹ For CI/CD use cases, this command failed with exit code 1
Pass --exit-zero-on-changes to succeed this command regardless of changes.
Pass --auto-accept-changes to succeed and automatically accept any changes.
Build 706 completed
    → Tested 153 stories across 97 components; captured 395 snapshots in 1 minute 51 seconds

On Github (Chromatic CLI v11.3.1) I see this output:

Started build 703
    → View build details at https://www.chromatic.com/build?appId=XXX&number=703
Running 395 tests
    → This may take a few minutes
ℹ Speed up Continuous Integration
Your project is linked to GitHub so Chromatic will report results there.
This means you can pass the --exit-once-uploaded flag to skip waiting for build results.
Read more here: https://www.chromatic.com/docs/cli#chromatic-options
    → [                    ] 1/395  
   ....
    → [====================] 395/395  
✔ Build 703 passed!
395 visual changes were found in this build.
ℹ View build details at https://www.chromatic.com/build?appId=XXX&number=703
Build 703 completed
    → Tested 153 stories across 97 components; captured 395 snapshots in 2 minutes 26 seconds

The Github action run has 395 changes but it still says "Build passed". Is this a bug in the action or maybe something I implemented wrong? I logged the outputs from the action and I see this:

{
    "code": "0",
    "url": "xxx",
    "buildUrl": "xxx",
    "storybookUrl": "xxx,
    "specCount": "153",
    "componentCount": "97",
    "testCount": "395",
    "changeCount": "395",
    "errorCount": "0",
    "interactionTestFailuresCount": "0",
    "actualTestCount": "395",
    "actualCaptureCount": "395",
    "inheritedCaptureCount": "0"
  }

[winkerVSbecks]

@aghuddleston are you using exitZeroOnChanges option?

https://www.chromatic.com/docs/github-actions/#command-exit-code-for-required-checks

If you are using pull request statuses as required checks before merging, you may not want your workflow step to fail if test snapshots render without errors (but with changes). To achieve this, pass the option exitZeroOnChanges to the with clause, and your workflow will continue in such cases.

[aghuddleston]

@winkerVSbecks Nope, I am not using exitZeroOnChanges. It should just be using the default option there.

I'm only using these options:

Run chromaui/action@latest
  with:
    projectToken: ***
    buildScriptName: sb:hc
    onlyChanged: false
    exitOnceUploaded: false

[winkerVSbecks]

What you described is the default Chromatic behaviour. Chromatic exits with 1 when there is a successful build, regardless of changes or not. To change that behaviour, please use the exitZeroOnChanges flag.

[danewilson]

What you described is the default Chromatic behaviour. Chromatic exits with 1 when there is a successful build, regardless of changes or not. To change that behaviour, please use the exitZeroOnChanges flag.

Why would Chromatic exit with 1 for a successful build? As it says in the docs, exitZeroOnChanges is for ignoring changes, essentially. The problem being described in this issue is that GitHub Actions seems to do this by default, and there's no way to tell GHA to fail if there are changes. Please reopen this issue.

[danewilson]

Actually it looks like the confusion here is that exitZeroOnChanges defaults to true in GHA, so explicitly setting it to false actually produces the desired result. This is not the behaviour described in the docs.