Fail when coverage is not met

[Clonex]

Is it possible to make the action fail when the coverage is not met?
I've been working on setting up the coverage report action in our repo, but it still goes green even if I dont meet the coverage goal.

Our config looks like this:

export default defineConfig({
	test: {
		watch: false,
		threads: false,
		poolOptions: {
			threads: {
				singleThread: true,
			},
		},
		environment: 'jsdom',
		setupFiles: ['dotenv/config'],
		coverage: {
			clean: true,
			cleanOnRerun: true,
			provider: 'v8',
			reporter: ['html', 'json-summary', 'json'],
			reportsDirectory: './coverage',

			lines: 40,
			branches: 40,
			functions: 40,
			statements: 40,
		},
	},
})

This is the steps that im doing to use the action.

- name: 'Coverage test - Generate'
  run: yarn vitest run --coverage

- name: 'Coverage test - Check'
  if: always()
  uses: davelosert/vitest-coverage-report-action@v2

It does however still go green

[davelosert]

Hi @Clonex,

thanks for the request!
I would have to test, but I thought that, if the coverage is not met, the vitest command itself should fail and the action should report failure for the workflow (that's why I put the if: always() there to still generate the coverage report).

I am wondering if this changed with a recent version of vitest - or do you have something in that dotenv/config that makes the command pass (return with 0) even if the coverage is not met?

[davelosert]

Hi @Clonex,

so I found the reason for this behavior: Only using the --coverage flag within the test-command will actually overwrite all coverage-configuration supplied in the vite config file. This seems to be a somewhat changing behavior in a recent upgrade of vitest (I was too lazy to look where exactly ;) ).

To fix this, you'll actually have to use the dot notation --coverage.enabled true, so in your case change the test command to in your action configuration to:

- name: 'Coverage test - Generate'
  run: yarn vitest run --coverage.enabled true

I've adjusted the documentation of this action in #318 accordingly.

Also please note two things:

• As you also specified the if: always() configuration in the report-aciton, you'll have to put the reportOnFailure: true in your vite configuration file as well
• The thresholds in vitest need to be configured under a thresholds property now (I believe it still works the way you have it, but it's now documented differently in the official vitest docs.

If you want to fix both, you'll have to alter your configuration to this:

export default defineConfig({
	test: {
		watch: false,
		threads: false,
		poolOptions: {
			threads: {
				singleThread: true,
			},
		},
		environment: 'jsdom',
		setupFiles: ['dotenv/config'],
		coverage: {
			clean: true,
			cleanOnRerun: true,
			provider: 'v8',
			reporter: ['html', 'json-summary', 'json'],
			reportsDirectory: './coverage',
                        reportOnFailure: true,
                        thresholds: {
			    lines: 40,
			    branches: 40,
			    functions: 40,
			    statements: 40,
                       },
		},
	},
})

[github-actions]

🎉 This issue has been resolved in version 2.2.0 🎉

The release is available on:

• GitHub release
• v2.2.0

Your semantic-release bot 📦🚀

[Clonex]

Hi @Clonex,

so I found the reason for this behavior: Only using the --coverage flag within the test-command will actually overwrite all coverage-configuration supplied in the vite config file. This seems to be a somewhat changing behavior in a recent upgrade of vitest (I was too lazy to look where exactly ;) ).

To fix this, you'll actually have to use the dot notation --coverage.enabled true, so in your case change the test command to in your action configuration to:

- name: 'Coverage test - Generate'
  run: yarn vitest run --coverage.enabled true

I've adjusted the documentation of this action in #318 accordingly.

Also please note two things:

• As you also specified the if: always() configuration in the report-aciton, you'll have to put the reportOnFailure: true in your vite configuration file as well
• The thresholds in vitest need to be configured under a thresholds property now (I believe it still works the way you have it, but it's now documented differently in the official vitest docs.

If you want to fix both, you'll have to alter your configuration to this:

export default defineConfig({
	test: {
		watch: false,
		threads: false,
		poolOptions: {
			threads: {
				singleThread: true,
			},
		},
		environment: 'jsdom',
		setupFiles: ['dotenv/config'],
		coverage: {
			clean: true,
			cleanOnRerun: true,
			provider: 'v8',
			reporter: ['html', 'json-summary', 'json'],
			reportsDirectory: './coverage',
                        reportOnFailure: true,
                        thresholds: {
			    lines: 40,
			    branches: 40,
			    functions: 40,
			    statements: 40,
                       },
		},
	},
})

I see, that clears it up! I appreciate the in-depth answer 😄
I've updated our action based on your comment, and it works as intended now!