Automatically detect the usage of parallel tests and modify setup

Question

Automatically detect the usage of parallel tests and modify setup

colszowka opened this issue 13 years ago · 13 comments

See #61: For parallel_tests, it is neccessary to set the command name to something random like this SimpleCov.command_name "RSpec #{rand(100000)}" so parallel runs do not overwrite each other.

This is not obvious to users of the library and should be handled by SimpleCov automatically.

As a bonus, it would be great if instead of random numbers, some replicable ID would be available for each parallel test run, for example using a MD5 of the invoked shell command so that reported line hits do not add up along the way when the same test runs twice.

Answer 1 · 2011-09-30T03:34:58.000Z

Hi,

Where should I put this command_name in the specs for it to work? Immediately after the .start command in spec_helper?

It seems to be generating the coverage in between the split processes, and not merging.

Answer 2 · 2011-09-30T07:33:33.000Z

All good, added it to a before each as follows:

config.before(:each) do
    SimpleCov.command_name "RSpec:#{Process.pid.to_s}#{ENV['TEST_ENV_NUMBER']}"
end

Answer 3 · 2012-03-16T21:12:36.000Z

Hi cesar, it should be placed in your spec_helper, in between your RSpec.configure block.

On 17/03/2012, at 7:36 AM, César Camachoreply@reply.github.com wrote:

All good, added it to a before each as follows:

config.before(:each) do
SimpleCov.command_name "RSpec:#{Process.pid.to_s}#{ENV['TEST_ENV_NUMBER']}"
end

Where should this block of code be placed? Thanks!

Reply to this email directly or view it on GitHub:
#64 (comment)

Answer 4 · 2012-03-16T21:14:55.000Z

My original question was where this code block should be placed? I apologize for deleting it. Thanks for the super quick response tommeier!

config.before(:each) do
    SimpleCov.command_name "RSpec:#{Process.pid.to_s}#{ENV['TEST_ENV_NUMBER']}"
end

Answer 5 · 2012-03-17T00:37:33.000Z

I've actually gone with a less 'brick to the face' method for it, now i've split up tasks, such as unit tests, etc by rake tasks. Then I have a Jenkins CI box that parallelizes out the tasks individually. Adding this to the Rakefile:

Rake::Task.class_eval do
  #Required to ensure SimpleCov doesn't overwrite coverage shared values
  alias :execute_without_simplecov_command :execute
  def execute_and_set_simplecov_command args=nil
    ENV['COV_COMMAND'] = "rspec:#{name}" if ENV['COVERAGE']
    execute_without_simplecov_command(args)
  end
  alias :execute :execute_and_set_simplecov_command
end

So simplecov only groups and applies coverage based on the rake task groups, ie: where the custom split has been applied for rake tasks.

Jenkins CI can then merge all these together (from a shared dir). Currently finding it as a very accurate method to merge parallelised specs with SimpleCov.

Answer 6 · 2012-08-04T20:39:35.000Z

For RSpec, I'm using

config.before(:each) { SimpleCov.command_name example.location }

to tag the coverage with a unique id per spec (namely the filename and line number, eg ./spec/models/something_spec.rb:123)

Answer 7 · 2012-09-14T02:26:25.000Z

How do you fail the build then? Here's what I do:

# spec/support/coverage.rb
if ENV['COV']
  puts "Running with test coverage"
  require 'simplecov'

  SimpleCov.start 'rails' do
    add_filter "lib/extensions"
    # bug: changing the coverage_dir here breaks coverage recording.
    at_exit do
      SimpleCov.result.format!
      threshold, actual = 97, SimpleCov.result.covered_percent
      if actual < threshold
        $stderr.puts "Coverage #{actual}% is below the threshold of #{threshold}%."
        exit 1
      end
    end
  end

  RSpec.configure do |config|
    config.before(:each) do
      SimpleCov.command_name "RSpec:#{Process.pid}#{ENV['TEST_ENV_NUMBER']}"
    end
  end
end

But unfortunately the results don't seem to be merged:


Coverage report generated for RSpec:711845, RSpec:711926, RSpec:711942, RSpec:711977 to proj/x/coverage. 2997 / 3817 LOC (78.52%) covered.
/Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse': 757: unexpected token at '' (MultiJson::DecodeError)
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json/adapters/json_common.rb:7:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json.rb:93:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:20:in `resultset'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:72:in `store_result'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov.rb:48:in `result'
    from /Users/dnagir/proj/x/spec/support/coverage.rb:11:in `block (2 levels) in <top (required)>'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `call'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `block in <top (required)>'
/Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse': 757: unexpected token at '' (MultiJson::DecodeError)
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json/adapters/json_common.rb:7:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json.rb:93:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:20:in `resultset'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:44:in `results'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:61:in `merged_result'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov.rb:49:in `result'
    from /Users/dnagir/proj/x/spec/support/coverage.rb:10:in `block (2 levels) in <top (required)>'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `call'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `block in <top (required)>'
/Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse': 757: unexpected token at '' (MultiJson::DecodeError)
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json/adapters/json_common.rb:7:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json.rb:93:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:20:in `resultset'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:44:in `results'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:61:in `merged_result'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov.rb:49:in `result'
    from /Users/dnagir/proj/x/spec/support/coverage.rb:10:in `block (2 levels) in <top (required)>'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `call'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `block in <top (required)>'
Coverage report generated for RSpec:711845, RSpec:711904, RSpec:711926, RSpec:711942, RSpec:711977 to /Users/dnagir/proj/x/coverage. 3349 / 3817 LOC (87.74%) covered.
/Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse': 757: unexpected token at '' (MultiJson::DecodeError)
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json/adapters/json_common.rb:7:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json.rb:93:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:20:in `resultset'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:72:in `store_result'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov.rb:48:in `result'
    from /Users/dnagir/proj/x/spec/support/coverage.rb:11:in `block (2 levels) in <top (required)>'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `call'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `block in <top (required)>'

I'm probably missing something, but not sure what exactly.

Answer 8 · 2012-11-12T05:33:16.000Z

Because after running this a few times I started encountering issues with it building the coverage report. I figured out that deleting the coverage folder would get it working again. So for now I've added:

config.before(:suite) do
rm -rf #{Rails.root}/coverage
end

Answer 9 · 2012-12-28T04:57:21.000Z

@dnagir you are seeing the same race condition that I'm seeing. Simplecov needs to lock the data file when it accesses it. I'm assuming in lib/simplecov/json.rb

Answer 10 · 2012-12-28T04:59:06.000Z

Scratch that json.rb is the wrong place. I'm too tired to hunt it down right now.

Answer 11 · 2012-12-28T05:00:34.000Z

It needs to be in lib/simplecov/result_merger.rb and maybe some other places.

Answer 12 · 2013-01-07T17:24:21.000Z

PR #185 should totally fix this issue.

Answer 13 · 2013-09-11T15:41:12.000Z

As @jshraibman-mdsol wrote an (embarassing) while ago, PR #185 should have fixed this problem. It will be released in a couple of minutes as part of version 0.8.0.pre2