Python tests does not count over `make cover`

[alanjds]

History:

As said in the discussion starting on #243 (comment), the Python tests does not count over make cover report.

Personally, this hinders the development speed by forcing people to write tests twice: one for Python (what we want working) and one for Golang (what we need to count on make cover).

The hypothesis is: Lowering the work needed to push code, more code will be pushed by more people.

Lets fix it to speed up the development!

Plan:

• Change grumpc to be able to produce module_test.go output from Python test files
• Make the make test to produce module_test.go files from grumpy/testing/*.py files
• Make the produced module_test.go files to be in such place that counts for coverage
• Write a change in a runtime .go file. Write a test the for the change in a .py
• Check that make cover counts the .py covering the .go changed

[alanjds]

@trotterdylan had said that:

Coverage is only calculated from tests within the runtime directory.

and

[is] Just the way go test works, it expects *_test.go files with the package name of that being tested. And then within those files it expects Test* functions. One way to do it would be to generate a module_test.go file with a single TestModule function in there that loads the module and executes it as main.

Then I am trying to make the grumpy/testing/*.py files to generate *_test.go files.

[alanjds]

From a Python file containing print 'Hello World!', grumpc does produce this module.go:

package __main__
import πg "grumpy"
var Code *πg.Code
func init() {
        Code = πg.NewCode("<module>", "print_hello.py", nil, 0, func(πF *πg.Frame, _ []*πg.Object) (*πg.Object, *πg.BaseException) {
                var πR *πg.Object; _ = πR
                var πE *πg.BaseException; _ = πE
                var πTemp001 []*πg.Object
                _ = πTemp001
                for ; πF.State() >= 0; πF.PopCheckpoint() {
                        switch πF.State() {
                        case 0:
                        default: panic("unexpected function state")
                        }
                        // line 1: print 'Hello World!'
                        πF.SetLineno(1)
                        πTemp001 = make([]*πg.Object, 1)
                        πTemp001[0] = πg.NewStr("Hello World!").ToObject()
                        if πE = πg.Print(πF, πTemp001, true); πE != nil {
                                continue
                        }
                }
                return nil, πE
        })
        πg.RegisterModule("__main__", Code)
}

I managed to add a flag to grumpc that, for the same Python input, produces:

package __main__
import πg "grumpy"
import πt "testing"
var Code *πg.Code
func init() {
        Code = πg.NewCode("<module>", "print_hello.py", nil, 0, func(πF *πg.Frame, _ []*πg.Object) (*πg.Object, *πg.BaseException) {
                var πR *πg.Object; _ = πR
                var πE *πg.BaseException; _ = πE
                var πTemp001 []*πg.Object
                _ = πTemp001
                for ; πF.State() >= 0; πF.PopCheckpoint() {
                        switch πF.State() {
                        case 0:
                        default: panic("unexpected function state")
                        }
                        // line 1: print 'Hello World!'
                        πF.SetLineno(1)
                        πTemp001 = make([]*πg.Object, 1)
                        πTemp001[0] = πg.NewStr("Hello World!").ToObject()
                        if πE = πg.Print(πF, πTemp001, true); πE != nil {
                                continue
                        }
                }
                return nil, πE
        })
        πg.RegisterModule("__main__", Code)
}
func TestRunCode(t *testing.T) {
        init()
        πg.ImportModule(grumpy.NewRootFrame(), "traceback")
        //πg.ImportModule(grumpy.NewRootFrame(), "__main__")
        πg.RunMain(Code)
}

to be saved as module_test.go

[alanjds]

Question: Is this an appropriate Go test to count for coverage, or should something be changed on the output yet?

The plan is to, after producing a reasonable output file, figure where should it be placed and how to put it there.

[trotterdylan]

I think it's reasonable to use the Python files in testing/ count toward coverage.

Rather than adding complexity to grumpc, could we make this a standalone utility that takes a module name and generates a _test.go file that can be run for this purpose?

[alanjds]

@trotterdylan Can this new tool be made later, before the merge?
The plan is to have it working, then do a pass cleaning the mess.

In fact, I am biased towards refactor grumpc to make it cleaner. You are right to worry about putting more stuff on tool composed of +70 lines of main().

Anyway,

• Do the attached output code seems right as a test file?
• Where should it live on the /build folder to count over the coverage?

(am I making the right questions, or got totally clueless?)

[trotterdylan]

Thinking about it a little more, what you really want is to do something similar to grumprun, except instead of generating a main() function, you will generate a TestRunCode() function. Unfortunately the logic is a bit different, because you really want to generate the _test.go file, not build and run a binary. I would write a new tool based on what's in grumprun (except much simpler, because it doesn't need invoke grumpc or the Go toolchain) that outputs something like this:

package foo
import (
	"os"
	"grumpy"
	<other transitive imports here>
)
func TestRunCode(t *testing.T) {
	grumpy.ImportModule(grumpy.NewRootFrame(), "traceback")
	if grumpy.RunMain(Code) != 0 {
		t.Fail()
	}
}

This file would be called foo_test.go and would sit alongside the code generated by grumpc for module foo. This would mean you could run the test like: go test __python__.foo. You can then pass -coverprofile to generate coverage data.

The second part of this is to merge coverage data for all the different sources. In particular, the coverage generated for the grumpy package with the coverage generated by each of the tests in testing/. This will require some work to coverparse to pull all that data together into a single coverage report.

[alanjds]

Thanks for pointing to a direction and for saying where the test file will end and how to run it.

I will take a look at the grumpyrun later. Maybe this week, maybe next one.

[alanjds]

I see.

You are suggesting to import the module and run it, instead of pasting all the code inside the test...

[trotterdylan]

Right. Except rather than importing it with ImportModule(), call RunMain() on it so that the module is treated as __main__.

[alanjds]

I am writing the new tool. Just realized that the Code is not described on the test file.

Where does it came from? Is imported from the module.go?

[trotterdylan]

In the example I wrote above, Code would come from the foo package that's being tested. In Go, the tests are part of the package under test and can access members of that package.

[alanjds]

On #389, the new tool gentest got able to produce *_test.go files.

Aside from formatting and this kind of details, do the tool is producing the right _test.go files on the right places?

If so, the next step would be make test to run the testfiles.

[alanjds]

On #389, I did not understand what caused this: https://travis-ci.org/google/grumpy/jobs/316438956#L938

/tmp/tmpsi9FK2/main.go:5:2: found packages time_test (module.go) and time (module_test.go) in /home/travis/gopath/src/github.com/google/grumpy/build/src/__python__/time_test

What had I made wrong?

[alanjds]

Gone ahead w/ the proposed test generator.

Results for functools.py module:

Chosen the functools to show as it is a very simple example. imputil.calculate_transitive_deps(...) got only _functools as dependency.

The result _test.go generated is:

package functools
import (
    "testing"
    "grumpy"
    _ "__python__/_functools"
    _ "__python__/traceback"
)
func TestRunCode(t *testing.T) {
    grumpy.ImportModule(grumpy.NewRootFrame(), "traceback")
    if grumpy.RunMain(Code) != 0 {
        t.Fail()
    }
}

ok  	__python__/functools	0.025s

However, I started to think that this transitive imports should be on the module.go itself instead of on the one trying to import it. Then there will be no need to, on the main package, detect the dependencies of the whole tree.

[alanjds]

Something strange is failing:

/tmp/tmptxx_OJ/main.go:5:2: found packages time_test (module.go) and time (module_test.go) in /home/travis/gopath/src/github.com/google/grumpy/build/src/__python__/time_test

But both files are of the pack time_test:

$ head -n 3 build/src/__python__/time_test/module.go
package time_test
import πg "grumpy"
var Code *πg.Code

$ head -n 10 build/src/__python__/time_test/module_test.go
package time_test
import (
	"testing"
	"grumpy"
	_ "__python__/__go__/time"
	_ "__python__/time"
	_ "__python__/traceback"
)
func TestRunCode(t *testing.T) {

$ head -n 3 build/src/__python__/time/module.go
package time
import πg "grumpy"
var Code *πg.Code

(there is no time/module_test.go)

Now I am stuck.
Any ideas about how to keep going?

[trotterdylan]

The package declaration in the _test.go file should just be package time. In Go, unit tests reside in the same package as the package being tested.

[alanjds]

On the file named time_test/module_test.go the package should be time?

[alanjds]

It worked after changed time_test/module.go and time_test/module_test.go files. Is this ok?

[alanjds]

Yay! Test passed!!

Now, how to count the coverage over?

[alanjds]

I took a look on the Makefile and on the -coverprofile flag. Got the impression that all the tests should be about a package.

If it is true, all the tests of TestRunCode stuff should be of grumpy package!

So should I generate every test on the same single folder to fulfil the Go desired directory structure?

[trotterdylan]

Yeah I guess you're right that we're interested in the coverage against the grumpy package. The problem with putting all the generated tests together in the same directory is that they have symbols that will collide (e.g. Code).

Looks like this issue is relevant: golang/go#6909

It looks like in Go 1.10 you can specify multiple packages to test with -coverprofile and the coverage for each package will be a union of the coverage data generated for all the tests being run. Maybe you could try running against the 1.10 beta to see if that works? Apparently it will be released in January.

[alanjds]

I see. Will take a look on the 1.10 tool.

Thanks for pointing.