'cricket-unittest run all' 922+ Fails [Window]
JaesungNa opened this issue · 22 comments
I am following 'Contributing to Batavia' steps to start to see where I can contribute to batavia project.
I am using python 3.4 and Widow 10. I followed up to 'Running the test suite' However, when I try to use cricket/cmd to test the code, every builtin~FunctionTests fails as below. (922 fails+)
Futhermore, I spend more than a hour waiting to get test results for all the test case. Is it normal things to happen?
I am completely lost what I have done wrong and where to start. please help.
BTW, my colleagues have succeed on same tests which I got failed on; He is using Mac and python 3.4
@JaesungNa Thanks for the report!
The test suite does take a long time to run - 10 hours seems a little long, but it's difficult to say without knowing details about your computer's speed.
However, the test suite should pass without any errors. It looks like you're getting consistent errors, so something is clearly going wrong.
My suggestion would be to run a single test - highlight a single test (say, test_int for BuiltinAbsFunctionTests), and see the test output. The error produced by that one test will likely be the same as all the other tests, and will give an indication of the underlying problem.
The most likely source of problems will be your Node and NPM version. You need to make sure you're running an LTS version of Node (something from the 6.9.X series) and NPM 4+.
However, if you've got the right versions of Node and NPM, we'll need to see the output of a single test failure to work out what is going on.
I've had success with the latest node and npm versions (on Linux though).
Also make sure you ran npm run build before running the tests.
@freakboy3742 @lielfr Thank you for the feedback!
I have thoroughly read your feedback, and it still throws errors which also takes ages to finish its test suite.
-
My laptop details
- i5-6200U
- RAM 8.00GB
- Window 10 Enterprise 64 bit
-
node/npm version check
-
npm run build
-
"py setup.py test -s tests.datatypes.test_str.BinaryStrOperationTests.test_add_bool" command -> fail
I am just guessing that warning that says 'no files found matching 'Makefile'', 'no files found matching 'requirements*.txt'' gives this test suite to throw error.
Am I on right track?
@JaesungNa Thanks for posting those logs - they're very helpful (although posting raw copy & pasted text, rather than images, is generally more helpful)
As I said last time - the long run time is expected. The test suite takes a long time to run. A very looooooong time. We usually only run the full test suite on our automated build hardware when you submit a pull request.
The "no files found matching Makefile" and "no file found matching requirements*.txt" aren't a problem - those are normal output that Batavia doesn't have any control over.
There are two details that point to the most likely source of the problem:
-
Firstly, the output is using
₩where I'd expect to see/or\. I don't know if this is a common thing to see on Korean terminal output, but it's quite different from what I'd normally expect to see - so it points at language encoding being the underlying problem -
When the test runs, it is outputting:
>>> x = ""
>>> y = True
>>> x + y
- /// <class 'TypeError'> : Can't convert 'bool' object to str implicitly
? ----
+ <class 'TypeError'> : Can't convert 'bool' object to str implicitly
This is 'diff' output, showing you the test code that was executed, the output that was actually received, and the output that was expected. In this case, the only difference is the /// at the start of each result line. I don't know where that output is coming from, though - it's not output I've ever seen before. I can only assume it's related to language encoding as well.
I don't have any experience with Korean language encoding, and even less experience with encodings on Windows; so this is an area where your experience will be invaluable. It may turn out that the problem is with the test suite - not Batavia itself. That is, the problem may be with how we're testing the code, not a bug in the code itself.
From here, you're going to need to go exploring to find the source of the bug. Happy hunting!
@freakboy3742 Thanks for the feedback! I found a clue where I get those error from.
1. Initial Error
- Looks like there is a problem arising from line 'self.assertEqual(js_out, py_out, context)' in def assertCodeExecution() in class TranspileTestCase(TestCase) in utils.py
(venv) C:\Users\Jaesung\Desktop\Data\Coding\OSS\batavia-dev\batavia>py setup.py test -s tests.builtins.test_abs.BuiltinAbsFunctionTests.test_bool
C:\Users\Jaesung\Desktop\Data\Coding\OSS\batavia-dev\batavia\venv\lib\site-packages\setuptools\dist.py:283: UserWarning: The version specified requires normalization, consider using '3.4.0.dev19' instead of '3.4.0-dev19'.
self.metadata.version,
running test
running egg_info
writing dependency_links to batavia.egg-info\dependency_links.txt
writing top-level names to batavia.egg-info\top_level.txt
writing batavia.egg-info\PKG-INFO
reading manifest file 'batavia.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'Makefile'
warning: no files found matching 'requirements*.txt'
writing manifest file 'batavia.egg-info\SOURCES.txt'
running build_ext
c:\users\jaesung\desktop\data\coding\oss\batavia-dev\batavia\tests
building 'batavia.js'
test_bool (tests.builtins.test_abs.BuiltinAbsFunctionTests) ... FAIL
======================================================================
FAIL: test_bool (tests.builtins.test_abs.BuiltinAbsFunctionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "c:\users\jaesung\desktop\data\coding\oss\batavia-dev\batavia\tests\utils.py", line 1217, in func
transform_output=transform_output,
File "c:\users\jaesung\desktop\data\coding\oss\batavia-dev\batavia\tests\utils.py", line 1254, in assertBuiltinFunction
transform_output=transform_output,
File "c:\users\jaesung\desktop\data\coding\oss\batavia-dev\batavia\tests\utils.py", line 497, in assertCodeExecution
self.assertEqual(js_out, py_out, context)
AssertionError: '>>> [27 chars]f(x)\n||| 1\n\n>>> f = abs\n>>> x = False\n>>> f(x)\n||| 0\n\n' != '>>> [27 chars]f(x)\n1\n\n>>> f = abs\n>>> x = False\n>>> f(x)\n0\n\n'
>>> f = abs
>>> x = True
>>> f(x)
- ||| 1
+ 1
>>> f = abs
>>> x = False
>>> f(x)
- ||| 0
+ 0
: Global context: Error running f(x)
----------------------------------------------------------------------
Ran 1 test in 5.036s
FAILED (failures=1)
2. result after modification
- So I have checked the result of each 'js_out' and 'py_out'
!!js_out = >>> f = abs
>>> x = True
>>> f(x)
||| 1
>>> f = abs
>>> x = False
>>> f(x)
||| 0
!!py_out = >>> f = abs
>>> x = True
>>> f(x)
1
>>> f = abs
>>> x = False
>>> f(x)
0
FAIL
-
The difference between these two result is that there is '|||' in 'js_out ' result, therefore I have eliminated the code('|||' and '///') in assertBinaryOperation() in class BuiltinFunctionTestCase(NotImplementedToExpectedFailure)
-
The modification give result at bottom (test suite working fine)
(venv) C:\Users\Jaesung\Desktop\Data\Coding\OSS\batavia-dev\batavia>py setup.py test -s tests.builtins.test_abs.BuiltinAbsFunctionTests.test_bool
C:\Users\Jaesung\Desktop\Data\Coding\OSS\batavia-dev\batavia\venv\lib\site-packages\setuptools\dist.py:283: UserWarning: The version specified requires normalization, consider using '3.4.0.dev19' instead of '3.4.0-dev19'.
self.metadata.version,
running test
running egg_info
writing dependency_links to batavia.egg-info\dependency_links.txt
writing top-level names to batavia.egg-info\top_level.txt
writing batavia.egg-info\PKG-INFO
reading manifest file 'batavia.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'Makefile'
warning: no files found matching 'requirements*.txt'
writing manifest file 'batavia.egg-info\SOURCES.txt'
running build_ext
c:\users\jaesung\desktop\data\coding\oss\batavia-dev\batavia\tests
building 'batavia.js'
test_bool (tests.builtins.test_abs.BuiltinAbsFunctionTests) ... ok
----------------------------------------------------------------------
Ran 1 test in 7.388s
OK
I have gone through 'cricket-unittest' and it went well
Do you Think this is the right solution for this issue?
You're narrowing down on the cause, but you have't got it yet. The ||| and /// markers are an important part of the output handling process - this code has the description of what they're doing.
There is clearly a bug related to this handling - but the fix isn't to just delete those prefixes.
I have a very similar problem and I'm using windows 10 as well. All of the BuiltinWhateverFunctionTests are failing. I'm also from the US so it might not be a language encoding issue. I'm using node version 6.9.1, npm version 5.6.0, and python version 3.4.
This is the error I get for the test BuiltinAbsFunctionTests>test_None:
Traceback (most recent call last):
File "c:\program files\python34\Lib\unittest\case.py", line 58, in testPartExecutor
yield
File "c:\program files\python34\Lib\unittest\case.py", line 580, in run
testMethod()
File "C:\Users\johns\GitRepos\batavia\tests\utils.py", line 1216, in func
transform_output=transform_output,
File "C:\Users\johns\GitRepos\batavia\tests\utils.py", line 1253, in assertBuiltinFunction
transform_output=transform_output,
File "C:\Users\johns\GitRepos\batavia\tests\utils.py", line 496, in assertCodeExecution
self.assertEqual(js_out, py_out, context)
File "c:\program files\python34\Lib\unittest\case.py", line 800, in assertEqual
assertion_func(first, second, msg=msg)
File "c:\program files\python34\Lib\unittest\case.py", line 1173, in assertMultiLineEqual
self.fail(self._formatMessage(msg, standardMsg))
File "c:\program files\python34\Lib\unittest\case.py", line 645, in fail
raise self.failureException(msg)
AssertionError: ">>> [28 chars](x)\n/// <class 'TypeError'> : bad operand typ[23 chars]\n\n" != ">>> [28 chars](x)\n<class 'TypeError'> : bad operand type fo[19 chars]\n\n"
>>> f = abs
>>> x = None
>>> f(x)
- /// <class 'TypeError'> : bad operand type for abs(): 'NoneType'
? ----
+ <class 'TypeError'> : bad operand type for abs(): 'NoneType'
: Global context: Error running f(x)
I'm also getting the /// as output. In addition, it looks like the error message is printing the newline character instead of making newlines.
This is for BuiltinAbsFunctionsTests>test_int:
Traceback (most recent call last):
File "c:\program files\python34\Lib\unittest\case.py", line 58, in testPartExecutor
yield
File "c:\program files\python34\Lib\unittest\case.py", line 580, in run
testMethod()
File "C:\Users\johns\GitRepos\batavia\tests\utils.py", line 1216, in func
transform_output=transform_output,
File "C:\Users\johns\GitRepos\batavia\tests\utils.py", line 1253, in assertBuiltinFunction
transform_output=transform_output,
File "C:\Users\johns\GitRepos\batavia\tests\utils.py", line 496, in assertCodeExecution
self.assertEqual(js_out, py_out, context)
File "c:\program files\python34\Lib\unittest\case.py", line 800, in assertEqual
assertion_func(first, second, msg=msg)
File "c:\program files\python34\Lib\unittest\case.py", line 1173, in assertMultiLineEqual
self.fail(self._formatMessage(msg, standardMsg))
File "c:\program files\python34\Lib\unittest\case.py", line 645, in fail
raise self.failureException(msg)
AssertionError: '>>> [25 chars](x)\n||| 3\n\n>>> f = abs\n>>> x = 0\n>>> f(x)[2594 chars]\n\n' != '>>> [25 chars](x)\n3\n\n>>> f = abs\n>>> x = 0\n>>> f(x)\n0\[2510 chars]\n\n'
>>> f = abs
>>> x = 3
>>> f(x)
- ||| 3
+ 3
>>> f = abs
>>> x = 0
>>> f(x)
- ||| 0
+ 0
>>> f = abs
>>> x = -5
>>> f(x)
- ||| 5
+ 5
>>> f = abs
>>> x = -3
>>> f(x)
- ||| 3
+ 3
>>> f = abs
>>> x = 5
>>> f(x)
- ||| 5
+ 5
>>> f = abs
>>> x = 1
>>> f(x)
- ||| 1
+ 1
>>> f = abs
>>> x = -1
>>> f(x)
- ||| 1
+ 1
>>> f = abs
>>> x = 9223372036854775807
>>> f(x)
- ||| 9223372036854775807
? ----
+ 9223372036854775807
>>> f = abs
>>> x = 9223372036854775808
>>> f(x)
- ||| 9223372036854775808
? ----
+ 9223372036854775808
>>> f = abs
>>> x = -9223372036854775807
>>> f(x)
- ||| 9223372036854775807
? ----
+ 9223372036854775807
>>> f = abs
>>> x = -9223372036854775808
>>> f(x)
- ||| 9223372036854775808
? ----
+ 9223372036854775808
>>> f = abs
>>> x = 18446744073709551615
>>> f(x)
- ||| 18446744073709551615
? ----
+ 18446744073709551615
>>> f = abs
>>> x = 18446744073709551616
>>> f(x)
- ||| 18446744073709551616
? ----
+ 18446744073709551616
>>> f = abs
>>> x = 18446744073709551617
>>> f(x)
- ||| 18446744073709551617
? ----
+ 18446744073709551617
>>> f = abs
>>> x = -18446744073709551615
>>> f(x)
- ||| 18446744073709551615
? ----
+ 18446744073709551615
>>> f = abs
>>> x = -18446744073709551616
>>> f(x)
- ||| 18446744073709551616
? ----
+ 18446744073709551616
>>> f = abs
>>> x = -18446744073709551617
>>> f(x)
- ||| 18446744073709551617
? ----
+ 18446744073709551617
>>> f = abs
>>> x = 1361129467683753853853498429727072845824
>>> f(x)
- ||| 1361129467683753853853498429727072845824
? ----
+ 1361129467683753853853498429727072845824
>>> f = abs
>>> x = -1361129467683753853853498429727072845824
>>> f(x)
- ||| 1361129467683753853853498429727072845824
? ----
+ 1361129467683753853853498429727072845824
>>> f = abs
>>> x = 179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356329624224137216
>>> f(x)
- ||| 179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356329624224137216
? ----
+ 179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356329624224137216
>>> f = abs
>>> x = -179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356329624224137216
>>> f(x)
- ||| 179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356329624224137216
? ----
+ 179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356329624224137216
: Global context: Error running f(x)
@johnsolo15 Hrm - interesting. My money is still on encoding of some kind, but it's clearly not as simple as "non-english language" encoding. Since you're on Windows as well, CRLF encoding would be my next guess - but I'd need to dig in to the code on a Windows box to work out more. Anything you can discover would be very helpful.
I think I found the problem.
In tests/utils.py, line 424:
lines1 = code1.split(os.linesep)
It seems like the output from javascript does not respect the native line separator difference. Changing it to:
lines1 = code1.split('\n')
seems to solve the problem, but I'm not sure it's the right solution.
@lielfr Thanks. That fixes most of the problems, although I still get 4 errors. I'm going to see if I can do a little more investigating.
Hi,
I got the same issue when running from windows the extra /// and |||.
What I did to get away with these errors I changed in utils.py as in attached image.
This worked fine in windows . I am not sure why we /// and ||| in print statement.
Let me check fix reported by @lielfr .
Btw I see a commit on this one
https://github.com/pybee/batavia/pull/700/files#diff-0
@smitar0y Yes - you're not the first to suggest this fix :-). If you check the CI results for that PR, you'll see what happens to the rest of the test suite.
Checked with fixes suggested by @lielfr . The tests looks good on windows. I think we should implement this change.
@smitar0y Can you submit a pull request implementing this fix?
Sure .Working on it.
Done.
I am too getting some failures, i am using python 3.4 and windows 10, i ran the test-suite three times now, twice with npm version 5.6.0 and node version 8.9.5 i got the same 50 failures, i tried running the tests one at a time but it still gave me failure, so i tried changing my node version and still got 50 failures
am i doing something wrong? Here are the test logs
(venv) D:\git-repos\batavia>python --version
Python 3.4.3
(venv) D:\git-repos\batavia>npm run build
> @pybee/batavia@3.4.0-dev.19 build D:\git-repos\batavia
> webpack --progress
Caching is enabled.
Hash: c201238713bf5222e136
Version: webpack 2.2.1
Time: 13027ms
Asset Size Chunks Chunk Names
batavia.js 5.52 MB 0 [emitted] [big] batavia
[0] ./batavia/types.js 7.18 kB {0} [built]
[1] ./batavia/core.js 1.21 kB {0} [built]
[3] ./batavia/builtins.js 4.64 kB {0} [built]
[6] ./~/bignumber.js/bignumber.js 96.4 kB {0} [built]
[8] ./batavia/modules/sys.js 1.58 kB {0} [built]
[11] ./batavia/modules/dis.js 5.94 kB {0} [built]
[12] ./batavia/builtins/repr.js 1.01 kB {0} [built]
[13] ./batavia/builtins/tuple.js 382 bytes {0} [built]
[14] ./batavia/core/constants.js 718 bytes {0} [built]
[18] ./batavia/modules.js 823 bytes {0} [built]
[19] ./batavia/stdlib.js 7.16 kB {0} [built]
[146] ./batavia/VirtualMachine.js 78.7 kB {0} [built]
[217] ./batavia/modules/base64.js 3.68 kB {0} [built]
[417] ./batavia/types/Zip.js 1.94 kB {0} [built]
[426] ./batavia/batavia.js 693 bytes {0} [built]
+ 412 hidden modules
(venv) D:\git-repos\batavia>npm -v
5.6.0
(venv) D:\git-repos\batavia>node -v
v6.9.5
- tried running tests.builtins.test_chr.BuiltinChrFunctionTests.test_int
one thing i noticed is that f(x) is called it returns missing value
Traceback (most recent call last):
File "C:\Python34\lib\unittest\case.py", line 58, in testPartExecutor
yield
File "C:\Python34\lib\unittest\case.py", line 577, in run
testMethod()
File "D:\git-repos\batavia\tests\utils.py", line 1216, in func
transform_output=transform_output,
File "D:\git-repos\batavia\tests\utils.py", line 1253, in assertBuiltinFunction
transform_output=transform_output,
File "D:\git-repos\batavia\tests\utils.py", line 496, in assertCodeExecution
self.assertEqual(js_out, py_out, context)
File "C:\Python34\lib\unittest\case.py", line 797, in assertEqual
assertion_func(first, second, msg=msg)
File "C:\Python34\lib\unittest\case.py", line 1170, in assertMultiLineEqual
self.fail(self._formatMessage(msg, standardMsg))
File "C:\Python34\lib\unittest\case.py", line 642, in fail
raise self.failureException(msg)
AssertionError: ">>> [525 chars]'> : signed integer is greater than maximum\n\[2219 chars]\n\n" != ">>> [525 chars]'> : Python int too large to convert to C long[2234 chars]\n\n"
>>> f = chr
>>> x = 3
>>> f(x)
�
>>> f = chr
>>> x = 0
>>> f(x)
>>> f = chr
>>> x = -5
>>> f(x)
<class 'ValueError'> : chr() arg not in range(0xXXXXXXXX)
>>> f = chr
>>> x = -3
>>> f(x)
<class 'ValueError'> : chr() arg not in range(0xXXXXXXXX)
>>> f = chr
>>> x = 5
>>> f(x)
�
>>> f = chr
>>> x = 1
>>> f(x)
�
>>> f = chr
>>> x = -1
>>> f(x)
<class 'ValueError'> : chr() arg not in range(0xXXXXXXXX)
>>> f = chr
>>> x = 9223372036854775807
>>> f(x)
- <class 'OverflowError'> : signed integer is greater than maximum
+ <class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = 9223372036854775808
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = -9223372036854775807
>>> f(x)
- <class 'OverflowError'> : signed integer is less than minimum
+ <class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = -9223372036854775808
>>> f(x)
- <class 'OverflowError'> : signed integer is less than minimum
+ <class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = 18446744073709551615
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = 18446744073709551616
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = 18446744073709551617
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = -18446744073709551615
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = -18446744073709551616
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = -18446744073709551617
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = 1361129467683753853853498429727072845824
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = -1361129467683753853853498429727072845824
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = 179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356329624224137216
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
>>> f = chr
>>> x = -179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356329624224137216
>>> f(x)
<class 'OverflowError'> : Python int too large to convert to C long
: Global context: Error running f(x)
I can repro @Souldiv 's results.
(for_batavia) C:\Users\bildz\OneDrive\Documents\batavia>npm -v
6.0.1
(for_batavia) C:\Users\bildz\OneDrive\Documents\batavia>node -v
v8.11.1
n.b. this version of node.js seems to be the current LTS.
Are you still able to reproduce this @Souldiv or @bildzeitung? It looks on the surface like there was a skew in the error messages that we were reporting when converting large numbers.
I'm going to close this for now. If you can reproduce this still, please reopen the issue.