Failed to read proto message: unexpected EOF

Question

Failed to read proto message: unexpected EOF

haroon-sheikh opened this issue 6 years ago · 55 comments

haroon-sheikh commented 6 years ago

Expected behavior

Tell us what should happen

Actual behavior

Continuously logs the following error message and stuck in a loop at the end of execution

Failed to read proto message: unexpected EOF
Failed to read proto message: unexpected EOF
Failed to read proto message: unexpected EOF
...

Gauge version

Gauge version: 1.0.4
Commit Hash: 3a9a647

Plugins
-------
html-report (4.0.7)
java (0.7.1)
screenshot (0.0.1)
spectacle (0.1.3)
xml-report (0.2.1)

Answer 1 · 2019-02-11T13:26:33.000Z

Sometimes see this warning instead.

Plugin [Html Report] with pid [200] did not exit after 30.00 seconds. Forcefully killing it.

Answer 2 · 2019-02-14T10:37:30.000Z

I'm getting a loop of following message thrown at the end of the execution:

[html-report] 2019/02/12 14:21:49 Failed to read proto message: proto: can't skip unknown wire type 6\n
[html-report] 2019/02/12 14:21:49 Failed to read proto message: proto: can't skip unknown wire type 6\n
[html-report] 2019/02/12 14:21:49 Failed to read proto message: proto: can't skip unknown wire type 6\n

Answer 3 · 2019-02-14T17:02:07.000Z

On some runs I'm seeing
[html-report] 2019/02/14 15:31:32 Failed to read proto message: proto: can't skip unknown wire type 4
repeated many, many times.

Gauge version: 1.0.4
Commit Hash: 3a9a647

Plugins
-------
html-report (4.0.7)
java (0.7.1)
xml-report (0.2.1)

Answer 4 · 2019-02-14T17:06:14.000Z

Looks like this is happening after the upgrade from 4.0.6 to 4.0.7

Answer 5 · 2019-02-14T17:11:15.000Z

yeah looks like 4.0.7 is having problems with killing itself.

Answer 6 · 2019-02-14T17:21:27.000Z

Looks like this is happening after the upgrade from 4.0.6 to 4.0.7

I'm switching back to HTML-report 4.0.6, then will run my regression suite then will keep an eye on console log file sizes, to confirm this is a 4.0.7 issue.

Answer 7 · 2019-02-21T06:23:30.000Z

@haroon-sheikh @ocary Could you please try to install latest nightlies of gauge and html-report and see if they fix this issue?

Answer 8 · 2019-02-21T20:35:26.000Z

@BugDiver -

Next week, I'll see what I can do to get a nightly installed in the env that was having issues. 50% chance of being successful due to other limitations.
There is an issue with html-report 4.0.7+ with the content that goes into xml-report. Defect opened in xml-report. getgauge/xml-report#16

Answer 9 · 2019-02-26T12:02:52.000Z

This issue was there with html-report 4.0.6 as well. i upgraded to 4.0.7 to see if the issue would be resolved. It did not.
This has happened to me on three separate machines.
Is this issue due to large number of test scenarios (1200+ in my case) being executed as part of a suite?
Also, when this issue happens, an hour+ long execution and its results are lost. is there a way where we can salvage the results by re-running the html-report plugin against some raw results or execution history being stored somewhere?

Answer 10 · 2019-03-05T15:29:15.000Z

any update on this?

Answer 11 · 2019-03-06T10:26:36.000Z

I am investigating this. I do see the issue occur intermittently when running on html-report 4.0.7 in a low-spec machine. But this is not consistent enough to yield a possible cause.

With html-report 4.0.6, I haven't been able to see this issue.

@alpha1592 - do you run your specs in parallel?

Answer 12 · 2019-03-06T16:25:54.000Z

Yes, i usually run the full suite in parallel mode. I have mostly seen this error when running in parallel and a large number of scenarios. However, I have seen this error in single threaded mode as well. This issue has also been seen on two very different hardware specs as well (below). I have never seen this issue when running a spec file here or there through intelliJ. Only seen this error when running from command line outside of IntelliJ. Hope this helps...

in \env\default\default.properties i have enable_multithreading = false

Command to Start the Test Suite (Single) - takes 1 hour to execute
mvn gauge:execute -DspecsDir=specs/

Command to Start the Test Suite (In Parallel) - takes 25 mins to execute
mvn gauge:execute -DspecsDir=specs/ -DinParallel=true -Dnodes=6

Hardware Specs
Server 1 (run only in single mode)

Windows Server 2012 VM x64
2 vCores
4 GB RAM

Server 2

Windows Server 2012 VM x64
8 vCores
16 GB RAM

Tests

35 Specs (Some specs are over 5,000 lines)
1,200 Scenarios

Answer 13 · 2019-03-07T05:13:10.000Z

@alpha1592 - what is the output of gauge -v on the machine where you see this issue?

Answer 14 · 2019-03-07T12:08:44.000Z

Gauge version: 1.0.4
Commit Hash: 3a9a647

Plugins
-------
flash (0.0.1)
html-report (4.0.7)
java (0.7.1)
json-report (0.2.2)
screenshot (0.0.1)
spectacle (0.1.3)
xml-report (0.2.1)

Manifest.json

{
  "Language": "java",
  "Plugins": [
    "html-report",
    "flash"
  ]
}

Java Version

java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

Answer 15 · 2019-03-07T13:58:39.000Z

I'm seeing a similar behaviour when it's a larger set of suites running in parallel.

Answer 16 · 2019-03-08T11:20:56.000Z

hi @alpha1592 @haroon-sheikh - can you please try the latest html-report nightly build and see if this is better?

Essentially we implemented a change (ref getgauge/gauge#1239) which allowed gauge to send data to the html-report in chunks. I have disabled this change, since I noticed that the chunks weren't getting through properly in certain cases. And this isn't consistent. I have disabled this capability.

Answer 17 · 2019-03-08T14:52:43.000Z

I'll try it today and let you know.

Answer 18 · 2019-03-08T17:01:43.000Z

I'll give it a try...
How is it expected to work now...
will html-report going to create the report as execution is going... or is it done all at the end of the test execution?

Answer 19 · 2019-03-11T02:39:15.000Z

html-report is still generated at the end of execution. There needs to be some changes in gauge to enable reports receiving data impromptu.

Answer 20 · 2019-03-12T12:00:19.000Z

@haroon-sheikh @alpha1592 Were you able to give this a try? If this fixes the unexpected EOF issue, we can look at making a html report plugin release.

Answer 21 · 2019-03-12T12:27:59.000Z

@nehashri I'm not able to reproduce the issue anymore with the nightly.

Answer 22 · 2019-03-12T13:50:52.000Z

I have run the suite twice and have not seen this issue either... Although, keep in mind, this was a intermittent issue. I will continue to run the tests a few more times to ensure its fully gone.

Answer 23 · 2019-03-12T15:12:30.000Z

after switching to the nightly build (gauge-java-0.7.2.nightly-2019-03-07), i am seeing a warning show up which wasn't coming up before.
Not sure why i am getting this warning now... as Gauge-Java-0.7.1 uses the same dependency, but never saw the warning before. I also switched to nighly html-report and gauge 0.0.5

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/.m2/repository/com/google/protobuf/protobuf-java/3.6.1/protobuf-java-3.6.1.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Maven Dependency

<dependency>
  <groupId>com.thoughtworks.gauge</groupId>
  <artifactId>gauge-java</artifactId>
  <version>0.7.2.nightly-2019-03-07</version>
</dependency>

Gauge version: 1.0.5.nightly-2019-03-08
Commit Hash: 96f9960

Plugins
-------
flash (0.0.1)
go (0.1.2)
html-report (4.0.8)  -> html-report-4.0.8.nightly-2019-03-08-windows.x86_64
java (0.7.2)
json-report (0.2.2)
screenshot (0.0.1)
spectacle (0.1.3)
xml-report (0.2.1)

Answer 24 · 2019-03-12T15:27:22.000Z

Didn't get the EOF error, but i have been getting errors like this one as well. This was outlined in my original request.

[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)
[html-report] 2019/03/12 09:59:05 Failed to read proto message: proto: gauge_messages.StepNamesResponse: illegal tag 0 (wire type 4)

Only updated Gauge and HTML-Reports to nightly build

Gauge version: 1.0.5.nightly-2019-03-08
Commit Hash: 96f9960

Plugins
-------
flash (0.0.1)
html-report (4.0.8.nightly-2019-03-08)
java (0.7.1)
json-report (0.2.2)
screenshot (0.0.1)
spectacle (0.1.3)
xml-report (0.2.1)

Answer 25 · 2019-03-19T18:47:12.000Z

any update on this?

Answer 26 · 2019-03-22T07:43:58.000Z

Managed to get the error with a huge suite that had around 3000 specs. We will be looking at fixing this soon by streaming results to report plugins instead of sending them together at the end.

Answer 27 · 2019-03-26T06:31:48.000Z

We have released a new version of html-report plugin 4.0.8 which disables capability to receive data in chunks, that reduces the probability of getting Failed to read proto message: unexpected EOF. Still the issue can be observed in very large test suites, which we will track as part of #218

Answer 28 · 2019-08-27T05:50:55.000Z

Still able to see this error. https://ci.appveyor.com/project/getgauge/gauge-dotnet/builds/26957493

Answer 29 · 2019-08-28T13:57:11.000Z

This continues to be an issue for us, and a pretty annoying one at that. We are on html-report v4.0.8.
any new releases for html-report scheduled soon?

Answer 30 · 2019-08-28T16:49:23.000Z

It started happening when I upgraded from gague 1.0.5 to 1.0.6.

Switched back to gauge 1.0.5 and html-report 4.0.6 and issue does not appear anymore.

Answer 31 · 2019-08-28T19:39:21.000Z

I am also now having this issue. It only just starts today after upgrading @getgauge/cli to 1.0.6

Plugins

flash (0.0.1)
html-report (4.0.8)
js (2.3.5)
json-report (0.3.1)
screenshot (0.0.1)
spectacle (0.1.3)
xml-report (0.2.1)

Answer 32 · 2019-08-28T19:53:33.000Z

I am still on 1.0.5. still having the issue.

Answer 33 · 2019-08-29T05:16:32.000Z

@bimalviswam @adea11 @alpha1592 we will try to fix this issue soon. We have also started to work on #221 to fix this issue in a proper way.

Answer 34 · 2019-09-20T09:19:25.000Z

html-report is fine for me. it's json-report that's producing this:

Answer 35 · 2019-09-20T09:23:29.000Z

@nehashri Do we have an update on this please. This is happening a lot more with the latest version of gauge. 1.0.5 seems to be more stable.

Answer 36 · 2019-09-23T09:47:16.000Z

chuv13 commented 5 years ago

Answer 37 · 2019-10-21T08:45:37.000Z

@chuv13 latest json-report is released, can you please see if it fails with the latest too?

Answer 38 · 2019-11-12T22:52:29.000Z

any updates on this?
Gauge updates have slowed down recently...? any reason why we don't see faster updates?

Answer 39 · 2019-11-13T08:51:03.000Z

To keep track of updates please follow https://github.com/orgs/getgauge/projects/6

Answer 40 · 2019-11-13T13:53:53.000Z

periodically i have the same issue, today i had it again with next configuration:

2019/11/13 03:53:29 Failed to read proto message: unexpected EOF
2019/11/13 03:53:29 Failed to read proto message: unexpected EOF
03:53:29 [Thread-1] INFO config.DefaultHooks - 13.11.2019 03:53:29   ### 'click ...' step started ###
2019/11/13 03:53:29 Failed to read proto message: unexpected EOF
2019/11/13 03:53:29 Failed to read proto message: unexpected EOF
2019/11/13 03:53:29 Failed to read proto message: unexpected EOF
2019/11/13 03:53:29 Failed to read proto message: unexpected EOF
[html-report] 2019/11/13 03:53:29 Failed to read proto message: proto: can't skip unknown wire type 6
[html-report] 2019/11/13 03:53:29 Failed to read proto message: proto: can't skip unknown wire type 6
[html-report] 2019/11/13 03:53:29 Failed to read proto message: proto: can't skip unknown wire type 6
[html-report] 2019/11/13 03:53:29 Failed to read proto message: proto: can't skip unknown wire type 6

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)


Gauge version: 1.0.6
Commit Hash: 2bc49db

Plugins
-------
flash (0.0.1)
html-report (4.0.8)
java (0.7.3)
screenshot (0.0.1)

started using maven plugin with args: -DinParallel=true -Dnodes=2 -Denv=ext -Dtags=regression

may be it's somehow connected with network, our company network is very complex and sometimes internet connection can be lost/switched...
I'm so tired about this problem and thinking about another BDD framework...

Answer 41 · 2019-11-13T13:57:37.000Z

@savin0ff can you try downgrading it to 1.0.5 till it's fixed?

Answer 42 · 2019-11-14T06:36:25.000Z

ok, i'll try to downgrade gauge...and again it happened with -Dnodes=1

[html-report] 2019/11/14 05:36:28 Failed to read proto message: proto: gauge_messages.ImplementationFileGlobPatternResponse: illegal tag 0 (wire type 3)
[html-report] 2019/11/14 05:36:28 Failed to read proto message: proto: gauge_messages.ImplementationFileGlobPatternResponse: illegal tag 0 (wire type 3)
[html-report] 2019/11/14 05:36:28 Failed to read proto message: proto: gauge_messages.ImplementationFileGlobPatternResponse: illegal tag 0 (wire type 3)
2019/11/14 05:36:28 Failed to read proto message: proto: bad wiretype for field gauge_messages.Message.MessageType: got wiretype 2, want 0
2019/11/14 05:36:28 Failed to read proto message: proto: bad wiretype for field gauge_messages.Message.MessageType: got wiretype 2, want 0
2019/11/14 05:36:28 Failed to read proto message: proto: bad wiretype for field gauge_messages.Message.MessageType: got wiretype 2, want 0
2019/11/14 05:36:28 Failed to read proto message: proto: bad wiretype for field gauge_messages.Message.MessageType: got wiretype 2, want 0

Answer 43 · 2019-11-15T12:48:19.000Z

We are working on a fix for this issue as disscused in getgauge/gauge#1476.

Answer 44 · 2019-12-27T06:13:59.000Z

We have pushed some fixes to Gauge, Gauge-Java and html-report to address this issue. Please give it a try and provide your valuable feedback.

Note: You will have to build all of them( Gauge, Gauge-Java and html-report) from source as we have not released them yet.

Build Gauge from source.
Prerequisite: Go lang 1.13.4

run go run build/make.go && go run build/make.go --install
Build Html-Report from source.
Prerequisite: Go lang 1.13.4, Gauge

run go run build/make.go && go run build/make.go --install
Build Gauge Java from source.

Answer 45 · 2020-01-06T15:57:28.000Z

We've been having issues similar to the ones listed here. We were originally getting a type 5 or 6 error. We've tried downgrading the various plugins and gauge install, and now we are back on gauge 1.0.4 java 0.6.9 and we are still getting the EOF error.

So today I started looking into building from source gauge and gauge java, however the environments we run gauge on are VMs which do not have access to the internet, only the intranet, so I have to make distros to test them out.

I've installed NSIS so the --distro option would have access to the makensis executable, however when I try to run this I get the following error:

$ go run build/make.go && go run build/make.go --distro
Command line defined: "PRODUCT_VERSION=1.0.7"
Command line defined: "GAUGE_DISTRIBUTABLES_DIR=D:\Users\***\Repositories\Github\gauge\deploy\gauge-1.0.7-windows.x86_64"
Command line defined: "OUTPUT_FILE_NAME=D:\Users\***\Repositories\Github\gauge\deploy\gauge-1.0.7-windows.x86_64.exe"
Processing config: D:\Users\***\Programs\NSIS\nsisconf.nsh
Processing script file: "build\install\windows\gauge-install.nsi" (ACP)
Error: unterminated string parsing line at macro:_IncludeStrFunction:7
Error in macro _IncludeStrFunction on macroline 7
!include: error in script: "EnvVarUpdate.nsh" on line 49
Error in script "build\install\windows\gauge-install.nsi" on line 21 -- aborting creation process
panic: exit status 1

goroutine 1 [running]:
main.runProcess(0x6ebcf6, 0xc, 0xc000097dd8, 0x4, 0x4)
        D:/Users/***/Repositories/Github/gauge/build/make.go:64 +0x19c
main.createWindowsInstaller()
        D:/Users/***/Repositories/Github/gauge/build/make.go:280 +0x4ca
main.createWindowsDistro()
        D:/Users/***/Repositories/Github/gauge/build/make.go:268 +0x3c
main.createDistro()
        D:/Users/***/Repositories/Github/gauge/build/make.go:263 +0x95
main.createGaugeDistributables(0xc000016900)
        D:/Users/***/Repositories/Github/gauge/build/make.go:256 +0x1af
main.main()
        D:/Users/***/Repositories/Github/gauge/build/make.go:190 +0x11c
exit status 2

Also as an additional question, it seems these issues are related to changes around screenshots? If so, would turning off screenshots prevent this issue until a fix has been officially released?

Answer 46 · 2020-01-09T06:34:29.000Z

Notes for testing: Make sure screenshots are available in the report for the following conditions.

When Gauge, runners, and HTML-Report are installed from the Github master branch.
When Gauge is installed from the Github master branch and runners and HTML-Report are the released one.

Note: Screenshots will not be available in the report when one of them( runners, HTML-Report) is from the Github master branch or released one.

New API to implement custom screenshots.
We have introduced a new custom_screenshot_writer API to take custom screenshots. While implementing the new API user should take care of capturing and writing the screenshot into the location configured by screenshots_dir env variable.

Answer 47 · 2020-01-16T08:16:05.000Z

This issue has been verified and found fixed against the below versions:

Gauge version: 1.0.7
Commit Hash: 50fb6484

Plugins
-------
html-report (4.0.9)
java (0.7.4)

Answer 48 · 2020-01-16T10:45:04.000Z

@Debashis9012 When is 1.0.7 planned to be released?

Answer 49 · 2020-01-17T06:26:13.000Z

@haroon-sheikh Most probably we will release gauge 1.0.7 by next week.

Answer 50 · 2020-01-17T07:23:50.000Z

Reopening it as we need to test gauge-python and gauge-ruby runners and also we need to update the gauge documentation for custom_screenshot_writer

Answer 51 · 2020-01-24T17:34:24.000Z

Hi folks, any news on the gauge 1.0.7 release?

Answer 52 · 2020-01-24T17:53:25.000Z

On the last mile of testing, the latest changes. This should come out by next week.

Answer 53 · 2020-01-29T11:55:29.000Z

An update: We detected some issues with the recent changes that sometimes messes up multi threaded parallel execution.

This is holding up the release(s). We are working on this.

Answer 54 · 2020-02-21T13:16:20.000Z

Still getting EOF failed to read proto with the following versions

Your Environment Information -----------
linux, 1.0.7, ed7b4fd6
html-report (4.0.9), java (0.7.3), spectacle (0.1.4), xml-report (0.2.3)
.......................

Running in 48 parallel

Answer 55 · 2020-03-04T00:55:47.000Z

hi @jonathangiber - please update gauge-java to 0.7.4 and try.