Thrift deserialization error in GetWorkflowExecutionHistoryResponse when getting history of archived workflows

[emrahs]

Quoting the slack conversation:
https://uber-cadence.slack.com/archives/CL22WDF70/p1586990821382300

--
hi team, cadence-web communication with cadence-server is throwing following error :

Expected Thrift encoded arg3 for endpoint WorkflowService::GetWorkflowExecutionHistory.
Got
 instead of Thrift.
Parsing error was: unexpected typeid 11 (STRING) for field "archived" with id 30 on GetWorkflowExecutionHistoryResponse; expected 2 (BOOL).
    at createError (/Users/cdeshpande/code/cadence-web/node_modules/error/typed.js:40:22)
    at Object.createError [as ThriftBodyParserError] (/Users/cdeshpande/code/cadence-web/node_modules/error/wrapped.js:99:19)
    at TChannelAsThrift.parse [as _parse] (/Users/cdeshpande/code/cadence-web/node_modules/tchannel/as/thrift.js:414:35)
    at handleResponse (/Users/cdeshpande/code/cadence-web/node_modules/tchannel/as/thrift.js:338:32)
    at gotArg23 (/Users/cdeshpande/code/cadence-web/node_modules/tchannel/request.js:243:13)
    at compatCall (/Users/cdeshpande/code/cadence-web/node_modules/tchannel/streaming_in_response.js:93:9)
    at end (/Users/cdeshpande/code/cadence-web/node_modules/run-parallel/index.js:18:15)
    at done (/Users/cdeshpande/code/cadence-web/node_modules/run-parallel/index.js:22:10)
    at each (/Users/cdeshpande/code/cadence-web/node_modules/run-parallel/index.js:28:7)
    at /Users/cdeshpande/code/cadence-web/node_modules/run-parallel/index.js:38:43
    at signalReady (/Users/cdeshpande/code/cadence-web/node_modules/ready-signal/index.js:21:26)
    at StreamArg.finish (/Users/cdeshpande/code/cadence-web/node_modules/tchannel/argstream.js:284:9)
    at StreamArg.emit (events.js:198:13)
    at endReadableNT (_stream_readable.js:1129:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  type: 'tchannel-thrift-handler.parse-error.body-failed',
  name: 'TchannelThriftHandlerParseErrorBodyFailedError',
  message:
   'Could not parse body (arg3) argument.\nExpected Thrift encoded arg3 for endpoint WorkflowService::GetWorkflowExecutionHistory.\nGot \f\u0000\u0000\f\u0000\n\u000f\u0000\n\f instead of Thrift.\nParsing error was: unexpected typeid 11 (STRING) for field "archived" with id 30 on GetWorkflowExecutionHistoryResponse; expected 2 (BOOL).\n',
  isSerializationError: true,
  endpoint: 'WorkflowService::GetWorkflowExecutionHistory',
  direction: 'in.response',
  ok: true,
  bodyBuf: <Buffer 0c 00 00 0c 00 0a 0f 00 0a 0c>,
  causeMessage:
   'unexpected typeid 11 (STRING) for field "archived" with id 30 on GetWorkflowExecutionHistoryResponse; expected 2 (BOOL)',
  origMessage:
   'unexpected typeid 11 (STRING) for field "archived" with id 30 on GetWorkflowExecutionHistoryResponse; expected 2 (BOOL)',
  fullType:
   'tchannel-thrift-handler.parse-error.body-failed~!~thrift-unexpected-field-value-typeid' }
ℹ ｢hot｣: WebSocket Client Connected
{ TchannelThriftHandlerParseErrorBodyFailedError: Could not parse body (arg3) argument.
Expected Thrift encoded arg3 for endpoint WorkflowService::GetWorkflowExecutionHistory.
Got

I have checked out latest cadence-web code and starting it using npm run dev

--

Hey @chitresh Deshpande, That sounds like a mismatch between the IDLs used by cadence-web and cadence server.
First of all, please make sure you checkout a released version both for the server and on the cadence-web (see the git tags). That would rule out random bugs that might be in the latest master.
If you checkout the latest versions from both repositories and still have the issue, please let me know what versions you are running.
If you are unable to upgrade cadence server, try using v3.13.2 of the cadence-web for now. I think I know which change in v3.14.0 is causing the incompatibility.

--

I tried both v3.13.2 and v3.14.0 tags for cadence-web and both are running into same issue. I can't upgrade the server, server versoin is v0.11.

Found the issue.. the above error is only when fetching history of archived workflows

[emrahs]

My first thought was an IDL mismatch between the server and cadence-web, however, that doesn't appear to be the reason. Based on what Chitresh said, the IDL versions are as follows:

cadence-web@v.3.13.2 => https://github.com/uber/cadence-idl/blob/db27bb7ac668de0b4ba582b4de9bbc280b5a1243/thrift/shared.thrift

cadence@v0.11 => https://github.com/uber/cadence/blob/v0.11.0/idl/github.com/uber/cadence/shared.thrift

Both have:

struct GetWorkflowExecutionHistoryResponse {
  10: optional History history
  20: optional binary nextPageToken
  30: optional bool archived
}

That struct was recently modified as follows, but that was only included in cadence@v0.12:

struct GetWorkflowExecutionHistoryResponse {
  10: optional History history
  11: optional list<DataBlob> rawHistory
  20: optional binary nextPageToken
  30: optional bool archived
}

I will ask Chitresh to double-check if they ever ran cadence@v0.12 or the master branch with a commit after v0.11. In the meantime, it would be nice to hear if @just-at-uber @yycptt or @meiliang86 have any other ideas.

[chitreshd]

Hi @emrahs : Thanks for looking into this. We recently upgraded from v0.10.x to v0.11 and never ran v0.12. Also, I am seeing the same error when querying workflow history of the workflows archived a day ago. I am sure these were executed solely on v0.11 version. Let me know, if you need any more information from our end.

[just-at-uber]

One thing to check for cadence-web is try running npm install to make sure the idl's are updated correctly. Otherwise it wont update these idls when pulling latest version of cadence-web

[emrahs]

Hey @chitreshd , can you also try using the Cadence CLI tool to try showing the history of the same archived workflow. If it works fine, we'll be more confident that this in fact is a cadence-web issue.

[chitreshd]

Cadence CLI works and i did to clean npm install to weed out any cache issues.

[just-at-uber]

Hi @chitreshd ,
Looks like cadence-web thrift version is ahead of cadence-server thrift version which is where the incompatibility is coming from.
You can try manually changing the thrift version to an earlier version and see if that works?

Basically you need to patch the package.json with this change:

"napa": {
"cadence-idl": "uber/cadence-idl#a2eea81f48247860ff0ec8d85ef218e99195bb7f"
}

then run npm install which will run napa and fetch the idl files for that commit version.

[just-at-uber]

Hi @chitreshd ,

I'm closing as we have posted a workaround for this issue.
If the workaround does not work for you, feel free to open and will take a second look.

Thanks!