jeremyong/exprotoc

Parser isn't working properly

Closed this issue · 1 comments

I'm writing a new RethinkDB client driver, and am running into an issue with the parser:

** (MatchError) no match of right hand side value: {:error, {10, :proto_grammar, ['syntax error before: ', 'x3f61ba36']}}
    lib/exprotoc/parser.ex:36: Exprotoc.Parser.parse/1
    lib/exprotoc.ex:5: Exprotoc.compile/3
    (elixir) lib/enum.ex:534: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:534: Enum.each/2
    (mix) lib/mix/tasks/compile.ex:68: anonymous fn/2 in Mix.Tasks.Compile.run/1
    (elixir) lib/enum.ex:962: Enum."-map/2-lc$^0/1-0-"/2
    (mix) lib/mix/tasks/compile.ex:67: Mix.Tasks.Compile.run/1
    (mix) lib/mix/cli.ex:64: Mix.CLI.run_task/2

The .proto file I'm attempting to parse is RethinkDB's ql2.proto, which can be found here. I also maintain a fork of elixir-protobuf, and haven't had an issue with parsing this file. I'm looking to deprecate elixir-protobuf and use exprotoc, otherwise I'd just use that. If I get some time tonight or tomorrow, I'll take a look and see if I can find the source of the problem, but I'm sure you'll know where to look much faster than I will.

It seems like the tokenizing is wrong with respect to the byte values for the Version and Protocol enums. Those byte values are little-endian 32-bit integers, but are being tokenized like so:

[{:message, 9}, {:var, 9, :VersionDummy}, {:"{", 9}, {:enum, 9},
 {:var, 9, :Version}, {:"{", 9}, {:var, 10, :V0_1}, {:=, 10}, {:integer, 10, 0},
 {:atom, 10, :x3f61ba36}, {:";", 10}, {:var, 11, :V0_2}, {:=, 11},
 {:integer, 11, 0}, {:atom, 11, :x723081e1}, {:";", 11}, {:var, 11, :V0_3},
 {:=, 11}, {:integer, 11, 0}, {:atom, 11, :x5f75e83e}, {:";", 11}, {:"}", 11},
 {:enum, 13}, {:var, 13, :Protocol}, {:"{", 13}, {:var, 14, :PROTOBUF},
 {:=, 14}, {:integer, 14, 0}, {:atom, 14, :x271ffc41}, {:";", 14},
 {:var, 15, :JSON}, {:=, 15}, {:integer, 15, 0}, {:atom, 15, :x7e6970c7},
 {:";", 15}, {:"}", 16}, {:"}", 17},...]