tree-sitter/tree-sitter-ruby

heredoc_content doesn't return a token

mjambon opened this issue · 1 comments

Input:

$ cat heredoc2.rb 
query = <<-SQL
SELECT * FROM #{table}
WHERE #{type} = true
SQL

The current tree-sitter parse doesn't show tokens:

$ tree-sitter parse heredoc2.rb 
(program [0, 0] - [4, 0]
  (assignment [0, 0] - [0, 14]
    left: (identifier [0, 0] - [0, 5])
    right: (heredoc_beginning [0, 8] - [0, 14]))
  (heredoc_body [0, 14] - [3, 3]
    (heredoc_content [0, 14] - [1, 14])
    (interpolation [1, 14] - [1, 22]
      (identifier [1, 16] - [1, 21]))
    (heredoc_content [1, 22] - [2, 6])
    (interpolation [2, 6] - [2, 13]
      (identifier [2, 8] - [2, 12]))
    (heredoc_content [2, 13] - [3, 0])
    (heredoc_end [3, 0] - [3, 3])))

The new output (work in progress https://github.com/ahlinc/tree-sitter/pull/4) show a missing token for heredoc_content:
image

We'd like to get its content in the CST like the other tokens (leaves of the tree) e.g. like heredoc_end=SQL.

I was mistaken here. It's just that multiline tokens aren't printed out by the tool yet.