Limit number of tokens in pattern ingester

[trevorwhitney]

Is your feature request related to a problem? Please describe.
The Pattern Ingesters OOM too often.

Describe the solution you'd like
In addition to limiting the number of patterns per stream, which we already do, we should also evict patterns with too many tokens, as that requires a lot of memory to store, and is less useful to a user anyway.

Describe alternatives you've considered

• Drop more patterns
• Run the pattern ingesters with more memory

[benclive]

Also worth revisiting the "state" generated by the line_tokenizer. It's used to reconstruct a LogQL matcher from the pattern, and I implemented it as an opaque interface{}, but in hindsight its always an int[] which can contain a lot of data so maybe we should cull patterns based on that too.