very long warmup on some queries

[trinity-1686a]

observed on Airmail, on a query of the form

query

{
  "type": "bool",
  "must": [
    { 
      "type": "range",
      "field": "timestamp",
      "lower_bound": "Unbounded",
      "upper_bound": {
        "Excluded": 123456
      }
    },
    { 
      "type": "term",
      "field": "field1",
      "value": "val1"
    },
    { 
      "type": "phrase_prefix",
      "field": "field2",
      "phrase": "val2",
      "max_expansions": 50,
      "params": { 
        "mode": {
          "type": "phrase"
        }
      }
    },
    { 
      "type": "user_input",
      "user_text": "my-query*",
      "default_fields": [
        "field3",
        "field4"
      ],
      "default_operator": "Or"
    }
  ]
}

many splits look fine, but some take more than 10s to warmup, including 100ms of compute during warmup, while the search takes 15ms. It's unclear if this is something particular about these splits and this query, or if this is sign of an overloaded system

[trinity-1686a]

i found very simple term queries with the same issue, sometime even worse. I'm ruling this as an overloaded system issue