managing index templates in elasticsearch
WassimDhib opened this issue · 9 comments
Hi,
When using this jmeter plugin, I noticed that the mapping in elasticsearch indexes was not consistent.
Sometimes date fields are mapped to long type.
"Timestamp" : {
"type" : "long"
},
I suggest to use an index template to define mapping explicitly.
PUT _template/jmeter-2.6.10
{
"index_patterns":[
"jmeter-2.6.10-*"
],
"aliases" : { },
"mappings" : {
"properties" : {
"AllThreads" : {
"type" : "long"
},
"BodySize" : {
"type" : "long"
},
"Bytes" : {
"type" : "long"
},
"ConnectTime" : {
"type" : "long"
},
"ContentType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"DataType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ElapsedTime" : {
"type" : "date"
},
"ErrorCount" : {
"type" : "long"
},
"FailureMessage" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"GrpThreads" : {
"type" : "long"
},
"IdleTime" : {
"type" : "long"
},
"InjectorHostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"Latency" : {
"type" : "long"
},
"ResponseCode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ResponseTime" : {
"type" : "long"
},
"SampleCount" : {
"type" : "long"
},
"SampleEndTime" : {
"type" : "date"
},
"SampleLabel" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"SampleStartTime" : {
"type" : "date"
},
"SentBytes" : {
"type" : "long"
},
"Success" : {
"type" : "boolean"
},
"TestElement" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"TestStartTime" : {
"type" : "date"
},
"ThreadName" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"Timestamp" : {
"type" : "date"
},
"URL" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"settings" : {
"index" : {
"number_of_shards" : "1",
"number_of_replicas" : "1"
}
}
}
Hi @WassimDhib,
The field timestamp should technically always be a long - it's a timestamp in millisecond. For a full list of types used when sending the documents to the index:
As well as JMeter's documentation:
https://jmeter.apache.org/api/org/apache/jmeter/samplers/SampleResult.html
Yes, from the point of view of jmeter, the timestamp is always a long
But, if you don't provide mapping information, elasticsearch won't be able to map this field to a date (the behavior may be inconsistent).
from elasticsearch docs :
You know more about your data than Elasticsearch can guess, so while dynamic mapping can be useful to get started, at some point you will want to specify your own explicit mappings.
It's a problem since your time series require a field with date type to create charts/visualizations
Are you using Kibana or Grafana to generate your dashboard? Because I have been using Grafana with the dynamic mapping and never really had any issues.
I use kibana
kibana being generally less aimed towards time series graphs, I'd suggest using Grafana to create your dashboards.
As for the change, you are more than welcome to suggest the changes through a pull request 😄
I had the same problem, solved it with a template. Would like to point out that while kibana is geared towards time series, its grouping/bucketing of graphs is quite ahead of Graphana.
It's not only a kibana problem, since the mapping is also responsible of optimizing the disk space allocated to indices in elasticsearch .
You can also specify index settings in template to improve indexing speed (number of primary shards, refresh interval ... ).
I'll take the time to make a PR to be able to manage templates.
Thanks, the timeout issue is resolved.
What would be the way to resolve the timestamp as it is displayed in ms to be used in Kibana?
E.G. currently "Timestamp": 1590995355655
Can we get the same format - yyyy-MM-dd'T'HH:mm:ss.SSSZZ
which would also apply to
"ElapsedTime": 1577829723000,
"TestStartTime": 1590995232368,
I'm new with ELK. I have already activated the k8s pods on ELK and Grafana. I have already created a test plan with the plugin. Grafana can communicate with ELK, but I don't know how to import this model index. Could someone help me?