Dashboard files in .ndjson format for ELK 8.x
Opened this issue · 7 comments
Is there a latest version of the dashboard JSON files in .ndjson format that is required for ELK 8.x?
Not at all but happy to accept any changes via PR! I haven't worked on this in a long while.
I converted the .json files to .ndjson. When trying to import any of the files in Management -> Saved Objects, I get the following error:
kibana_1 | [2023-10-11T08:28:14.618+00:00][WARN ][process] TypeError: Cannot destructure property 'title' of 'obj.attributes' as it is undefined.
kibana_1 | at /usr/share/kibana/node_modules/@kbn/core-saved-objects-import-export-server-internal/src/import/lib/collect_saved_objects.js:38:7
kibana_1 | at Transform.transform [as _transform] (/usr/share/kibana/node_modules/@kbn/utils/src/streams/filter_stream.js:20:33)
kibana_1 | at Transform._write (node:internal/streams/transform:175:8)
kibana_1 | at writeOrBuffer (node:internal/streams/writable:392:12)
kibana_1 | at _write (node:internal/streams/writable:333:10)
kibana_1 | at Transform.Writable.write (node:internal/streams/writable:337:10)
kibana_1 | at Transform.ondata (node:internal/streams/readable:766:22)
kibana_1 | at Transform.emit (node:events:514:28)
kibana_1 | at addChunk (node:internal/streams/readable:324:12)
kibana_1 | at readableAddChunk (node:internal/streams/readable:297:9)
kibana_1 | at Transform.Readable.push (node:internal/streams/readable:234:10)
kibana_1 | at node:internal/streams/transform:182:12
kibana_1 | at Transform.transform [as _transform] (/usr/share/kibana/node_modules/@kbn/core-saved-objects-import-export-server-internal/src/import/lib/create_limit_stream.js:26:7)
kibana_1 | at Transform._write (node:internal/streams/transform:175:8)
kibana_1 | at writeOrBuffer (node:internal/streams/writable:392:12)
kibana_1 | at _write (node:internal/streams/writable:333:10)
kibana_1 | at Transform.Writable.write (node:internal/streams/writable:337:10)
kibana_1 | at Readable.ondata (node:internal/streams/readable:766:22)
kibana_1 | at Readable.emit (node:events:514:28)
kibana_1 | at Readable.read (node:internal/streams/readable:539:10)
kibana_1 | at flow (node:internal/streams/readable:1023:34)
kibana_1 | at resume_ (node:internal/streams/readable:1004:3)
kibana_1 | at processTicksAndRejections (node:internal/process/task_queues:82:21)
I am using release-7.x branch from https://github.com/deviantony/docker-elk/tree/release-7.x. Any suggestions on how to fix the same? The ui-dashboards.ndjson file (cannot attach due to unsupported file type)
{"_id":"b6b2aea0-788f-11e8-a41e-1930b01c3fee","_type":"dashboard","_source.title":"[Filebeat Postfix] Overview","_source.hits":0,"_source.description":"","_source.panelsJSON":"[{\"panelIndex\":\"4\",\"gridData\":{\"x\":0,\"y\":8,\"w\":48,\"h\":11,\"i\":\"4\"},\"embeddableConfig\":{},\"id\":\"0609c7b0-7893-11e8-a41e-1930b01c3fee\",\"title\":\"Status over time\",\"type\":\"visualization\",\"version\":\"6.3.0\"},{\"panelIndex\":\"7\",\"gridData\":{\"x\":0,\"y\":0,\"w\":10,\"h\":8,\"i\":\"7\"},\"embeddableConfig\":{\"spy\":null},\"id\":\"2261e870-78ac-11e8-a41e-1930b01c3fee\",\"title\":\"Messages total\",\"type\":\"visualization\",\"version\":\"6.3.0\"},{\"panelIndex\":\"8\",\"gridData\":{\"x\":10,\"y\":0,\"w\":30,\"h\":8,\"i\":\"8\"},\"embeddableConfig\":{},\"id\":\"618e79d0-78ae-11e8-a41e-1930b01c3fee\",\"title\":\"Message status totals (click to filter)\",\"type\":\"visualization\",\"version\":\"6.3.0\"},{\"panelIndex\":\"9\",\"gridData\":{\"x\":0,\"y\":31,\"w\":48,\"h\":17,\"i\":\"9\"},\"embeddableConfig\":{},\"id\":\"4cdea1d0-7891-11e8-a41e-1930b01c3fee\",\"title\":\"Logs\",\"type\":\"search\",\"version\":\"6.3.0\"},{\"panelIndex\":\"10\",\"gridData\":{\"x\":40,\"y\":0,\"w\":8,\"h\":8,\"i\":\"10\"},\"title\":\"Manual (click to filter)\",\"embeddableConfig\":{},\"id\":\"480e8b10-78b0-11e8-a41e-1930b01c3fee\",\"type\":\"visualization\",\"version\":\"6.3.0\"},{\"panelIndex\":\"11\",\"gridData\":{\"x\":24,\"y\":19,\"w\":24,\"h\":12,\"i\":\"11\"},\"embeddableConfig\":{},\"id\":\"49274ce0-7896-11e8-a41e-1930b01c3fee\",\"title\":\"Top senders\",\"type\":\"visualization\",\"version\":\"6.3.0\"},{\"panelIndex\":\"12\",\"gridData\":{\"x\":0,\"y\":19,\"w\":24,\"h\":12,\"i\":\"12\"},\"embeddableConfig\":{},\"id\":\"cf5cb2f0-7891-11e8-a41e-1930b01c3fee\",\"title\":\"Hosts\",\"type\":\"visualization\",\"version\":\"6.3.0\"}]","_source.optionsJSON":"{\"darkTheme\":false,\"hidePanelTitles\":false,\"useMargins\":true}","_source.version":1,"_source.timeRestore":false,"_source.kibanaSavedObjectMeta.searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[],\"highlightAll\":true,\"version\":true}"}
I don't have a running version to test with, but I think we need to adjust the format/content of the ndjson to match the newer Kibana structure:
{
"attributes": {
"title": "[Filebeat Postfix] Overview",
"hits": 0,
"description": "",
"panelsJSON": "…",
"optionsJSON": "…",
"version": 1,
"timeRestore": false,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "…"
}
},
"id": "b6b2aea0-788f-11e8-a41e-1930b01c3fee",
"type": "dashboard"
}Really appreciate you working to get things fixed up, btw.
With release-7.x branch, I get:
Sorry, there was an error
The file could not be processed due to error: "Bad Request: Unexpected end of JSON input"
Is there a way to validate the .ndjson file before trying to import the same?
If you've got a shell with Python handy you could do something like:
import json
from pathlib import Path
def validate_ndjson(file_path):
try:
# Reading and parsing each line as JSON
[json.loads(line) for line in Path(file_path).read_text().splitlines()]
return "Valid NDJSON"
except json.JSONDecodeError:
return "Invalid NDJSON"
# Validating the NDJSON file
validate_ndjson("foo.ndjson")I have created a PR that cleanly imports api-combined.ndjson, ui-dashboards.ndjson, and ui-visualizations.ndjson files in Elasticsearch v 8.10.2.
But, when I try to open Dashboards -> [Filebeat Postfix] Overview it throws the following error:
Cannot read properties of undefined (reading 'searchSourceJSON')
I removed the searchSourceJSON field from kibanaSavedObjectMeta key. Is this mandatory?