Handler fails deserialising event.json file output from Downloader module

[lawrencegripper]

k logs 8d645dbd335548d48f2dcb6707c76a4e-v0-s6tx7
time="2018-07-04T10:41:49Z" level=info msg="Starting handler"
time="2018-07-04T10:41:49Z" level=info msg="Committing module's environment to the data plane" correlationID=987b2dee-57a4-4d35-b670-2cad6867dc58 eventID=a03ab51d-6939-4bb9-876d-ba64e79a6161 name=downloader timestamp="2018-07-04 10:41:49.290936294 +0000 UTC m=+0.172202852"
time="2018-07-04T10:41:49Z" level=info msg="Committed blob data" correlationID=987b2dee-57a4-4d35-b670-2cad6867dc58 eventID=a03ab51d-6939-4bb9-876d-ba64e79a6161 name=downloader timestamp="2018-07-04 10:41:49.35799116 +0000 UTC m=+0.239257718"
time="2018-07-04T10:41:49Z" level=info msg="Cleaning up handler" correlationID=987b2dee-57a4-4d35-b670-2cad6867dc58 eventID=a03ab51d-6939-4bb9-876d-ba64e79a6161 name=downloader timestamp="2018-07-04 10:41:49.358141265 +0000 UTC m=+0.239407823"
panic: Error during commit Error committing events: failed to unmarshal map 'event-0.json' with error: 'json: cannot unmarshal object into Go value of type common.KeyValuePairs'

goroutine 1 [running]:
github.com/lawrencegripper/ion/internal/app/handler.Run(0x7ffcf7aefe44, 0x6, 0x98669f, 0x2, 0xc42009e640, 0x7ffcf7aefd57, 0xf, 0xc42009e680, 0xc42009e6c0, 0xc42008c960, ...)
	/go/src/github.com/lawrencegripper/ion/internal/app/handler/run.go:86 +0x7e3
main.NewStartCommand.func2(0xc420184000, 0xc4200ca9a0, 0x0, 0x16)
	/go/src/github.com/lawrencegripper/ion/cmd/handler/start.go:60 +0x9d
github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra.(*Command).execute(0xc420184000, 0xc4200ca6e0, 0x16, 0x16, 0xc420184000, 0xc4200ca6e0)
	/go/src/github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200b1b80, 0xc4200b1b80, 0xc420184280, 0xc420184000)
	/go/src/github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra/command.go:846 +0x30a
github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200b1b80, 0x8c7460, 0xc420070058)
	/go/src/github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra/command.go:794 +0x2b
main.main()
	/go/src/github.com/lawrencegripper/ion/cmd/handler/main.go:10 +0x27

[lawrencegripper]

As the downloader has an integration test which checks the format of the event.json file I think this may be an issue with the handler but I need to dig further.

[lawrencegripper]

This was related to the downloader using the wrong format for writing insights information. It requires the data to be an array of key value items

[{"key":"eventType","value":"file_downloaded"},{"key":"files","value":"file.raw"}]

Editted to correct formatting...

[lawrencegripper]

Getting closer new error now, again formatting issue in the downloader.

time="2018-07-04T13:39:27Z" level=info msg="Starting handler"
time="2018-07-04T13:39:27Z" level=info msg="Committing module's environment to the data plane" correlationID=c19cbdef-5722-4fc8-939c-528a35484037 eventID=eede9893-d331-4b91-8b2e-528e37f76862 name=downloader timestamp="2018-07-04 13:39:27.326392684 +0000 UTC m=+0.171126431"
time="2018-07-04T13:39:27Z" level=info msg="Committed blob data" correlationID=c19cbdef-5722-4fc8-939c-528a35484037 eventID=eede9893-d331-4b91-8b2e-528e37f76862 name=downloader timestamp="2018-07-04 13:39:27.392110016 +0000 UTC m=+0.236843663"
time="2018-07-04T13:39:27Z" level=info msg="Committed insights data" correlationID=c19cbdef-5722-4fc8-939c-528a35484037 eventID=eede9893-d331-4b91-8b2e-528e37f76862 name=downloader timestamp="2018-07-04 13:39:27.402755829 +0000 UTC m=+0.247489476"
time="2018-07-04T13:39:27Z" level=info msg="Cleaning up handler" correlationID=c19cbdef-5722-4fc8-939c-528a35484037 eventID=eede9893-d331-4b91-8b2e-528e37f76862 name=downloader timestamp="2018-07-04 13:39:27.402859032 +0000 UTC m=+0.247592679"
panic: Error during commit Error committing events: all events must contain an 'eventType' field, error: '<nil>'

goroutine 1 [running]:
github.com/lawrencegripper/ion/internal/app/handler.Run(0x7ffd5eee9e44, 0x6, 0x98669f, 0x2, 0xc420062680, 0x7ffd5eee9d57, 0xf, 0xc4200626c0, 0xc420062700, 0xc42007e960, ...)
	/go/src/github.com/lawrencegripper/ion/internal/app/handler/run.go:86 +0x7e3
main.NewStartCommand.func2(0xc42016c000, 0xc4200b6b00, 0x0, 0x16)
	/go/src/github.com/lawrencegripper/ion/cmd/handler/start.go:60 +0x9d
github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra.(*Command).execute(0xc42016c000, 0xc4200b6840, 0x16, 0x16, 0xc42016c000, 0xc4200b6840)
	/go/src/github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc42009fb80, 0xc42009fb80, 0xc42016c280, 0xc42016c000)
	/go/src/github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra/command.go:846 +0x30a
github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra.(*Command).Execute(0xc42009fb80, 0x8c7460, 0xc420070058)
	/go/src/github.com/lawrencegripper/ion/vendor/github.com/spf13/cobra/command.go:794 +0x2b
main.main()
	/go/src/github.com/lawrencegripper/ion/cmd/handler/main.go:10 +0x27

[lawrencegripper]

Fixed!