possible move to Jackson from Gson for kafka

[a1shadows]

based on discussion with @baulea and @authorjapps , we need to investigate why Gson was used and if we should move fully to Jackson.

discussed in #648

[baulea]

Hi @a1shadows ,
I could have a look at that next week.

[baulea]

Hi @a1shadows
GSON internally uses a ToNumberPolicy.DOUBLE and converts JSON numbers into Double.

E.g. in kafka-testing/src/test/resources/kafka/produce/test_kafka_produce_int_key.json, the request contains an "Integer" key "101" in the produce operation, but during the unload operation the assertion checks for a "Double" key.

{
    "scenarioName": "Produce a message to kafka topic - int Key",
    "steps": [
        {...
            "operation": "produce",
            "request": {
                "recordType" : "RAW",
                "records": [
                    {
                        **"key": 101,**
                        "value": "Hello World"
                    }
                ]
            }, ...
        },
        {  ...
            "operation": "unload",
     ...
            "assertions": {
                "size": 1,
                "records": [
                    {
                        **"key": 101.0,**
                        "value": "Hello World"
                    }
                ]
            }
        }
    ]
}

Moving to Jackson, I did not find a suitable build-in replacement for this mechanism. So we would have recreate this behaviour programmatically to prevent the tests from failing.

I don't know whether this implicit conversion Integer->Double just sneaked in because GSON had been used or if it is an important feature.

Could we change "key" in the produce section of the unit tests to double? This would break the current functionality, but it would be more consistent in my opinion.

[a1shadows]

Hey, I'll take a look at this and get back to you

[authorjapps]

Hey Guys @baulea, @a1shadows, Thanks for looking at it. But imo gson is still fine as long as it doesn't do any harm, also it's great where it has solved a usecase easily.
I don't think we will gain a lot by replacing this with jackson as currently this only does the serde tasks around 1% of the scenarios(e.g. kafka headers etc).

So my advice would be to keep this as a low proprity TECHDEBT or withdraw this ticket until it blocks anyone or any problem scenario.
It's worth checking other functional issues which have ben raised recently as well the old ones which might benefit the framework's overall capability.

Thanks for understanding...