[FEA]: Support different output data types in the `HttpSourceStage`

[mdemoret-nv]

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

Currently, the HttpSourceStage only supports outputting a MessageMeta object. This is very limiting in situations where some additional metadata needs to be attached to the message that is only available with the HTTP request object.

Describe your ideal solution

Support outputting different data types by the HttpSourceStage class. Ideally, it should be capable of outputting any type which is returned by the endpoint processing lambda. At a minimum, ControlMessages should be able to be generated by this stage.

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[dagardner-nv]

Ideally, it should be capable of outputting any type which is returned by the endpoint processing lambda.

Both HttpServerSourceStage and HttpClientSourceStage support a payload_to_df_fn method, however the output is a DataFrame.

If we wanted to do this I think there are two options:

1. Break API compatibility and change this to a request_to_msg_fn method
2. Add a second optional method to convert both the payload and the DataFrame to a message.

However currently for HttpServerSourceStage specifying payload_to_df_fn forces the Python impl of the stage (although the server itself is still implemented in C++).

I'm thinking instead doing something similar to what the DeserializeStage does and do:

1. Add a message_type & task_type constructor arguments
2. Add a request_to_task_payload_fn constructor argument. When None a default C++ impl will be used. When not-none the Python source stage impl will be used.