Integrating Milter with kafka

[iGaurav4]

In order to scale the application for analysis, we were trying to integrate kafka with milter.
The simplest way, i could found was to send milterContext to kafka topic for further analysis and then context would be ask postfix to deliver to users or quarantine it as shown in the image.
The initial issue in doing so, was serialization issue of MilterContext.
@nightcode
Could you please guide me, how can we achieve it?

[iGaurav4]

@nightcode Need your help on this.

[nightcode]

Hello, @iGaurav4
I can suggest you do not serialize MilterContext but collect all the necessary data about the email during SMFIC_* phases and send all the collected data to Kafka in the MilterCommands.eob() method.

[iGaurav4]

@nightcode
When i tried collecting all the necessary data about the email during SMFIC_* phases as EmailTopic. When this emailTopic passed to kafka, milter threads will still be in wait state to get the updated event(deliver, hold, modifications in email).
So, one thread in wait and kafka processing, it rather add more resource utilization which can be avoided.
What are your views in this, am i doing something wrong here ?

[nightcode]

@iGaurav4, have you tried using CompletableFuture and ThreadPools to solve your problems?
In my opinion, you should go in that direction.

[iGaurav4]

@nightcode ThreadPools would be able to solve a minor issue of scalability, while kafka could introduce vital improvement.
Is there any chance to include kafka in the architecture?

[nightcode]

@iGaurav4, the library does not impose any restrictions on use with Kafka. The question is beyond the functionality of the library and more related to design of your application. If you need a reference implementation of using Kafka with the library, it can be discussed separately, not as part of the discussion of the library.