How to hand off data to another thread for processing?

[rhd]

Hi, maybe this isn't possible. But I really like the API of this library and would really like to use it.

I want to use it to handle all of my network IO and simple message parsing/framing before that data gets handed off to a real thread for processing.

pseudo code:

while true:
    read 4 bytes
    parse length from those bytes
    read N more bytes based on length
    put data into thread safe queue for processing

This generates 2 questions.

1. Is this possible?
2. How to deal with the data after it's been processed and we need to respond back to client.

Does this make sense?

[raedwulf]

libmill isn't intended to be used this way and you should be designing your server more like this: http://libmill.org/tutorial.html#step7
EDIT: My bad, I misread it initially - if the processing thread is heavy, this is a valid use-case.

Having said that, it is still possible:

• Set up UNIX socket to listen. Use mfork. In child process, connect to socket. Do real processing in child process and communicate (it is bidirectional) with parent process (doing the network processing) using the UNIX socket. EDIT: Other combinations exist of course, work can be done in any process.
• Use pipes instead of UNIX sockets when it gets implemented: it is being considered in #158

[sustrik]

I think the design does not make sense. Parsing is cheap. No need to have a separate thread/cpu-core assigned to it. Just read the data from socket and process it in a single thread. (Btw, note that actual network packets are received asynchronously by the kernel and stored in socket's rx buffer.)

[rhd]

@sustrik Right, parsing is cheap - that's why I would do it within the context of libmill's coroutines. Processing, OTOH, is not - that's why I would shovel the data into a threadsafe queue for someone else to process.

@raedwulf Ok, I saw step7 - but it's a bit more awkward that I want to do. I have an app that reads data from several sockets, processes the heck out of it, lays it to disk and responds to the caller. Basically, it's a multi-stage pipeline - each stage is connected via a thread-safe queue.

I just really like the socket API of libmill and wish I could use it for the socket portion of my app. But it doesn't seem possible to get the data to/from the processing portion of the app without forking.

Thanks for responding so quickly - I appreciate it. I'm going to close this issue. If I'm not understanding something, please let me know.

[sustrik]

You can still open a thread and pass data to it via a socketpair. No surprises there. Standard POSIX.