Using scramjet to concate and split data received with socket

[timvol]

I'm trying to pipe incoming data from a socket to a BufferStream, split it and call a function for each split. But I've no idea how I can implement this using scramjet. What I've is the following:

const net = require('net')
const { BufferStream } = require('scramjet')

const mySocket = new net.Socket()
mySocket.pipe(new BufferStream()).split(Buffer.from([0x11, 0x22, 0x22, 0x11])).each(console.log)

But this doesn't work. The first time I recieve data starting with [0x11, 0x22, 0x22, 0x11], I see data on the console. But for all subsequent data I receive, I don't see any output. My messages are having the following structure:

<4 Bytes Message Splitter><20 Bytes Command><4 Bytes Payload-Length><4 Bytes Checksum>

What I want is that the incoming data are concatenated and splitted based on the message splitter ([0x11, 0x22, 0x22, 0x11]). Any idea how I can do this using scramjet?

[MichalCz]

Hi @timvol,

Thanks for filing the issue. :)

We'll take a look at this during the weekend and come back to you. I don't think that the above case is tested so there might be a problem, so we'll start by implementing tests for the above case.

M.

[MichalCz]

BTW, if that's not a big trouble, can you also post a simple socat or netcat invocation from bash so we can quickly reproduce your problem?

[timvol]

Thank you @MichalCz, I'll try to provide a simple example next week because I'll have no time this weekend, sorry.
Maybe you already solved the problem I'm facing before I can provide an example :-D

[MichalCz]

Ok, I set up some tests and tried it out, but I don't seem to see the issue...

I'll drill in on this tomorrow or the day after again. :)

[timvol]

I created a very simple Node.js app:

const net = require('net')
const { BufferStream } = require('scramjet')

const server = net.createServer(function (socket) {
  for (let i = 0; i < 10; i++) {
    socket.write(new Uint8Array([0x11, 0x22, 0x22, 0x11])) // Splitter
    socket.write(new Uint8Array(0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x74, 0x65, 0x73, 0x74)) // Command
    socket.write(new Uint8Array([0x00, 0x10, 0x00, 0x00])) // Payload Length
    socket.write(new Uint8Array(1024 * 1024)) // Message
  }

  socket.pipe(socket)

  console.log('Buffer sent!')
})

server.listen(1337)

const client = new net.Socket()
client.pipe(new BufferStream()).split(Buffer.from([0x11, 0x22, 0x22, 0x11])).each(console.log)
client.connect(1337, '127.0.0.1')

When I run this application, I see the following output:

Buffer sent!
<Buffer >

I hope this helps :-)

[MichalCz]

Thanks, will check.

[MichalCz]

Hi @timvol

Instead of socket.pipe(socket) I think there should be socket.end() - can you check?

[timvol]

When I replace socket.pipe(socket) by socket.end() it shows me the following:

Buffer sent!
<Buffer >
<Buffer 00 00>

In my "real" application I'm just implementing the client-side. It's a p2p network and other client mostly written C/C++, Python or similar. So I don't know the "real" implementation... The Server side in this example is demonstrating purposes. But it shows exactly the issue I'm facing :-(

BUT! I commented out socket.write(new Uint8Array(1024 * 1024)) line and it works perfectly when I call socket.end():

Buffer sent!
<Buffer >
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>

When I use socket.pipe(socket) instead of socket.end() and socket.write(new Uint8Array(1024 * 1024)) is still commented out, it just shows me:

Buffer sent!
<Buffer >
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 73 74 00 10 00 00>

[MichalCz]

Sorry, closed by mistake...

[MichalCz]

@timvol can you check with 4.36.1? This has been just published. I'll be checking this issue again today though.