assafmo/xioc

Support lines bigger than 64k bytes

philpraxis opened this issue · 3 comments

philpraxis commented 
$ python3 -c "print('A'*6553)" | ~/go/bin/xioc -o "ip4,domain,url" 
$

OK !

$ python3 -c "print('A'*65536)" | ~/go/bin/xioc -o "ip4,domain,url" 
2019/01/21 13:18:49 bufio.Scanner: token too long
$

NOT OK

philpraxis commented

By the way, congrats, good project! :)

On input, if non-allocating aka streaming search support could be implemented (it looks like not it's not, I didn't look at the source), that would be a huge plus since you will have arbitrarily long input lines in your input.

  • 👍1
assafmo commented

@philpraxis Thanks for helping! I'll fix this.

assafmo commented

On input, if non-allocating aka streaming search support could be implemented

What do you mean by "streaming search"?
I've tried to do this by reading line-by-line from stdin (with some bugs, apparently).

I guess you meant searching byte-by-byte without allocating potentially huge memory chunks.
I think a simpler solution for this is to limit the allocation size to (lets say) 5 MiB, which seems reasonable to me, but I'll also check the byte-by-byte streaming route.

Thanks again for your feedback. :-)