81 of 100?
Closed this issue · 8 comments
#!/bin/bash
export TOPIC="test10"
export ENTRIES=100
curl -G --silent "http://localhost:19092/topics/${TOPIC}/messages?group=bar"
function pushit() {
curl --silent -X POST "http://localhost:19092/topics/${TOPIC}/messages?sync" -H "Content-Type: text/plain" -d "Entry $1"
}
export -f pushit
time (seq $ENTRIES | parallel -j10 pushit)
curl -X POST -H'Content-Type: application/json' -d'[{"partition":0,"offset":0}]' "http://localhost:19092/topics/${TOPIC}/offsets?group=bar"
echo
sleep 2
function pullit() {
value=$(curl -G --silent "http://localhost:19092/topics/${TOPIC}/messages?group=bar" | tr "\n" " ")
dec=$(echo $value | jq '.value' -r | base64 --decode)
echo "$dec $value"
}
export -f pullit
time (seq $ENTRIES | parallel -j10 pullit)
So, the level of parallelism is 10:
Entry 1 { "key": null, "value": "RW50cnkgMQ==", "partition": 0, "offset": 0 }
Entry 2 { "key": null, "value": "RW50cnkgMg==", "partition": 0, "offset": 1 }
...(skip)...
Entry 74 { "key": null, "value": "RW50cnkgNzQ=", "partition": 0, "offset": 73 }
Entry 75 { "key": null, "value": "RW50cnkgNzU=", "partition": 0, "offset": 74 }
Entry 76 { "key": null, "value": "RW50cnkgNzY=", "partition": 0, "offset": 75 }
Entry 77 { "key": null, "value": "RW50cnkgNzc=", "partition": 0, "offset": 76 }
Entry 79 { "key": null, "value": "RW50cnkgNzk=", "partition": 0, "offset": 77 }
Entry 78 { "key": null, "value": "RW50cnkgNzg=", "partition": 0, "offset": 78 }
Entry 80 { "key": null, "value": "RW50cnkgODA=", "partition": 0, "offset": 79 }
Entry 81 { "key": null, "value": "RW50cnkgODE=", "partition": 0, "offset": 80 }
��e { "error": "long polling timeout" }
��e { "error": "long polling timeout" }
��e { "error": "long polling timeout" }
��e { "error": "long polling timeout" }
I get back only 81 entries, 4 timeouts and quite a few empty lines :)
This behaviour repeats only occasionally, not on each run. It's best to change the TOPIC on each run
This one I cannot reproduce, probably due to differences between our Kafka cluster setups. But my guess would be that something was going one on the production side. So, could you please post complete log that includes the producer output. Or just email it to my github address. Thanks.
I'm not sure how that works. Is that horkhe at github... com ? Because on your profile your e-mail is not public.
It seems that I even crashed pixy a few times :)
Sep 8 12:06:15.778 ERROR </cons[0]/G:bar[1]/manager[0]/rebalancer[2]> paniced: runtime error: invalid memory address or nil pointer dereference, stack=goroutine 306 [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
/opt/go/src/runtime/debug/stack.go:24 +0x80
github.com/mailgun/kafka-pixy/actor.Spawn.func1.1(0xc82061f680)
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/actor/actor.go:63 +0x5b
panic(0x857840, 0xc82000e070)
/opt/go/src/runtime/panic.go:443 +0x4e9
github.com/mailgun/kafka-pixy/consumer/groupcsm.(*T).rewireMuxAsync(0xc820076dc0, 0xc82062a1e0, 0xc82045f540, 0x0, 0x0, 0x0, 0x0)
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/consumer/groupcsm/groupcsm.go:244 +0x2d9
github.com/mailgun/kafka-pixy/consumer/groupcsm.(*T).runRebalancer(0xc820076dc0, 0xc82061f680, 0xc820628120, 0xc820628090, 0xc8204e4a80)
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/consumer/groupcsm/groupcsm.go:212 +0x345
github.com/mailgun/kafka-pixy/consumer/groupcsm.(*T).runManager.func1()
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/consumer/groupcsm/groupcsm.go:181 +0x44
github.com/mailgun/kafka-pixy/actor.Spawn.func1(0x0, 0xc82061f680, 0xc820628180)
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/actor/actor.go:68 +0x112
created by github.com/mailgun/kafka-pixy/actor.Spawn
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/actor/actor.go:69 +0x67
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x6aef29]
goroutine 306 [running]:
panic(0x857840, 0xc82000e070)
/opt/go/src/runtime/panic.go:481 +0x3e6
github.com/mailgun/kafka-pixy/actor.Spawn.func1.1(0xc82061f680)
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/actor/actor.go:64 +0x19c
panic(0x857840, 0xc82000e070)
/opt/go/src/runtime/panic.go:443 +0x4e9
github.com/mailgun/kafka-pixy/consumer/groupcsm.(*T).rewireMuxAsync(0xc820076dc0, 0xc82062a1e0, 0xc82045f540, 0x0, 0x0, 0x0, 0x0)
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/consumer/groupcsm/groupcsm.go:244 +0x2d9
github.com/mailgun/kafka-pixy/consumer/groupcsm.(*T).runRebalancer(0xc820076dc0, 0xc82061f680, 0xc820628120, 0xc820628090, 0xc8204e4a80)
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/consumer/groupcsm/groupcsm.go:212 +0x345
github.com/mailgun/kafka-pixy/consumer/groupcsm.(*T).runManager.func1()
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/consumer/groupcsm/groupcsm.go:181 +0x44
github.com/mailgun/kafka-pixy/actor.Spawn.func1(0x0, 0xc82061f680, 0xc820628180)
/home/vagrant/goworld/src/github.com/mailgun/kafka-pixy/actor/actor.go:68 +0x112
created by github.com/mailgun/kafka-pixy/actor.Spawn
Looks like you are hitting #64. Please use version v0.11.1 for your testing. The bug is fixed there.
My email is horkhe at гмеил дот ком. Something tells me that you will be able to decipher that :).
Yep, your assumption is correct :) Ok, I'll re-try the tests with 0.11.1 and 1 request on start and if the bug is still there - I'll send the log.
Yep, when using 0.11.1 and 50 concurrent connections and sleeping 20 seconds before doing the "set offset" call - everything works as expected! Thanks, Maxim!