connector is not receiving all changes
Closed this issue · 12 comments
I used the couchbase docker(port 8091, 8092, 11211, 11210 exported) image to start a new couchbase bucket;
Then use the connector sample code to connector it to an Kafka cluster;
when the connector is running, I use python scripts to update the couchbase documents.
I updated 10000 documents in a few seconds, but form the Kafka, I can see only a few(~30) message pop out.
No error outputs.
Hi, I am seeing this issue as well. I have also followed a similar pattern as within this article (http://nosqlgeek.blogspot.de/2015/05/dcp-magic.html) to create a more basic version of a DCPRequest receiver, and am seeing the same thing. It seems to me that there must be some fundamental issues with DCP events. Some changes come through, others don't , seemingly arbitrarily. Could someone comment on this behaviour and whether an issue can be opened to address please (this issue has not seemingly been reported in https://issues.couchbase.com/projects/KAFKAC/)
Actually, a Jira ticket was created for this item but it's not obvious from the description that it's the same one: https://issues.couchbase.com/projects/KAFKAC/issues/KAFKAC-12. I will update that with a link to this issue.
Great thanks, so this fix is expected around 10th November?
Would you also be able to comment on this please? https://forums.couchbase.com/t/couchbase-kafka-connector-environments/5640
As suggested, please use Jira for further questions/comments.
@will, can you comment on the question from forums? Thanks.
While I used to work on the ruby couchrest gem, I dont know much about kafka.
From what I know it seems like kafka would be a good place to send the changes feed in couch, so it's probably a good idea! 💯
Hi,
I see you have now closed this issue, but i am unable to comment on the JIRA issue itself.
I have applied the changeset as seen here http://review.couchbase.org/#/c/56422/1/src/main/java/com/couchbase/kafka/CouchbaseReader.java which is the supposed fix for https://issues.couchbase.com/projects/KAFKAC/issues/KAFKAC-12, but this has not had any effect that i can see.
Some keys still do not come through. Unless I am missing something and the fix was bigger than the changeset above (in which case i'd appreciate you pointing me to it), I think this issue needs re-opening.
Cheers,
Tom
@waynesiu How can I post on your JIRA please? My credentials for the forum do not work there, are non-couchbase-ers able to raise and comment on tickets?
Our current plan is to issue a Couchbase Kafka Connector 1.2 release on November 10th. The Couchbase JVM Core Lib also includes some necessary improvements that Kafka Connector 1.2 depends on. In addition, some users may need to change their RxJava configuration if they are running more than 128 streams.
@tk6022 : can you confirm this is now working for you with Connector 1.2?
Wanted to go the couchbase -> Kafka route, but this is blocking
@gebrits, yes the 1.2.0 release appears to have sorted this as far as I can tell, but haven't extensively tested yet