Error Code 50002 while trying to get records
andy2046 opened this issue · 4 comments
andy2046 commented
Hi,
Got 50002 error while trying to get records, cannot see the consumer group created in kafka while doing bin/kafka-consumer-groups.sh --bootstrap-server PLAINTEXT://bootstrap.kafka:9092 --list
steps to produce the issue as shown below:
➜ curl -i -X POST -H "Content-Type: application/vnd.kafka.v2+json" -H "Accept: application/vnd.kafka.v2+json" \
--data '{"records": [{ "key": "a2V5", "value": "Y29uZmx1ZW50" }, { "value": "bG9ncw==" }]}' \
http://$RESTIP:8082/topics/kafka-01
HTTP/1.1 200 OK
Date: Mon, 09 Jul 2018 08:21:01 GMT
Content-Type: application/vnd.kafka.v2+json
Content-Length: 173
Server: Jetty(9.2.24.v20180105)
{"offsets":[{"partition":0,"offset":0,"error_code":null,"error":null},{"partition":0,"offset":1,"error_code":null,"error":null}],"key_schema_id":null,"value_schema_id":null}%
➜ curl -i -X POST -H "Content-Type: application/vnd.kafka.v2+json" -H "Accept: application/vnd.kafka.v2+json" \
--data '{"name": "consumer-01", "format": "binary", "auto.offset.reset": "earliest"}' \
http://$RESTIP:8082/consumers/consumer-group-03
HTTP/1.1 200 OK
Date: Mon, 09 Jul 2018 08:21:32 GMT
Content-Type: application/vnd.kafka.v2+json
Content-Length: 118
Server: Jetty(9.2.24.v20180105)
{"instance_id":"consumer-01","base_uri":"http://xxx.xxx.xxx.xxx:8082/consumers/consumer-group-03/instances/consumer-01"}%
➜ curl -i -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"topics":["kafka-01"]}' \
http://$RESTIP:8082/consumers/consumer-group-03/instances/consumer-01/subscription
HTTP/1.1 204 No Content
Date: Mon, 09 Jul 2018 08:22:04 GMT
Server: Jetty(9.2.24.v20180105)
➜ curl -i -X GET -H "Accept: application/vnd.kafka.v2+json" \
http://$RESTIP:8082/consumers/consumer-group-03/instances/consumer-01/records
HTTP/1.1 500 Internal Server Error
Date: Mon, 09 Jul 2018 08:22:19 GMT
Content-Type: application/vnd.kafka.v2+json
Content-Length: 650
Server: Jetty(9.2.24.v20180105)
{"error_code":50002,"message":"Kafka error: null io.confluent.rest.exceptions.RestServerErrorException: Kafka error: null
io.confluent.rest.exceptions.RestServerErrorException: Kafka error: null
at io.confluent.kafkarest.Errors.kafkaErrorException(Errors.java:188)
at io.confluent.kafkarest.v2.KafkaConsumerManager$1.onCompletion(KafkaConsumerManager.java:295)
at io.confluent.kafkarest.v2.KafkaConsumerReadTask.finish(KafkaConsumerReadTask.java:167)
at io.confluent.kafkarest.v2.KafkaConsumerReadTask.doPartialRead(KafkaConsumerReadTask.java:153)
at io.confluent.kafkarest.v2.KafkaConsumerWorker.run(KafkaConsumerWorker.java:112)"}
yams file for deployment as shown below:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zk
namespace: kafka
labels:
app: zk
spec:
serviceName: "zk-headless-svc"
replicas: 1
selector:
matchLabels:
app: zk
template:
metadata:
labels:
app: zk
replicas: "1"
spec:
containers:
- name: k8s-zk
imagePullPolicy: IfNotPresent
image: confluentinc/cp-zookeeper:4.1.1
resources:
requests:
memory: "2Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "500m"
ports:
- containerPort: 2181
name: client
- containerPort: 2888
name: server
- containerPort: 3888
name: leader-election
env:
- name: ZOOKEEPER_CLIENT_PORT
value: "2181"
- name: ZOOKEEPER_TICK_TIME
value: "2000"
volumeMounts:
- name: data
mountPath: /var/lib/zookeeper
volumeClaimTemplates:
- metadata:
name: data
spec:
storageClassName: zookeeper
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kafka
namespace: kafka
spec:
selector:
matchLabels:
app: kafka
serviceName: "broker"
replicas: 1
updateStrategy:
type: OnDelete
template:
metadata:
labels:
app: kafka
annotations:
spec:
terminationGracePeriodSeconds: 30
initContainers:
- name: init-config
image: solsson/kafka-initutils@sha256:18bf01c2c756b550103a99b3c14f741acccea106072cd37155c6d24be4edd6e2
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
command: ['/bin/bash', '/etc/kafka-configmap/init.sh']
volumeMounts:
- name: configmap
mountPath: /etc/kafka-configmap
- name: config
mountPath: /etc/kafka
containers:
- name: broker
image: solsson/kafka:1.0.1@sha256:1a4689d49d6274ac59b9b740f51b0408e1c90a9b66d16ad114ee9f7193bab111
env:
- name: KAFKA_LOG4J_OPTS
value: -Dlog4j.configuration=file:/etc/kafka/log4j.properties
ports:
- name: inside
containerPort: 9092
- name: outside
containerPort: 9094
command:
- ./bin/kafka-server-start.sh
- /etc/kafka/server.properties
resources:
requests:
cpu: 800m
memory: 2048Mi
readinessProbe:
tcpSocket:
port: 9092
timeoutSeconds: 1
volumeMounts:
- name: config
mountPath: /etc/kafka
- name: data
mountPath: /var/lib/kafka/data
volumes:
- name: configmap
configMap:
name: broker-config
- name: config
emptyDir: {}
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: kafka-broker
resources:
requests:
storage: 100Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rest-proxy
namespace: kafka-test
spec:
replicas: 1
selector:
matchLabels:
app: kafka-rest
template:
metadata:
labels:
app: kafka-rest
spec:
containers:
- name: kafka-rest
image: solsson/kafka-cp@sha256:2797da107f477ede2e826c29b2589f99f22d9efa2ba6916b63e07c7045e15044
env:
- name: KAFKAREST_LOG4J_OPTS
value: -Dlog4j.configuration=file:/etc/kafka-rest/log4j.properties
command:
- kafka-rest-start
- /etc/kafka-rest/kafka-rest.properties
readinessProbe:
httpGet:
path: /
port: 8082
livenessProbe:
httpGet:
path: /
port: 8082
ports:
- containerPort: 8082
volumeMounts:
- name: config
mountPath: /etc/kafka-rest
volumes:
- name: config
configMap:
name: rest-config
---
kind: ConfigMap
metadata:
name: rest-config
namespace: kafka-test
apiVersion: v1
data:
schema-registry.properties: |-
port=8081
listeners=http://0.0.0.0:8081
kafkastore.bootstrap.servers=PLAINTEXT://kafka-client-svc:9092
kafkastore.topic=_schemas
debug=false
# https://github.com/Landoop/schema-registry-ui#prerequisites
access.control.allow.methods=GET,POST,PUT,OPTIONS
access.control.allow.origin=*
kafka-rest.properties: |-
#id=kafka-rest-test-server
listeners=http://0.0.0.0:8082
bootstrap.servers=PLAINTEXT://kafka-client-svc:9092
# schema.registry.url=http://avro-schemas.kafka:8082
# https://github.com/Landoop/kafka-topics-ui#common-issues
access.control.allow.methods=GET,POST,PUT,DELETE,OPTIONS
access.control.allow.origin=*
zookeeper.connect=zk-client-svc:2181
consumer.threads=20
host.name=xxx.xxx.xxx.xxx
consumer.request.timeout.ms=5000
consumer.request.max.bytes=300000
consumer.instance.timeout.ms=600000
debug=true
simpleconsumer.pool.timeout.ms=5000
simpleconsumer.pool.size.max=50
consumer.iterator.backoff.ms=1000
consumer.iterator.timeout.ms=100
client.zk.session.timeout.ms=600000
client.timeout.ms=60000
log4j.properties: |-
log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.logger.kafka=WARN, stdout
log4j.logger.org.apache.zookeeper=WARN, stdout
log4j.logger.org.apache.kafka=WARN, stdout
log4j.logger.org.I0Itec.zkclient=WARN, stdout
log4j.additivity.kafka.server=false
log4j.additivity.kafka.consumer.ZookeeperConsumerConnector=false
log4j.logger.org.apache.kafka.clients.Metadata=DEBUG, stdout
log4j.logger.org.apache.kafka.clients.consumer.internals.AbstractCoordinator=INFO, stdout
solsson commented
I think this issue needs more context, what you're trying to do and how it relates to this repository, and regarding the yaml please fork so I can see a proper diff.
golonzovsky commented
As produce request worked we can assume that cluster is healthy and connection between rest proxy and cluster is fine.
50002 is a Rest-proxy generic error code, which might be timeout.
Check rest proxy logs for more details about this error.
Rest proxy might not have enough resources or request might be too big - then you might try to give it more resources, increase query timeout (with timeout query param), and maybe get a smaller chunk (with max_bytes query param)
curl -i -X GET -H "Accept: application/vnd.kafka.v2+json" \
http://$RESTIP:8082/consumers/consumer-group-03/instances/consumer-01/records?max_bytes=3000&timeout=30000
You might want to increase timeout defaults / reduce max size of request for rest-proxy deployment as well. see docs
But I would agree that question might not have relation to this repo, and seems you have same issues with confluent images as well here and confluentinc/kafka-rest#445
solsson commented
Thanks a lot @golonzovsky
andy2046 commented
@golonzovsky thanks for the reply, actually I did try to limit the max_bytes to 50 and timeout to 30,000 before I raised this issue, but it's still the same error
since this repo is quite active and I also use the image built by @solsson, hope to get a solution here
I am looking at the source code to find out why, pls update me if you have any suggestion on this issue
thanks a lot