confluentinc/confluent-kafka-python

Could not build wheels for confluent-kafka error when doing pip install of the package

galaxy79 opened this issue · 3 comments

galaxy79 commented

Description

I'm installing the package in a python 3.9 docker image using pip and is throwing this below error,

[pipenv.exceptions.InstallError]: Failed to build confluent-kafka
[pipenv.exceptions.InstallError]: error: subprocess-exited-with-error
[pipenv.exceptions.InstallError]:   
[pipenv.exceptions.InstallError]:   × python setup.py bdist_wheel did not run successfully.
[pipenv.exceptions.InstallError]:   │ exit code: 1
[pipenv.exceptions.InstallError]:   ╰─> [36 lines of output]
[pipenv.exceptions.InstallError]:       running bdist_wheel
[pipenv.exceptions.InstallError]:       running build
[pipenv.exceptions.InstallError]:       running build_py
[pipenv.exceptions.InstallError]:       creating build
[pipenv.exceptions.InstallError]:       creating build/lib.linux-x86_64-cpython-39
[pipenv.exceptions.InstallError]:       creating build/lib.linux-x86_64-cpython-39/confluent_kafka
[pipenv.exceptions.InstallError]:       copying confluent_kafka/__init__.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka
[pipenv.exceptions.InstallError]:       creating build/lib.linux-x86_64-cpython-39/confluent_kafka/kafkatest
[pipenv.exceptions.InstallError]:       copying confluent_kafka/kafkatest/__init__.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/kafkatest
[pipenv.exceptions.InstallError]:       copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/kafkatest
[pipenv.exceptions.InstallError]:       copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/kafkatest
[pipenv.exceptions.InstallError]:       copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/kafkatest
[pipenv.exceptions.InstallError]:       creating build/lib.linux-x86_64-cpython-39/confluent_kafka/avro
[pipenv.exceptions.InstallError]:       copying confluent_kafka/avro/__init__.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/avro
[pipenv.exceptions.InstallError]:       copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/avro
[pipenv.exceptions.InstallError]:       copying confluent_kafka/avro/load.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/avro
[pipenv.exceptions.InstallError]:       copying confluent_kafka/avro/error.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/avro
[pipenv.exceptions.InstallError]:       creating build/lib.linux-x86_64-cpython-39/confluent_kafka/admin
[pipenv.exceptions.InstallError]:       copying confluent_kafka/admin/__init__.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/admin
[pipenv.exceptions.InstallError]:       creating build/lib.linux-x86_64-cpython-39/confluent_kafka/avro/serializer
[pipenv.exceptions.InstallError]:       copying confluent_kafka/avro/serializer/__init__.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/avro/serializer
[pipenv.exceptions.InstallError]:       copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-cpython-39/confluent_kafka/avro/serializer
[pipenv.exceptions.InstallError]:       warning: build_py: byte-compiling is disabled, skipping.
[pipenv.exceptions.InstallError]:       
[pipenv.exceptions.InstallError]:       running build_ext
[pipenv.exceptions.InstallError]:       building 'confluent_kafka.cimpl' extension
[pipenv.exceptions.InstallError]:       creating build/temp.linux-x86_64-cpython-39
[pipenv.exceptions.InstallError]:       creating build/temp.linux-x86_64-cpython-39/confluent_kafka
[pipenv.exceptions.InstallError]:       creating build/temp.linux-x86_64-cpython-39/confluent_kafka/src
[pipenv.exceptions.InstallError]:       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/app/.local/share/virtualenvs/repo-rmKx9mUF/include -I/usr/local/include/python3.9 -c confluent_kafka/src/Admin.c -o build/temp.linux-x86_64-cpython-39/confluent_kafka/src/Admin.o
[pipenv.exceptions.InstallError]:       In file included from confluent_kafka/src/Admin.c:17:
[pipenv.exceptions.InstallError]:       confluent_kafka/src/confluent_kafka.h:22:10: fatal error: librdkafka/rdkafka.h: No such file or directory
[pipenv.exceptions.InstallError]:          22 | #include <librdkafka/rdkafka.h>
[pipenv.exceptions.InstallError]:             |          ^~~~~~~~~~~~~~~~~~~~~~
[pipenv.exceptions.InstallError]:       compilation terminated.
[pipenv.exceptions.InstallError]:       error: command '/usr/bin/gcc' failed with exit code 1
[pipenv.exceptions.InstallError]:       [end of output]
[pipenv.exceptions.InstallError]:   
[pipenv.exceptions.InstallError]:   note: This error originates from a subprocess, and is likely not a problem with pip.
[pipenv.exceptions.InstallError]:   ERROR: Failed building wheel for confluent-kafka

As per documentaion, this package automatically download and install the binary librdkafka during the pip install which seems not to be happening. I'm trying to install this in a corporate network and it may be that download is restricted due to proxy. I want to know if there is any option to configure the binary link url and then install the package

Reliable - It's a wrapper around [librdkafka](https://github.com/edenhill/librdkafka) (provided automatically via binary wheels) which is widely deployed in a diverse set of production scenarios. It's tested using [the same set of system tests](https://github.com/confluentinc/confluent-kafka-python/tree/master/src/confluent_kafka/kafkatest) as the Java client [and more](https://github.com/confluentinc/confluent-kafka-python/tree/master/tests). It's supported by [Confluent](https://confluent.io/).

confluent-kafka-python : 2.4.0
Python version : 3.9

Checklist

Please provide the following information:

  • [x ] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
  • Apache Kafka broker version:
  • Client configuration: {...}
  • Operating system:
  • Provide client logs (with 'debug': '..' as necessary)
  • Provide broker log excerpts
  • Critical issue
pranavrth commented

Can you please provide following detail?

  • Operating System information
  • pip version
  • Are you installing from source distribution? command used to install

I suspect that the pip version is old. Try upgrading the pip. Can you please provide the above details even if it works for you as it would be helpful for anybody else that faces this problem?

pranavrth commented

Can you please update this issue? If the issue is solved, let us know so that I can close the issue.

pranavrth commented

I think the issue is resolve. Closing this. Please feel free to open a new issue if required.