awslabs/amazon-kinesis-client-python

Pip install from cache does not include KCL jars

whummer opened this issue · 7 comments

whummer commented

We noticed that after the recent changes (fix of #28) the amazon_kclpy package has a bug regarding downloads of JAR files.

Steps to reproduce:

$ cd /tmp
$ virtualenv .venv
$ . .venv/bin/activate
$ pip install --no-cache amazon_kclpy
$ ls .venv/lib/python*/site-packages/amazon_kclpy/jars
__init__.py amazon-kinesis-client-1.7.2.jar aws-java-sdk-cloudwatch-1.11.14.jar ...
$ pip uninstall -y amazon_kclpy
$ pip install amazon_kclpy
$ ls .venv/lib/python*/site-packages/amazon_kclpy/jars
__init__.py

After the last step, the JAR files are missing in the jars directory. Would be great to get this fixed because most people will probably not use the --no-cache option on pip install, then the library becomes unusable.

pfifer commented

Thanks for the report. I'm looking into it.

pfifer commented

What OS are you see this on? I do see this on macOS, while Amazon Linux appears to work fine.

whummer commented

Sorry, forgot to mention. Tested on MacOS 10.11.6, Python version 2.7.11

pfifer commented

Thanks.
It seems to work fine on Amazon Linux. Install fails on macOS 10.11 and 10.12, and Windows (2016 Datacenter is what I'm testing on).

I have a fix, but it's somewhat hacky. It works by downloading the jars during the creation of the wheel, which appears to occur on the first installation of amazon_kclpy to the virtual environment.

whummer commented

Thanks for the fix, looks good to me. Any easy way I can test this branch on my machine via "pip install ..."?

pfifer commented

I hope to have the release ready soon, but for testing you can do this:

  1. Download the source for the module, and run python setup.py sdist.
    This will create dist/amazon_kclpy-1.4.0.tar.gz
  2. Create and activate the virtual environment as normal
  3. Install the KCL dependencies
  pip install boto
  pip install argparse
  pip install mock
  pip install amazon_kclpy --no-index --find-links <path to the dist directory>

The dependencies need to be installed separately since the installation of the amazon_kclpy won't be able to resolve dependendencies from PyPI.

pfifer commented

Version 1.4.1 has been released that fixes this:
Release Page
PyPI Page