Upload failed (500): INTERNAL SERVER ERROR

Question

Upload failed (500): INTERNAL SERVER ERROR

fusionbeam opened this issue 10 years ago · 9 comments

Hi,
First of all thanks for your work on this project.
I am unable to upload packages to the private repo and nothing shows up in the error logs that could help me with the troubleshooting.
the upload command manages to add Package and Release entries but it fails to upload the actual archive.
Any hints on what might be wrong?

Thank you in advance,
Rares

Answer 1 · 2015-02-18T12:56:49.000Z

never mind, i've figured it out.

Answer 2 · 2015-05-26T21:30:05.000Z

@fusionbeam What was the problem? I'm running into this issue now.

Answer 3 · 2015-05-27T09:27:31.000Z

Hi @adams-sarah , could you try the develop branch? I've refactored a lot of stuff so this might work for you now. Thanks

Answer 4 · 2015-05-27T23:09:52.000Z

Yep, I'm using the develop branch.
I seem to have escaped the 1st 500 error I got. Now I get a new one (but only sometimes..).
Here's what I've got so far:

Context:

Server/localshop-side:

I've got nginx running as a reverse proxy (https -> http) so that pip is happy (pip seems to want https)
I've signed the ssl cert with my own certificate authority.

My ~/.pypirc looks like this:

[distutils]
index-servers =
   localpypi

[localpypi]
username: USER-KEY
password: SECRET-KEY
repository: http://127.0.0.1:8000/simple/

Client-side:

My ~/.pip/pip.conf looks like this:

[global]
index-url = https://MY-LOCALSHOP-ACCESS-KEY:MY-LOCALSHOP-SECRET-KEY@ec2-instance-ip-address/simple/
cert = /ABSOLUTE/PATH/TO/ca.crt

The error:

High-level: What I've noticed:

The error seems to happen only IF I am installing a package (with pip install my-private-pkg) that has a setup.py which requires a python dependency that is not yet known by localshop.
pip craps out when trying to install this dependency (let's call it flask) -- see stack trace 1 below.
BUT! If I re-run my install command (pip install my-private-pkg), pip is able to find this dependency (flask)! However, it then craps out at the next dependency it can't find locally.. and so on -- see stack trace 2 below, run directly after 1.

So, I assume the 'fetch' from pypi.python.org of flask just hasn't completed at the time that localshop tries to find it.

What I don't understand is how the SSL stuff plays a part.

Perhaps is localshop forwarding my CA "bundle" (only one CA, mine) to pypi.python.org? I just don't know.

Stack trace 1:

$ pip install .
Collecting Flask<0.11,>=0.10 (from my-private-pkg==0.0.1.dev0)
/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
  Downloading https://ec2-instance-ip-address/packages/Flask/download/2169/Flask-0.10.1.tar.gz (544kB)
Collecting gunicorn==19.1.1 (from my-private-pkg==0.0.1.dev0)
  DEPRECATION: Failed to find 'gunicorn' at https://MY-LOCALSHOP-ACCESS-KEY:MY-LOCALSHOP-SECRET-KEY@ec2-instance-ip-address/simple/gunicorn/. It is suggested to upgrade your index to support normalized names as the name in /simple/{name}.
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 223, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 291, in run
    wb.build(autobuilding=True)
  File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 704, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 317, in prepare_files
    functools.partial(self._prepare_file, finder))
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 304, in _walk_req_to_install
    more_reqs = handler(req_to_install)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 469, in _prepare_file
    session=self.session)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 825, in unpack_url
    session,
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 673, in unpack_http_url
    from_path, content_type = _download_http_url(link, session, temp_dir)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 857, in _download_http_url
    stream=True,
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 373, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 594, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 196, in resolve_redirects
    **adapter_kwargs
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 46, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

Stack trace 2:

$ pip install .
Collecting Flask<0.11,>=0.10 (from my-private-pkg==0.0.1.dev0)
/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
  Downloading https://ec2-instance-ip-address/packages/Flask/download/2169/Flask-0.10.1.tar.gz (544kB)
Collecting gunicorn==19.1.1 (from my-private-pkg==0.0.1.dev0)
  DEPRECATION: Failed to find 'gunicorn' at https://MY-LOCALSHOP-ACCESS-KEY:MY-LOCALSHOP-SECRET-KEY@ec2-instance-ip-address/simple/gunicorn/. It is suggested to upgrade your index to support normalized names as the name in /simple/{name}.
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 223, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 291, in run
    wb.build(autobuilding=True)
  File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 704, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 317, in prepare_files
    functools.partial(self._prepare_file, finder))
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 304, in _walk_req_to_install
    more_reqs = handler(req_to_install)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 469, in _prepare_file
    session=self.session)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 825, in unpack_url
    session,
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 673, in unpack_http_url
    from_path, content_type = _download_http_url(link, session, temp_dir)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 857, in _download_http_url
    stream=True,
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 373, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 594, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 196, in resolve_redirects
    **adapter_kwargs
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 46, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

Stack trace 3:

$ pip install .
Collecting Flask<0.11,>=0.10 (from my-private-pkg==0.0.1.dev0)
/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
  Downloading https://ec2-instance-ip-address/packages/Flask/download/2169/Flask-0.10.1.tar.gz (544kB)
Collecting gunicorn==19.1.1 (from my-private-pkg==0.0.1.dev0)
  Downloading https://ec2-instance-ip-address/packages/gunicorn/download/2213/gunicorn-19.1.1-py2.py3-none-any.whl (104kB)
Collecting gevent==1.0.1 (from my-private-pkg==0.0.1.dev0)
  Downloading https://ec2-instance-ip-address/packages/gevent/download/502/gevent-1.0.1.tar.gz (1.5MB)
Collecting plyvel<1.0,>=0.9 (from my-private-pkg==0.0.1.dev0)
  DEPRECATION: Failed to find 'plyvel' at https://MY-LOCALSHOP-ACCESS-KEY:MY-LOCALSHOP-SECRET-KEY@ec2-instance-ip-address/simple/plyvel/. It is suggested to upgrade your index to support normalized names as the name in /simple/{name}.
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 223, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 291, in run
    wb.build(autobuilding=True)
  File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 704, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 317, in prepare_files
    functools.partial(self._prepare_file, finder))
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 304, in _walk_req_to_install
    more_reqs = handler(req_to_install)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 469, in _prepare_file
    session=self.session)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 825, in unpack_url
    session,
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 673, in unpack_http_url
    from_path, content_type = _download_http_url(link, session, temp_dir)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 857, in _download_http_url
    stream=True,
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 373, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 594, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 196, in resolve_redirects
    **adapter_kwargs
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 46, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

My setup.py:

#!/usr/bin/env python
# vim: fileencoding=utf8:et:sw=4:ts=8:sts=4

from setuptools import setup


VERSION = (0, 0, 1, "dev")


setup(
    name="my-private-pkg",
    description="REST service for receiving data from ATS or HR systems",
    packages=["my-private-pkg"],
    version=".".join(filter(None, map(str, VERSION))),
    url="http://github.com/my-company/my-private-pkg",
    install_requires=["Flask<0.11,>=0.10", "gunicorn==19.1.1", "gevent==1.0.1", "plyvel<1.0,>=0.9", "suds<0.5,>=0.4", "boto>=2.33.0", "other-private-pkg"],
)

Thanks so much for your help.

This project is awesome btw. Cheers!

Answer 5 · 2015-05-28T05:35:17.000Z

Oh wow, thanks for the all the information. I'll dive into this problem this weekend. Thanks!

Answer 6 · 2015-05-28T20:58:54.000Z

Okay I can reproduce the SSL error (although it doesn't raise an exception on my side). Which version of PIP are you using?

Basically the problem is that in your pip.conf you specify the cert bundle to validate against, but since localshop redirects you to pypi if the package isn't available yet the pip client isn't accepting the ssl cert from pypi.python.org.

Answer 7 · 2015-05-28T22:05:11.000Z

Thanks for helping me with this.

My env:

OS: OS X 10.10.3
python: 2.7.10
pip: 7.0.1

So should I pass the whole CA bundle from my mac with my personal CA cert appended? (instead of just sending my own CA cert)
Or, what would you suggest?

I feel like this must be a common problem, right? pip seems to prefer ssl..
Do you know what other people do? Or do most folk just set up localshop on their LAN?

Answer 8 · 2015-05-28T22:06:05.000Z

Oh, the ec2 instance is running debian jessie. I'm not sure what versions of pip / python it has. Let me know if you need that info.

Answer 9 · 2015-05-30T09:27:51.000Z

Yes you need to add your own ca cert to the default ca bundles. I've just tested the following method:

Creating self signed certificate with a self signed ca: http://blog.didierstevens.com/2008/12/30/howto-make-your-own-cert-with-openssl/

I've always used real certificates instead of self-signed so never ran into this issue