pip install -r requirements.txt fails to "setuptools-scm was unable to detect version for '/tmp'"

Question

pip install -r requirements.txt fails to "setuptools-scm was unable to detect version for '/tmp'"

tuukkamustonen opened this issue 9 years ago · 31 comments

Version: 1.15.0

I have requirements declared in setup.py, so with requirements.txt file of:

pip install -r requirement.txt:

Processing /home/musttu/Code/lib/wires/aws
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-pgsLbN-build/setup.py", line 31, in <module>
        'pastry-common==3.2.1.dev30',
      File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools/dist.py", line 318, in __init__
        _Distribution.__init__(self, attrs)
      File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
        self.finalize_options()
      File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools/dist.py", line 376, in finalize_options
        ep.load()(self, ep.name, value)
      File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools_scm/integration.py", line 19, in version_keyword
        dist.metadata.version = get_version(**value)
      File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 117, in get_version
        parsed_version = _do_parse(root, parse)
      File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 95, in _do_parse
        "use git+https://github.com/user/proj.git#egg=proj" % root)
    LookupError: setuptools-scm was unable to detect version for '/tmp'.

I don't believe it's possible to fix this within setuptools-scm, however. If pip uses /tmp for temporarily building/handling the package, there's simply no version information available and nothing to be done. Correct? Would you have any guesses if it's possible to make it work on pip side?

There is a related ticket in #46 but I think that was closed because user tried to install an sdist there.

Answer 1 · 2016-12-08T10:32:23.000Z

now at first glance that looks like a pip bug, but what strikes me as more odd, is that the root setuptools_scm found was /tmp

do you per chance have a .git folder in /tmp?

Answer 2 · 2016-12-08T10:34:41.000Z

@RonnyPfannschmidt Nope, lots of stuff there but nothing related to git.

Answer 3 · 2016-12-08T11:15:48.000Z

i believe we may figure more from verbose output + setting SETUPTOOLS_SCM_DEBUG=1 in the environment

Answer 4 · 2016-12-08T12:02:55.000Z

Running pip==9.0.1 (I tried with a few older version also, did not seem to make difference):

13:54 $ SETUPTOOLS_SCM_DEBUG=1 pip install -v -r requirements.txt 
Processing /home/musttu/Code/lib/wires/common
  Running setup.py (path:/tmp/pip-bBgpZD-build/setup.py) egg_info for package from file:///home/musttu/Code/lib/wires/common
    Running command python setup.py egg_info
    root '/tmp'
    looking for ep setuptools_scm.parse_scm /tmp
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-bBgpZD-build/setup.py", line 18, in <module>
        'pathlib >= 1.0.1'
      File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools/dist.py", line 272, in __init__
        _Distribution.__init__(self,attrs)
      File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
        self.finalize_options()
      File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools/dist.py", line 327, in finalize_options
        ep.load()(self, ep.name, value)
      File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools_scm/integration.py", line 19, in version_keyword
        dist.metadata.version = get_version(**value)
      File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 117, in get_version
        parsed_version = _do_parse(root, parse)
      File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 95, in _do_parse
        "use git+https://github.com/user/proj.git#egg=proj" % root)
    LookupError: setuptools-scm was unable to detect version for '/tmp'.

    Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

    For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
Cleaning up...
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-bBgpZD-build/
Exception information:
Traceback (most recent call last):
  File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/pip/req/req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/pip/req/req_install.py", line 439, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-bBgpZD-build/

I don't know, it just seems pip runs stuff under /tmp so there's no git information available.

It seems this problem occurs on version 1.6.0+.

On 1.5.0, pip install -r requirements.txt succeeds but version number is set to 0.0.0. Bit of changelog from 1.6.0:

bail out early if the scm is missing

this brings issues with git tarballs and older devpi-client releases to light, before we would let the setup stay at version 0.0, now there is a ValueError

Interesting is that python setup.py develop gets correct version on any setuptools-scm version.

Also, pip install -e . works just fine on any version.

In the end, I don't know if it makes sense to do what I am doing here. If I want package in development mode, I can pip install -e ., and if I want to install it (from local checkout) I can pip install ..

Answer 5 · 2016-12-08T12:13:41.000Z

the complete output of setuptools_scm during setup is missing, please get it in some way (i don't recall how that works)

Answer 6 · 2016-12-08T12:20:42.000Z

Not sure what you mean, but I tried:

$ SETUPTOOLS_SCM_DEBUG=1 python setup.py --version
root '/home/musttu/Code/lib/wires'
looking for ep setuptools_scm.parse_scm /home/musttu/Code/lib/wires
found ep .git = setuptools_scm.git:parse
cmd 'git rev-parse --verify --quiet HEAD'
out '4f99f4e7cc40c6d3a8f3a67fbf92bf70f9b76ab1\n'
cmd 'git status --porcelain --untracked-files=no'
out ' M common/requirements.txt\n M common/setup.py\n M common/tox.ini\n'
cmd 'git describe --tags --long --match *.*'
out '2.2.0-4-g4f99f4e\n'
tag 2.2.0
version <Version('2.2.0')>
version 2.2.0
scm version <ScmVersion 2.2.0 d=4 n=g4f99f4e d=True x={}>
config {'version_scheme': 'guess-next-dev', 'local_scheme': 'node-and-date'}
ep ('setuptools_scm.version_scheme', 'guess-next-dev')
ep ('setuptools_scm.local_scheme', 'node-and-date')
version 2.2.1.dev4
local_version +ng4f99f4e.d20161208
2.2.1.dev4+ng4f99f4e.d20161208

And:

14:17 $ SETUPTOOLS_SCM_DEBUG=1 python -m setuptools_scm
root '/home/musttu/Code/lib/wires/common'
looking for ep setuptools_scm.parse_scm /home/musttu/Code/lib/wires/common
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/musttu/Code/virtualenvs/wires/lib/python2.7/site-packages/setuptools_scm/__main__.py", line 9, in <module>
    print('Guessed Version', get_version())
  File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 117, in get_version
    parsed_version = _do_parse(root, parse)
  File "/home/musttu/Code/virtualenvs/wires/local/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 95, in _do_parse
    "use git+https://github.com/user/proj.git#egg=proj" % root)
LookupError: setuptools-scm was unable to detect version for '/home/musttu/Code/lib/wires/common'.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj

Answer 7 · 2016-12-08T12:22:50.000Z

The latter probably fails, because I'm in a sub-directory inside my git repo (I'm hosting multiple components within same git repo). When I run python -m setuptools_scm on git root, it works (and gives the same output as with python setup.py --version.

I have set:

use_scm_version={'root': '..'},

...in my setup.py.

Answer 8 · 2016-12-08T13:13:43.000Z

i see, then thats the reason for looking in /tmp, - it would be helpfull to get the setuptools_scm output while a pip install actually runs

Answer 9 · 2016-12-10T14:20:13.000Z

@RonnyPfannschmidt I couldn't find out how to get more output, but I stepped over some of the code and I think here's the problem:

First, when setup.py and requirements.txt reside on git repository root and I have use_scm_version=True, things work ok. Pip (or maybe it's setuptools, I don't know) seems to copy the git repository to /tmp/pip-<hash>-build, and version is detected succesfully.
When I put setup.py and requirements.txt to sub-directory (let's call it subdir here) and set use_scm_version={'root': '..'}, pip again copies the git repository to /tmp/pip-<hash>-build, but working directory is also set to /tmp/pip-<hash>-build and not /tmp/pip-<hash>-build/subdir. So, when setuptools_scm tries to detect version, it navigates to .., ending up in /tmp and fails.
Here, if I set use_scm_version={'root': '..', 'relative_to': 'subdir/'}, then pip copies only the subdir/* into /tmp/pip-<hash>-build. The version would be correctly determined here after https://github.com/pypa/setuptools_scm/blob/master/setuptools_scm/__init__.py#L112-L114, but as pip copied only subdir/* contents, there's no .git available and setuptools_scm fails to determine version.

I would assume the fault is at pip's (or setuptools) end here - should it set the working directory to /tmp/pip-<hash>-build/subdir instead of just /tmp/pip-<hash>-build? Then, I assume, things would work.

Answer 10 · 2016-12-10T14:28:51.000Z

please try "relative_to" : __file__

Answer 11 · 2016-12-10T14:31:42.000Z

That fails also, setuptools_scm resolves root to just /tmp and only contents of the sub-directory get copied to /tmp/pip-<hash>-build anyway.

Answer 12 · 2016-12-10T14:35:58.000Z

oh, i see, then i missunderstood,

yes, then technically pip is breaking things appart and breaking the build

Answer 13 · 2016-12-12T08:39:45.000Z

@RonnyPfannschmidt I will open up a ticket on pip's tracker on this. However, would you first have a pointer or explanation on how pip determines that it will copy the parent directory with .git/ (and not the current working dir, which would be subdir/) to /tmp by default and why setuptools_scm's relative_to has impact on it? Does setuptools_scm alter pip behavior somehow?

Answer 14 · 2016-12-13T06:37:36.000Z

no, setuptools_scm is completely unaware of pip

Answer 15 · 2016-12-13T08:15:37.000Z

Weird, I thought so because I got different behavior when I set relative_to. Well, let's what pip folks say.

Answer 16 · 2016-12-25T14:51:42.000Z

@tuukkamustonen can you link the pip issue here please

Answer 17 · 2016-12-26T16:24:10.000Z

It's visible already there above: pypa/pip#4180

Answer 18 · 2017-01-29T20:34:23.000Z

@tuukkamustonen i found a possible solution using the pip-egg-info

Answer 19 · 2017-01-29T21:12:39.000Z

i uncovered a mistake from fixing #45 and #43 in 0f1c98d about 2 years ago

Answer 20 · 2017-01-31T06:31:01.000Z

@RonnyPfannschmidt Nice! Not a very critical regression then if no one bumped into it during those 2 years :)

Do you think pypa/pip#4180 is invalid and should be closed?

Answer 21 · 2017-01-31T06:35:25.000Z

@tuukkamustonen the feature never ever worked as intended, so its quite an oversight

Answer 22 · 2017-01-31T06:36:20.000Z

@tuukkamustonen pypa/pip#4180 is still valid, but should receive some extra infos

Answer 23 · 2018-03-16T20:16:14.000Z

Can anyone please help me to fix this bug ? I am aving trouble installing this module.
Following is the error while installing this package:

[root@linux3 setuptools_scm-1.15.4]# python setup.py  install
Traceback (most recent call last):
  File "setup.py", line 115, in <module>
    arguments.update(scm_config())
  File "setup.py", line 48, in scm_config
    root=here, parse=parse, **config))
  File "/home/aceadm/shivam/1603/setuptools_scm-1.15.4/setuptools_scm/__init__.py", line 119, in get_version
    parsed_version = _do_parse(root, parse)
  File "/home/aceadm/shivam/1603/setuptools_scm-1.15.4/setuptools_scm/__init__.py", line 97, in _do_parse
    "use git+https://github.com/user/proj.git#egg=proj" % root)
LookupError: setuptools-scm was unable to detect version for '/home/aceadm/shivam/1603/setuptools_scm-1.15.4'.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj

Answer 24 · 2018-03-16T21:00:22.000Z

how did you obtain the folder you install from?

Answer 25 · 2018-03-16T21:19:50.000Z

I downloaded the folder from this link https://github.com/pypa/setuptools_scm/tree/v1.15.4.

Answer 26 · 2018-03-17T08:27:28.000Z

well, then the error explains you very exactly what you did wrong - git and github have no way to make the direct tarballs able to provide the required metadata, there is a documented 3rd party plugin that can add support for tags, but setuptools_scm doenst use it

Answer 27 · 2018-03-20T16:35:04.000Z

So how can i get it installed.

As suggested on the main github page , what values should i use in setup.py so that i can get in installed.

On the github page this is given:

Setup.py usage
To use setuptools_scm just modify your project's setup.py file like this:

Add 'setuptools_scm' to the setup_requires parameter

Add the use_scm_version parameter and set it to True

E.g.:

from setuptools import setup
setup(
...,
use_scm_version=True,
setup_requires=['setuptools_scm'],
...,
)
Arguments to get_version() (see below) may be passed as a dictionary to use_scm_version. For example:

from setuptools import setup
setup(
...,
use_scm_version = {"root": "..", "relative_to": file},
setup_requires=['setuptools_scm'],
...,
)
Access the version number in your package via pkg_resources

E.g. (PEP-0396):

from pkg_resources import get_distribution, DistributionNotFound
try:
version = get_distribution(name).version
except DistributionNotFound:
# package is not installed
pass

What should i modify in setup.py ?

Answer 28 · 2018-03-20T16:50:01.000Z

@skshivammahajan setuptools_scm is unable to support github provided tarballs/zip files because neither git nor git-hub cares to provide the metadata required to get any idea whats going on

to support tarballs of tags you need to use https://pypi.python.org/pypi/setuptools_scm_git_archive

its still strongly suggested to actually do releases on pypi instead

Answer 29 · 2018-03-21T12:02:23.000Z

@RonnyPfannschmidt
I tried as per your suggestion but I am still facing the same issue. I downloaded the source from https://pypi.python.org/pypi/setuptools_scm_git_archive as suggested by you.

Here is the error log .

[root@linux3 dateutil-2.7.0]# python setup.py install
setup.py:19: UserWarning: python_requires requires setuptools version > 24.3
  UserWarning)
/usr/local/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
your setuptools is too old (<12)
setuptools_scm functionality is degraded
Traceback (most recent call last):
  File "setup.py", line 67, in <module>
    "test": Unsupported
  File "/usr/local/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/local/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/dist.py", line 225, in __init__
    _Distribution.__init__(self,attrs)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/usr/local/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/dist.py", line 258, in finalize_options
    ep.load()(self, ep.name, value)
  File "/usr/local/lib/python2.7/site-packages/setuptools_scm-1.17.0-py2.7.egg/setuptools_scm/integration.py", line 22, in version_keyword
    dist.metadata.version = get_version(**value)
  File "/usr/local/lib/python2.7/site-packages/setuptools_scm-1.17.0-py2.7.egg/setuptools_scm/__init__.py", line 119, in get_version
    parsed_version = _do_parse(root, parse)
  File "/usr/local/lib/python2.7/site-packages/setuptools_scm-1.17.0-py2.7.egg/setuptools_scm/__init__.py", line 97, in _do_parse
    "use git+https://github.com/user/proj.git#egg=proj" % root)
LookupError: setuptools-scm was unable to detect version for '/home/aceadm/shivam/1603/dateutil-2.7.0'.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
[root@linux3 dateutil-2.7.0]#

Answer 30 · 2018-03-21T12:12:00.000Z

@RonnyPfannschmidt

I removed all the version of setuptools_scm. Now I am getting this error.

[root@linux3 dateutil-2.7.0]# python setup.py install
setup.py:19: UserWarning: python_requires requires setuptools version > 24.3
  UserWarning)
Download error on http://pypi.python.org/simple/setuptools_scm/: [Errno -3] Temporary failure in name resolution -- Some packages may not be found!
Download error on http://pypi.python.org/simple/setuptools-scm/: [Errno -3] Temporary failure in name resolution -- Some packages may not be found!
Couldn't find index page for 'setuptools_scm' (maybe misspelled?)
Download error on http://pypi.python.org/simple/: [Errno -3] Temporary failure in name resolution -- Some packages may not be found!
No local packages or download links found for setuptools-scm
Traceback (most recent call last):
  File "setup.py", line 67, in <module>
    "test": Unsupported
  File "/usr/local/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/local/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/dist.py", line 221, in __init__
    self.fetch_build_eggs(attrs.pop('setup_requires'))
  File "/usr/local/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/dist.py", line 245, in fetch_build_eggs
    parse_requirements(requires), installer=self.fetch_build_egg
  File "/usr/local/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/pkg_resources.py", line 588, in resolve
    dist = best[req.key] = env.best_match(req, self, installer)
  File "/usr/local/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/pkg_resources.py", line 833, in best_match
    return self.obtain(req, installer) # try and download/install
  File "/usr/local/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/pkg_resources.py", line 845, in obtain
    return installer(requirement)
  File "/usr/local/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/dist.py", line 295, in fetch_build_egg
    return cmd.easy_install(req)
  File "/usr/local/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/command/easy_install.py", line 598, in easy_install
    raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('setuptools-scm')

Where can i find setuptools version > 24.3 ?

Answer 31 · 2018-03-21T13:02:51.000Z

dateutil provides both a pypi rele4ase and a full git repo, either get a full git repo or use the tarball from pypi
again - please use a git checkout or real release,
the plugin for tags doesnt work unless the repo in question adds support for it
please format your code blocks as code - else its unreadable