chaoss/grimoirelab-graal

CoLic throwing path error

altsalt opened this issue · 4 comments

altsalt commented

Hello, I am participating in the MSR Hackathon and have attempted to install Graal a few times to no avail. Most recently, I set up a clean Debian 11 VM to verify that it wasn't due to prior system configuration, incorrect permissions, etc. Alas, the same error is appearing:

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from graal.backends.core.colic import CoLic
>>> repo_uri = 'https://github.com/chaoss/grimoirelab-graal'
>>> repo_dir = 'grimoirelab-graal'
>>> cl = CoLic(uri=repo_uri, git_path=repo_dir)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/dist-packages/graal-0.2.9-py3.9.egg/graal/backends/core/colic.py", line 79, in __init__
    if not GraalRepository.exists(exec_path):
  File "/usr/local/lib/python3.9/dist-packages/graal-0.2.9-py3.9.egg/graal/graal.py", line 411, in exists
    return os.path.exists(dest)
  File "/usr/lib/python3.9/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
>>>

I have tried with both a blank directory and having pre-created the grimoirelab-graal folder (not that this should be a blocker, but looking for anything that might be...)

Any pointers or guidance would be very much appreciated. Thanks!

altsalt commented

I've spent hours trying to get graal and scancode to play nice to no avail. I think this is largely due to my python path setup, but given that I have a fresh VM that only being used for this, there is likely an issue with the build instructions.

My latest attempted has graal being built into scancode-toolkit/bin using --prefix. This command is then successful:
(scancode-toolkit) $ graal colic https://github.com/chaoss/grimoirelab-graal --exec-path /home/salt/MSR-Magna_Carta/scancode-toolkit/bin/scancode --category code_license_scancode > ./scancode_test

However, as a script it continues to throw a ModuleNotFoundError. I've tried loading it directly via /home/salt/scancode-toolkit/bin/python3 test.py with or without the virtualenv active. I've also tried a variety of sys.path.append calls. I'm just not sure what I'm missing here....

The test script

#!/home/salt/MSR-Magna_Carta/scancode-toolkit/bin/python3

#import sys
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/bin')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/lib')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/python3.9/site-packages/')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/python3.9/site-packages/graal-0.2.9-py3.9.egg/graal/')

from graal.backends.core.colic import CoLic

# URL for the git repo to analyze
repo_uri = 'https://github.com/chaoss/grimoirelab-graal'

# directory where to mirror the repo
repo_dir = 'grimoirelab-graal'

# CoLic object initialization
cl = CoLic(uri=repo_uri, git_path=repo_dir)

# fetch all commits
commits = [commit for commit in cl.fetch()]

with open("graal_test_commits", "w") as f:
  f.write(commits)

Some of the commands and errors which have gone by

(scancode-toolkit) $ graal colic https://github.com/chaoss/grimoirelab-graal --exec-path /home/salt/MSR-Magna_Carta/scancode-toolkit/etc/scripts/scancli.py  --category code_license_scancode_cli > ./scancode_cli_test
[2021-12-07 11:40:16,258] - Starting the quest for the Graal.
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
[2021-12-07 11:40:16,592] - Error!: Command '['/home/salt/MSR-Magna_Carta/scancode-toolkit/configure']' returned non-zero exit status 1.
[2021-12-07 11:40:16,593] - Quest completed.
valeriocos commented

Hi @altsalt,

I followed the steps here https://github.com/chaoss/grimoirelab-graal#how-to-installcreate-the-executables, section ScanCode

cd /tmp
git clone https://github.com/nexB/scancode-toolkit.git
cd scancode-toolkit
git checkout -b test_scancli 96069fd84066c97549d54f66bd2fe8c7813c6b52
./scancode --help

pip install simplejson execnet

The execution from command line and python were OK

  • command line
graal colic https://github.com/chaoss/grimoirelab-graal --git-path /tmp/test --exec-path /tmp/scancode-toolkit/etc/scripts/scancli.py --category code_license_scancode_cli
  • python
from graal.backends.core.colic import CoLic

# URL for the git repo to analyze
repo_uri = 'https://github.com/chaoss/grimoirelab-graal'

# directory where to mirror the repo
repo_dir = '/tmp/grimoirelab-graal'

# CoLic object initialization
cl = CoLic(uri=repo_uri, git_path=repo_dir, exec_path='/tmp/scancode-toolkit/etc/scripts/scancli.py')

# fetch all commits
commits = [commit for commit in cl.fetch(category='code_license_scancode_cli')] <--- this call may take a lot of time, you may want to save each commit in the file below.

with open("graal_test_commits", "w+") as f:
  f.write(commits)

Hope this helps

EDIT:

Also the category code_license_scancode seems to work fine

  • command line
graal colic https://github.com/chaoss/grimoirelab-graal --git-path /tmp/test --exec-path /tmp/scancode-toolkit/scancode --category code_license_scancode
  • python
from graal.backends.core.colic import CoLic

# URL for the git repo to analyze
repo_uri = 'https://github.com/chaoss/grimoirelab-graal'

# directory where to mirror the repo
repo_dir = '/tmp/grimoirelab-graal'

# CoLic object initialization
cl = CoLic(uri=repo_uri, git_path=repo_dir, exec_path='/tmp/scancode-toolkit/scancode')

# fetch all commits
commits = []
for commit in cl.fetch(category='code_license_scancode'):
  commits.append(commit)
  print(commit)

with open("graal_test_commits", "w+") as f:
  f.write(commits)
altsalt commented

An error has been thrown in step 1. I'll install the missing package and keep pushing through, but it may point to part of the problem.

salt@debianclivm:/tmp/scancode-toolkit$ ./scancode --help
* Configuring ScanCode for first use...
/tmp/scancode-toolkit/configure: line 24: python2.7: command not found
./scancode: line 114: /tmp/scancode-toolkit/bin/scancode: No such file or directory

FWIW, I had been building ScanCode from the main branch rather than checking one out.