Getting error in checkmate issue

Question

Getting error in checkmate issue

Closed this issue 5 months ago · 12 comments

Hi,
I trying to add betterscan in Azure devops pipeline using cli image. But getting error at last step
docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate issues'

Traceback (most recent call last):
File "/usr/local/bin/checkmate", line 33, in
sys.exit(load_entry_point('checkmate==0.2.0', 'console_scripts', 'checkmate')())
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/scripts/manage.py", line 114, in main
result = command.run()
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/management/commands/issues.py", line 65, in run
if issue["line"]==1:
File "/usr/local/lib/python3.8/site-packages/blitzdb3_ce-4.0.0-py3.8.egg/blitzdb/document.py", line 191, in getitem
return self.attributes[key]
KeyError: 'line'

Could you please help

Answer 1 · 2023-02-07T13:10:46.000Z

@sohan1812 Hi, maybe your state is messed up.

Please remove from git repo .checkmate folder or in your run, to make sure it is created from scratch.

Answer 2 · 2023-02-13T06:46:25.000Z

@marcinguy Hi, I have tried after removing everything still the same error.

docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate git analyze --branch main'
Analyzing the 1 most recent commits in branch main (offset: 0)
Excluding 0 file revisions
Analyzing 1 new file revisions (0 are already analyzed)
Analyzing and saving: 0 - 1 (1 remaining)
0%| | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/lib/code/environment.py", line 561, in analyze_file_revision
analyzer_results = analyzer.analyze(file_revision)
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/contrib/plugins/all/fluidattacksscanner/analyzer.py", line 72, in analyze
next(reader)
StopIteration

100%|█████████████████████████████████████████████████████████████████████████| 1/1 [03:26<00:00, 206.54s/it]
Annotating and saving file revisions...
Summarizing file revisions...
Saving snapshot...
Done analyzing snapshot 73fbe0d667b6463b8a459b6361ed313e

docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate issues'
Traceback (most recent call last):
File "/usr/local/bin/checkmate", line 33, in
sys.exit(load_entry_point('checkmate==0.2.0', 'console_scripts', 'checkmate')())
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/scripts/manage.py", line 114, in main
result = command.run()
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/management/commands/issues.py", line 65, in run
if issue["line"]==1:
File "/usr/local/lib/python3.8/site-packages/blitzdb3_ce-4.0.0-py3.8.egg/blitzdb/document.py", line 191, in getitem
return self.attributes[key]
KeyError: 'line'

Answer 3 · 2023-02-13T08:14:46.000Z

@sohan1812 That helps more. Do you have internet connection active? Seems like fluidattacksscanner does not produce results.

Can you make sure you remove all the docker images and get the latest one?

Answer 4 · 2023-02-13T13:32:37.000Z

@marcinguy Hi, Yes it was due to internet connection & fluidattacksscanner is fixed. Now when I run "docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate git analyze --branch main'" on cloned git repo it works fine but when I run this in Azure Devops pipeline it is showing following error

Traceback (most recent call last):
File "/usr/local/bin/checkmate", line 33, in
sys.exit(load_entry_point('checkmate==0.2.0', 'console_scripts', 'checkmate')())
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/scripts/manage.py", line 114, in main
result = command.run()
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/contrib/plugins/git/commands/analyze.py", line 354, in run
analyzed_snapshots, diffs = self.analyze_and_generate_diffs(branch_name,
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/contrib/plugins/git/commands/analyze.py", line 104, in analyze_and_generate_diffs
git_settings = self.project.git.get_settings()
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/contrib/plugins/git/models.py", line 53, in get_settings
latest_commit = self.repository.get_commits(
IndexError: list index out of range

PS- I am using azure classic pipeline with selfhosted agent & betterscan as bash task as following

export CODE_DIR=${PWD}
cd $CODE_DIR
docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate init'
docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate git init'
docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate git analyze --branch $(Build.SourceBranchName)'
docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate issues html'

Answer 5 · 2023-02-13T13:41:37.000Z

@sohan1812 Add this before cd $CODE_DIR everywhere

git config --global --add safe.directory $CODE_DIR
or
git config --global --add safe.directory $(Build.SourcesDirectory)

See this:

https://github.com/marcinguy/betterscan-ce#azure-devops-integration

or

https://github.com/marcinguy/betterscan-ce/blob/master/cli-html.sh

Answer 6 · 2023-02-13T14:04:18.000Z

@sohan1812

I recommend above but you can also try this:

export CODE_DIR=${PWD}
cd $CODE_DIR
docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'git config --global --add safe.directory $CODE_DIR && cd $CODE_DIR && checkmate init'
docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'git config --global --add safe.directory $CODE_DIR && cd $CODE_DIR && checkmate git init'
docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'git config --global --add safe.directory $CODE_DIR && cd $CODE_DIR && checkmate git analyze --branch $(Build.SourceBranchName)'
docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'git config --global --add safe.directory $CODE_DIR && cd $CODE_DIR && checkmate issues html'

git config --global --add safe.directory $CODE_DIR is needed due to recent change in git

Answer 7 · 2023-02-13T15:14:59.000Z

@marcinguy Hi, I have changed the step as recommended by you but still same error

Analyzing the 1 most recent commits in branch main (offset: 0)
Traceback (most recent call last):
File "/usr/local/bin/checkmate", line 33, in
sys.exit(load_entry_point('checkmate==0.2.0', 'console_scripts', 'checkmate')())
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/scripts/manage.py", line 114, in main
result = command.run()
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/contrib/plugins/git/commands/analyze.py", line 354, in run
analyzed_snapshots, diffs = self.analyze_and_generate_diffs(branch_name,
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/contrib/plugins/git/commands/analyze.py", line 104, in analyze_and_generate_diffs
git_settings = self.project.git.get_settings()
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/contrib/plugins/git/models.py", line 53, in get_settings
latest_commit = self.repository.get_commits(
IndexError: list index out of range

Answer 8 · 2023-02-13T15:40:06.000Z

@sohan1812 Try using the branch name that exist, maybe there is something wrong with passing branch name, i.e

docker run -e CODE_DIR -v ${PWD}:${PWD} -ti scanmycode/scanmycode3-ce:worker-cli /bin/sh -c 'git config --global --add safe.directory $CODE_DIR && cd $CODE_DIR && checkmate git analyze --branch main'

Answer 9 · 2023-02-15T05:47:30.000Z

@marcinguy Hi, Yes you are right this was due to branch name. I have added "git checkout -b $(Build.SourceBranchName)" & now its doing analyzing but stops at fluidattacksscanner even I have internet connectivity.

Analyzing 13 new file revisions (0 are already analyzed)
Analyzing and saving: 0 - 10 (13 remaining)

0% 0/10 [00:00<?, ?it/s]Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/lib/code/environment.py", line 561, in analyze_file_revision
analyzer_results = analyzer.analyze(file_revision)
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/contrib/plugins/all/fluidattacksscanner/analyzer.py", line 72, in analyze
next(reader)
StopIteration

10% 1/10 [05:23<48:34, 323.82s/it]Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/lib/code/environment.py", line 561, in analyze_file_revision
analyzer_results = analyzer.analyze(file_revision)
File "/usr/local/lib/python3.8/site-packages/checkmate-0.2.0-py3.8.egg/checkmate/contrib/plugins/all/fluidattacksscanner/analyzer.py", line 72, in analyze
next(reader)
StopIteration

Answer 10 · 2023-02-15T10:36:17.000Z

@sohan1812

You hit some odd issue. Works on my end.

Do you use Azure? Or standalone windows server? What is you IaaS? What location?

Make sure you pull always the latest dockerhub image

I added a change that it should not stop on Fludattacksscanner error (if it happens) and continue further (without Fluidattacksscanner though)

Answer 11 · 2023-02-15T11:17:02.000Z

@marcinguy Hi, Yes I am using Azure Linux(Red Hat Enterprise Linux release 8.7) VM , Location eastus2. Also I am using podman instead of docker.

Answer 12 · 2024-04-15T09:24:23.000Z

Closing due to inactivity