Add "tar.gz" for local PyPi package scanning supports
coffeehb opened this issue · 6 comments
I have a lot of Pypi native packages of the source file type, when I try to scan it I get an error.
./run_analysis.sh -ecosystem pypi -package djanga -local /opt/pypi/evalpackages/djanga-0.3.tar.gz
The reason for the error is that the sandbox environment does not support source code installation.
/src/internal/log/writer.go:40
2023-03-10T02:50:48.607Z WARN log/writer.go:63 subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '--pre', '/djanga-0.3.tar.gz')' returned non-zero exit status 1. {"args": ["/usr/local/bin/analyze-python.py", "--local", "/djanga-0.3.tar.gz", "install", "djanga"]}
github.com/ossf/package-analysis/internal/log.WriteTo
/src/internal/log/writer.go:63
github.com/ossf/package-analysis/internal/log.Writer.func1
/src/internal/log/writer.go:40
2023-03-10T02:50:48.730Z DEBUG sandbox/sandbox.go:240 podman {"args": ["--cgroup-manager=cgroupfs", "--events-backend=file", "stop", "64c62e04a7c86fc8f47df72ca15aea5e4f9f3c8eb9b83ce4f50753637c47778d"]}
2023-03-10T02:50:53.946Z WARN log/writer.go:63 destroying container: stopping container: gofer is still running {"args": ["/usr/local/bin/analyze-python.py", "--local", "/djanga-0.3.tar.gz", "install", "djanga"]}
github.com/ossf/package-analysis/internal/log.WriteTo
/src/internal/log/writer.go:63
github.com/ossf/package-analysis/internal/log.Writer.func1
/src/internal/log/writer.go:40
2023-03-10T02:50:53.980Z WARN log/writer.go:63 Error: error removing container 64c62e04a7c86fc8f47df72ca15aea5e4f9f3c8eb9b83ce4f50753637c47778d from runtime: `/usr/local/bin/runsc_compat.sh delete --force 64c62e04a7c86fc8f47df72ca15aea5e4f9f3c8eb9b83ce4f50753637c47778d` failed: exit status 128 {"args": ["/usr/local/bin/analyze-python.py", "--local", "/djanga-0.3.tar.gz", "install", "djanga"]}
github.com/ossf/package-analysis/internal/log.WriteTo
/src/internal/log/writer.go:63
github.com/ossf/package-analysis/internal/log.Writer.func1
/src/internal/log/writer.go:40
Are you sure it's a valid package? The logs you posted don't have an explicit error message, so perhaps pip failed to install the package because it was invalid.
Package Analysis definitely supports source packages, as well as installing from local archives.
What happens if you try the requests source package? Here is the direct download link to the archive.
When I download this and run scripts/run_analysis.sh -ecosytem pypi -package requests -local /path/to/requests-2.28.2.tar.gz, it runs fine :)
The error log of the scanned malicious package is as follows。
error.log
Sometimes the malicious package we want to analyze is not necessarily a valid package, it just needs to achieve its purpose during the installation process.
the package like this:
djanga-0.3.tar.gz
Aha, this is a regression bug in error handling. Thank you!
You are right that a package does not have to be valid to be malicious. This is not an error condition, and should not cause early termination. However, a recent change in the local analysis code resulted in this happening. I have pushed a fix :)
Note, until the fix is merged, a workaround is to modifying scripts/run_analysis.sh to not check the exit code of the docker process and remove the results dirs. In other words, simply delete lines 197 and 202-212 inclusive.
By the way, I would like to ask how should I scan a list of local files?
does -list supports set a file directory?
You can use the scripts/analyse-tarballs.sh file. Currently it only supports static analysis but I will push a change that enables both dynamic and static analysis. You will be able to edit the script to disable static analysis if you desire.
Please note: even though we run analysis inside a sandbox, the firewall may not block access to other hosts on your network. If you are running a lot of local analysis, it would be ideal to run it on a network which is isolated from other devices.
PR is in #687