Submodules causing issues

[jaapmarcus]

For my project "we" include some submodules with:

git submodule update --init --recursive

When importing the "main" project

[WARNING / 2022-02-14 13:16:50] Cannot read source file: test/test_helper/bats-assert [ERROR / 2022-02-14 13:16:50] Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/checkmate-0.2.0-py2.7.egg/checkmate/lib/code/environment.py", line 546, in analyze_file_revision analyzer_results = analyzer.analyze(file_revision) File "/usr/local/lib/python2.7/site-packages/checkmate-0.2.0-py2.7.egg/checkmate/contrib/plugins/all/semgrep/analyzer.py", line 46, in analyze f.write(file_revision.get_file_content()) File "/usr/local/lib/python2.7/site-packages/checkmate-0.2.0-py2.7.egg/checkmate/lib/models.py", line 155, in get_file_content return self._file_content() File "/usr/local/lib/python2.7/site-packages/checkmate-0.2.0-py2.7.egg/checkmate/contrib/plugins/git/models.py", line 109, in <lambda> file_revision._file_content = lambda commit_sha = commit_sha, file_revision = file_revision: self.repository.get_file_content(commit_sha,file_revision.path) File "/usr/local/lib/python2.7/site-packages/checkmate-0.2.0-py2.7.egg/checkmate/contrib/plugins/git/lib/repository.py", line 517, in get_file_content raise IOError IOError

Example:
https://github.com/hestiacp/hestiacp/tree/main/test/test_helper

[marcinguy]

Hi @jaapmarcus! Thanks for your issue.

Looked briefly into this.

You can look here: https://github.com/marcinguy/scanmycode-ce/blob/0ad67062a87f14f8847e3380fe7abb0b31d8d4bc/quantifiedcode/plugins/git/backend/tasks/analyze.py#L51

Before this you can check if submodules are there:

https://github.com/marcinguy/checkmate-ce/blob/fc33c7c27bc640ab4db5dbda274a0edd3b3db218/checkmate/contrib/plugins/git/lib/repository.py#L411

And if yes, make a call to via a method def init_submodules(self) with maybe something along the lines you mention:

git submodule update --init --recursive

You can look here how git is called:

https://github.com/marcinguy/checkmate-ce/blob/fc33c7c27bc640ab4db5dbda274a0edd3b3db218/checkmate/contrib/plugins/git/lib/repository.py#L328

Feel free to make a PR if you manage to achieve it. When I have time I can look into this or anybody else can.

[marcinguy]

@jaapmarcus You can use git log to get commits for each submodules like this:

    def get_commits_sub(self,branch = None,offset = 0,limit = 0,shas = None,params = None,
                    from_to = None,args = None, **kwargs):

        split_sequence = '---a4337bc45a8fc544c03f52dc550cd6e1e87021bc896588bd79e901e2---'

        try:
            base_args = ["git",
                         "submodule",
                         "foreach",
                         "git",
                         "--no-pager",
                         "log",
                         "--date=raw",
                         "--pretty=format:%H:-:%ct:-:%cn:-:%ce:-:%at:-:%an:-:%ae:-:%P:-:%T:-:%n%B%n"+split_sequence+"%n"]
            extra_args = []

By fetching you can init the submodules in:

def fetch(self,remote = "origin",branch = None,ssh_identity_file = None,git_config = None,git_credentials = None):

Somewhere at the end:

return_code,stdout = self.call(["git","submodule","update","--init","--recursive"],env = env,timeout = 120)

Or in Fetch Task:

/srv/scanmycode/quantifiedcode/plugins/git/backend/tasks/fetch.py

There could be also other things involved.

Let me know if you are still interested/working on this.

Maybe somebody else using this feature can also help.

Thanks,

[marcinguy]

Closing due inactivity