espressif/clang-tidy-runner

OSError: [WinError 193] %1 is not a valid Win32 application (RDT-381)

VitorBFreitas opened this issue · 13 comments

VitorBFreitas commented

I'm following the steps in https://docs.espressif.com/projects/esp-idf/en/v4.4.2/esp32/api-guides/tools/idf-clang-tidy.html in a Windows 10, using vscode esp-idf extension terminal, when I try to run 'idf.py clang-check' i get this error:

Executing action: clang-check
Running command: "idf.py -B build reconfigure"...
Traceback (most recent call last):
File "c:\esp\esp-idf\tools\idf.py", line 816, in
main()
File "c:\esp\esp-idf\tools\idf.py", line 734, in main
cli(sys.argv[1:], prog_name=PROG, complete_var='_IDF.PY_COMPLETE')
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1130, in call
return self.main(*args, **kwargs)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1055, in main
rv = self.invoke(ctx)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1689, in invoke
return _process_result(rv)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1626, in _process_result
value = ctx.invoke(self._result_callback, value, **ctx.params)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "c:\esp\esp-idf\tools\idf.py", line 636, in execute_tasks
task(ctx, global_args, task.action_args)
File "c:\esp\esp-idf\tools\idf.py", line 217, in call
self.callback(self.name, context, global_args, **action_args)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\idf_extension.py", line 25, in call_runner
runner()
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 184, in call
self._run(folder, log_fs, output_dir)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 155, in _run
func(folder, log_fs, output_dir)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 198, in _f
return func(self, *args, **kwargs)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 223, in idf_reconfigure
run_cmd(
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\utils.py", line 37, in run_cmd
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs)
File "subprocess.py", line 854, in init
File "subprocess.py", line 1307, in _execute_child
OSError: [WinError 193] %1 não é um aplicativo Win32 válido

hfudev commented

Hi @VitorBFreitas, have you changed your run-clang-tidy.py into an executable file? Please have a try!

VitorBFreitas commented

@hfudev Thanks for the quick reply.
I have changed it into executable through pyinstaller utilizing pyinstaller --oneline path/to/run-clang-tidy.py, I don't know if that's the correct way. I also don't know if needs to include to $PATH the executable file folder, the python script one or both.
I tried changed this line p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) at utils.py in pyclang folder, adding an addtional argument shell=True (kinda reverting this commit 5b584d1) , and executing idf.py clang-check got further, but ends with this error:

Traceback (most recent call last):
File "c:\esp\esp-idf\tools\idf.py", line 816, in
main()
File "c:\esp\esp-idf\tools\idf.py", line 734, in main
cli(sys.argv[1:], prog_name=PROG, complete_var='_IDF.PY_COMPLETE')
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1130, in call
return self.main(*args, **kwargs)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1055, in main
rv = self.invoke(ctx)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1689, in invoke
return _process_result(rv)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1626, in _process_result
value = ctx.invoke(self._result_callback, value, **ctx.params)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "c:\esp\esp-idf\tools\idf.py", line 636, in execute_tasks
task(ctx, global_args, task.action_args)
File "c:\esp\esp-idf\tools\idf.py", line 217, in call
self.callback(self.name, context, global_args, **action_args)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\idf_extension.py", line 25, in call_runner
runner()
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 184, in call
self._run(folder, log_fs, output_dir)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 155, in _run
func(folder, log_fs, output_dir)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 198, in _f
return func(self, *args, **kwargs)
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 314, in run_clang_tidy
self.run_clang_tidy_py,
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 144, in run_clang_tidy_py
if not self.run_clang_tidy_py:
File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 139, in run_clang_tidy_py
if os.path.isfile(os.path.realpath(self._run_clang_tidy_py)):
File "ntpath.py", line 626, in realpath
File "ntpath.py", line 452, in normpath
TypeError: expected str, bytes or os.PathLike object, not NoneType

hfudev commented

@VitorBFreitas Sorry if I misled you. I'm not a Windows user and I don't know much about it. What I meant is actually check the executable flag, not turning it into an .exe file. According to this stackoverflow answer, seems like this issue is quite common on Windows.

Could you provide more info for debugging?

  1. check if you could run the file directly with python, by
$ python.exe run-clang-tidy.py -h
  1. your python version, by
$ python.exe -VV

BTW, based on my python experience, using popen without shell=True would be much better when supporting multiple OSes. IMO that won't be the root cause of this issue.

VitorBFreitas commented

@hfudev
I tried run it with python.exe and without it. Got this:

C:\Windows\System32>python.exe run-clang-tidy.py -h
python.exe: can't open file 'C:\Windows\System32\run-clang-tidy.py': [Errno 2] No such file or directory

C:\Windows\System32>run-clang-tidy.py -h
usage: run-clang-tidy.py [-h] [-allow-enabling-alpha-checkers] [-clang-tidy-binary PATH]
                         [-clang-apply-replacements-binary PATH] [-checks CHECKS]
                         [-config CONFIG | -config-file CONFIG_FILE] [-header-filter HEADER_FILTER]
                         [-line-filter LINE_FILTER] [-j J] [-fix] [-format] [-style STYLE] [-use-color [USE_COLOR]]
                         [-p BUILD_PATH] [-extra-arg EXTRA_ARG] [-extra-arg-before EXTRA_ARG_BEFORE] [-quiet]
                         [-load PLUGINS]
                         [files ...]

Runs clang-tidy over all files in a compilation database. Requires clang-tidy and clang-apply-replacements in $PATH or
in your build directory.

positional arguments:
  files                 files to be processed (regex on path)

optional arguments:
  -h, --help            show this help message and exit
  -allow-enabling-alpha-checkers
                        allow alpha checkers from clang-analyzer.
  -clang-tidy-binary PATH
                        path to clang-tidy binary
  -clang-apply-replacements-binary PATH
                        path to clang-apply-replacements binary
  -checks CHECKS        checks filter, when not specified, use clang-tidy default
  -config CONFIG        Specifies a configuration in YAML/JSON format: -config="{Checks: '*', CheckOptions: {x: y}}"
                        When the value is empty, clang-tidy will attempt to find a file named .clang-tidy for each
                        source file in its parent directories.
  -config-file CONFIG_FILE
                        Specify the path of .clang-tidy or custom config file: e.g. -config-
                        file=/some/path/myTidyConfigFile. This option internally works exactly the same way as -config
                        option after reading specified config file. Use either -config-file or -config, not both.
  -header-filter HEADER_FILTER
                        regular expression matching the names of the headers to output diagnostics from. Diagnostics
                        from the main file of each translation unit are always displayed.
  -line-filter LINE_FILTER
                        List of files with line ranges to filter thewarnings.
  -j J                  number of tidy instances to be run in parallel.
  -fix                  apply fix-its
  -format               Reformat code after applying fixes
  -style STYLE          The style of reformat code after applying fixes
  -use-color [USE_COLOR]
                        Use colors in diagnostics, overriding clang-tidy's default behavior. This option overrides the
                        'UseColor' option in .clang-tidy file, if any.
  -p BUILD_PATH         Path used to read a compile command database.
  -extra-arg EXTRA_ARG  Additional argument to append to the compiler command line.
  -extra-arg-before EXTRA_ARG_BEFORE
                        Additional argument to prepend to the compiler command line.
  -quiet                Run clang-tidy in quiet mode
  -load PLUGINS         Load the specified plugin in clang-tidy.

About the version:

C:\esp\llvm-project-main\clang-tools-extra\clang-tidy\tool>python.exe -VV
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]
igrr commented

I think the logic we are looking for is:

  • find the clang-tidy in directories listed in PATH
  • derive the full path of run-clang-tidy.py from the path of clang-tidy
  • call {sys.interpreter} {full_path}

Or, we could stop using run-clang-tidy.py. The logic from it which we actually need seems to be about 100 lines of code... We could copy it and call clang-tidy directly.

VitorBFreitas commented

Ok, it's still kinda confusing to me, so I'll list how I did all those steps:

  • find the clang-tidy in directories listed in PATH:
    I went to Windows Environment Variables -> System Variable section -> PATH -> Edit -> Copy C:\esp\llvm-project-
    main\clang-tools-extra\clang-tidy\tool

  • derive the full path of run-clang-tidy.py from the path of clang-tidy:
    I went to copied directory -> Copy the full path of run-clang-tidy.py (C:\esp\llvm-project-main\clang-tools-extra\clang-
    tidy\tool\run-clang-tidy.py)

  • call {sys.interpreter} {full_path}:
    Opened CMD.exe then,

C:\Windows\System32>py C:\esp\llvm-project-main\clang-tools-extra\clang-tidy\tool\run-clang-tidy.py
   Error: could not find compilation database.
igrr commented

Sorry for not being clear. That was a comment about what we should do in the code of this project, not an instruction for you to follow.

As your message indicates, run-clang-tidy.py was executed correctly, but produced an error because there were no arguments supplied.

VitorBFreitas commented

@igrr
Oh, ok. I supposed could be that, but was not sure.
So should I wait for a patch for this clang-tidy feature integrated to esp-idf or it's still a workaround to make it work?

hfudev commented

I think the logic we are looking for is:

  • find the clang-tidy in directories listed in PATH
  • derive the full path of run-clang-tidy.py from the path of clang-tidy
  • call {sys.interpreter} {full_path}

Or, we could stop using run-clang-tidy.py. The logic from it which we actually need seems to be about 100 lines of code... We could copy it and call clang-tidy directly.

@igrr Here it is using the full path: https://github.com/espressif/clang-tidy-runner/blob/v0.2.2/pyclang/runner.py#L143

hfudev commented

idf.py is not using [sys.executable, full_path, ...] approach. Created one PR to fix it.

hfudev commented

@VitorBFreitas The new bugfix version is published. please run pip install -U pyclang to upgrade the version.

VitorBFreitas commented

@hfudev

After upgrade I got this:

Executing action: clang-check
Running command: "c:\esp\tools\python_env\idf4.4_py3.8_env\Scripts\python.exe c:\esp\tools\tools\idf-exe\1.0.3\idf.py.EXE -B build reconfigure"...
  File "c:\esp\tools\tools\idf-exe\1.0.3\idf.py.EXE", line 1
SyntaxError: Non-UTF-8 code starting with '\x90' in file c:\esp\tools\tools\idf-exe\1.0.3\idf.py.EXE on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
Traceback (most recent call last):
  File "c:\esp\esp-idf\tools\idf.py", line 816, in <module>
    main()
  File "c:\esp\esp-idf\tools\idf.py", line 734, in main
    cli(sys.argv[1:], prog_name=PROG, complete_var='_IDF.PY_COMPLETE')
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1689, in invoke
    return _process_result(rv)
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 1626, in _process_result
    value = ctx.invoke(self._result_callback, value, **ctx.params)
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "c:\esp\esp-idf\tools\idf.py", line 636, in execute_tasks
    task(ctx, global_args, task.action_args)
  File "c:\esp\esp-idf\tools\idf.py", line 217, in __call__
    self.callback(self.name, context, global_args, **action_args)
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\idf_extension.py", line 25, in call_runner
    runner()
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 187, in __call__
    self._run(folder, log_fs, output_dir)
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 158, in _run
    func(folder, log_fs, output_dir)
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 201, in _f
    return func(self, *args, **kwargs)
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\runner.py", line 226, in idf_reconfigure
    run_cmd(
  File "c:\esp\tools\python_env\idf4.4_py3.8_env\lib\site-packages\pyclang\utils.py", line 56, in run_cmd
    raise RuntimeError(f'command "{cmd_str}" failed with exitcode {returncode}')
RuntimeError: command "c:\esp\tools\python_env\idf4.4_py3.8_env\Scripts\python.exe c:\esp\tools\tools\idf-exe\1.0.3\idf.py.EXE -B build reconfigure" failed with exitcode 1
hfudev commented

@VitorBFreitas I think the reason is that the idf.py.exe is messing things up while finding idf.py. I've created #29 to track it. Thanks for reporting.

  • 👍1