Standalone script errors.
assumptionsoup opened this issue · 20 comments
Out of the box:
>>> python script.py
Clang directory = C:\Program Files\LLVM\bin
Platform: ('32bit', 'WindowsPE')
Python version: 3.5.1
filename = C:\Program Files\LLVM\bin\libclang.dll
Traceback (most recent call last):
File "C:\Users\jordan\work\test-complete-plugin\clang\cindex38.py", line 3615, in get_cindex_library
library = cdll.LoadLibrary(self.get_filename())
File "C:\Program Files (x86)\Python35-32\lib\ctypes\__init__.py", line 425, in LoadLibrary
return self._dlltype(name)
File "C:\Program Files (x86)\Python35-32\lib\ctypes\__init__.py", line 347, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 193] %1 is not a valid Win32 application
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "script.py", line 21, in <module>
tu = TU.from_source(filename=filename, args=clang_flags)
File "C:\Users\jordan\work\test-complete-plugin\clang\cindex38.py", line 2344, in from_source
index = Index.create()
File "C:\Users\jordan\work\test-complete-plugin\clang\cindex38.py", line 2231, in create
return Index(conf.lib.clang_createIndex(excludeDecls, 0))
File "C:\Users\jordan\work\test-complete-plugin\clang\cindex38.py", line 144, in __get__
value = self.wrapped(instance)
File "C:\Users\jordan\work\test-complete-plugin\clang\cindex38.py", line 3594, in lib
lib = self.get_cindex_library()
File "C:\Users\jordan\work\test-complete-plugin\clang\cindex38.py", line 3620, in get_cindex_library
raise LibclangError(msg)
clang.cindex38.LibclangError: [WinError 193] %1 is not a valid Win32 application. To provide a path to libclang use Config.set_library_path() or Config.set_library_file().
I have these prebuilt binaries installed: http://llvm.org/releases/3.8.0/LLVM-3.8.0-win64.exe
Other interesting things of note:
Clang directory = C:\Program Files\LLVM\bin # Correct
Platform: ('32bit', 'WindowsPE') # INCORRECT. 64bit, Windows 10 - this was wrong through cmd.exe and GitBash
Python version: 3.5.1 # Correct
filename = C:\Program Files\LLVM\bin\libclang.dll # Correct
Yes, the error occurs when you are using 32 bit python. Please make sure you install python of 64 bit architecture if you are using 64 bit clang and windows.
Ah, okay. Good point. I'll grab the 64 bit version and try it out. Thanks.
Okay, yeah, I'm getting everything as expected now. Standalone works:
>>> py -3.5 ./script.py
Clang directory = C:\Program Files\LLVM\bin
Platform: ('64bit', 'WindowsPE')
Python version: 3.5.1
filename = C:\Program Files\LLVM\bin\libclang.dll
59
59
Sublime Text 3 does not.
Clang directory = C:\Program Files\LLVM\bin
Platform: ('64bit', 'WindowsPE')
Python version: 3.3.6
filename = C:\Program Files\LLVM\bin\libclang.dll
Traceback (most recent call last):
File "C:\Program Files\Sublime Text 3\sublime_plugin.py", line 109, in reload_plugin
m = importlib.import_module(modulename)
File "./python3.3/importlib/__init__.py", line 90, in import_module
File "<frozen importlib._bootstrap>", line 1584, in _gcd_import
File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
File "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
File "<frozen importlib._bootstrap>", line 1022, in load_module
File "<frozen importlib._bootstrap>", line 1003, in load_module
File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
File "<frozen importlib._bootstrap>", line 868, in _load_module
File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
File "C:\Users\jordan\AppData\Roaming\Sublime Text 3\Packages\test-complete-plugin\script.py", line 21, in <module>
tu = TU.from_source(filename=filename, args=clang_flags)
File "C:\Users\jordan\AppData\Roaming\Sublime Text 3\Packages\test-complete-plugin\clang\cindex38.py", line 2372, in from_source
raise TranslationUnitLoadError("Error parsing translation unit.")
clang.cindex38.TranslationUnitLoadError: Error parsing translation unit.
And later EasyClangComplete will error twice. This one:
Traceback (most recent call last):
File "C:\Program Files\Sublime Text 3\sublime_plugin.py", line 530, in on_activated_async
callback.on_activated_async(v)
File "C:\Users\jordan\AppData\Roaming\Sublime Text 3\Packages\EasyClangComplete\EasyClangComplete.py", line 92, in on_activated_async
if not completer.needs_init(view):
AttributeError: 'Completer' object has no attribute 'needs_init'
And the more expected:
ERROR:EasyClangComplete.plugin.completion.lib_complete: cannot complete. No translation unit for view 31
I gotta admit, this error is super weird. I was curious if something was wrong with the ctypes in sublime text. I tried swapping the ctypes directory in the sublime text install with one from python 3.3 and 3.5. They produce the same error, which is also kind of surprising because I half expected a different error from 3.5 with this Frankenstein python install. I also added a print statement to the ctypes library, so I'm sure this version was being picked up by at least one script in sublime text.
So, I guess that rules out a problem with cytpes.
@assumptionsoup thanks, that's a great input. I was blaming ctypes to be honest. Now it got even weirder. I will update my stack overflow question with your comment.
Cool, thanks. Where's the stack overflow question? I'm curious what the responses are (if any).
OH. So I had tried replacing ctypes in the python3.3.zip that sublime text uses. This zip file appears to hold the Lib/ directory of a python install. This means it does not include the libs/ directory. Ctypes heavily imports from _ctypes, which is a dll shipped with python in libs/. So, there's still a chance that the _ctypes library compiled into python33.dll(?) has something wrong with it.
I wonder if there's some way to recreate python33.dll... or point sublime text's python at a real python install.
Here is the question on Stack Overflow: link
I wonder if there is python33.dll or alike somewhere in the system for the real install. dll has to be a dynamic library, so theoretically we should be able to move it around.
Sorry, I meant to say, I don't know if sublime is packaging these python dlls into python33.dll, or into the executable. Despite trying to use a c++ package, I am very much a newbie to c++, so I don't know how plausible these cases are.
To test if ctypes is the problem, I would need to somehow make your test case use a different _ctypes dll. Here's what I've tried:
>>> import _ctypes
>>> _ctypes
<module '_ctypes' (built-in)>
>>> importlib.machinery
>>> _ctypes = importlib.machinery.SourceFileLoader('_ctypes','C:\\Python33\\DLLs\\_ctypes.pyd').load_module()
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
File "<frozen importlib._bootstrap>", line 1022, in load_module
File "<frozen importlib._bootstrap>", line 1003, in load_module
File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
File "<frozen importlib._bootstrap>", line 853, in _load_module
File "<frozen importlib._bootstrap>", line 980, in get_code
File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
TypeError: source code string cannot contain null bytes
Uhhh. Huh.
Oh wait. Got it.
import importlib.machinery
_ctypes = importlib.machinery.ExtensionFileLoader('_ctypes','C:\\Python33\\DLLs\\_ctypes.pyd').load_module()
import sys
sys.modules['_ctypes'] = _ctypes
I put this into the top of the ctypes.__init__ file. But I still get the same error in the test script. I'm about 90% sure that this is picking up the real ctypes dll.
reloading plugin test-complete-plugin.script
we are on 'Windows'
searching for: 'libclang.dll'
libclang search output = 'C:\Program Files\LLVM\bin\'
real output: C:\Program Files\LLVM\bin\
architecture: ('64bit', 'WindowsPE')
found libclang dir: 'C:\Program Files\LLVM\bin'
found library file: 'libclang.dll'
Clang directory = C:\Program Files\LLVM\bin
Platform: ('64bit', 'WindowsPE')
Python version: 3.3.6
filename = C:\Program Files\LLVM\bin\libclang.dll
0
Traceback (most recent call last):
File "C:\Program Files\Sublime Text 3\sublime_plugin.py", line 109, in reload_plugin
m = importlib.import_module(modulename)
File "./python3.3/importlib/__init__.py", line 90, in import_module
File "<frozen importlib._bootstrap>", line 1584, in _gcd_import
File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
File "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
File "<frozen importlib._bootstrap>", line 1022, in load_module
File "<frozen importlib._bootstrap>", line 1003, in load_module
File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
File "<frozen importlib._bootstrap>", line 868, in _load_module
File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
File "C:\Users\jordan\AppData\Roaming\Sublime Text 3\Packages\test-complete-plugin\script.py", line 21, in <module>
tu = TU.from_source(filename=filename, args=clang_flags)
File "C:\Users\jordan\AppData\Roaming\Sublime Text 3\Packages\test-complete-plugin\clang\cindex38.py", line 2372, in from_source
raise TranslationUnitLoadError("Error parsing translation unit.")
clang.cindex38.TranslationUnitLoadError: Error parsing translation unit.
You rock man!
Maybe we should do it vise versa? Like try breaking the script when run from PowerShell? Trying to make it use sublime internal stuff?
Oh yeah, python is awesome like that! You can shoot yourself in the foot before you know you pulled the trigger :D
The reason I want to find out if it's ctypes is I want to make a bug report to sublime text if it is. While it might be possible to break it the other way around, I'm not sure how useful it would be. I've made a thread on the sublime text forum too: https://forum.sublimetext.com/t/can-anyone-confirm-that-the-ctypes-module-works-correctly/21465
I hacked a little more at trying to get sublime text to run the not-shipped cytpes unit tests:
import sys
import importlib
_testcapi = importlib.machinery.ExtensionFileLoader('_testcapi','C:\\Python33\\DLLs\\_testcapi.pyd').load_module()
sys.modules['_testcapi'] = _testcapi
_ctypes_test = importlib.machinery.ExtensionFileLoader('_ctypes_test','C:\\Python33\\DLLs\\_ctypes_test.pyd').load_module()
sys.modules['_ctypes_test'] = _ctypes_test
# import ctypes
sys.argv = ['','-v']
# _ctypes_test = ctypes.test
import unittest
import ctypes.test
ctypes.test.main(ctypes.test)
test_noargs (ctypes.test.test_win32.FunctionCallTestCase) ... ok
test_struct_by_value (ctypes.test.test_win32.Structures) ... ok
test_winerror (ctypes.test.test_win32.TestWinError) ... ok
test_COMError (ctypes.test.test_win32.TestWintypes) ... ok
test_HWND (ctypes.test.test_win32.TestWintypes) ... ok
test_PARAM (ctypes.test.test_win32.TestWintypes) ... ok
test_qsort (ctypes.test.test_libc.LibTest) ... ok
test_sqrt (ctypes.test.test_libc.LibTest) ... ok
test_chararray (ctypes.test.test_delattr.TestCase) ... ok
test_simple (ctypes.test.test_delattr.TestCase) ... ok
test_struct (ctypes.test.test_delattr.TestCase) ... ok
test_simple (ctypes.test.test_pickling.PickleTest) ... ok
test_struct (ctypes.test.test_pickling.PickleTest) ... ok
test_unpickable (ctypes.test.test_pickling.PickleTest) ... ok
test_wchar (ctypes.test.test_pickling.PickleTest) ... ok
test_simple (ctypes.test.test_pickling.PickleTest_1) ... ok
test_struct (ctypes.test.test_pickling.PickleTest_1) ... ok
test_unpickable (ctypes.test.test_pickling.PickleTest_1) ... ok
test_wchar (ctypes.test.test_pickling.PickleTest_1) ... ok
test_simple (ctypes.test.test_pickling.PickleTest_2) ... ok
test_struct (ctypes.test.test_pickling.PickleTest_2) ... ok
test_unpickable (ctypes.test.test_pickling.PickleTest_2) ... ok
test_wchar (ctypes.test.test_pickling.PickleTest_2) ... ok
test_incomplete_example (ctypes.test.test_incomplete.MyTestCase) ... ok
test_char_array (ctypes.test.test_slicing.SlicesTestCase) ... ok
test_char_ptr (ctypes.test.test_slicing.SlicesTestCase) ... ok
test_char_ptr_with_free (ctypes.test.test_slicing.SlicesTestCase) ... ok
test_getslice_cint (ctypes.test.test_slicing.SlicesTestCase) ... ok
test_setslice_cint (ctypes.test.test_slicing.SlicesTestCase) ... ok
test_wchar_ptr (ctypes.test.test_slicing.SlicesTestCase) ... ok
test_basic (ctypes.test.test_pointers.PointersTestCase) ... ok
test_basics (ctypes.test.test_pointers.PointersTestCase) ... ok
test_bug_1467852 (ctypes.test.test_pointers.PointersTestCase) ... ok
test_c_void_p (ctypes.test.test_pointers.PointersTestCase) ... ok
test_callbacks_with_pointers (ctypes.test.test_pointers.PointersTestCase) ... ok
test_change_pointers (ctypes.test.test_pointers.PointersTestCase) ... ok
test_charpp (ctypes.test.test_pointers.PointersTestCase)
Test that a character pointer-to-pointer is correctly passed ... ok
test_from_address (ctypes.test.test_pointers.PointersTestCase) ... ok
test_other (ctypes.test.test_pointers.PointersTestCase) ... ok
test_pass_pointers (ctypes.test.test_pointers.PointersTestCase) ... error: plugin_host has exited unexpectedly, plugin functionality won't be available until Sublime Text has been restarted
Which was all well and good, until I realized this doesn't prove anything, since I now have this freak of a python install...
So at this point, either:
- Something is wrong with ctypes. Not much we can do other than try to report it to the sublime text team and cross our fingers they fix it.
- Something is wrong with cindex. Somewhat unlikely since it works with the normal windows install. Maybe we could ask the clang team if they know what's going on. Maybe they'll have more experience.
That's all I got. In any case, if it's unfixable in sublime text, the "best" solution would probably be to start up a local python server and use that to communicate clangs results. I think this would be a crazy amount of work, for what would probably be a rather fragile system - I've used other plugins that did this and they always seemed to have problems leaving processes around. But maybe you think it's workable.
Oh, I though C++ was for shooting one's legs off :)
Jokes aside, yep, I guess running a separate python process sucks and is an overkill. I would then rather just hack something in C++ that would bind to libclang. But the whole purpose of this plugin was to avoid compilation and ease up the things for the user. This will not happen with C++ enabled plugin. + we get all the nice issues with making it cross-platform. Not the most fun thing I can think of to be honest :)
I may try to open issues in clang tracker. The problem is that I doubt they will be instantly convinced that this issue is big because it is only Sublime Text related and every developer always tries to put the blame off himself and NOT fix the issue if he can avoid it :)
As for Sublime Text forum - thanks a lot! That is cool, but I have doubts about how much it will bring. There are two developers in the whole Sublime Text office and one of them works solely on Package Control. This is definitely not the priority issue for them (unless we use some jedi tricks to force them into thinking otherwise) and with the speed of development of sublime text I doubt anything is going to happen. :(
However, the plugin is only one month old. I think with patience and persistent trying out things we will eventually either solve this or at least find what exactly is the issue and maybe circumvent it.
I have a really busy Summer unfortunately and I am slightly scared of Windows. I haven't worked in it since like 6 years already. So the developments on this issue from my side will be quite sporadic, but I will be on vacations soon and I plan to hack something together at that time. Maybe replacing the python dll while breaking sublime text will give hints on what is wrong there. Also I wonder if there is any way to get any stacktrace from this. Let's see.
And they seem to get libclang_x64.dll on 64 bit Windows.... Hmmmmm....
I saw that. It's seems worth looking into. I don't know why they were looking for the 64 bit, but shipping the 32 bit one (the 64 bit dll isn't named that anymore btw). I suspect the 32 bit versions were shipped because clang precompiled binaries on windows are a relatively new thing.
I agree with you about the level of support we can expect from the sublime text team. I was actually looking at using Atom because of that. But Atom is slower than Sublime, and has a bunch of tiny issues that make the switch less immediately appealing.
I'm going to be busy for the next few days, but I may have a chance to dig into this more after that.
Yeah, an open source Sublime Text would have been nice :)
Atom is fine, but it makes stuff a bit harder with libclang as the bingdings are still Python, so one either has to use some python bindings for CoffeeScript or do the same for C++ or write Javascript bindings to libcalng library. Which is an adventure I don't currently have will or time for.
Let's see if I find some time over the weekend to play around with the file I linked.
I have opened an issue in technical support and it gained attention of sublime devs. However, it seems it is a bug of python 3.3 and we will be stuck with this :(
https://forum.sublimetext.com/t/ctypes-from-sublime-text-3-python-broken-on-windows/21490
Ok, so as pointed out by wbond in sublime forums I had a bug in the code with relative path to the file. This should be fixed now and this test plugin does its thing on my machine having clang 3.8.1 both from powershell and from sublime text. Can anyone try this out too?
The original EasyClangComplete plugin still crashes the host, when trying to open the same file as in this small test plugin. So my next step would be to reduce as much as possible and gradually build it up until I see the host crashing again. Because clearly, parsing and reparsing should be working just fine as shown in this repo.
It seems that there is something wrong reading the diagnostics of the parsed translation unit, so I have updated this plugin to reflect this issue. Again, the issue only occurs when using the script from sublime text and not from power shell