Building Python 2.7 with OpenSSL 1.0.1
Opened this issue · 8 comments
Hi, sorry for opening new issue, but this is only place where I can reach to you. If you have discord, linkedin or something else please send me pm to add you.
These days I was trying to compile Python 2.7.11 with OpenSLL 1.0.1 but without any success. I cloned XBMC4Xbox SVN repo and I located Python at xbmc/lib/libPython. Inside that folder I fixed download link inside download_requisites.sh and succesfully build Python 2.7.11 with OpenSSL 0.9.8zg using Visual Studio 2003 solution inside XBOXbuild folder.
Now I want to do the same but with OpenSSL 1.0.1 and I can't. I see that BuZz was aplying some patches to OpenSSL (located in patches folder) but these pathce doesn't work with v1.0.1. I manged to understand how .diff works and patch these files manually but cmll_locl.h and VC-32.pl are somehow different and I don't know what to patch inside of them.
Do you maybe know or have any experience with this?
You can Join the xbox scene on discord if you wish, there is a dev section where someone can possibly help you.
Hi @antonic901
To help troubleshoot this I would need to re-create what you have done and replicate it, without doing this I'm not sure sorry, I would also suggest rasing on the Discord as @professor-jonny posted above.
Regards
Hi @brentdc-nz! Thanks for quick response and sorry for late replay.. I will give my best to explain what I want and what problems I have. So let's begin.
Why am I trying to accomplish?
I want to build Python 2.7.11 with OpenSSL 1.0.1 for Original Xbox (XBMC4Xbox)
Why am I doing this?
Becase current Python which is shipped with XBMC4Xbox is builded with OpenSSL 0.9.8zg which unfurtunately for us doesn't support TLS1.2+
Why this is a problem?
Because more and more APIs are dropping support for HTTP and unsecure HTTPS (SSL, TLS1.0, TLS1.1) and our Xbox is unable to make request to them. One example of such API was OpenSubtitle REST but developers of that API were kind enough to allow TLS1.0 when I asked them. Another example is Invidious API where I needed to create simple Proxy in Node.js to make succesfully call.
What I have tried?
Here are the steps that I tried:
- I have cloned XBMC4Xbox repo
- I have Virtual Machine configured with Windows XP in which I have installed Microsoft Visual Studio .NET 2003 and XDK
- I have tried to build XBMC4Xbox from source just to confirm that everything works as it should
- Everything was alright, so I could go to the next step, building Python
How did I build Python?
Inside xbmc/lib/libPython is everythin required for the Python building. Here are the steps how to build it:
- Run download_requisites.sh script which downloads openssl, sqlite and bzip. It also patches sqlite and openssl with patches from patches folder. This patches basically replaces some line of code, they are nothing special.
- After that go to /python/XBOXBuild and open pcbuild.sln.
- Select Release XBOX and build project
- After succesfull build, inside /python/XBOXBuild you will find Release XBOX folder in which are all builded modules and libs (python27.dll, _ssl.pyd, _sqlite.pyd etc.). Inside this folder there is obj folder which have many folder which coresponds to each module log build.
What was broken before I could execute first succesfull build of Python 2.7.11 with OpenSSL 0.9.8zg?
- links from where download_requisites.sh script fetches openssl, sqlite and bzip were broken. A fixed that by replacing old links with new ones that works. After this, I succesfully run this script which downloaded this modules and patches them without any problem. After that I was able to build Python and used it on Xbox
What I want now and what problems am I getting?
- I want to do the same, but with OpenSSL 1.0.1(h) which supports TLS1.2. I downloaded openssl-1.0.1 and put that instead of openssl-0.9.8zg, but now I don't know how to patch this 1.0.1 version to make it comatible with Xbox. A take a closer look at this patches inside patches folder and I released that he is replacing some lines of files inside OpenSSL. But two files are completely different in v1.0.1 and these files are cmll_locl.h and VC-32.pl. I patched these four files and left these two as they are but after trying building python I'm getting error and _ssl.pyd (openssl) failes to build. These are the errors that I am getting:
_hashlib fatal error LNK1181: cannot open input file '..\..\openssl\out32\libeay32.lib' inc32\openssl\mdc2.h(69): fatal error C1189: #error : MDC2 is disabled. inc32\openssl\mdc2.h(69): fatal error C1189: #error : MDC2 is disabled. _ssl fatal error LNK1181: cannot open input file '..\..\openssl\out32\libeay32.lib' _ssl fatal error U1077: 'cl' : return code '0x2' _hashlib fatal error U1077: 'cl' : return code '0x2' c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Include\Python-ast.h(481): warning C4005: 'Yield' : macro redefinition c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(838): warning C4018: '<' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(838): warning C4018: '<' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3206): warning C4018: '<' : signed/unsigned mismatch C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Objects\bufferobject.c(416): warning C4018: '<=' : signed/unsigned mismatch C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\zlibmodule.c(482): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(2905): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(2913): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(2922): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(2985): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3032): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3049): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3073): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3077): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3105): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3117): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3130): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3149): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3179): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3213): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3228): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3168): warning C4018: '>=' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3177): warning C4018: '>=' : signed/unsigned mismatch C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Objects\fileobject.c(480): warning C4047: 'function' : 'int' differs in levels of indirection from 'FILE *' C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\sqlite\sqlite3.c(65536): warning C4049: compiler limit : terminating line number emission C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Python\dynload_win.c(170): warning C4101: 'import_python' : unreferenced local variable C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Objects\unicodeobject.c(4223): warning C4244: '=' : conversion from 'long' to 'Py_UNICODE', possible loss of data C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Objects\unicodeobject.c(4224): warning C4244: '=' : conversion from 'long' to 'Py_UNICODE', possible loss of data sqlite3 warning LNK4218: non-native module found; restarting link with /LTCG
- I also want to build Python 2.7.18 for Xbox. I first tried to build Python 2.7.11 from source dowloaded from here because I know for sure that 2.7.11 can be build for Xbox. But again, there probably were also some patches to source code of Python to make it to compile for Original Xbox. Here are the errors that I'me getting when I want to compile Python 2.7.11 from source with OpenSSL 0.9.8zg:
C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\posixmodule.c(7199): error C2065: '_MAX_ENV' : undeclared identifier inc32\openssl\mdc2.h(69): fatal error C1189: #error : MDC2 is disabled. inc32\openssl\mdc2.h(69): fatal error C1189: #error : MDC2 is disabled. _ssl fatal error LNK1181: cannot open input file '..\..\openssl\out32\libeay32.lib' _hashlib fatal error LNK1181: cannot open input file '..\..\openssl\out32\libeay32.lib' _socket fatal error LNK1181: cannot open input file '.\Release XBOX\python27.lib' python fatal error LNK1181: cannot open input file '.\Release XBOX\python27.lib' unicodedata fatal error LNK1181: cannot open input file '.\Release XBOX\python27.lib' select fatal error LNK1181: cannot open input file '.\Release XBOX\python27.lib' pyexpat fatal error LNK1181: cannot open input file '.\Release XBOX\python27.lib' pythonw fatal error LNK1181: cannot open input file '.\Release XBOX\python27.lib' _elementtree fatal error LNK1181: cannot open input file '.\Release XBOX\python27.lib' _sqlite3 fatal error LNK1181: cannot open input file '.\Release XBOX\python27.lib' bz2 fatal error LNK1181: cannot open input file '.\Release XBOX\python27.lib' _ctypes fatal error LNK1181: cannot open input file '.\Release XBOX\python27.lib' _ssl fatal error U1077: 'cl' : return code '0x2' _hashlib fatal error U1077: 'cl' : return code '0x2' c:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\PlatformSDK\Include\WinSock2.h(434): warning C4005: 'INVALID_SOCKET' : macro redefinition C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Python\pystrtod.c(102): warning C4013: '__control87_2' undefined; assuming extern returning int C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Objects\floatobject.c(1165): warning C4013: '__control87_2' undefined; assuming extern returning int C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\posixmodule.c(7199): warning C4018: '<' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(838): warning C4018: '<' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(838): warning C4018: '<' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3206): warning C4018: '<' : signed/unsigned mismatch C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Objects\bufferobject.c(416): warning C4018: '<=' : signed/unsigned mismatch C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\zlibmodule.c(482): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(2905): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(2913): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(2922): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(2985): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3032): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3049): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3073): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3077): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3105): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3117): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3130): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3149): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3179): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3213): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3228): warning C4018: '>' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3168): warning C4018: '>=' : signed/unsigned mismatch c:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\_sre.c(3177): warning C4018: '>=' : signed/unsigned mismatch C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\sqlite\sqlite3.c(65536): warning C4049: compiler limit : terminating line number emission C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Modules\posixmodule.c(4443): warning C4101: 'opath' : unreferenced local variable C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Objects\unicodeobject.c(4223): warning C4244: '=' : conversion from 'long' to 'Py_UNICODE', possible loss of data C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Objects\unicodeobject.c(4224): warning C4244: '=' : conversion from 'long' to 'Py_UNICODE', possible loss of data C:\Documents and Settings\Nikola Antonic\Desktop\xbmc4xbox\xbmc\lib\libPython\python\Python\import.c(1025): warning C4244: 'return' : conversion from '__int64' to 'time_t', possible loss of data sqlite3 warning LNK4218: non-native module found; restarting link with /LTCG
If you have time I hope you can take a look on this. Maybe it isn't something hard to fix. I can also upload my Virtual Machine and send it to you where all is installed for Xbox development.
Undeclared identifier usually means you have missing includes, mis spelt varables, or includes before VS specific includes, you need to look at contents of the files mentioned with errors.
I also see ctypes error this could be related to includes also.
It also looks like there is missing files or files that have not been built and have been skipped or something?
There is also a few datatype errors that may not be all that abnormal but could prove problamitic under specific conditions.
It would be a process of stepping through each error and looking at the causes there is a lot of good resourses online.
Of note Buzz on the xbmc4xbox forum might be worth a pm as he customised python for xbmc4xbox and I believe he limited it at this version even tho newer variants were available at the time as newer builds were heaver on resourses and it was to provide a better experience as a whole on the xbox, over compatibility with upstream plugins.
he might be able to offer asistance or sugest to patch tls into the current version rather than update python.
Also for the siginificantly modified files and patches it could be handy to look at the history and the differences between files to see what is going on looking at main stream kody form a snapshot back in time may offer assistance too.
I get in touch with BuZz (main developer of XBMC4Xbox) and he managed to build Python 2.7.18 with OpenSSL 1.0.2u. Now we have working TLS1.2 on Original Xbox. I also fixed broken weather inside XBMC4Xbox and in couple of days there will be new release of XBMC4Xbox (probably 3.7).
@brentdc-nz There is a discord group called Xbox-Scene where we have separate channel for XBMC development. BuZz is also there. If you have any questions regarding XBMC4Xbox he is the right person to ask. I will leave join link here:
https://discord.gg/CWkdVdc
Nice, look forward to updating my XBMC4XBOX.
Sounds good, thank you
How much is left for you to have complete GUILIB working? I could maybe try porting Weather window, but don't know how much difference in C++ is between OG Xbox and Xbox 360
I've just fixed my roadblock font color issue so I can now move onto more fun things like weather, I had been putting off doing anything until the font issue was resolved, so I should be more active here now.
Still have a lot to do for the guilib but getting there, since the large update a few commits back it introduced a few bugs I need to fix as well.