MPF-MC audio fails on Windows
toomanybrians opened this issue · 13 comments
Something is broken in the MPF-MC custom audio code build process for Windows. MPF-MC will install fine, but it will crash when attempting to play audio. This only affects Windows. Mac and Linux are fine.
This bug slipped through since we do not actually test audio on the GiHub cloud builds since the GitHub hosted runners don't support audio, and (at least for me), I only use Mac and Linux so I didn't even know that audio wasn't working on the Windows builds.
This issue only occurs with the current builds. The most recent build that does NOT have this problem is 0.56.0.dev33. The temporary workaround is for Windows users to use dev33.
The dev33 build is from Aug 22, 2022. (I tested this with Python 3.9 only, I assume 3.8 and 3.7 have the same issues but have not verified.)
dev34 was posted Sept 9, 2022 (2 weeks after dev33) and is the first build that is broken. The only change for dev34 was to remove MPF as a hard coded dependency, shown here.
I do not believe this change is causing the problem directly. Rather my guess is something changed in the build environment, most likely one of the dependencies or the compiler tool chain changed? Unfortunately because the dev33 build is more than 90 days old, we no longer have access to the build logs to see what changed.
I have since built dev35 and dev36, but those do not fix this audio error. Also, dev36 rolls back the MPF dependency change, which, as expected, does not fix the issue. (Keeping MPF as a requirement for MPF-MC is fine moving forward too, so we can close the book on that.)
I analyzed the differences between the dev33 and dev36 Python 3.9 Windows wheels from the MPF-MC project on PyPI. The only files that are different (apart from the version file) are the 6 compiled .pyd files (5 audio and 1 bitmap font). This makes sense, since this is the component that is broken.
What's interesting is that the associated Cython-generated .c files (which are included in the repo) have NOT changed. (They have not changed since Nov 2021.)
So this is why I believe our issue is some kind of dependency or build environment change. The Windows MPF-MC installer gets its gstreamer and SDL libraries from PyPI, I noticed that some of these have more recent changes:
- kivy-deps.sdl2 and kivy-deps.sdl2-dev updated from 0.4.5 (Feb 2022) to 0.5.0 on Sept 10, 2022, and 0.5.1 on Oct 2022.
- kivy-deps.angle and kivy-deps.angle-dev updated to 0.3.3 on Oct 27, 2022
- (other deps, like gstreamer and glew, have not changed since before this issue started)
The next thing to test would be to pin the Windows build back to kivy-deps.sdl2 0.4.5 and see if that fixes it. I can try that at some point, but I wanted to post here so we could track this properly. Assuming this works, I will pin all the kivy deps packages in the installer config so we don't have this issue in the future.
I attempted to fix this issue by pinning back to an older version of kivy-deps.sdl2 (0.4.5 versus 0.5.1), but that did not fix the issue. (In hindsight, the broken dev34 build is from Sept 9 but kivy-deps.sdl2 didn't move to 0.5.0 until Sept 10, so it's no surprise that this didn't work.
Here are the dependencies I pinned:
dependencies = [
"ruamel.yaml == 0.15.42", # 0.15.x newer than this requires compiled c lib, currently no Mac arm versions, 0.16+ require code refactor
"mpf >= 0.56.0.dev33",
"kivy == 2.1.0", # Mar 6, 2022
"psutil == 5.9.4", # Nov 7, 2022
"Pygments == 2.14.0", # Jan 1, 2023
"kivy_deps.sdl2 == 0.4.5; platform_system=='Windows'", # Feb 15, 2022. 0.5.0 (Sept 2002) and 0.5.1 (Oct 2022) cause audio playback issues
"kivy_deps.glew == 0.3.1; platform_system=='Windows'", # Feb 15, 2022
"kivy_deps.gstreamer == 0.3.3; platform_system=='Windows'", # Jan 11, 2022
"ffpyplayer == 4.3.5" # Mar 29, 2022
]
Here is the crash from the MC:
2023-01-10 18:13:54,257 : kivy : Factory: 189 symbols loaded
2023-01-10 18:13:55,228 : kivy : ImageLoaderFFPy: Using ffpyplayer 4.3.5
2023-01-10 18:13:55,230 : kivy : Image: Providers: img_tex, img_dds, img_sdl2, img_pil, img_ffpyplayer
2023-01-10 18:13:55,586 : kivy : VideoGstplayer: Using Gstreamer 1.18.5.0
2023-01-10 18:13:55,587 : kivy : Video: Provider: gstplayer
2023-01-10 18:13:55,839 : kivy : mpfmc.core.audio library could not be loaded. Audio features will not be available
2023-01-10 18:13:55,839 : kivy : Loading MPF-MC controller
2023-01-10 18:13:55,847 : YamlMultifileConfigLoader : Machine config file #1: config
2023-01-10 18:13:55,847 : ConfigProcessor : Loading config from file C:\Users\Brian Madden\.local\pipx\venvs\mpf\lib\site-packages\mpfmc\mcconfig.yaml.
2023-01-10 18:13:55,896 : ConfigProcessor : Loading config: C:\Users\Brian Madden\.local\pipx\venvs\mpf\lib\site-packages\mpfmc\mcconfig.yaml
2023-01-10 18:13:55,896 : ConfigProcessor : Loading config from file c:\mpf-examples\mc_demo\config\config.
2023-01-10 18:13:56,005 : ConfigProcessor : Loading config: c:\mpf-examples\mc_demo\config\config
2023-01-10 18:13:56,008 : ConfigProcessor : Config file cache created: C:\Users\BRIANM~1\AppData\Local\Temp\fd0a4f9ff86b870212d95b5a542b5d1c.mpf_cache
2023-01-10 18:13:56,008 : ConfigProcessor : Loading config from cache: C:\Users\BRIANM~1\AppData\Local\Temp\a8f7d298b06fc8aa46bda66631f724e9.mpf_cache
2023-01-10 18:13:56,013 : ConfigProcessor : Loading config from cache: C:\Users\BRIANM~1\AppData\Local\Temp\d4a89d54d93a51ad2d0c26455314101f.mpf_cache
2023-01-10 18:13:56,017 : mpfmc : Mission Pinball Framework Media Controller v0.56.0-dev38
2023-01-10 18:13:56,017 : mpfmc : Mission Pinball Framework Game Engine v0.56.0-dev33
2023-01-10 18:13:56,017 : mpfmc : Machine path: c:\mpf-examples\mc_demo
2023-01-10 18:13:56,017 : mpfmc : Starting clock at 60.0Hz
2023-01-10 18:13:56,080 : kivy : Text: Provider: sdl2
2023-01-10 18:13:56,102 : kivy : DLL load failed while importing sound_file: The specified procedure could not be found.
Traceback (most recent call last):
File "C:\Users\Brian Madden\.local\pipx\venvs\mpf\lib\site-packages\mpfmc\commands\mc.py", line 225, in __init__
MpfMc(options=vars(args),
File "C:\Users\Brian Madden\.local\pipx\venvs\mpf\lib\site-packages\mpfmc\core\mc.py", line 151, in __init__
create_config_collections(self, self.machine_config['mpf-mc']['config_collections'])
File "C:\Users\Brian Madden\.local\pipx\venvs\mpf\lib\site-packages\mpfmc\core\config_collection.py", line 82, in create_config_collections
imported_module = import_module(module)
File "C:\Users\Brian Madden\AppData\Local\Programs\Python\Python39\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "C:\Users\Brian Madden\.local\pipx\venvs\mpf\lib\site-packages\mpfmc\config_collections\sound_loop_set.py", line 1, in <module>
from mpfmc.core.audio.audio_exception import AudioException
File "C:\Users\Brian Madden\.local\pipx\venvs\mpf\lib\site-packages\mpfmc\core\audio\__init__.py", line 7, in <module>
from mpfmc.core.audio.audio_interface import AudioInterface
File "mpfmc/core/audio/audio_interface.pyx", line 1, in init mpfmc.core.audio.audio_interface
ImportError: DLL load failed while importing sound_file: The specified procedure could not be found.
2023-01-10 18:13:56,110 : kivy : Stopping child threads... (0 remaining)
2023-01-10 18:13:56,111 : kivy : All child threads stopped.
At this point there's nothing more that I know how to do. So I'm hoping for help from someone who knows more than me? I will update the docs to inform Windows users to use dev33 until further notice.
Pinging @qcapen to see if you have any ideas and/or availability to work on this? Note that the dependency pinning I just did is in a branch called windows-audio-fix if you (or anyone) wants to work from there?
Trying the build on my system I don't get any crashes, but I'm also not getting any sound. I will look into this some more and see if I can come up with a solution.
I chatted with @qcapen a bit on this. He built the latest MPF-MC on Windows locally on his laptop and sent me the wheel, and it works fine. (mc_demo test has audio). So maybe there's an issue with the GitHub runner only and/or maybe something changed? I looked at the diff of the readme for the Windows 2022 runner we use from the 8/21 release through today. That diff is here:
Search the page for "images/win/Windows2022-Readme.md" to find the file.
Does any of this matter? I have no idea, maybe someone else knows?
Anyway since we can build the wheel locally (at least for Python 3.9), I'm learning towards doing a release of MPF-MC 0.56, then manually building the Windows wheel. (And manually building the Mac ARM wheel, since GH doesn't have hosted Mac ARM runners), and then we will have a release for Python 3.9 which will be stable for the time being.
Maybe this release lasts years? At some point we'll have to figure out how to get to Python 3.10+, but since I don't use Windows (and I also don't use MPF-MC), I won't be of much help, but I'll make sure the 0.56 release of MPF-MC works with future versions of MPF, and if someone who knows more about Python, Cython, SDL, Gstreamer, and Windows building and compiling comes along, maybe they'll be able to figure this out for real?
BTW, here's a pip list snapshot of all the installed modules that were used to produce the working wheel:
Cython 0.29.32
Kivy 2.1.0
certifi 2022.9.24
charset-normalizer 2.1.1
docutils 0.19
ffpyplayer 4.3.5
idna 3.4
kivy-deps.angle 0.3.3
kivy-deps.angle-dev 0.3.3
kivy-deps.glew 0.3.1
kivy-deps.glew-dev 0.3.1
kivy-deps.gstreamer 0.3.3
kivy-deps.gstreamer-dev 0.3.3
kivy-deps.sdl2 0.4.5
kivy-deps.sdl2-dev 0.4.5
kivy-garden 0.1.5
mpf 0.56.0.dev33
mpf-mc 0.56.0.dev38
packaging 21.3
pip 22.2.2
psutil 5.9.1
pygments 2.14.0
pyparsing 3.0.9
pypiwin32 223
pyserial 3.5
pyserial-asyncio 0.4
pywin32 304
requests 2.28.1
ruamel.yaml 0.15.42
setuptools 65.6.3
sortedcontainers 2.3.0
urllib3 1.26.12
wheel 0.37.1
Visual Studio 2019 is installed, and the build command is python setup.py bdist_wheel
@toomanybrians Is this why Windows requires 3.9 to run? If so, can it be built for 3.8 and published to allow Windows users to use 3.8 instead of only 3.9? If that were the case, then people that are stuck with 3.8 could benefit from the newest MPF dev version, instead of being stuck on 0.56-dev33.
Yes, this is why. I would assume someone could figure out how to build it locally for Python 3.8 as well. If anyone can get it working, they can send me the wheel and I can upload it to PyPI.
Okay, that at least solves the confusion I had around why windows had to be on 3.9. It doesn't really, but with what you had above something changed, and it was only manually built for 3.9. Obviously at some point that needs fixed, so we don't have to build for every version that is every published.
I don't know enough presently to be that person, nor do I have time right now. But if left unanswered down the road, I would be willing to take a peek at what is going on and see if I can learn a thing or two.
I looked for just a couple minutes, it is as easy as just running python -m pip wheel . ? I have Python 3.9 and ran that and it generated this wheel. If this looks correct, file name, content, etc. then I can run it for 3.7 and 3.8 for inclusion.
...well it won't let me load a wheel file, but it's called mpf_mc-0.56.1-cp39-cp39-win_amd64.whl and is 15,517 KB. If name and size seem right I can try for 3.8.
The command I run is python setup.py bdist_wheel, (I think, according to earlier messages in this thread.) Maybe the one you used would work too.
The issue with Windows though was not building the wheel. That always worked. It was that when you installed the MC from that wheel, audio wouldn't work. To make things more annoying, the GitHub cloud runners don't have audio, so the audio tests are skipped on GitHub which means even the broken wheels "pass" on GitHub. (That's why I removed building those from the automated workflows, because they were getting built and passing the tests and then uploaded PyPI, then people would install from them and their audio wouldn't work.)
So the wheel you built needs to be tested to see if audio works.
Anyway, yeah, we need a Windows + Python + Cython expert to look into this. I guess they would need GitHub workflow skills too.
BTW Python 3.7 went officially end-of-life yesterday, so I wouldn't even try to mess with that, and we should drop 3.7 when we dig into this. I did manage to get Python 3.10 and 3.11 working now for MPF (in my own fork, not in the mainstream yet), so that should provide the path to updating MPF-MC for 3.10/3.11 too. It is not a small task though. There's a reason MPF has been stuck on 3.9 for the past 3 years. 🤣
That makes more sense, seemed FAR too easy! I saw the note that its only tested with .c files generated note you added here: b84c218#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711
Anything to dig into there as a culprit, since the .pyx files throw all kinds of errors in pycharm when downloading the wheels. Not sure if they do that for other OS, but windows does not like it at all. Could be a red hearing, but wanted to run it past you for thoughts.
How familiar are you with Cython? Personally I don't know too much about it. I don't know C or C++ at all, and I have only a vague notion of how compiling and linking works.
Anyway the .pyx files in the repo are used to generate the .c files, and then the Python wheel building process compiles those .c files into the DLLs or dylibs or whatever the local OS needs. (Even if you install MPF-MC from source, that installation process first builds a temporary wheel locally and then installs from that.
If the .pyx files don't change, then you don't need to generate new .c files. If you include the generated .c files in the repo (as we do), then people should be able to build and install the package without needing Cython or any of its dependencies, etc.
Can you explain more about
.pyx files throw all kinds of errors in pycharm when downloading the wheels
I guess the pyx files are in the wheel? But they wouldn't be needed, since the whole point of the wheel is it contains the platform-specific compiled binaries. (So technically a Windows wheel would only need the .DLL files and not the source .c and .pyx files.)
When I first ran into this problem (in January I guess based on this thread), I did try to install Cython locally to make new .c files. I was able to get that to work (using both the version of Cython that generated our original .c files, as well as trying with newer versions of Cython to generate new .c files). But the ultimate result didn't change (e.g. locally built wheels were fine, github-generated wheels had non working audio on Windows.)
Now I use ChatGPT a lot which I didn't back in Jan. So my next step would be to make a fork to mess around with and try a bunch of things and get as much ChatGPT help as I can. I don't think the pyx files are the issue. I really think it's something with the Cython or wheel building (maybe with paths or linking or dynamic links versus static or ... ???)
My knowledge there is limited, I am not myself a developer, but run a team of developers. I know how to read code, and can get myself around with writing/extending, but not an expert outside of the "beaten path" stuff.
There was the thread somewhere of all my issues trying to get MPF-MC in editable mode, and it was specifically yelling about the this file: https://github.com/orgs/missionpinball/discussions/67#discussioncomment-4871222 So not sure if that's the C file in editable, building from the pyx file, or truly what is happening there. I know Quinn said he gets it, but he is compiling himself, so not sure that anyone really knows how it's supposed to magically work outside of him.
So my understanding stops there at what files are actually being pulled and used when building wheels, installing locally, and installing in editable. The second I install in editable, it breaks, and I have to remove everything (Python included) and start again, or it gets hung up in random places. So somehow editable vs install are different as well.
Either way, here is my file showing the errors. Even when I import the modules per the tooltip, it still doesn't change anything. File path of: Core>Audio>audio_interface.pyx
Looks like @ericselkpc fixed this in #452 !! 🎉