Atrocious HDMI audio
Closed this issue · 30 comments
When I tested Eye of the Beholder to double-check if my .crt loading regression is "really really fixed", I stumbled over strange background noises in the HDMI audio stream. It was subtle, not super prominent, and I ignored it at first.
But then I saw this commit 2b7dd65 and this commit message:
M2M: Refactor audio clock speed
Replace the 30 MHz clock signal with a 12.288 MHz clock.
The 30 MHz valus is completely arbitrary, whereas 12.288 MHz is exactly
256 times the audio sample rate (48 kHz).
So my working assumption is, that something went wrong here.
I then did some more tests: The subtle distortion that I hear in Eye of the Beholder is for some reason 100x amplified when I listen to the song "Excalibur" or "Game of Thrones": I recorded the issue, listen to "Excalibur" here: excalibur.zip
Maybe the reason for the amplification of the problem is that Eye of the Beholder is Mono (one SID) and the two songs "Excalibur" and "Game of Thrones" are Stereo? But independent from Mono and Stereo: The problem is always there. Just different degrees of amplifications.
I hope you can reproduce this (i.e. the speakers of your monitor are sensible enough; my HiFi for sure is ;-)), so that you can test the bugfixing progress independently from me. But of course I am here to support and to test in case you can't. It just takes a bit longer currently due to my extreme end-of-year business.
Reproduction steps:
- Download this music disk
- Mount it and
LOAD "*",8,1 - Choose Dual Sid, D420 in the music player's startup menu
- Configure the OSM to Stereo SID by choosing "L:8580 R:8580" D420
- Audio Improvements On
- Play Excalibur
What I find interesting is, that "Audio Improvements" amplify the problem.
Closing thoughts:
-
I plan to make a build of the core just one commit before the very commit 2b7dd65 to see if this commit really introduced the problem. Not sure when I find time for this (I hope today - otherwise might take a while)
-
I have a faint memory of a comment in Paul's original code that I used and adapted for an early version of the C64 core. He was speaking of the necessity of oversampling - but not sure if and how this has anything to do with 30 MHz vs 12.288 MHz
-
Maybe the "Audio Improvements" are broken
Update: I built and tested the very commit before the audio refactoring (c5f2d0a) and there everything is OK, so it seems indeed that the refactoring introduced a problem.
Thank you for the detailed report. I've tried to reproduce and I have some doubts.
- On the analog output I hear no distortion.
- On my HDMI monitor I hear no distortion.
However, when I do a recording from the analog output to my desktop computer (via a 3.5 mm audio cable) and record using Audacity I notice a difference between the left and right channels. The left channel sounds great, but the right channel is indeed distorted.
I tried the same setup using commit c5f2d0a, but same result, i.e. left channel sounds great and right channel is heavily distorted.
So, could you please confirm that you hear a difference between the left and right channels? I would like to determine whether my observation is just an artifact in my setup.
OK - just to make sure we have the same setup: you have "Improve Audio = ON"? Because this seems to amplify the effect and maybe the tube-amplifier and the HiFi speakers I use are more sensitive to this kind of stuff than your equipment?
Unfortunately I have no analog audio recording abilities at this moment, so I am exclusively testing HDMI audio.
I re-tested, here is the result (using the music disk I provided to you above):
Commit c5f2d0a (the one before you refactored):
-
Excalibur and Game of Thrones sounds fine. Audio improvement ON or OFF does not amplify any distortion, because there is none. Hint: The "shhhhhhh" sound at the beginning of Excalibur is intented. I guess this shall be "wind" or "mist" or something :-)
-
Double-checked if the intended "shhh" sounds the same on the officially released V5: It does
-
But now cames the WTF moment: I tried "Odyssey_2SID" on the very same music disk and it has heavy distortions ON BOTH SPEAKERS.
-
I then changed the HDMI Display mode from 720p 50 Hz to 4:3 576p while "Odyssey" was running and the distortions where gone ??! 🤯
-
I then changed the HDMI Display mode from 576p back to 720p and the distortions REMAIN GONE 🤯
So this is the proof that the problem seems to be somewhere else and not directly related to the "refactor commit 2b7dd65
-
I then continued with the "one before refactor" commit c5f2d0a and started excalibur again: SOFT (very softly) I can hear distortions now (after the back and forth of the screen output resolutions) while I am in 720p. I now switch to 576p and the distortions are gone again.
-
Switched back to 720p and used the
.crtloading mechanism and loaded Eye of the Beholder, MONO SID, 8580: Distortion on both speakers while being in 720p. Switched to 4:3 or 5:4 576p: Distortions gone. Now switched back to 720p: Distortions REMAIN GONE (just as above in the audio player).
WOW - this one is interesting!
Interesting indeed. When I find time (within a few days I hope) I will attempt to recreate the tests you have described, to see whether I can reproduce your observations.
Update on this statement from above:
What I find interesting is, that "Audio Improvements" amplify the problem.
Actually, there are situations where "Audio Improvements" hides / covers the problem and there are situations where "Audio Improvements" emphasizes the problem.
@MJoergen As you asked for, I did a kind of "binary search" to narrow down the commit where the sound bug was introduced. My challenge is that one build takes about 47mins... so for now I can only offer two results for now:
✅ Alpha 3: Audio OK. Commit #c099921, remember Alpha 3 introduced: "Reconfigurable (and completely refactored) MMCM video clock, refactored reset, ASCAL reduced burst length that might fix some rare REU bugs"
❌ 3030cb7 "Fix HDMI audio microcuts": I was able to reproduce the problem with the song Excalibur and Odyssey and Game of Thrones, but only after switching back and forth between display modes
Thank you so much. This is already very helpful information.
There have been one other HDMI audio related fix (specifically fixing the problem on my monitor). So to test whether this is the cause, please try with these two bit-files, generated from two close commits.
✅ mega65_r3_17f06ab.bit: Sound OK - was not able to reproduce the problem.
✅ mega65_r3_18d9349.bit: Sound OK - was not able to reproduce the problem.
Your results narrow down the list of commits to less than ten. Most of these commits are harmless, but one commit (3030cb7) does indeed touch the HDMI audio. So this is likely the culprit.
Just to be completely sure, I've completed a build based on the very previous commit:
mega65_r3_1981141.bit.gz
EDIT: I've made two additional builds I would like tested as well:
mega65_r3_0fed2c6.bit.gz
mega65_r3_f8cbef0.bit.gz
When you find time, please try this build.
Thanks @MJoergen - I will not be able to test this in the next 7 days or so. Hopefully you can reproduce it.
@MJoergen here are the test results:
✅ mega65_r3_1981141.bit: Sound OK - was not able to reproduce the problem.
✅ mega65_r3_0fed2c6.bit: Sound OK - was not able to reproduce the problem.
✅ mega65_r3_f8cbef0.bit: Sound OK - was not able to reproduce the problem.
@MJoergen on Skype you wrote:
Perhaps you could spend a few minutes to reproduce the issue on the first image reported (3030cb7). You've previously tested this image, and shown the audio fails on that image.
❌ 3030cb7 - Sound not OK - I was able to reproduce the problem multiple times. using 3030cb7.
But what I find stunning is that sometimes it takes like 30min of playing around with the core and switching back and forth screen resolutions and stuff to reproduce it and then the problem is only very faintly audible. But then, sometimes it happens DIRECTLY after I use the m65 -q mega65_r3.bit command line to program the FPGA and play the first song. And in this case, it is not faintly audible it is loud, ugly, atrocious. Listen to this recording of a situation, where the problem directly arose after I programmed the FPGA: audibug-3030cb7.zip "Audio Improvements ON" is emphasizing the problem but it is also audible when OFF.
So in short:
-
Sometimes it takes 30min to reproduce and I need to switch back and forth the screen resolution multiple times and even then I can only faintly hear the bug.
-
Sometimes the problem is IMMEDIATELY audible after programming the FPGA via
m65and THEN it is not faintly but super loud as you can hear: audibug-3030cb7.zip -
I did some experiments: It takes 7 to 10 times of re-programming the FPGA via
m65to provoke the "loud" version of the problem as described in (2). It seems to be easier to reproduce the problem by just re-programming the FPGA while it stays in the 720p @ 50 Hz mode all the time (instead of trying this strenuous manual switching of screen-resolutions).
Disclaimer: Everything I wrote so far is about commit 3030cb7.
This leads to the following question I have for you - maybe we can do "debugging by thinking":
Can it be that this is some kind of an "FPGA logic initialization" bug?
You also wrote on Skype:
Of the three new images, I expect two of them to work, but I do expect f8cbef0 to fail in the same way as 3030cb7.
❌ f8cbef0 - Sound not OK - Yes, this time I was able to use my newly discovered "re-program the FPGA" technique to reproduce the problem, so f8cbef0 failed, too (in contrast to my claim above, that it works). In this specific instance, it happened directly after the initial programming. No re-programming needed this time.
So this is what you expected. Now let's try the re-prgram technique on the two others that you do expect to be stable.
✅ 1981141 - Sound OK - was not able to reproduce after 12 attempts of re-programming
✅ 0fed2c6 - Sound OK - was not able to reproduce after 12 attempts of re-programming
I also tested your very latest commit 12a36d9 in develop from December 20, which does not have branch "mfj_fix_issue_116" included:
✅ 12a36d9 - Sound OK - was not able to reproduce after 12 attempts of re-programming
Is this expected? It does not include the fixes from the specialized branch, but 1981141 does also not include these fixes and works, too.
I think we would be able to close this issue as everything seems to work with the latest commit 12a36d9 and also with the "remove HDMI text messaging" commit 0fed2c6. But do we have any explanation why the bug went away?
Thank you for your detailed testing and reporting. That the latest commit 12a36d9 works is indeed unexpected!
Between the commits f8cbef0 (from December 4th) and 12a36d9 (from December 20th) there are only approximately ten other commits. And looking at the file difference statistics shows the following:
mike@mike-desktop2 ~/git/MJoergen/C64MEGA65 (develop=) $ git diff --stat f8cbef0 12a36d9
CORE/CORE-R3.xpr | 6 +-
CORE/CORE-R4.xpr | 6 +-
CORE/CORE-R5.xpr | 6 +-
CORE/vhdl/config.vhd | 10 +--
CORE/vhdl/main.vhd | 20 +++--
CORE/vhdl/prg_loader.vhd | 194 ++++++++++++++++++++++------------------------
M2M/vhdl/controllers/M65/audio.vhd | 8 +-
M2M/vhdl/framework.vhd | 26 +++++--
M2M/vhdl/i2c/cpu_to_i2c_master.vhd | 67 +++++++++++++---
M2M/vhdl/i2c/i2c_controller.vhd | 11 ++-
M2M/vhdl/i2c/i2c_master.vhd | 9 +++
M2M/vhdl/i2c/rtc.vhd | 95 -----------------------
M2M/vhdl/i2c/rtc_controller.vhd | 251 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
M2M/vhdl/i2c/rtc_master.vhd | 323 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
M2M/vhdl/i2c/rtc_reader.vhd | 190 ---------------------------------------------
M2M/vhdl/i2c/{rtc_i2c.vhd => rtc_wrapper.vhd} | 59 ++++++++------
M2M/vhdl/qnice_arbit.vhd | 6 +-
M2M/vhdl/top_mega65-r3.vhd | 14 ++--
M2M/vhdl/top_mega65-r4.vhd | 17 ++--
M2M/vhdl/top_mega65-r5.vhd | 23 +++---
doc/inofficial.md | 1 +
21 files changed, 861 insertions(+), 481 deletions(-)
Nothing in the above list can explain the sudden disappearance of the HDMI audio bug - nothing is related to HDMI audio. The file M2M/vhdl/controllers/M65/audio.vhd is used for the audio DAC.
I do still believe this HDMI audio bug is related to upgrading from Tyto 1 to Tyto 2. This follows from the commit 3030cb7 where Tyto 2 is introduced, while the very previous commit 1981141 does not have the HDMI audio bug.
Originally, we upgraded from Tyto 1 to Tyto 2 to solve the problem of "micro cuts" (see issue #13). However, this same upgrade seems to have a side effect of worsening the HDMI audio.
So we are at a cross-roads: Should we keep Tyto 2 and hope that the HDMI audio bug won't reappear? Or should we downgrade to Tyto 1 and potentially re-introduce the "micro cuts"?
I can add that comparing the difference between Tyto 1 and Tyto 2 can be done with the command git diff -w 0fed2c6 12a36d9 M2M/vhdl/controllers/HDMI/vga_to_hdmi.vhd. The differences seem to be limited to:
- Refactoring of the Clock Domain Crossings.
- A very specific change related to checking for HSYNC/VSYNC before sending AVI InfoFrame.
So it is entirely possible that the second bullet above may be the real culprit ?!
@MJoergen Here is an update:
The latest Alpha 5 release, which does not have the branch mfj_fix_issue_116 included did indeed fail. Interestingly enough upon the first power-on of the MEGA65. Here is a recording how it sounds: audibug-3030cb7.zip
This new is kind of "good" news, since a few days ago I ran into the strange situation, that I was not able to reproduce the bug with this commit 12a36d9 which is similar to "Alpha 5", see here: #116 (comment) - and this also confused you. This is now a proof that 12 re-tries might not always be enough to reproduce the problem. And that sometimes it "just happens" when you power-on the MEGA65 (I have flashed Alpha 5 to run when I power on the machine).
Might this problem be temperature related? My MEGA65 was ice-cold. It was cold in the room over night.
More findings:
-
A LONG reset does not mitigate the problem. If it is there, it is here to stay...
-
A power cycle removes the problem
Since I was not (so far) able to reproduce the problem with the branch mfj_fix_issue_116, I will now merge this branch to develop and make an Alpha 6 for us, which will also contain MiSTer's drive-switching-fix. We can then, before releasing an RC1, test a bit by ourselves (including @paich64) if we can reproduce the bug in Alpha 6.
- Test performed on Mega65 R3 board using core V5.1A6-R3
- Audio stream grabbed from HDMI output and redirected to Stereo external audio equipement
The following actions have been repeated 40 times without encountering the Mentionned Audio Issue
On PC (40 times):
./m65 -q /home/paich64/tools/mega65tools/c64core/04_audio_issue/C64MEGA65-V5.1A6-R3.bit
On Mega65(40 times) C64 core which has just started
and which is configured with
- Dual SID L:8580 R:8580
- "Improve Audio" is ON
execute the following steps:
- Mount D420.d64
- Start SIDPLAY64 (load"*",8,1 then run)
- Yes for second SID, configured on $d420
- Load and listen to "Excalibur" tune
Result : Audio issue never occured
@sy2002 @MJoergen
in addition to my previous update (and as discussed during our call) :
-
Just noticed "sizzling/crackling" while running Wonderland XIV demo (over both HDMI, have to double check for analog since not sure at all) but I can't reproduce.
And it's amplified when "Audio Improvement" is ON. -
Fix for the disk swap issue
Before the swap disk issue was fixed
Booze Design 2001 Royal Arte 100% (Mister) KO (Mega65) KO <=====
Plush 1997 +H2K (Mister) KO (Mega65) OK
Shape 2017 The Shores of Reflection (Mister) OK (Mega65) KO
Resource 2001 Soiled Legacy (Mister) KO (Mega65) OK
Coma 1997 Void (Mister) KO (Mega65) OK
After the swap disk issue was fixed
Booze Design 2001 Royal Arte 100% (Mister) OK (Mega65) OK <=====
Plush 1997 +H2K (Mister) OK (Mega65) OK
Shape 2017 The Shores of Reflection (Mister) OK (Mega65) KO
Resource 2001 Soiled Legacy (Mister) OK (Mega65) OK
Coma 1997 Void (Mister) OK (Mega65) OK
Mega65 was better than the Mister.
And the fix
- helped a lot the Mister
- helped the Mega65 with "Royal Arte 100%"
- did not introduce regression for the Mega65 but did not fix "The Shores of Reflection" which has always worked on mister.
Anyway the fix is obviously to be kept as it fixed "Royal Arte" and running other demos does not seem (so far) to have introduced disk swap regressions.
Other issues fixed in MiSTer-devel/C64_MiSTer#160
- Plush 1997 +H2K : all Nasty big glitches are fixed with V5.1A6 => OK !!!
- Fairlight 2023 The Space is Broken : Issue fixed on Mister is also fixed with V5.1A6 => OK !!!
- Bonzai 2020 Christmas Megademo : Issue introduced when fixing "The Space is Broken" is fixed with V5.1A6 => OK !!!
- Booze Design 2008 Edge of Disgrace : Glitch related to "CMOS" composant type is fixed thanks to hard coding "OldHMos" value in V5.1A6 => OK !!!
- Censor Design 2023 Wonderland IV : Hblank fix does not seem to be implemented - discussed during our chat - => Should be fixed in V5.1A7
@paich64 Here is Alpha 7 C64MEGA65-V5.1A7.zip
- It is our next attempt to fix this audio problem, please pressure-test as hard as you can
- It is supposed to fix the HBlank issue
- Please ask Tonton to test, if the random micro-cuts of sound is still fixed with this Alpha 7
@sy2002 @MJoergen the audio issue captured : https://youtu.be/6-1ZefC8ObM
additional comments :
-
note that starting from the ~120nd execution i switched from 576p to 720p so it could be its more likely to occure when on 720p. I hardly believe it was all ok during 141 executions and suddenly it fails. switching to 720p at the 120nd execution might be considered as being related to the issue.
-
Now that i think about yesterday when i tested 40 times using V5.1A6, i was using 576p resolution. But when i got the Audio issue it was when running the Wonderland XIV demo on my 16:9 screen (using 720p resolution) ..... so 720p might be involved in the issue ....
Maybe we should consider testing using 720p ( a 80 iterations run) and the using 576p (a 80 iteration run).
We may come to the conclusion that it's only occuring using 720p (just a guess ...)
@paich64 WOW - Thank you for this incredibly thorough testing cycle! You have proven a ton of stamina.
@paich64 We know that this is a huge ask. Sorry for that! Can you please re-test using this Alpha 8 version:
C64MEGA65-V5.1A8.zip Feel free to use 720p - maybe it provokes the issue faster.
@paich64 We know that this is a huge ask. Sorry for that! Can you please re-test using this Alpha 8 version: C64MEGA65-V5.1A8.zip Feel free to use 720p - maybe it provokes the issue faster.
I want to be in the exact same conditions (having enough time ahead of me to run as many iterations as possible). Because of a job constraint i will do it tomorrow starting from the end of afternoon ;)
@sy2002 I ran a new bench session with V5.1A8 and unfortunately it failed again after 31 attempts.
It seems to happen faster when indeed using 720p.
@paich64 Please perform 150 (or more) tests with this Alpha 9 version: C64MEGA65-V5.1A9.zip
Adam Barnes (@amb5l ) found a connection between the clock speed/resolution and the problem, which is why I encourage you to continue to test using 720p (which lead to faster failure in past). If you manage to do 150 tries without the problem happening then we would consider it as resolved.
FYI, if you're interested in the details, here is a quote from Adam: "I have a theory. PCM sample pairs are latched into signals called iec_* by the PCM process which is clocked by pcm_clk (qualified by pcm_clken). iec_req is asserted for each PCM sample pair. The way this is done has changed recently - it is now generated by the P_IEC_REQ process - but the timing is the same. The iec_* signals pass through a CDC synchroniser so they can be safely sampled in the vga_clk domain. This sychroniser has 2 stages. So does the synchroniser for iec_req -> vga_iec_en. This means there is a chance that vga_iec_en could be asserted while the signals it is supposed to qualify are still updating. This chance will vary as the phase relationship between the PCM and VGA clock domains changes. Which will happen when there are resolution/clock changes. See iec_req_s, vga_iec_en and iec_ack for changes. Basically vga_iec_en is delayed by a couple of clocks to allow all the iec_* synchroniser outputs from SYNC3 to settle."
