frame based I2S master bug

Question

frame based I2S master bug

Closed this issue 3 years ago · 17 comments

QuinnWang commented 8 years ago

Problem: sometimes noise, sometimes volume up, sometimes volume down. Just like some bits are shifted.

How to reproduce:

run I2S frame master loopback on MC
then using xrun --dump-state your.xe (for reference: normally, just a short break, then continue playing)

The issue was originally found by a customer:

power off for over 10min, then power on, there is about 90% probability to reproduce.
because 10min is too long to know the issue, then found the xrun --dump-state way to reproduce, but didn't find a solution yet, then using the normal I2S master as a workaround, since the requirement is only 48k sample rate, however, in this project, there are 3 kind I2S drivers:

normal I2S master in -> normal I2S master out, for 48kHz only SR;
I2S slave in -> I2S frame master out, for supporting up to 192kHz SR;
UAC2 in -> UAC2 FW native I2S core out

Answer 1 · 2017-01-25T10:28:45.000Z

Hi Quinn,
I helped develop I2S frame based. It's very low overhead and so can deal with quite long back-pressure on the callbacks (total time of all callbacks must be less than the frame period) and has been working at 768KHz, so 48KHz should be trivial. However, like all of the I2S implementations, it will only hold synch if the callbacks do not delay the I2S loop beyond the limit. If this happens then LR clock and/or data will be shifted (BCLK is free running). So to progress this bug, we need to be sure that the application does not assert a significant delay causing I2S to break. Can you insert a timing assertion to test this?

Something like this in the restart case:

time_old = time_now;
t :> time_now;
if (time_now - time_old > 2084) {
debug_printf("timing assertion fail in i2s handler%d\n", time_now - time_old );
__builtin_trap();
}

Answer 2 · 2017-01-25T15:07:54.000Z

Hi Ed

It is not the callback delay issue.
I can understand your concern, because when I met the problem originally, I doubted the delay (or the efficiency of the interface call) as well, so at that time, I tried to:

change the interface call to the direct channel transfer, no improvement;
use the hard-coded audio samples to replace the real in samples, found, especially, the I2S frame master in has problem, but can't find out the root cause.
finally, using the simplest loopback testing, found it is the problem of the lib_i2s:
if using the simplest loopback, means I2S frame master in, then store in a local sample buffer, then using I2S frame master out from the buffer, the issue is easy to reproduce with the xrun --dump-state.

It is worth noting:

in the loopback testing, the application callback can't be a problem anymore. And in fact, the issue can be reproduced at low sample rate: 48kHz when found the issue.
using "xrun --dump-state" can speed up to reproduce the issue.
You may not be able to see this issue normally, means most of time, the I2S frame master works well. In the customer project, when power-off for 10min, it just has 90% probability to reproduce.
And another thing can prove it is not because of the callbacks delay, since if using the normal I2S master (not the frame one. Can't support 192kHz), the loopback testing and customer project (48kHz required only) won't have the issue.

Answer 3 · 2017-01-25T17:13:17.000Z

Hi Quinn, Thanks for the great level of detail. That does seem to point to some other error than callback back pressure. Can I ask? - Did you reproduce on xmos hardware or is this just seen on the customer hardware? - Do you have the simple test case program to share? Obviously the first thing we have to do is make it fail locally so we can then find the root cause of the issue. The main difference between i2s master and frame i2s master is (apart from frequency and timing of callbacks) one uses BCLK divider in the clock block and the other uses a software divider by outputting patterns to p_bclk. In software divider case, we can sometimes see signal integrity problems on the BCLK pin - relfections can cause false clocks. However, this would be worse on the software divider case than clockblock divider case. Anyhow, please confirm the above 2 points. Regards, Ed On 25 Jan 2017, at 15:07, QuinnWang <notifications@github.com<mailto:notifications@github.com>> wrote: Hi Ed It is not the callback delay issue. I can understand your concern, because when I met the problem originally, I doubted the delay (or the efficiency of the interface call) as well, so at that time, I tried to: * change the interface call to the direct channel transfer, no improvement; * use the hard-coded audio samples to replace the real in samples, found, especially, the I2S frame master in has problem, but can't find out the root cause. finally, using the simplest loopback testing, found it is the problem of the lib_i2s: * if using the simplest loopback, means I2S frame master in, then store in a local sample buffer, then using I2S frame master out from the buffer, the issue is easy to reproduce with the xrun --dump-state. It is worth noting: * in the loopback testing, the application callback can't be a problem anymore. And in fact, the issue can be reproduced at low sample rate: 48kHz when found the issue. * using "xrun --dump-state" can speed up to reproduce the issue. You may not be able to see this issue normally, means most of time, the I2S frame master works well. In the customer project, when power-off for 10min, it just has 90% probability to reproduce. * And another thing can prove it is not because of the callbacks delay, since if using the normal I2S master (not the frame one. Can't support 192kHz), the loopback testing and customer project (48kHz required only) won't have the issue. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#26 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAmLbREs1fb1YWDkrav5Ew7EoMMewA4Cks5rV2VKgaJpZM4Lrw9e>.

Answer 4 · 2017-01-25T20:04:37.000Z

Hi QUinn, Regarding the "xrun --dump-state” case. Do you mean you can cause a running system to fail by typing that command from the host? If so, this is exactly what we would expect. Just like print over JTAG, the JTAG operation interrupts the entire tile. It will do this to interrogate all of the state and report back to the host. So I would absolutely expect real-time to be broken if you try to dump the state of a running device. This would almost certainly cause alignment issues on I2S. Regards, Ed On 25 Jan 2017, at 15:07, QuinnWang <notifications@github.com<mailto:notifications@github.com>> wrote: Hi Ed It is not the callback delay issue. I can understand your concern, because when I met the problem originally, I doubted the delay (or the efficiency of the interface call) as well, so at that time, I tried to: * change the interface call to the direct channel transfer, no improvement; * use the hard-coded audio samples to replace the real in samples, found, especially, the I2S frame master in has problem, but can't find out the root cause. finally, using the simplest loopback testing, found it is the problem of the lib_i2s: * if using the simplest loopback, means I2S frame master in, then store in a local sample buffer, then using I2S frame master out from the buffer, the issue is easy to reproduce with the xrun --dump-state. It is worth noting: * in the loopback testing, the application callback can't be a problem anymore. And in fact, the issue can be reproduced at low sample rate: 48kHz when found the issue. * using "xrun --dump-state" can speed up to reproduce the issue. You may not be able to see this issue normally, means most of time, the I2S frame master works well. In the customer project, when power-off for 10min, it just has 90% probability to reproduce. * And another thing can prove it is not because of the callbacks delay, since if using the normal I2S master (not the frame one. Can't support 192kHz), the loopback testing and customer project (48kHz required only) won't have the issue. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#26 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAmLbREs1fb1YWDkrav5Ew7EoMMewA4Cks5rV2VKgaJpZM4Lrw9e>.

Answer 5 · 2017-01-26T04:48:34.000Z

Hi Ed

Did you reproduce on xmos hardware or is this just seen on the customer hardware?
[Quinn] Both.
For the testing on MC, seems github can't support to upload the zip and xc file(?), I will send you the testing software on the MC:
it based on the AN00162_i2s_loopback_demo, and supports both normal i2s master and frame i2s master via the build option, so you can test on MC easily:

xrun xxx.xe, you can hear the normal loopback
then xrun --dump-state xxx.xe, you can hear the noise with AN00162_i2s_loopback_demo_frame_i2s_master.xe; but it is normal with AN00162_i2s_loopback_demo_normal_i2s_master.xe

Do you have the simple test case program to share?
[Quinn] Yes, please find the separate mail, thanks
Regarding the "xrun --dump-state” case. Do you mean you can cause a running system to fail by typing that command from the host?
[Quinn] Yes, since it is the easiest way to reproduce the customer project issue.
Though I am not sure if the dump-state is related to the real issue, but if the dump-state can cause the issue, then can see the issue on the real project; and if using the normal i2s, the dump-state can't cause the issue, then on the real project, can't see the issue anymore. So from the simple logic, I suppose they are related.
If so, this is exactly what we would expect. Just like print over JTAG, the JTAG operation interrupts the entire tile. It will do this to interrogate all of the state and report back to the host. So I would absolutely expect real-time to be broken if you try to dump the state of a running device. This would almost certainly cause alignment issues on I2S.
[Quinn] So this is my concern, firstly because the normal I2S can't reproduce that issue. When running the normal i2s with dump-state, it will recover the normal playback.
IMO, since xCORE is the i2s master, means the I2S timing is controlled by xCORE, then there should be no alignment issue rather than I2S slave, if there is alignment issue with I2S master mode, then we should be able to re-sync anyway. So this refreshed my world view :), just like another ESD issue, of course, this is another story.
BTW, seems it is the issue of the master in, master out seems work normally.
And in fact, in the real project, the issue will happen with I2S frame master anyway at a low freq without dump-state and JTAG.

BTW, now this is not a block issue on customer side, but it is worth finding out the root cause when you have time, since it is indeed an issue for who is using this lib.
Thanks for your attention.

Cheers,
Quinn

Answer 6 · 2017-01-26T04:59:05.000Z

Just for testing

Answer 7 · 2017-01-26T12:47:02.000Z

Hi Quinn, Please see comments below +++ "Though I am not sure if the dump-state is related to the real issue, but if the dump-state can cause the issue, then can see the issue on the real project; and if using the normal i2s, the dump-state can't cause the issue, then on the real project, can't see the issue anymore. So from the simple logic, I suppose they are related." +++The fact that normal I2S is not broken by dumpstate (interrupting the tile) is that it uses a different clocking scheme. BCLK is generated via a pattern on a buffered port so if the processor stops, BCLK stops. This also means that LRCLK and data stop. The master may be able stop and restart without losing synch relative to data, LRCLK and BCLK. Frame based has a free-running BCLK generated by the clockblock (much more efficient). However, if the processor is interrupted, then the I2S slave will continue to receive the BCLK from the clock block. When it is resumed, there is only a 1/64 chance of alignment being correct. Neither component was designed to be interrupted so the fact one copes with it better than the other is interesting, but not useful. So whilst this causes an issue that looks like the reported problem, it is not re-creatign the problem locally - the root cause for the customer problem is different. +++We need to look elsewhere to get to the real root cause. This is interesting: "BTW, seems it is the issue of the master in, master out seems work normally. And in fact, in the real project, the issue will happen with I2S frame master anyway at a low freq without dump-state and JTAG.” +++To me, this suggests that the slave (CODEC I assume) is receiving an extra BCLK occasionally which will cause it to mis-align. The master/data out does not misalign because those signals are locked internally. My feeling is that this could be a signal integrity problem on the BCLK line - perhaps another signal interfering with BCLK or perhaps reflections on the trace? I think we should focus on this to solve the issue. Have you reviewed the schematic/layout? Regards, Ed

Answer 8 · 2017-02-06T08:19:30.000Z

Hi Ed

Disregarding the dumpstate, the fact on the real customer project, if using the normal I2S, then customer didn't find the random startup noise anymore.
That shows:

the hardware is the same.
only the software "frame I2S -> normal I2S" can solve the problem.

So the simple logic above should be able to show:

it is the software issue?
the hardware has no problem?

Best Regards
Quinn

Answer 9 · 2017-02-06T18:09:05.000Z

Hi Quinn, yes, we should completely forget dumpstate - this is misleading and will not bring us to the root cause. As I explained before, there is a difference in how the port works between the two implementations: - i2s normal -> BCLK is an output AND an input. Any noise on BCLK may falsely clock the DAC or falsely trigger our chip too.. - i2s frame -> BCLK is output only I was wondering if the a poor BCLK signal may cause both XMOS and DAC to get a false edge, but actually stay in synch. This is just an idea at the moment because I cannot think of any other reason for this behaviour. I am not yet sure of the root cause. To systematically find root cause we have to: 1) Recreate the problem locally 2) Debug the observed issue Because you cannot reproduce this on an XMOS board, my feeling is that this hardware related. I could be wrong because I do not have a system I can interrogate to verify this. This is just my feeling from many years of writing and debugging I2S systems. I cannot see why after millions of loops, the I2S frame master would suddenly fail (especially as you say there is no application timing delay) from a code perspective. It has been used in a few places also and we have not had reports that it is broken. There could be a bug, but probably a small chance. Could you get a scope capture from the BCLK pin near the DAC? It would be interesting to see what the egdes look like and perhaps the probe loading may make the problem disappear? Do you see how, if it is only the input that misaligns occasioanlly, that it is most likely an issue of the BCLK reaching the DAC? Thanks, Ed

Answer 10 · 2017-02-08T06:46:00.000Z

H Ed As I explained before, there is a difference in how the port works between the two implementations: - i2s normal -> BCLK is an output AND an input. Any noise on BCLK may falsely clock the DAC or falsely trigger our chip too.. - i2s frame -> BCLK is output only Sorry, here I can’t understand that, why in i2s normal, BCLK is an output AND an input? Because the hardware is the same, then the roles of the I2S ends are fixed, no matter using what kind of I2S master. Yes, the original issue has not been reproduced on the XMOS demo board yet. So let me try to recreate the problem on the demo board in order to solve 2 strange items: - why the issue can’t reproduce on the XMOS board - why on the same hardware, the issue can be solved with the I2S normal, while I2S frame has problem BTW, here is the original issue info, just FYI: [How to reproduce the issue] - power off the board - after 10min, then power on the board again, then there is about 90% probability that the audio analog out will output noise [The firmware audio flow] I2S master in -> DSP -> I2S master out [The firmwares loading flow in the project] 1\ audio input mode choose Analog in, power off 2\ power on 3\ loader, run default firmware “UAC2 in -> DSP -> I2S master out”; then detect the mode selection is not UAC2 in, is Analog in; do software reboot 4\ loader, run Analog in firmware “I2S master in -> DSP -> I2S master out” Best Regards Quinn

Answer 11 · 2017-02-08T09:35:42.000Z

Hi Quinn, “Sorry, here I can’t understand that, why in i2s normal, BCLK is an output AND an input? Because the hardware is the same, then the roles of the I2S ends are fixed, no matter using what kind of I2S master. “ Yes it does seem odd, but it really does work like that for i2s normal. The clocking scheme looks like this: [cid:1D2A9197-5185-437F-9161-B25A950020AF@xmos.local] The key here is that the bclk clock block is clocked from the output of the BCLK port (bclk is generated by sending 0xaaaa etc. to the BCLK pin inside i2s). HOWEVER, the design of the chip is such that the input to the bclk clock block comes from the *physical bclk pin*, NOT an internal signal. i.e. it goes through the buffer to the physical pin and then back inside again. This means that, if you short p_bclk, then the bclk clock block will not get a clock. Or, if the BCLK port has bad reflections on it due to PCB layout issues like stubs, then bclk clock block may will also see extra clocks. I have seen this on prototype hardware where BCLK is badly routed. I am not 100% sure that this is the reason for the difference between i2s normal/frame IP, but it is an area worth investigating. (I am assuming that you have ruled out timing issues from back-pressure on intrefaces etc, as per previous comments). This is why I strongly suggest to check signal integrity of BCLK using a high speed scope on the customer hardware to see if there are reflections, or if the scope probe capacitive loading improves the situation by dampening out the signal. Regards, Ed

Answer 12 · 2017-02-08T09:59:23.000Z

Hi Ed “The key here is that the bclk clock block is clocked from the output of the BCLK port (bclk is generated by sending 0xaaaa etc. to the BCLK pin inside i2s). HOWEVER, the design of the chip is such that the input to the bclk clock block comes from the *physical bclk pin*, NOT an internal signal. i.e. it goes through the buffer to the physical pin and then back inside again. This means that, if you short p_bclk, then the bclk clock block will not get a clock. Or, if the BCLK port has bad reflections on it due to PCB layout issues like stubs, then bclk clock block may will also see extra clocks. I have seen this on prototype hardware where BCLK is badly routed.” This is interesting, because from above, looks like the I2S normal is susceptible by the bad bclk signal integrity, however, the real case shows only the I2S frame has issue. Best Regards Quinn

Answer 13 · 2017-02-08T10:06:53.000Z

Hi Quinn, If there is a glitch on BCLK then perhaps, in normal I2S case, both master (xmos) and slave (DAC/ADC) all receive so they are still in synch. But in the case of I2S frame, a glitch on BCLK line may cause I2S slaves (DAC/ADC) to receive an extra clock, but not the master (xmos). So in the frame case, they could be out of sync. This is just a theory, but like i say, worth checking in the absence of any better ideas! Regards, Ed On 8 Feb 2017, at 09:59, QuinnWang <notifications@github.com<mailto:notifications@github.com>> wrote: Hi Ed “The key here is that the bclk clock block is clocked from the output of the BCLK port (bclk is generated by sending 0xaaaa etc. to the BCLK pin inside i2s). HOWEVER, the design of the chip is such that the input to the bclk clock block comes from the *physical bclk pin*, NOT an internal signal. i.e. it goes through the buffer to the physical pin and then back inside again. This means that, if you short p_bclk, then the bclk clock block will not get a clock. Or, if the BCLK port has bad reflections on it due to PCB layout issues like stubs, then bclk clock block may will also see extra clocks. I have seen this on prototype hardware where BCLK is badly routed.” This is interesting, because from above, looks like the I2S normal is susceptible by the bad bclk signal integrity, however, the real case shows only the I2S frame has issue. Best Regards Quinn — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#26 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAmLbTySIJotCLhJUjR8Jff6HQeuzRBmks5raZH8gaJpZM4Lrw9e>.

Answer 14 · 2018-04-18T08:14:44.000Z

Hi Quinn, is this still an issue (if so we need a repeatable failure case) or can we close it?

Answer 15 · 2018-04-20T09:12:11.000Z

This topic can be closed now, thanks very much for your input about the thinking of solving this problem. Best Regards Quinn

Answer 16 · 2020-07-01T08:32:23.000Z

@ed-xmos, can this be closed now?

Answer 17 · 2021-10-25T12:32:42.000Z

@QuinnWang @ed-xmos looking through all of the I2S issues currently - going to close this issue as it appears to have been resolved; if it does need reopening due to further as-yet-unrecorded-here observations let me know.