Wishbone, 2 writes followed by colliding read returns incorrect result, write stuck in FIFO

[epsilon537]

I'm investigating an issue where a specific write-write-read sequence returns an incorrect result.
My LiteDRAM core is configured for Arty A7 with a 32-bit wishbone port.
I was able to reproduce the issue in simulation. The transaction sequence that goes wrong is the following:

0. Word at address 0 contains value 'h13121100.
1. Write word 'h0432D459 to address 'h16.
2. Write byte 'h10 to address 'h4080 (WB sel=1)
3. Read word from address 'h4080. The returned value is 'h0342D459, not 13121110.

Looking at the trace, the LiteDRAMNativePortConverter's cmd_we signal is deasserted shortly after the read word is issued. At this point the write to 'h16 has gone out, but the write to 'h4080 is stuck in the wdata_fifo. By the time read transaction completes (with incorrect data), the write transaction is still stuck in the wdata_fifo.

I can see that the read-write collision is detected (rw_collision=1). However, the collision is cleared when the write to 'h16 is finished, not the write to 'h4080 which actually causes the collision. As a result, the read command is allowed to proceed before the colliding write command has even started.

Stepping back a bit, I see the following in wishbone.py:

self.comb += [
port.cmd.addr.eq(wishbone.adr - offset),
port.cmd.we.eq(wishbone.we),
port.cmd.last.eq(~wishbone.we), # Always wait for reads.
port.flush.eq(~wishbone.cyc) # Flush writes when transaction ends.
]

Here, ~CYC is used to flush writes. However, in my case CYC is asserted throughout the entire sequence. The Wishbone spec does not require CYC to be deasserted between writes and read. In fact, the spec explicitly states that CYC may remain asserted indefinitely (permission 3.05).

[epsilon537]

The above sequence goes wrong because the LiteDRAMNativePortConverter wdata_fifo gets into a bad state because of the three preceding write commands:

1. a word write to 'h18
2. a halfword write to h'19 (wb_sel=3)
3. a byte write to h'19 (wb_sel=4)

The trace shows the wdata_fifo initially empty. Then the three writes enter the FIFO. However, only two entries are subsequently read from the FIFO. The last write operation (to h'19 with wb_sel=4) gets stuck.

[epsilon537]

I further simplified the failing sequence:

1. Write word 0xffffffff to 0x19
2. Immediately followed by a write word 0x11111111 to the same address.
3. Read word from 0x19 -> returns 0xffffffff

I.e. the 2nd word write is ignored. Again the wdata FIFO shows two write commands entering the FIFO but only one write command leaving the FIFO.

Two back-to-back writes to the same address may seem like a degenerate case, but when the two write commands enable different byte lanes, it's not that uncommon.

The big hammer workaround I'm currently using is to flush all wishbone transactions. In wishbone.py:

self.comb += [
port.cmd.addr.eq(wishbone.adr - offset),
port.cmd.we.eq(wishbone.we),
port.cmd.last.eq(~wishbone.we), # Always wait for reads.
port.flush.eq(1) # Always flush.
]

Here is the trace with the workaround enabled. You can see both write command entering and leaving the write FIFO.