greiman/SdFat

ExFatLogger vs LowLatencyLogger different behaviour

d3-f4ult opened this issue · 7 comments

d3-f4ult commented

Let me preface this by saying I know the examples are still a work in progress and especially the ones under examplesV1 may not be compatible with v2.

I read here that "the ExFatLogger example it replaced the LowLatencyLogger example. It works without raw writes." and while it is true that it achieves the same thing, perhaps even better, it behaves differently on a power cut.

I tried them both on a Nano Every by running them for some time then resetting the board, after observing the data on the file with ImHEX these were my results:

  • LowLatencyLogger: file exists on the SD, filesize depends on runtime, data is written but the last block may be corrupt (as expected).
  • ExFatLogger: file exists on SD, filesize is always the preallocated size, no data is written. Same result even when writing data beyond the preallocated size (I measured how much time it takes on successful run, then ran it for the same time but reset the board at the end instead of interrupting through serial).

If I understand the code correctly, ExFatLogger moved away from the raw writes and favours a simpler approach for the user by using truncate() and sync(), however these 2 functions are called only after a serial input.

I have a couple of questions:

  1. Would calling sync() more often achieve the same behaviour as LowLatencyLogger?
  2. Does sync() autonomously handle when to actually write data to the SD in the most optimal way (eg. by holding back data until the buffer is full)?
  3. How does calling sync() more often affect performance and how often should I call it for optimal results? In my project I gather very little data each iteration (a few bytes) so calling sync() each time seems very wasteful. Should I wait to have a multiple of 512 or something along those lines (see question 2)?
  4. Is the first empty dummy sector necessary / why do we need it?
  5. If I can write files bigger than the preallocation why should I preallocate? Does performance beyond the allocated space drop off?

I've tried reading the docs but they are not very descriptive, so I had to dive directly into the code slowing things down considerably.

greiman commented

ExFatLogger: file exists on SD, filesize is always the preallocated size, no data is written

The data is written but the file is not closed so the directory shows an empty file for exFAT SD cards. FAT16/FAT32 should have data but the file won't be truncated.

Calling sync() updates the directory entry but will cause a big delay. It also interrupts the SD write sequence for data.

Is the first empty dummy sector necessary / why do we need it?

This insures the SD write pipeline is started.

If I can write files bigger than the preallocation why should I preallocate? Does performance beyond the allocated space drop off?

You will see random delays while a search for free clusters occurs.

You may not need any of these optimizations depending on your SD and sample rate and speed of your board.

There is a better way to do a FIFO style logger but I only have an example for Teensy SDIO. It is easy to modify this example for SPI. It can be uses for binary or text.

d3-f4ult commented

  • You may not need any of these optimizations depending on your SD and sample rate and speed of your board.

    The original data source is a CAN bus running at 500MB/s 500Kb/s which is way to high for my SDXC UHS-I, but I've managed to reduce the speed to write at to just under the SD specs limit.

  • There is a better way to do a FIFO style logger but I only have an example for Teensy SDIO. It is easy to modify this example for SPI. It can be uses for binary or text.

    Will definitely look into it, thanks. edit: I see it uses a circular buffer which I was already thinking about and suits my needs perfectly.

  • The data is written but the file is not closed so the directory shows an empty file for exFAT SD cards. FAT16/FAT32 should have data but the file won't be truncated.

    So the data has been written to the SD but it's not accessible because the FAT Region hasn't been updated. Is it possible to recover the data/update the directory after the power turns back on? (similar to LowLatencyLoggers's file recovery.

    Calling sync() updates the directory entry but will cause a big delay. It also interrupts the SD write sequence for data

    Does sync() handle it or should I write my code?

greiman commented

So the data has been written to the SD but it's not accessible because the FAT Region hasn't been updated. Is it possible to recover the data/update the directory after the power turns back on? (similar to LowLatencyLoggers's file recovery.

The problem is the directory and there is no way to recover the length that has been written since it is stored in the directory entry.

Does sync() handle it or should I write my code?

You can't write your own code. sync() is called by close to update the directory entry.

lowLatencyLogger was designed for FAT32/FAT16. FAT only has a file length entry in the directory so it behaves differently than exFAT.

exFAT has an allocated length and a valid length. Preallocate sets the allocated length but unless valid length is set, you can't read the data.

With FAT16/FAT32 you can read the data. There is junk in the unwritten part of the FAT file. It appears you recovered the data.

If you use a FAT32 SD, ExFatLogger should behave like lowLatencyLogger.

Note that Windows may delete preallocated clusters beyond valid length on a exFAT SD. That is why the file should be truncated to valid length.

greiman commented

Because of the structure of exFAT, raw writes to a preallocated file won't work on exFAT. The valid length field won't be set.

If you try to rewrite an existing file the result will be extremely slow unless you write a multiple of 512 byte chunks on 512 byte boundaries. the reason is that unchanged data in a block must be read, updated, and rewritten.

Allowing a crash while writing files is not safe on any computer. FAT and exFAT are especially bad since recovery may not be possible because of limited information in the file structures.

greiman commented

The original data source is a CAN bus running at 500MB/s

That's really amazing Gigabit Ethernet is only about 125 MB/s

A gigabit network can be incredibly fast. 1 gigabit (Gb) is equal to 125 megabytes (MBs), so a gigabit network offering a speed of 1 Gbps could transfer 125 megabytes of data per second.

I bet Bosch would like to know how you go 200 times faster than CAN_XL. CAN_XL is 20Mbit/s or 2.5 MB/sec.

You could easily record CAN_XL with a Teensy 4.1. I get 22 MB/Sec with Teensy SDIO.

d3-f4ult commented

I bet Bosch would like to know how you go 200 times faster than CAN_XL. CAN_XL is 20Mbit/s or 2.5 MB/sec.

This is exceedingly funny, I must have the fastest bus this side of the Moon.

I made multiple mistakes, the CAN bus works at 500Kb/s(or a mere 0.0625MB/s), but up to 1Mb/s . However the SD shares SPI with other devices so I have to minimize the amount of time the SD holds the bus for.
I explained myself poorly, what I meant is that on the MCU's side I managed to process data in batches such that if I write at sufficient speed I will have minimum downtime and maximum data storage (low data loss).

I'll definitely study the subjects more, especially the underlying exFAT structures. Thank for the insights.

greiman commented

Forget everything I told you if you are using shared SPI. Don't bother me if you have performance problems.

You can forget using a fast logger if you share SPI. SD cards cancel fast multi-block transfer mode when SD_CS goes high.

I have a Due on my desk from a previous issue. I ran the bench example to illustrate.

Dedicated SPI:

Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0X1B
OEM ID: SM
Product: 00000
Revision: 1.0
Serial number: 0X390D14D2
Manufacturing date: 6/2015

FILE_SIZE_MB = 5
BUF_SIZE = 512 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
4533.09,2175,110,111
4537.21,127,110,111

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
4524.89,113,110,111
4533.09,113,110,111

Shared SPI:

Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0X1B
OEM ID: SM
Product: 00000
Revision: 1.0
Serial number: 0X390D14D2
Manufacturing date: 6/2015

FILE_SIZE_MB = 5
BUF_SIZE = 512 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
458.72,34850,950,1114
451.96,35999,949,1131

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1063.83,1795,339,479
1062.70,1802,339,480