Created file but cannot remove it

[cfeied]

I'm using FIFO_SDIO and SdFs (Fat16, Fat32, and exFat) for the built-in SD adaptor on the teensy 3.6.

Everything has been working as I work through testing over the past few weeks. I can read from and write to both Fat32 and exFat cards without difficulty.

Now I've encountered a single occurrence in which I created a file that cannot be deleted.

For my project, files are created on an exFat card by pre-allocating 1 GB, collecting multi-sensor data and writing to the file for a period of time, and then truncating the file. This seems to work correctly, and these files can be read and deleted normally.

I could not tell from the documentation whether the same code would work on Fat32 cards, or whether I would need to keep track of the bytes written and truncate the file to a specified size.

To test this, the exFat card was swapped out for a Fat32 card already containing a few dozen small files. The same code was executed. A 1 GB file was created, however the file was not truncated and the file size still shows as 1,073,741,824. I cannot guarantee that the file was closed or the truncation request was made, because I did have some sessions that were terminated abnormally for unrelated reasons, and I don't know during which session this file was created.

Now I cannot remove that file from the FAT32 card. I can rename it, but remove() fails with DBG_FAIL at FatPartition.cpp.214:

  // error if reserved cluster of beyond FAT
  if (cluster < 2 || cluster > m_lastCluster) {
    DBG_FAIL_MACRO;
    goto fail;
  }

FWIW, using MTP mode I can copy the file from the SD card to another drive, but MTP also fails when I attempt to delete it from the original SD card.

I can reformat the card, of course, but I'm wondering what else I can do to track down and/or resolve the problem.

My project code is fairly large and I haven't yet tried to build a small repro, but I'll do that and post it as soon as I can. For the moment, any thoughts or suggestions would be helpful.

= = = = =
Using SdFat-2.0.0-beta.8
Board is Teensy 3.6
Most current Arduino & Teensyduino core libraries
Most current Visual Studio 2019 + visual micro on Windows10.

[greiman]

The file system is corrupt. The most common cause is a shutdown or a crash with an open file. As soon as a file system is corrupt, the problem will spread.

It is not practical to build a file system check and repair program in the limited memory of a micro-controller.

[cfeied]

In my scenario I can assume that a session file size exactly matching the preallocated size means the file was not properly closed or truncated. Will that reliably corrupt the file system?

If not, is there anything I can do to detect a corrupted file system?

[greiman]

A file can appear OK but if it was created in a corrupt system it may become corrupt. Once the FAT is corrupt, allocation units may be in the allocation chain for two files.

There is no easy way to detect a corrupt file. You need to check that every allocation unit has the correct value. This requires lot of memory, usually for bitmaps to remember the state of every allocation unit.

[jerabaul29]

A small question here: I have been using the default Arduino SD library in the following way on a logger application: each 15 minutes, create a new file with incremented name, and use it to store the next 15 minutes of data. This way, I was getting some series of 15 minutes files, loosing only the last file for which the logger had been switched off in the middle of writing on an opened file. This approach has served me well, I do not think I experienced any loss of data on the files already written and closed, and I was able to just resume execution of logging the next day by using the next file.

But do you mean that doing things this way may corrupt the whole FAT and therefore also already written data? Was I just lucky that it did not happen? Or is the default SD library doing some extra actions to avoid corrupting the whole filesystem, that may not be implemented in the present code repo?

[greiman]

There is no way to avoid corruption of a FAT file system if a crash happens at a bad time. If you have a crash, use a PC to verify/repair the SD to prevent escalation of the problem.

It is not possible to verify an SD on Arduino because of the small memory.

The default SD library is a wrapper for an eleven year old version of SdFat and has less code to prevent corruption.