Help with corrupted FIT file
fulmar2 opened this issue · 7 comments
Hi Adrian - Thank you for your excellent tools. This is not an issue with php-fit-file-analysis, but instead a request for help. I am trying to write a PHP activity file using krwes/fit-php, and after some modification, I have gotten it to work (php-fit-file-analysis reads the file just fine). When I upload to Garmin or Strava, the file is rejected, but if I run it through fitfiletools, the export is repaired and accepted by Garmin/Strava. When comparing the debug info using your tool between the two files (the one I create/the one exported by fitfiletools), everything is the same, the header is the same, the records are the same. Not sure where else to look for what is causing the error. I'm sure it is a quickly identified issue by someone who understands FIT better. I have attached the test file in case it is easy for you to recognize the problem with this file. Thanks in advance for any help you may have!
test.fit.zip
Hi @fulmar2
I assume you have the FIT SDK for the details of the protocol etc?
I recommend using the FIT CSV Tool...
I copied your test file into the java folder and ran the command (after installing Java runtime environment v1.8): java -jar FitCSVTool.jar -b test.fit test.csv that gave the below output. Looks like something to do with the CRC?
FIT CSV Tool - Protocol 2.0 Profile 21.40 Release
Exception in thread "main" java.lang.RuntimeException: com.garmin.fit.FitRuntimeException: FIT decode error: File CRC failed. Error at byte: 530
at com.garmin.fit.csv.CSVTool.run(CSVTool.java:246)
at com.garmin.fit.csv.CSVTool.main(CSVTool.java:332)
Caused by: com.garmin.fit.FitRuntimeException: FIT decode error: File CRC failed. Error at byte: 530
at com.garmin.fit.Decode.read(Decode.java:577)
at com.garmin.fit.Decode.resume(Decode.java:371)
at com.garmin.fit.Decode.read(Decode.java:348)
at com.garmin.fit.csv.CSVTool.run(CSVTool.java:220)
... 1 more
The FIT protocol used to be in a PDF in the SDK but I see it is provided online now: https://developer.garmin.com/fit/protocol/
Are you including a two-byte CRC at the end of the file? I see the above error is Error at byte: 530 and the file is 531 bytes so perhaps there is either no CRC there or it is invalid?
Hope this helps - let me know how you get on.
Adrian.
Adrian - Thanks! That is extremely helpful; the fitcsvtool is providing error messages that are a lot less cryptic than Strava's "malformed file" or Garmin's "File not accepted"! Really appreciate you taking the time to do this. I had been putting in 0 for the CRC because it seems that in the 14 byte header, it is optional... but I think that I need to calculate the CRC for the end of the file, so got to figure out how to do that. That link you sent lays out the code in Java, so I guess I need to just translate it to PHP. Thanks again for your help. If I figure out how to calc the CRC, I'll post the code here in case others care. I assume your php-fit-file-analysis ignores the CRC which is why your tool can read the files, but some other tools cannot?
I came here to post a link to the code for calculating the CRC but I see you've already found it. I'd be interested in seeing your PHP implementation when you translate it.
Steve - Thanks! Yes, I found this on the FIT SDK page:
FIT_UINT16 FitCRC_Get16(FIT_UINT16 crc, FIT_UINT8 byte)
{
static const FIT_UINT16 crc_table[16] =
{
0x0000, 0xCC01, 0xD801, 0x1400, 0xF001, 0x3C00, 0x2800, 0xE401,
0xA001, 0x6C00, 0x7800, 0xB401, 0x5000, 0x9C01, 0x8801, 0x4400
};
FIT_UINT16 tmp;
// compute checksum of lower four bits of byte
tmp = crc_table[crc & 0xF];
crc = (crc >> 4) & 0x0FFF;
crc = crc ^ tmp ^ crc_table[byte & 0xF];
// now compute checksum of upper four bits of byte
tmp = crc_table[crc & 0xF];
crc = (crc >> 4) & 0x0FFF;
crc = crc ^ tmp ^ crc_table[(byte >> 4) & 0xF];
return crc;
}
I did convert the computations to PHP and that seems to be working. Now I'm struggling with figuring out what the inputs are. I don't really know what I'm doing 😉, but thanks to this tool: https://fit2tcx.runalyze.com/temp/ I have been able to determine what the CRC should be for my file. Now I'm just playing around with what I input into my PHP to see if I can get the expected CRC as a result. There is probably a better way to do this...
This is great news - I am glad you are making progress!
That code you posted looks like C/C++ and given its similarities to PHP, a quick and dirty:
function FitCRC_Get16($crc, $byte)
{
$crc_table =
[
0x0000, 0xCC01, 0xD801, 0x1400, 0xF001, 0x3C00, 0x2800, 0xE401,
0xA001, 0x6C00, 0x7800, 0xB401, 0x5000, 0x9C01, 0x8801, 0x4400
];
$tmp;
// compute checksum of lower four bits of byte
$tmp = $crc_table[$crc & 0xF];
$crc = ($crc >> 4) & 0x0FFF;
$crc = $crc ^ $tmp ^ $crc_table[$byte & 0xF];
// now compute checksum of upper four bits of byte
$tmp = $crc_table[$crc & 0xF];
$crc = ($crc >> 4) & 0x0FFF;
$crc = $crc ^ $tmp ^ $crc_table[($byte >> 4) & 0xF];
return $crc;
}
The bitwise operators are the same.
Have not used CRCs before (and yes the phpfitfileanalysis class doesn't do anything with it), but perhaps the Wikipedia page will help with how to use the above function: https://en.wikipedia.org/wiki/Cyclic_redundancy_check
The SDK example code should also demonstrate what you feed into the function as $crc and $byte
Thanks, Adrian! You and I both translated the C++ to PHP exactly the same way... so that is good news. The part I'm stuck with is what to feed into the function. I know the "right answer" thanks to the https://fit2tcx.runalyze.com/temp/ tool. And I found some code here: https://github.com/gpsbabel/gpsbabel/blob/master/garmin_fit.cc ... it looks like they iterate through the file and pass the current byte + the previous CRC to the function. That isn't giving me the expected CRC at the end, so I'm still plugging away. Thanks again for taking the time to help with this. I'll post my final code once I figure it out in case it helps other people.
Adrian - Thanks for being so kind and helpful. Your link to the Java tools in your first reply accelerated me towards diagnosing the problem, and then later coming up with a solution. The main problem I couldn't figure out: what to feed to that function. Anyway, here is the code that I finally came up with that is now working (in case it helps anyone else out trying to write FIT files).
protected function FitCRC_Get16($crc, $byte){
$crc_table =[0x0000, 0xCC01, 0xD801, 0x1400, 0xF001, 0x3C00, 0x2800, 0xE401,0xA001, 0x6C00, 0x7800, 0xB401, 0x5000, 0x9C01, 0x8801, 0x4400];
// compute checksum of lower four bits of byte
$tmp = $crc_table[$crc & 0xF];
$crc = ($crc >> 4) & 0x0FFF;
$crc = $crc ^ $tmp ^ $crc_table[$byte & 0xF];
// now compute checksum of upper four bits of byte
$tmp = $crc_table[$crc & 0xF];
$crc = ($crc >> 4) & 0x0FFF;
$crc = $crc ^ $tmp ^ $crc_table[($byte >> 4) & 0xF];
return $crc;
}
protected function writeFileClosure($filepath) {
//mark current writing position
$offset = $this->writer->getOffset();
//move to header and write the size of the data
$this->writer->setOffset(4); //GO TO 4th POSITION TO WRITE IN DATA SIZE
$data_size = $this->writer->getSize() - 14; //WAS MINUS 12
$this->writer->writeUInt32LE($data_size); // Does not include file header or crc. Little endian format.
//BEGIN CALCULATE CRC
$handle = @fopen($filepath, "r");
if ($handle) {
$crc=0;
while (!feof($handle)) {
$hex = bin2hex(fread ($handle , 1 ));
$byte=hexdec('0x'.$hex);
if ($hex){
$crc = $this->FitCRC_Get16($crc, $byte);
}
}
fclose($handle);
}
//END CALCULATE CRC
//write 2 byte crc check at end of file
$this->writer->setOffset($offset); //MOVE TO END OF FILE
$this->writer->writeUInt16LE($crc); //PUT IN CRC
return $this;
}