Firmware diff feature isn't detecting all changes

[n0x08]

Describe the bug
While testing the firmware diff feature against two firmware images, it didn't detect at least 2 changes I knew were in them.

To Reproduce
Steps to reproduce the behavior:

1. EMBA installation: standard install, fully updated as of Nov 1.
2. Use the firmware available here: [(https://www.racom.eu/eng/support/dw/archiv-midge.html)] - you want midge2-fw-4.6.40.102.img and midge2-fw-4.6.40.103.img
3. Start EMBA with the following parameters: sudo ./emba.sh -f ~/midge2-fw-4.6.40.102.img -o ~/midge2-fw-4.6.40.103.img -l ~/midge_diff
4. Check the d10_firmware_diffing.html page and search for gnssAutoAlign.php and sdkJobs.php; they don't exist.
5. Check the extracted firmware images and compare the two versions (paths truncated for brevity):

nate@emba2:~/midge_diff$ diff ./IMAGE1/home/www-data/admin/gnssAutoAlign.php ./IMAGE2/home/www-data/admin/gnssAutoAlign.php
36c36
<         exec("/usr/local/sbin/www-scripts/various/doAutoAlignment " . $device_id . " > /dev/null &");
---
>         exec("/usr/local/sbin/www-scripts/various/doAutoAlignment " . escapeshellarg($device_id) . " > /dev/null &");

nate@emba2:~/midge_diff$ diff ./IMAGE1/home/www-data/admin/include/sdkJobs.php ./IMAGE2/home/www-data/admin/include/sdkJobs.php
254c254
<         $name = trim($_POST['scriptName']);
---
>         $name = basename(trim($_POST['scriptName']));

Expected behavior
I would expect it to detect these changes; it found some differences between the images including some other PHP files which changed, not sure how it missed these.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Ubuntu 22.04 Server
• EMBA version: current
• Installation method: default with up to date docker image

Priority issue
Are you already a Sponsor? - Yes

Additional context
Add any other context about the problem here.

[m-1-k-3]

is this the correct firmware you have used for testing? https://www.racom.eu/eng/support/dw/archiv-midge.html

[n0x08]

is this the correct firmware you have used for testing? https://www.racom.eu/eng/support/dw/archiv-midge.html

Yes, sorry the link didn't come through initially.

[m-1-k-3]

I will do some tests and come back

[m-1-k-3]

┌──(m1k3㉿emba)-[~/firmware-stuff/emba_forked]
└─$ ssdeep -d -s logs/d05_firmware_diffing_extractor/extractor_firmware2/firmware2_extract/191-32165988.gzip_extract/gzip.uncompressed_extract/home/www-data/admin/gnssAutoAlign.php logs/d05_firmware_diffing_extractor/extractor_firmware/firmware_extract/191-31707726.gzip_extract/gzip.uncompressed_extract/home/www-data/admin/gnssAutoAlign.php
/home/m1k3/firmware-stuff/emba_forked/logs/d05_firmware_diffing_extractor/extractor_firmware/firmware_extract/191-31707726.gzip_extract/gzip.uncompressed_extract/home/www-data/admin/gnssAutoAlign.php matches /home/m1k3/firmware-stuff/emba_forked/logs/d05_firmware_diffing_extractor/extractor_firmware2/firmware2_extract/191-32165988.gzip_extract/gzip.uncompressed_extract/home/www-data/admin/gnssAutoAlign.php (99)

ssdeep is our main tool for fuzzy hashing. It gives us a 99% match. Our current threshold is 90%. I will adopt this value and add a better configuration possibility.

Thanks for figuring out that our threshold is not ideal :)

[m-1-k-3]

The PR #860 addresses this and some more issues. The threshold is now at 95% and an variable for adjusting the threshold is introduced. Further tests are currently running ...