BUG - renaming files corrupt them
juleslagarde opened this issue · 9 comments
Context
ntfs-3g version: 2022.10.3 external FUSE 29 (Configuration type 7, XATTRS are on, POSIX ACLS are on)
storage device: 2TB USB HDD using ntfs
OS: Arch Linux
Problem
When I rename a file using mv I get an Input/Output Error when using ls
root@ThinkPad-server:/media/usb_2TB# ls -l | grep testfile
-rwxrwxrwx 1 root root 8738 juil. 23 2019 testfile_old.dat
root@ThinkPad-server:/media/usb_2TB# mv testfile_old.dat testfile2_old.dat
root@ThinkPad-server:/media/usb_2TB# ls -l | grep testfile
ls: cannot access 'testfile_old.dat': Input/output error
-????????? ? ? ? ? ? testfile2_old.dat
But, it works fine on newly created files:
root@ThinkPad-server:/media/usb_2TB# dd if=/dev/urandom of=testfile_new.dat bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 1,01336 s, 103 MB/s
root@ThinkPad-server:/media/usb_2TB# mv testfile_new.dat testfile2_new.dat
root@ThinkPad-server:/media/usb_2TB# ls -l | grep testfile
ls: cannot access 'testfile_old.dat': Input/output error
-????????? ? ? ? ? ? testfile2_old.dat
-rwxrwxrwx 1 root root 104857600 janv. 30 12:09 testfile2_new.dat
debug
What would be the process to debug such issues ?
@juleslagarde The main information that can be collected is from the syslog. If there's an error then there's usually one or more lines in the syslog with more information. (Depending on your distribution and setup you may find these messages in /var/log/messages, /var/log/syslog or through invoking journalctl -a.)
here is the log of ntfs-3g in /var/log/syslog
Jan 31 10:16:04 ThinkPad-server ntfs-3g[585]: Version 2021.8.22 integrated FUSE 28
Jan 31 10:16:04 ThinkPad-server ntfs-3g[585]: Mounted /dev/sdb1 (Read-Write, label "YOLO", NTFS 3.1)
Jan 31 10:16:04 ThinkPad-server ntfs-3g[585]: Cmdline options: rw,relatime,user_id=0,group_id=0,allow_other
Jan 31 10:16:04 ThinkPad-server ntfs-3g[585]: Mount options: user_id=0,group_id=0,allow_other,allow_other,nonempty,relatime,rw,fsname=/dev/sdb1,blkdev,blksize=4096
Jan 31 10:16:04 ThinkPad-server ntfs-3g[585]: Ownership and permissions disabled, configuration type 7
I tried the "debug" option, and got this in the terminal:
unique: 3898, opcode: LOOKUP (1), nodeid: 2, insize: 57
LOOKUP /testfile2_old.dat
unique: 3898, error: -5 (Input/output error), outsize: 16
unique: 3974, opcode: LOOKUP (1), nodeid: 2, insize: 58
LOOKUP /testfile2_new.dat
NODEID: 150
unique: 3974, error: 0 (Success), outsize: 144
unique: 3976, opcode: GETATTR (3), nodeid: 150, insize: 56
unique: 3976, error: 0 (Success), outsize: 120
unique: 3978, opcode: GETXATTR (22), nodeid: 150, insize: 65
unique: 3978, error: -61 (No data available), outsize: 16
unique: 3980, opcode: GETXATTR (22), nodeid: 150, insize: 72
unique: 3980, error: 0 (Success), outsize: 16
so error code -5, probably from libntfs-3g
That's odd, I would have expected some errors in the syslog.
So I understand that this is reproducible for files created on Windows (?) but not for files created by ntfs-3g in Linux? Or does the problem also occur for files created by ntfs-3g, although less recently (e.g. files created during a previous mount/unmount session)?
Would you please post the output of ntfsinfo for a file that is behaving in this strange way and for its parent directory, e.g. for /dev/sdb1 and testfile2_old.dat in the root directory:
ntfsinfo -F "/testfile2_old.dat" -v /dev/sdb1
ntfsinfo -F "/" -v /dev/sdb1
The files where created on Windows 4-5 years ago, yeah. And it is reproducible on theses olds files yes.
root@ThinkPad-server:~# ntfsinfo -F "/testfile2_old.dat" -v /dev/sdb1
Error loading node: Input/output error
root@ThinkPad-server:~# ntfsinfo -F "/" -v /dev/sdb1
Unnormalized path /
Dumping Inode 292 (0x124)
Upd. Seq. Array Off.: 48 (0x30)
Upd. Seq. Array Count: 3 (0x3)
Upd. Seq. Number: 1450 (0x5aa)
LogFile Seq. Number: 0x20d43e8
MFT Record Seq. Numb.: 1 (0x1)
Number of Hard Links: 1 (0x1)
Attribute Offset: 56 (0x38)
MFT Record Flags: IN_USE DIRECTORY
Bytes Used: 512 (0x200) bytes
Bytes Allocated: 1024 (0x400) bytes
Next Attribute Instance: 10 (0xa)
MFT Padding: 00 00
Dumping attribute $STANDARD_INFORMATION (0x10) from mft record 292 (0x124)
Attribute length: 96 (0x60)
Resident: Yes
Name length: 0 (0x0)
Name offset: 0 (0x0)
Attribute flags: 0x0000
Attribute instance: 0 (0x0)
Data size: 72 (0x48)
Data offset: 24 (0x18)
Resident flags: 0x00
ReservedR: 0 (0x0)
File Creation Time: Tue Jul 23 21:22:22 2019 UTC
File Altered Time: Tue Jan 30 11:11:27 2024 UTC
MFT Changed Time: Tue Jan 30 11:11:27 2024 UTC
Last Accessed Time: Tue Jan 30 11:12:27 2024 UTC
File attributes: (0x00000000)
Maximum versions: 0
Version number: 0
Class ID: 0
User ID: 0 (0x0)
Security ID: 280 (0x118)
Quota charged: 0 (0x0)
Update Sequence Number: 0 (0x0)
Dumping attribute $FILE_NAME (0x30) from mft record 292 (0x124)
Attribute length: 104 (0x68)
Resident: Yes
Name length: 0 (0x0)
Name offset: 0 (0x0)
Attribute flags: 0x0000
Attribute instance: 4 (0x4)
Data size: 78 (0x4e)
Data offset: 24 (0x18)
Resident flags: 0x01
ReservedR: 0 (0x0)
Parent directory: 5 (0x5)
File Creation Time: Tue Jul 23 21:22:22 2019 UTC
File Altered Time: Sat Dec 9 19:58:04 2023 UTC
MFT Changed Time: Sat Dec 9 19:58:04 2023 UTC
Last Accessed Time: Tue Jun 13 11:52:08 2023 UTC
Allocated Size: 0 (0x0)
Data Size: 0 (0x0)
Filename Length: 6 (0x6)
File attributes: I30_INDEX (0x10000000)
Namespace: Win32 & DOS
Filename: 'movies'
Dumping attribute $OBJECT_ID (0x40) from mft record 292 (0x124)
Attribute length: 40 (0x28)
Resident: Yes
Name length: 0 (0x0)
Name offset: 0 (0x0)
Attribute flags: 0x0000
Attribute instance: 5 (0x5)
Data size: 16 (0x10)
Data offset: 24 (0x18)
Resident flags: 0x00
ReservedR: 0 (0x0)
Object ID: 11d83fe8-aaf8-11e9-b705-5ce0c5c4314a
Birth Volume ID: missing
Birth Object ID: missing
Domain ID: missing
Dumping attribute $INDEX_ROOT (0x90) from mft record 292 (0x124)
Attribute length: 88 (0x58)
Resident: Yes
Name length: 4 (0x4)
Name offset: 24 (0x18)
Attribute name: '$I30'
Attribute flags: 0x0000
Attribute instance: 9 (0x9)
Data size: 56 (0x38)
Data offset: 32 (0x20)
Resident flags: 0x00
ReservedR: 0 (0x0)
Indexed Attr Type: DIRECTORY_I30
Collation Rule: 1 (0x1)
Index Block Size: 4096 (0x1000)
Clusters Per Block: 1 (0x1)
Entries Offset: 16 (0x10)
Index Size: 40 (0x28)
Allocated Size: 40 (0x28)
Index header flags: 0x01
Dumping index root:
Entry length: 24 (0x18)
Key length: 0 (0x0)
Index entry flags: 0x03
Subnode VCN: 5 (0x5)
End of index block reached
Index entries total: 1
Dumping attribute $INDEX_ALLOCATION (0xa0) from mft record 292 (0x124)
Attribute length: 80 (0x50)
Resident: No
Name length: 4 (0x4)
Name offset: 64 (0x40)
Attribute name: '$I30'
Attribute flags: 0x0000
Attribute instance: 6 (0x6)
Lowest VCN 0 (0x0)
Highest VCN: 23 (0x17)
Mapping pairs offset: 72 (0x48)
Compression unit: 0 (0x0)
Data size: 98304 (0x18000)
Allocated size: 98304 (0x18000)
Initialized size: 98304 (0x18000)
Runlist: VCN LCN Length
0x0 0xabef3 0x18
Corrupt attribute 0xa0 in inode 292
Failed to read $INDEX_ALLOCATION attribute: Value too large for defined data type
Dumping attribute $BITMAP (0xb0) from mft record 292 (0x124)
Attribute length: 40 (0x28)
Resident: Yes
Name length: 4 (0x4)
Name offset: 24 (0x18)
Attribute name: '$I30'
Attribute flags: 0x0000
Attribute instance: 7 (0x7)
Data size: 8 (0x8)
Data offset: 32 (0x20)
Resident flags: 0x00
ReservedR: 0 (0x0)
End of inode reached
Total runs: 1 (fragments: 1)
thoses lines are outputed to stderr instead of stdout:
Corrupt attribute 0xa0 in inode 292
Failed to read $INDEX_ALLOCATION attribute: Value too large for defined data type
Also the new file is still new (meaning no problem) even after umount/mount.
It looks like the $INDEX_ALLOCATION attribute hits a consistency check... it seems that this attribute is never expected to be larger than 64 KiB, but in this case it's 96 KiB. I don't know what prompted the 64 KiB limit in the first place so I'll have to ask around. Just to be sure, could you possibly check this NTFS drive with chkdsk in Windows and paste the output here? If the data is valid and accepted by Windows then we should reevaluate.
Last time I plugged the drive on windows it removed the corrupted file.
C:\Windows\System32>chksdk D:
'chksdk' n’est pas reconnu en tant que commande interne
ou externe, un programme exécutable ou un fichier de commandes.
C:\Windows\System32>chkdsk D:
Le type du système de fichiers est NTFS.
Le nom de volume est YOLO.
AVERTISSEMENT ! Le paramètre /F n'a pas été spécifié.
Exécution de CHKDSK en mode lecture seule.
Étape 1 : Examen de la structure du système de fichiers de base...
1467 enregistrements de fichier traités. este : 0:00:03 ...
La vérification des fichiers est terminée.
Durée de la phase (Vérification des enregistrements de fichiers) : 1.03 secondes.
281 enregistrements de grand fichier traités. reste : 0:00:01
Durée de la phase (Récupération des enregistrements de fichiers orphelins) : 4.19 millisecondes.
0 enregistrements de fichier incorrect traités. : 0:00:01 .
Durée de la phase (Vérification des enregistrements de fichiers incorrects) : 4.94 millisecondes.
Étape 2 : Examen de la liaison des noms de fichiers...
Un nom de fichier non valide Parasite.2019.1080p.HDRip.X264.AC3-EVO.mkv:Zone.Identifier (5AD) a été détecté dans le répertoire 5AB.
Aucun nom du fichier 5AD n’est valide.
Des erreurs mineures de nom de fichier ont été détectées dans le fichier 5AD.
146 enregistrements d’analyse traités. reste : 0:00:01 ..
L’entrée Parasite.2019.1080p.HDRip.X264.AC3-EVO.mkv de l’index $I30 du fichier 124 n’est pas correcte.
L’entrée Source Code2.mkv de l’index $I30 du fichier 124 n’est pas correcte.
L’entrée Contagion.2011.1080p.BluRay.x264.AAC5.1-[YTS.MX]OLD.srt de l’index $I30 du fichier 2A4 n’est pas correcte.
L’entrée Parasite.2019.1080p.HDRip.X264.AC3-EVO.mkv:Zone.Identifier de l’index $I30 du fichier 5AB n’est pas correcte.
1787 entrées d’index traitées. : 0:00:01 ...
La vérification des index est terminée.
Durée de la phase (Vérification de l'index) : 235.50 millisecondes.
Erreurs trouvées. CHKDSK ne peut pas continuer en mode lecture seule.
I tried to change the windows language but couldn't arrive to a solution quickly.
Here is a Google Translate:
C:\Windows\System32>chksdk D:
'chksdk' is not recognized as an internal command
or external, an executable program or batch file.
C:\Windows\System32>chkdsk D:
The file system type is NTFS.
The volume name is YOLO.
WARNING ! The /F parameter was not specified.
Running CHKDSK in read-only mode.
Step 1: Examine the basic file system structure...
1467 file records processed. este: 0:00:03 ...
File verification is complete.
Duration of phase (Checking file records): 1.03 seconds.
281 large file records processed. remainder: 0:00:01
Phase duration (Recovery of orphaned file records): 4.19 milliseconds.
0 bad file records processed. : 0:00:01 .
Phase duration (Checking bad file records): 4.94 milliseconds.
Step 2: Examining File Name Binding...
An invalid file name Parasite.2019.1080p.HDRip.X264.AC3-EVO.mkv:Zone.Identifier (5AD) was detected in the 5AB directory.
None of the 5AD file names are valid.
Minor filename errors were detected in the 5AD file.
146 analysis records processed. rest: 0:00:01 ..
The entry Parasite.2019.1080p.HDRip.X264.AC3-EVO.mkv in index $I30 of file 124 is not correct.
The Source Code2.mkv entry in index $I30 of file 124 is not correct.
The entry Contagion.2011.1080p.BluRay.x264.AAC5.1-[YTS.MX]OLD.srt in index $I30 of file 2A4 is not correct.
The Parasite.2019.1080p.HDRip.X264.AC3-EVO.mkv:Zone.Identifier entry in index $I30 of file 5AB is not correct.
1787 index entries processed. : 0:00:01 ...
The index check is complete.
Phase duration (Index Check): 235.50 milliseconds.
Errors found. CHKDSK cannot continue in read-only mode.
All issues seems to be have been fixed with chkdsk D: /F on windows
Very strange, thanks for your time.
As a final comment I think what happened was that there was some inconsistency on the NTFS volume which was repaired by chkdsk. Due to other issues we weren't able to find enough information about the root cause to know exactly what it was, but whatever it was, chkdsk was able to fix it.