The SCSI fault injection test tool
Revision History:
rev 1.00 Jan 18 2008 K.Tanaka
- Initial Release.
1. Introduction
================
This tool enables to test error handling routines related with the filesystem
and block I/O of the Linux system by injecting a SCSI fault on the system.
This tool generates "pseudo" faults in the SCSI mid-layer. This could be
a more realistic SCSI device faults simulation. For example, device faults
resulting in scsi command timeout, and media faults which could be corrected by
writing data to the failed sector could be simulated.
User can designate a device currently connected to the system and the target
location within the designated device to inject a fault, so that test program
using this tool could generate an error with a specific access.
Internally, this tool uses SystemTap. This tool rewrite the status code and
sense data of SCSI command using SytemTap and pass it to the upper layer.
So the real error handling routine of the upper layer for I/O request can be tested.
This is originally created to test software RAID (md/dm-mirror)
on Linux. But any upper layer app/driver using the SCSI mid-layer can also apply
this tool.
2. Requirements
================
- Linux system (x86 architecture) compiled with debuginfo.
- SystemTap (v5.14.1 or later) installed to the Linux system.
Please see http://sourceware.org/systemtap/index.html for SystemTap.
3. Supported SCSI fault type
============================
This test tool supports to inject the following type of faults.
3-1. Permanent read error simulation
This type of fault simulates a permanent media error
on a particular sector. Any read access to the sector fails,
but write will succeed.
3-2. Permanent read /write error simulation
This type of fault simulates a severe media error.
Both read and write fails on the particular sector permanently.
3-3. Read error correctable by write simulation
This type of fault simulates a media fault which could be
corrected by writing data to the failed sector. After writing
to the sector, subsequent reads and writes will both succeed.
3-4. Temporary read error simulation
This type of fault simulates an accidental fault, just once.
3-5. Temporary write error simulation
This type of fault simulates an accidental fault, just once.
3-6. Temporary no response on a read access simulation
This type of fault simulates a situation,such as a congestion,
resulting in scsi command timeout on a read request. After
the congestion disappears, both read and write will succeed.
3-7. Temporary no response on a write access simulation
This type of fault simulates a situation,such as a congestion,
resulting in scsi command timeout on a write request. After
the congestion disappears, both read and write will succeed.
3-8. Permanent no response on both read and write access simulation
This type of fault simulates a device fault resulting in scsi
command timeout on a write request.
Both read and write fails on the particular sector permanently.
Each fault type can be injected by using SystemTap scripts(*.stp) for each type.
script name | fault type description
-------------------+-------------------------------------------------------
disk_rerr.stp | permanent read error simulation
disk_rwerr.stp | permanent read /write error simulation
sector_rerr.stp | read error correctable by write simulation
temporary_rerr.stp | temporary read error simulation
temporary_werr.stp | temporary write error simulation
r_timeout.stp | temporary no response on read access simulation
w_timeout.stp | temporary no response on write access simulation
rw_timeout.stp | permanent no response on both read and write access simulation
disk_rwerr.stp, disk_rerr.stp, sector_rerr.stp, temporary_rerr.stp, temporary_werr.stp
return a "fake" sense data(sense key = 3, ASC = 11, ASCQ = 4) to the upper layer.
This means a medium error.
The following scripts are also included, but they are for internal use only.
scsi_fault_injection_common.stp - common routine for fault injection
scsi_timeout_injection_common.stp - common routine for timeout injection
4. Usage
=========
The flow is as follows.
For more detailed example, see the sample_usage.txt.
1. Setup
1-1 Install SystemTap and kernel with debuginfo.
(For more details, see Appendix A)
1-2 Decompress toolset and move them to an arbitrary directory.
2. Decide the type and target of the fault
2-1 Fault type
Choose a fault type you want to inject described above.
2-2 Target
Choose a target device to inject a fault.
You need to know the (major, minor) number of the target SCSI device.
If you want to inject a fault by accessing particular file,
you need to know the inode number of the file. (e.g. by ls -i command)
Also, if you want to inject a fault by accessing particular
LBA(Logical Block Address) of the target SCSI device, instead
of accessing a file, you need to choose the LBA number.
NOTE. The file, with which you want to inject a fault may be cached
on memory, but an access to the target device needs to be generated
to inject a fault. For confirmation, you should drop all page
caches by "echo 1 > /proc/sys/vm/drop_caches".
3. Run the stap command
Run a script to setup a SystemTap hook to cause a fault. Refer to section
"6. Scripts usage details" for more details.
By running a script, the hook for injecting a fault is made.
If SystemTap shows the string "Pass 5: starting run.", it's ready.
4. Access the target device
Running the SystemTap script is just only to set a trap. So an access to the target
device is needed to inject the fault.
e.g. If you run "disk_rerr.stp" script, you need to read from the target device
You can make a read access by a cat command.
If you run "temporary_werr.stp you need to write to the target device.
You can make a write access by redirection to the file and a sync command.
5. Check the result
Check the result caused by the fault injection and stop the script.
If the fault is successfully injected, the script prints the message
beginning with "scsi_decide_disposition" or "scsi_add_timer" depending
on the fault type.
The "read/write error response" type of fault scripts (disk_rwerr.stp,
disk_rerr.stp, sector_rerr.stp, temporary_rerr.stp, temporary_werr.stp)
will show "scsi_decide_disposition ..." on successfully injected faults.
The "no response" type of fault scripts (r_timeout.stp, w_timeout.stp, rw_timeout.stp)
will show "scsi_add_timer ..." on successfully injected faults
The running SystemTap script can't stop automatically. So you need to
stop that by "Ctrl-C" or "pkill stap" command.
5. Scripts usage details
========================
SYNOPSIS
stap <script name> major minor-min minor-max inode/LBAflag value -g -I <lib directory>
DESCRIPTION
Set up a hook to inject a given type of fault by using SystemTap
Parameters as follows.
script name: Particular type of fault
major: Major number of the target device. Generally, 8 for HDD.
minor-min: Minimum minor number of the target device candidate.
minor-max: Maximum minor number of the target device candidate.
inode/LBA flag : 0 or 1. 0 -"value" means LBA, 1- "value" means inode
value: inode or LBA value depends on "inode/LBA flag"
Note. LBA is not equal to a "block number" managed by a filesystem.
Block number is a logical location, but LBA is a physical
location in the device.
lib directory: Path to the common routine directory.
You need to choose one of the following, depending on your environment.
fault_injection_common_scsi_RAID56 - target is SCSI disk using md RAID4/5/6
fault_injection_common_scsi - target is SCSI disk otherwise
fault_injection_common_sata_RAID56 - target is SATA disk using md RAID4/5/6
fault_injection_common_sata - target is SATA disk otherwise
Rules about the range of target device:
As described in the introduction, this tool is originally for a software RAID evaluation.
Target device should be given as a range because the software raid does load balancing,
and the next access can't be predicted precisely.
1. Target device candidates can be given by the range from minor-min to minor-max of the given
major number. The minor numbers are used to distinguish physical disk.
The first access within the range after SystemTap hook is ready,
if the destination condition (given by LBA or inode number) met, the accessed
device will be the target device and the accessed sector will be the target sector.
The target sector and the target device never change until stopping the script.
e.g. When (major, minor-min, minor-max) = (8, 0, 49), sda, sdb, sdc, sdd will
be target candidates.
2. Once the target device and the target sector is determined, fault injection
will only occur on access to the target sector of the target device.
This means that the "permanent" type of faults can only be injected repeatedly
on access to the target sector of the target device. Other accesses complete
successfully.
3. If you need to designate a particular target, you should give the same value to
minor-min and minor-max.
4. As described earlier, the minor-min and minor-max are used to distinguish physical disk.
Each scripts can't distinguish disk partition by minor number in the same disk.
e.g. If (minor-max, minor-min) = (17, 18) given, it means that the target device
will be /dev/sdb.
EXAMPLES
1. A temporary read error simulation
#stap ./temporary_rerr.stp 8 1 17 1 5000747 -g -I ./fault_injection_common_sata/
-> Set a temporary read error simulation hook to SCSI HDD(8) /dev/sda(1) or /dev/sdb(17).
A read access to the file with inode 5000747 on sda or sdb will cause a read error.
2. A read error correctable by write simulation
#stap ./sector_rerr.stp 8 33 49 0 40496541 -g -I ./fault_injection_common_scsi/
-> Set a correctable read error simulation hook to SCSI HDD(8) /dev/sdc(33) or /dev/sdd(49).
This case the target sector is given by (0, 40496541), means LBA 40496541.
A read access to one of these disks of LBA=40496541 will cause a read error.
After read error injected to the LAB=40496541, a write access to the LAB=40496541
assumed to fix the error. Then, following any access to LBA=40496541 will
complete successfully.
6. Known Issues and limitations
================================
- The read and write to inject a fault should be a "mapped" access.
Direct I/O is not supported yet.
- Currently run on x86 architecture.
===============================================================================
Appendix A.
Preparing for using systemtap on Fedora8
------------------------------------------
1. Decompress kernel and move them to an arbitrary location.
# tar jxf linux-2.6.23.12.tar.bz2
2. Some setting to compile the kernel
At least, the following configuration is needed.
Loadable module support
Enable loadable module support [Y]
Module unloading [Y]
General setup
Kernel->user space relay support [Y]
Instrumentation Support
Kprobes [Y]
Kernel hacking
Kernel debugging [Y]
Compile the kernel with debug info [Y]
But you can reuse Fedora's config file.
# cd linux-2.6.23.12
# make mrproper
# cp /boot/config-2.6.23.1-42.fc8 .config
# make oldconfig
# make prepare
3. Build the kernel
# make -j 4 bzImage
# make -j 4 modules
4. Install the kernel
# make modules_install
# make install
5. Create symlinks for SystemTap
# mkdir -p /usr/lib/debug/lib/modules/2.6.23.12/
# ln -s /lib/modules/2.6.23.12/kernel /usr/lib/debug/lib/modules/2.6.23.12/
# ln -s /lib/modules/2.6.23.12/build/vmlinux /usr/lib/debug/lib/modules/2.6.23.12/
6. reboot the system to boot the new kernel.
johnarok/scsi_fault_injection_test_tool
Fault injection test tool, forked from http://sourceforge.net/projects/scsifaultinjtst/ with support for 3.2 kernels
ShellNOASSERTION