# Illinois Open Source License # # University of Illinois/NCSA # Open Source License # # Copyright © 2009, University of Illinois. All rights reserved. # # Developed by: # # Innovative Systems Lab # National Center for Supercomputing Applications # http://www.ncsa.uiuc.edu/AboutUs/Directorates/ISL.html # # Permission is hereby granted, free of charge, to any person obtaining a copy of # this software and associated documentation files (the "Software"), to deal with # the Software without restriction, including without limitation the rights to use, # copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the # Software, and to permit persons to whom the Software is furnished to do so, subject # to the following conditions: # # * Redistributions of source code must retain the above copyright notice, this list # of conditions and the following disclaimers. # # * Redistributions in binary form must reproduce the above copyright notice, this list # of conditions and the following disclaimers in the documentation and/or other materials # provided with the distribution. # # * Neither the names of the Innovative Systems Lab, the National Center for Supercomputing # Applications, nor the names of its contributors may be used to endorse or promote products # derived from this Software without specific prior written permission. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR # PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE CONTRIBUTORS OR COPYRIGHT HOLDERS BE # LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT # OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER # DEALINGS WITH THE SOFTWARE. # This softare tests GPU memory for hardware errors and soft errors using CUDA or OpenCL =======================================Compile and run ================================================== (I) Compile The default cuda install path is /usr/local/cuda. If your install path is not the default, you will need to set CUDA_INSTALL_PATH to the correct path. Simply typing make should compile the program cuda_memtest %make If your GPUs does not have cuda capability 1.3, compile it with 1.0 %make cuda_memtest_sm10 Ignore the warning messsages "Advisory: Cannot tell what pointer points to, assuming global memory space" We've added the opencl implementation for the memory test that works on both Nvidia and AMD GPUs. To compile the opencl memtest, open the Makefile and edit the opencl install path, then %make ocl_memtest Run it with --help for usage information. (II) Run %cuda_memtest The default behavior is running the test on all the GPUs available infinitely. There are options to change the default behavior. %cuda_memtest --disable_all --enable_test 10 %cuda_memtest --stress This runs test 10 (the stress test). "--stress" is equivalent to "--disable_all --enable_test 10 --exit_on_error" %cuda_memtest --stress --num_iterations 100 --num_passes 1 This one does a quick sanity check for GPUs with a short run of test 10. More on this later. See help message by %cuda_memtest --help (III) Sanity check There is a simple script sanity_check.sh in the directory. This script does a quick check if one GPU or all GPUs are in bad health. Example usage: ./sanity_check.sh 0 //check GPU 0 ./sanity_check.sh 1 //check GPU 1 ./sanity_check.sh //check All GPUs in the system (IV) Known issue * If your machine is cuda 2.2, killing the program while it is running test 10 (the memory stress test) could result in your GPUs in bad state. This is a bug from the nvidia driver. A detailed description can be found in http://forums.nvidia.com/index.php?showtopic=97379. We have filed a bug report to nvidia. Rebooting or reloading the nvidia driver will put the GPUs back to clean state. ========================================Test descriptions ================================================ (I) list of all tests Running %cuda_memtest --list_tests will print out all tests and their short descriptions, as of 6/18/2009, we implemented 11 tests Test0 [Walking 1 bit] Test1 [Own address test] Test2 [Moving inversions, ones&zeros] Test3 [Moving inversions, 8 bit pat] Test4 [Moving inversions, random pattern] Test5 [Block move, 64 moves] Test6 [Moving inversions, 32 bit pat] Test7 [Random number sequence] Test8 [Modulo 20, random pattern] Test9 [Bit fade test] ==disabled by default== Test10 [Memory stress test] (II) The general algorithm First a kernel is launched to write a pattern. Then we exit the kernel so that the memory can be flushed. Then we start a new kernel to read and check if the value matches the pattern. An error is recorded if it does not match for each memory location. In the same kernel, the compliment of the pattern is written after the checking. The third kernel is launched to read the value again and checks against the compliment of the pattern. (III) detailed description Test0 [Walking 1 bit] This test changes one bit a time in memory address to see it goes to a different memory location. It is designed to test the address wires. Test1 [Own address test] Each Memory location is filled with its own address. The next kernel checks if the value in each memory location still agrees with the address. Test 2 [Moving inversions, ones&zeros] This test uses the moving inversions algorithm with patterns of all ones and zeros. Test 3 [Moving inversions, 8 bit pat] This is the same as test 1 but uses a 8 bit wide pattern of "walking" ones and zeros. This test will better detect subtle errors in "wide" memory chips. Test 4 [Moving inversions, random pattern] Test 4 uses the same algorithm as test 1 but the data pattern is a random number and it's complement. This test is particularly effective in finding difficult to detect data sensitive errors. The random number sequence is different with each pass so multiple passes increase effectiveness. Test 5 [Block move, 64 moves] This test stresses memory by moving block memories. Memory is initialized with shifting patterns that are inverted every 8 bytes. Then blocks of memory are moved around. After the moves are completed the data patterns are checked. Because the data is checked only after the memory moves are completed it is not possible to know where the error occurred. The addresses reported are only for where the bad pattern was found. Test 6 [Moving inversions, 32 bit pat] This is a variation of the moving inversions algorithm that shifts the data pattern left one bit for each successive address. The starting bit position is shifted left for each pass. To use all possible data patterns 32 passes are required. This test is quite effective at detecting data sensitive errors but the execution time is long. Test 7 [Random number sequence] This test writes a series of random numbers into memory. A block (1 MB) of memory is initialized with random patterns. These patterns and their complements are used in moving inversions test with rest of memory. Test 8 [Modulo 20, random pattern] A random pattern is generated. This pattern is used to set every 20th memory location in memory. The rest of the memory location is set to the complimemnt of the pattern. Repeat this for 20 times and each time the memory location to set the pattern is shifted right. Test 9 [Bit fade test, 90 min, 2 patterns] The bit fade test initializes all of memory with a pattern and then sleeps for 90 minutes. Then memory is examined to see if any memory bits have changed. All ones and all zero patterns are used. This test takes 3 hours to complete. The Bit Fade test is disabled by default Test10 [memory stress test] Stress memory as much as we can. A random pattern is generated and a kernel of large grid size and block size is launched to set all memory to the pattern. A new read and write kernel is launched immediately after the previous write kernel to check if there is any errors in memory and set the memory to the compliment. This process is repeated for 1000 times for one pattern. The kernel is written as to achieve the maximum bandwidth between the global memory and GPU. This will increase the chance of catching software error. In practice, we found this test quite useful to flush hardware errors as well.