walterchris/fwts

README for the FirmWare Test Suite (fwts)
=========================================

Quick Start Guide
=================

1) Dependencies:

Ubuntu Distro x86_64
	autoconf automake libglib2.0-dev libtool libpcre3-dev
	flex bison dkms libfdt-dev libbsd-dev

Ubuntu Distro ppc64el
	autoconf automake libglib2.0-dev libtool libpcre3-dev
	flex bison dkms libfdt-dev device-tree-compiler libpci-dev libbsd-dev

RHEL Distro ppc64el
	autoconf automake kernel-devel libtool flex flex-devel bison dkms
	libfdt libfdt-devel dtc pcre-devel pcre2 pcre2-devel
	pcre2-utf16 pcre2-utf32 glib2 glib2-devel pciutils pciutils-devel zlib-devel
	make libbsd-devel

Ubuntu Distro ARM64
	autoconf automake libglib2.0-dev libtool libpcre3-dev
	flex bison dkms libfdt-dev libbsd-dev

2) Kernel configuration

Some tests depend on special Linux kernel configuration settings. These include:

* CONFIG_CGROUP_FREEZER=y
* CONFIG_DMI_SYSFS=y
* CONFIG_EFI_TEST=m

3) To build and install (only if building from source)

	autoreconf -ivf
	./configure
	make
	sudo make install

To get help:

	fwts --help

To run all the default tests, use:

	sudo fwts

..this will dump the results into results.log

To see the available tests, use:

	fwts --show-tests

Advanced Options Guide
======================

Running Specific Tests
----------------------

Run specific tests, e.g. check syntax of DSDT:

	sudo fwts syntaxcheck

You can run multiple tests too:

	sudo fwts syntaxcheck fan battery

To see all the available tests:

	sudo fwts --show-tests-full

Changing the output
-------------------

To dump the results to another file:

	sudo fwts --results-output=myresults.log

or the short form:

	sudo fwts -r myresults.log

or to stderr or stdout and redirect:

	sudo fwts -r stderr
	sudo fwts -r stdout > mylog.log

or a shorter form for stdout:

	sudo fwts - > mylog.log

If the output is written to stdout, the width of the log is based on the
width of the tty.  The default width when writing to a log file may be
changed to N characters wide using -w N or --log-width=N, e.g.

	sudo fwts --log-width=90 - > mylog.log

The log has various columns that can be enabled, here is an example:

00012 13/12/12 12:55:04 wakealarm      Test ACPI Wakealarm.
00013 13/12/12 12:55:04 wakealarm      -------------------------------------------------------------
00014 13/12/12 12:55:04 wakealarm      Test 1 of 4: Check existence of /sys/class/rtc/rtc0
00015 13/12/12 12:55:04 wakealarm      /wakealarm.
00016 13/12/12 12:55:04 wakealarm      PASSED: Test 1, /sys/class/rtc/rtc0/wakealarm found.
00017 13/12/12 12:55:04 wakealarm      
00018 13/12/12 12:55:04 wakealarm      Test 2 of 4: Trigger wakealarm for 1 seconds in the future.
00019 13/12/12 12:55:04 wakealarm      Trigger wakealarm for 1 seconds in the future.
00020 13/12/12 12:55:04 wakealarm      PASSED: Test 2, RTC wakealarm was triggered successfully.
00021 13/12/12 12:55:04 wakealarm      
00022 13/12/12 12:55:04 wakealarm      Test 3 of 4: Check if wakealarm is fired.
00023 13/12/12 12:55:07 wakealarm      PASSED: Test 3, RTC wakealarm triggered and fired
00024 13/12/12 12:55:07 wakealarm      successfully.
00025 13/12/12 12:55:07 wakealarm      
00026 13/12/12 12:55:07 wakealarm      Test 4 of 4: Multiple wakealarm firing tests.
00027 13/12/12 12:55:07 wakealarm      Trigger wakealarm for 1 seconds in the future.
00028 13/12/12 12:55:09 wakealarm      Trigger wakealarm for 2 seconds in the future.
00029 13/12/12 12:55:12 wakealarm      Trigger wakealarm for 3 seconds in the future.
00030 13/12/12 12:55:16 wakealarm      Trigger wakealarm for 4 seconds in the future.
00031 13/12/12 12:55:21 wakealarm      PASSED: Test 4, RTC wakealarm triggered and fired
00032 13/12/12 12:55:21 wakealarm      successfully.
00033 13/12/12 12:55:21 wakealarm      
00034 13/12/12 12:55:21 wakealarm      =============================================================
00035 13/12/12 12:55:21 wakealarm      4 passed, 0 failed, 0 warnings, 0 aborted, 0 skipped, 0 info
00036 13/12/12 12:55:21 wakealarm      only.
00037 13/12/12 12:55:21 wakealarm      ============================================================
^     ^        ^        ^
|     |        |        |
|     |        |        +--- %owner (name of test that ran)
|     |        |
|     |        +--- %time
|     |
|     +--- %date 
|
+--- %line (line number)

Also, there is an optional %field that indicates the type of message being logged:

	sudo fwts wakealarm --log-format="%line %date %time %field %owner"

..this generates results.log containing:

00013 13/12/12 12:53:11 HED wakealarm      Test ACPI Wakealarm.
00014 13/12/12 12:53:11 SEP wakealarm      ---------------------------------------------------------
00015 13/12/12 12:53:11 INF wakealarm      Test 1 of 4: Check existence of /sys/class/rtc/rtc0
00016 13/12/12 12:53:11 INF wakealarm      /wakealarm.
00017 13/12/12 12:53:11 PAS wakealarm      PASSED: Test 1, /sys/class/rtc/rtc0/wakealarm found.
00018 13/12/12 12:53:11 NLN wakealarm      
00019 13/12/12 12:53:11 INF wakealarm      Test 2 of 4: Trigger wakealarm for 1 seconds in the
00020 13/12/12 12:53:11 INF wakealarm      future.
00021 13/12/12 12:53:11 INF wakealarm      Trigger wakealarm for 1 seconds in the future.
00022 13/12/12 12:53:11 PAS wakealarm      PASSED: Test 2, RTC wakealarm was triggered successfully.
00023 13/12/12 12:53:11 NLN wakealarm      
00024 13/12/12 12:53:11 INF wakealarm      Test 3 of 4: Check if wakealarm is fired.
00025 13/12/12 12:53:14 PAS wakealarm      PASSED: Test 3, RTC wakealarm triggered and fired
00026 13/12/12 12:53:14 PAS wakealarm      successfully.
00027 13/12/12 12:53:14 NLN wakealarm      
00028 13/12/12 12:53:14 INF wakealarm      Test 4 of 4: Multiple wakealarm firing tests.
00029 13/12/12 12:53:14 INF wakealarm      Trigger wakealarm for 1 seconds in the future.
00030 13/12/12 12:53:16 INF wakealarm      Trigger wakealarm for 2 seconds in the future.
00031 13/12/12 12:53:19 INF wakealarm      Trigger wakealarm for 3 seconds in the future.
00032 13/12/12 12:53:23 INF wakealarm      Trigger wakealarm for 4 seconds in the future.
00033 13/12/12 12:53:28 PAS wakealarm      PASSED: Test 4, RTC wakealarm triggered and fired
00034 13/12/12 12:53:28 PAS wakealarm      successfully.
00035 13/12/12 12:53:28 NLN wakealarm      
00036 13/12/12 12:53:28 SEP wakealarm      =========================================================
00037 13/12/12 12:53:28 SUM wakealarm      4 passed, 0 failed, 0 warnings, 0 aborted, 0 skipped, 0
00038 13/12/12 12:53:28 SUM wakealarm      info only.
00039 13/12/12 12:53:28 SEP wakealarm      =========================================================
                         ^
                         |
 +-----------------------+
 |
RES = result (PASSED or FAILED)
ERR = error (something fatal happened!)
WRN = warning (not a critical failure, but may need looking at)
DBG = debug output (deprecated)
INF = information fields
SUM = summary
NLN = new line (empty field)
ADV = advice information
HED = heading
TAG = tag information
SEP = separator (just pretty printing)
PAS = Test Passed
FAL = Test Failed
SKP = Test Skipped
ABT = Test Aborted
INO = Information only 

use:
	fwts --log-fields to show all available fields

One can filter on these fields using --log-filter, e.g.

	sudo fwts --log-filter=RES,SUM  

.. just collect results and summary of tests

	sudo fwts --log-filter=ALL,~SEP

.. all fields except the separators

One can also re-arrange or modify the log format using --log-format, e.g.

	sudo fwts --log-format="%line %field %time: " mtrr -

this produces:

00011 SEP 12:13:07: --------------------------------------------------------------------------------
00012 INF 12:13:07: MTRR overview
00013 INF 12:13:07: -------------
00014 INF 12:13:07: Reg 0: 0x00000000ffc00000 - 0x0000000100000000 (     4 MB)   Write-Protect
00015 INF 12:13:07: Reg 1: 0x0000000000000000 - 0x0000000080000000 (  2048 MB)   Write-Back
00016 INF 12:13:07: Reg 2: 0x0000000080000000 - 0x00000000c0000000 (  1024 MB)   Write-Back
00017 INF 12:13:07: Reg 3: 0x00000000c0000000 - 0x00000000e0000000 (   512 MB)   Write-Back
00018 INF 12:13:07: Reg 4: 0x00000000dc000000 - 0x00000000e0000000 (    64 MB)   Uncached
00019 INF 12:13:07: Reg 5: 0x00000000db000000 - 0x00000000dc000000 (    16 MB)   Uncached
00020 INF 12:13:07: Reg 6: 0x0000000100000000 - 0x0000000180000000 (  2048 MB)   Write-Back
00021 INF 12:13:07: Reg 7: 0x0000000180000000 - 0x00000001a0000000 (   512 MB)   Write-Back
00022 INF 12:13:07: Reg 8: 0x000000019f000000 - 0x00000001a0000000 (    16 MB)   Uncached
00023 INF 12:13:07: Reg 9: 0x000000019e800000 - 0x000000019f000000 (     8 MB)   Uncached
00024 NLN 12:13:07: 
00025 INF 12:13:07: Test 1 of 3: Validate the kernel MTRR IOMEM setup.
00026 PAS 12:13:07: PASSED: Test 1, Memory ranges seem to have correct attributes.
00027 NLN 12:13:07: 
00028 INF 12:13:07: Test 2 of 3: Validate the MTRR setup across all processors.
00029 PAS 12:13:07: PASSED: Test 2, All processors have the a consistent MTRR setup.
00030 NLN 12:13:07: 
00031 INF 12:13:07: Test 3 of 3: Check for AMD MtrrFixDramModEn being cleared by the BIOS.
00032 SKP 12:13:07: SKIPPED: Test 3, CPU is not an AMD, cannot test.
00033 NLN 12:13:07: 
00034 SEP 12:13:07: ================================================================================
00035 SUM 12:13:07: 2 passed, 0 failed, 0 warnings, 0 aborted, 1 skipped, 0 info only.
00036 SEP 12:13:07: ================================================================================

Available fields are:

%line   - log line number
%time	- current time
%date 	- current date
%owner  - name of the test program (the owner of the the current test)
%field  - the 3 letter record identifier field (e.g. SEP, RES, etc..)

Test Progress Feedback
======================

Use the --show-progress to dump out the test progress to stderr. Output is as follows:

Test: Test ACPI Wakealarm.                                                  
  Check existence of /sys/class/rtc/rtc0/wakealarm.       1 passed                                 
  Trigger wakealarm for 1 seconds in the future.          1 passed                                 
  Check if wakealarm is fired.                            1 passed                                 
  Multiple wakealarm firing tests.                        1 passed   

PASSED or FAILED?
=================

The --stdout-summary flag outputs to stdout PASSED or FAILED if test(s) passed or failed. It's essentially
for one-shot test modes where you want to run one test and see if it passed or failed without having to
look at the results log, e.g.

sudo fwts mtrr --stdout-summary

The exit code returns 1 for test failure and 0 for test success, so the
test suite can be run from a shell script and one can detect if the test worked or not, e.g.

#!/bin/bash
LOG=stdout
sudo fwts dmi_decode --results-output=$LOG
failed=$?

Resources
=========

fwts git repository:
	git://kernel.ubuntu.com/hwe/fwts.git
	https://kernel.ubuntu.com/git/hwe/fwts.git/

fwts reference guide:
	https://wiki.ubuntu.com/FirmwareTestSuite/Reference

fwts LiveCD:
	http://fwts.ubuntu.com/fwts-live/

fwts-devel mailing list:
	fwts-devel@lists.ubuntu.com
	https://lists.ubuntu.com/mailman/listinfo/fwts-devel