PFTT is a test tool for the PHP ecosystem designed for convenience, thoroughness and speed. It supports PHPUnit tests for applications and frameworks, WebDriver based UI Testing for User Interfaces and PHPT tests for the PHP Core interpreter. It can test multiple scenarios including running PHPT and PhpUnit tests on real web servers like Apache to actually test what will happen in production.
- Download binary from http://131.107.220.66/PFTT-Results/PFTT/current/ and run it.
- The Install Wizard will install PFTT to your local system. a. If you have installed PFTT before, you MUST delete c:\php-sdk\PFTT\current so that it does NOT exist before running the installer b. If you set the environment variable PHP_SDK, PFTT will use that place of C:\php-sdk (including for c:\php-sdk\PFTT\current) i. You can create a batch script named \PFTT\current\bin\internal\set_php_sdk.cmd where you can SET PHP_SDK, which will be used whenever you run PFTT instead of the default PHP_SDK value
- If you want to test with Apache, currently you’ll need to manually decompress/copy it a. Look in c:\php-sdk\PFTT\current\cache\dep\Apache (Windows) b. Each of the .7z files is a different Apache build (built by apachelounge.com) c. You’ll need to decompress them to c:<Apache Build> so you’ll have i. C:\Apache244-VC11-OpenSSL1.0.1e-x64\bin\httpd.exe ii. C:\Apache244-VC11-OpenSSL1.0.1e-x86\bin\httpd.exe iii. C:\Apache244-VC9-OpenSSL0.9.8y-x86\bin\httpd.exe iv. C:\ Apache224-VC9-OpenSSL0.9.8y-x86\bin\httpd.exe d. PFTT will assume they’re there and fail if they aren’t
Windows:
Click on PFTT Shell in \php-sdk\PFTT\current\bin.
Add it to your taskbar or start menu for next time.
Linux:
> PFTT_HOME/bin/set_env
List releases (by build, branch, both, or all)
> release_list 5.3
> release_list NTS
> rl nts 5.3
> rl
Download specific release (test-pack and build)
> pftt rg MASTER TS rc165a0b
Run all PHPTs from the test-pack
> core_all C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567
Run all PHPTs from the test-pack on Apache
> core_all -c apache C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567
Run all PHPTs from the test-pack with Opcache on CLI and Apache
> core_all -c apache,cli,opcache C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567
Run all PHPTs from the test-pack with Opcache and with no code cache on CLI and Apache and PHP's builtin web server
> core_all -c apache,cli,builtin_web,opcache,no_code_cache C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567
Run all Symfony's PhpUnit tests on Apache with Opcache and with no code cache
> app_all -c symfony,apache,opcache,no_code_cache C:\php-sdk\php-5.4.7-win32-vc9-x86
Runs PHPTs only from list in SKIP_list_file (can add comments to lists with a ; or #)
> core_list C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567 SKIP_list_file
Runs PHPTs that contain 021 (name fragment) in their name
> core_named C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567 021
Opens a GUI to efficiently review large amounts of tests
> ca -gui C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567 pftt -gui -config config_file core_list C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567 pftt -gui -config config_file core_named C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567
Shows help info (additional commands, etc...)
> pftt -help
- Create a .groovy file in /conf/app
- Extend PhpUnitSourceTestPack
- implement #getSourceRoot
- find all the phpunit.dist.xml files in the test-pack
- implement #openAfterInstall
- (required) create corresponding PhpUnitDists by calling PhpUnitSourceTestPack#addPhpUnitDist and provide all the information from the phpunit.dist.xml file. the Javadoc on the PhpUnitDist methods explains which method matches which XML tag.
- provide files and directories to include to PhpUnitSourceTestPack (optional)
- check if any tests are non-thread-safe (NTS) and if so, add their file names or partial file names to the list returned by PhpUnitSourceTestPack#getNonThreadSafeTestFileNames To speed test running, making testing more convenient and thus done more frequently and thoroughly, test running is threaded, so multiple tests are run at the same time except for NTS tests.
- you may provide some additional info to PhpUnitSourceTestPack (optional; mainly, its just doing steps 5 and 6)
To re-run tests in exactly the same order, use -thread_count 1 console option with app_list or core_list commands.
- Create a list of the tests that are crashing or might be crashing
- Run with -debug_all and core_list
This will run all the tests in a Debugger (WinDebug for Windows and GDB for Linux). Do 1 first to reduce the number of tests to run since they'll run much slower in a debugger.
When testing with Apache, this will run the Debugger on Apache to help trace Apache related problems.
For tests that crash occasionally
run tests 100 times each, 32 threads, no thread safety = so multiple instances of a test may be run at the same time (depending on size of list/speed of test)
> cl –run_test_times_all 100 –no_thread_safety –thread_count 32 –c apache,opcache php-5.5 php-5.5.0beta4-nts-Win32-VC11-x64 php-test-pack-5.5.0beta4 list_of_crashing_tests.txt
Stress, running listed tests 1000 times and all other tests once each
> ca -run_test_times_list 1000 wincache_crash.txt -ignore_output -randomize_order -no_thread_safety -thread_count 192 -c wincacheu_file_and_user,opcache,cli -debug_pack php-debug-pack-5.5-nts-windows-vc11-x86-rdb61685 php-5.5-nts-windows-vc11-x86-rdb61685 php-test-pack-5.5-ts-windows-vc11-x86-rdb61685
On Windows, with the Debugging Tools installed, you can use Time-Travel-Tracing (TTT) to trace bugs
> ca -debugger ttt -run_test_times_list 1000 wincache_crash.txt -ignore_output -randomize_order -no_thread_safety -thread_count 192 -c wincacheu_file_and_user,opcache,cli -debug_pack php-debug-pack-5.5-nts-windows-vc11-x86-rdb61685 php-5.5-nts-windows-vc11-x86-rdb61685 php-test-pack-5.5-ts-windows-vc11-x86-rdb61685
These options may be useful with core_list or core_all or core_named:
-no_restart_all - will not restart any web server. this allows the web server to be stressed more and for longer amounts of time (this will invalidate any functional testing results!). -run_test_times_all - runs each test N times in a row/consecutively -run_test_times_list - runs tests in that list N times. if used with -run_test_times_all, tests not in list can be run different number of times from tests in list (ex: run listed tests 5 times, run all other tests 2 times). -debug_list - runs tests in list in Debugger -run_group_times_all - runs all groups of tests N times (in same order every time, unless -randomize used) -run_group_times_list - like grun_group_times_All and run_test_times_list (but for groups of tests) -no_thread_safety - runs tests in any thread, regardless of thread-safety. This can increase load/stress, but may lead to false FAILS/ERRORs, especially in file or database tests.
When a PHP (or Apache) process crashes, PFTT will try running it a second time to confirm that that specific test really caused a crash before reporting the test as CRASH.
Use taskkill /im:php.exe /f /t or taskkill /im:httpd.exe /f /t to kill off remaining php or apache processes
Check net use and use net use z: /delete to take care of remaining remote file systems
Run WinCacheU, with file-cache and user-cache, only file-cache, only user-cache and with opcache and with no code cache (all on CLI)
> ca -c wincacheu_file_and_user,wincacheu_only_user,wincacheu_only_file,opcache,no_code_cache,cli php-5.5.6RC1-nts-Win32-VC11-x86 php-test-pack-5.5.6RC1
Edit conf/internal_example/mssql11 and save it to conf/internal/mssql11 (also optionally mssql10). Then you can run this:
> ca -c mssql11,mssql10,cli php-5.5.6RC1-nts-Win32-VC11-x86 php_sqlsvr-test-pack-2013-09-03
Note: The MSSQL extension and its test-pack are not in PHP Core. They are currently in Microsoft's Codeplex.
Be sure to use -auto console option to signal that its being run automatically. This will supress blocking warnings and change some other settings to make sure that it runs without you having to click any buttons, etc...
On Windows, be sure to run PFTT under Admistrator privileges or you'll get a bunch of UAC popups that will block automation.
BSD-3 Clause. See LICENSE.txt