Introduction --------------- The load automation framework is designed to allow flexibility and stability to traditional load testing strategies. This utility will allow you to dynamically configure your load testing environment. The framework is made up of 3 parts. Driver Utility: Dynamically sets up your load environment, configuration and tests. Load Testing API: ruby api to develop your load tests with Load Application: the application that runs the load and generates the statistics, currently Tsung Setup --------------- To effectively run the load testing framework you'll need at least 2 machines, though more are supported. Your driver machine is the machine you will be running the Driver Utility on. This machine can also be one of your client machines (described below). Your client machine is the machine where your load is generated. You can add multiple machines to your client pool to distribute load generation. Your server or System Under Test (SUT) is the machine running the web application you wish to test. This should not be a client as well, otherwise your statistics will be significantly flawed. Driver --------------- Unix/OSX machine Requires Erlang v5.6.1+ Requires Tsung v1.3.1+ (see dependencies) Requires Ruby v1.8.6+ rdoc required for documentation Requires Perl v5.8.8+ Template.pm module required to generate html statistics Checkout code: Testing Framework Builds Environment variables: DRB_PORT=9000 Client --------------- Unix/OSX machine Requires Erlang v5.6.1+ Requires Tsung v1.4.1+ (see dependencies) User running Tsung needs ssh access to all clients with no passphrase If running into errors starting remote client see Tsung FAQ for possible solutions Server --------------- Requires web server running application you wish to test. Port should be accessible by external machines. Directory Structure Kuali-Sakai-Performance-Test-Automation-Framework/ config/ (all configuration files for the framework) config.yaml import/ (data files for dynamic attributes) users.csv (list of username/password pairs) users.format (format for username/user_password) tests (the default location for .xml files created when executing driver.rb) doc/ (documentation generated by rdoc) lib/ (contains all API libraries) [Products] (contains API libraries for supported products) log/ (default location for log files and xml generated by drivery utility) suites/ (suite files the load framework has the option to run) [Products] tests/ (contains all tests that the suite files can call) [Products] driver.rb (Driver Utility) Config --------------- Inside the config directory you will find the following files: - config.yaml The config.yaml file consists of 3 distinct sections: - The Products we currently support section contains the products that are currently supported. Lines starting with # are ignored. Example: # Format: "product reference": "directory name in framework" ks: 'ks' oae: 'oae' - The Directory listing for test data in use in the tests section is currently only partially used. It contains information that is extracted from the file for use in the tests. Lines starting with # are ignored. - The Agent List section of the config.yaml file contains all the browser agents available for use in your load configuration. The driver utility will read this file. You can add whatever agents you like in this file. Lines starting with # are ignored. Example: # IE 7.0 - XP Mozilla/5.0 (Windows; MSIE 7.0; Windows NT 6.0; en-US) # IE 6.0 - Vista Mozilla/4.0 (Windows; MSIE 6.0; Windows NT 6.0) # FireFox 3.5.7 - Vista Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) # FireFox 3.5.6 - OSX 10.6 (Snow Leopard) Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Doc --------------- After you have checked out the code/tag you can generate this documentation by running 'rdoc'. This folder and all the documentation for the entire framework will be generated for you. You can open index.html inside of the doc folder to navigate to all documentation. Log --------------- This is the default location for all log and xml files generated by the Driver Utility. You can specify your own output locations through the Driver Utility options. Suites --------------- This folder contains all the suite files available for the framework to run. You can add as many suite files as you want here, likewise you can give them any name that makes sense for you. The format of the suite files is: test name,probability Each suite can have multiple tests, but the probability for all tests must total 100. This gives you flexibility in how you want your sessions to be executed. Example: authentication.rb,50 create_proposal.rb,25 approve_proposal.rb,25 The test name must refer to the exact filename of a test residing in the tests directory. driver.rb --------------- This is the Driver Utility, more on this below. Driver Utility --------------- Usage Usage: driver.rb [OPTIONS] -h, --help Display help screen -c, --config FILE path to xml config file for everything after <tsung> and before <sessions> : NOT CURRENTLY SUPPORTED -p, --product PRODUCT supported products: ["ks", "oae"] -s, --suite SUITE suite file -o, --output FILE path to xml output --contexts PRIMARY_CONTEXT,SECONDARY_CONTEXT contexts for both primary and secondary servers -l, --log FILE path to log output from this framework -x, --execute start the load after generating the XML -v, --verbose dumps http traffic from tsung --tsung_log_level LEVEL sets the log level for tsung. Available levels: emergency, critical, error, warning, notice (default), info, debug -d, --debug enable debug logging -t, --thinktime enable thinktime sleeps Interactive Shell --------------- When you initiate the Driver Utility you will enter the interactive shell to setup your configuration. There are specific examples in the Kuali-Sakai-Performance-Test-Automation-Framework/ directory. The following is a generic example showing the interactive session: ./driver.rb Your log file is here: /../Kuali-Sakai-Performance-Test-Automation-Framework/log/1318879134.log Let's setup your test run config... # Select the product that you are testing What product are we testing? [select from product listing] What suite are we testing? #Enter the name of the suite you want to test #the name of the suite that you choose must exist in the suites/[product] directory What suite are we testing? test_suite What do you want to name your output xml file? #Name your output xml file What do you want to name your output xml file? demo.xml Your xml file will be here: /../Kuali-Sakai-Performance-Test-Automation-Framework/config/tests/demo.xml #Specify your clients. Must be comma separated What clients are you driving your tests from? [hostname1,hostname2,...] localhost #Specify the server host you want to test What primary servers do you want to test? [hostname1:port,hostname2:port,...] server.com:80 #Specify any secondary web application host Secondary servers? [reference_name1:hostname1:port,reference_name2:hostname2:port,...] reference_for_this_server_that_is_used_in_the_api:secondary_host.com:80 #Specify the primary server context What is the primary server context? context_for_primary_server #Specify the secondary server context if there is a secondary server What is the secondary server context? context_for_secondary_server Let's setup your test scenario or phases, you can define multiple phases Phase 1: How many minutes? #A load test is made up of one or more test phases. The total duration of your load test will be the total duration of all of your phases. Specify how many minutes you want your first (and possibly only) test phase to be. You will have the option of adding more phases shortly. Phase 1: How many minutes? 5 At what interval should users be created at this phase? (x seconds per user) #For this phase specify at what rate user sessions should be created. Any float value will do. To generate 1 user a second you'd specify 1, to generate 5 users every second you'd specify 0.2 At what interval should users be created at this phase? (x seconds per user) 0.2 Do you want to add another phase? [y/n] #Specify whether you want to add another phase or not. If y you'll be prompted with same steps as in phase 1. Do you want to add another phase? [y/n] n Let's setup what user agents you'd like to test with Here are the agents you can specify... 1: Mozilla/5.0 (Windows; MSIE 7.0; Windows NT 6.0; en-US) 2: Mozilla/4.0 (Windows; MSIE 6.0; Windows NT 6.0) 3: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) 4: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Specify the number of the agent you'd like to use: #Agent list is read in from the Agents List section of the config.yaml file in the config directory. Choose which agent you'd like to use. You'll be able to specify multiple agents and #probability of their usage. Specify the number of the agent you'd like to use: 1 Specify the probability that this agent will be used: [1-100] #Specify the probability this agent will be used. Any value less than 100 will prompt you to add another agent. This will repeat until your total probability equals 100. Specify the probability that this agent will be used: [1-100] 100 Do you want to insert user thinktime? [y/n] #Thinktime causes the test to insert "sleeps" that simulate a real life user session. Do you want to insert user thinktime? [y/n] y Do you want to execute tsung now? [y/n] #Specify whether you want to run tsung now or not. Do you want to execute tsung now? [y/n] y Do you want to dump http traffic from tsung? [y/n] #Specify whether or not you want to dump http traffic. Only select yes if this is a debug run. This will greatly reduce performance. Do you want to dump http traffic from tsung? [y/n] n Test: login.rb Probability: 100 login.rb: Logging in as admin/admin login.rb: Logging out Starting tsung with command [tsung -f /Users/test/work/Kuali-Sakai-Performance-Test-Automation-Framework/config/tests/demo.xml start] Starting Tsung "Log directory is: /Users/test/.tsung/log/20111017-19:55" Statistics --------------- Tsung Logs --------------- In the log directory that was printed to stdout you will see the following files: % ls kctest.xml tsung.log match.log tsung_controller@KC-MBP.log kctest.xml: is the xml file used for the test. This is the same file as generated by the Driver Utility match.log: contains logging info for match expressions inside of your test. If none are used then nothing will be here. tsung.log: contains all the http statistics for the load run tsung_controller[KULSTG:...].log: Contains logging info from the client controllers. Generating Tsung HTML report --------------- Tsung comes with a perl utility for generating html reports with graphing, the utility is called tsung_stats.pl and should be in the lib directory where you've installed Tsung. For example if you have Tsung installed in /opt/local/bin/tsung, then the utility will be in /opt/local/lib/tsung/bin/tsung_stats.pl. To run just execute cd into your log directory and execute 'tsung_stats.pl' % tsung_stats.pl creating subdirectory data creating subdirectory images creating subdirectory gnuplot_scripts warn, last interval (1) not equal to the first, use the first one (10) No data for Match No data for Event No data for Async No data for Errors The script will generate 2 html files, both link to each other internally so just open one of them. report.html graph.html You can get a detailed description of all stats here. Tsung HTML Report Screenshots --------------- System Resource Monitoring --------------- Please refer to the System Resource Monitoring project located at the following location in github: https://github.com/kcampos/Open-Synchronized-System-Resource-Monitoring.git --------------- FAQ --------------- 1. If you see the following error when invoking driver.rb, then add a "log" directory in the Kuali-Sakai-Performance-Test-Automation-Framework directory: Kuali-Sakai-Performance-Test-Automation-Framework/lib/tsung-api.rb:277:in `initialize': No such file or directory - /Users/test/work/Kuali-Sakai-Performance-Test-Automation-Framework/log/1318877356.log (Errno::ENOENT) License --------------- Copyright 2012 rSmart Licensed under the Educational Community License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.osedu.org/licenses/ECL-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
kcampos/Open-Performance-Automation-Framework
Performance test automation framework designed to test the Kuali and Sakai product family.
RubyNOASSERTION