CITestAnalysis
Python scripts for crawling Apache CI test results, storing the results in MongoDB, and doing analysis.
usage: apacheCrawler.py [-h] --project PROJECT
Store Log info into db.
optional arguments: -h, --help show this help message and exit
--project PROJECT Name of the apache project listed on Jenkins (check https://builds.apache.org/view/All/)
usage: generateDBApache.py [-h] --input INPUT [--batch] --db DB
Store test result info into db.
optional arguments: -h, --help show this help message and exit
--input INPUT File that contains the test info
--batch Batch insert. Used only with empty database
--db DB Name of DB.
usage: analysis.py [-h] [--sortByFail] [--getDuration] [--getPassRate] [--getBuildQuality] [--prioritizeTests] [--support SUPPORT] [--checkNewTestPerf] [--checkNewTestPerfARIMA] [--perfTrend] [--testDeviation] [--testStepDeviation TESTSTEPDEVIATION] [--testDeviationEntropy] [--arimaModel] [--examineAllHistory] [--compare COMPARE [COMPARE ...]] [--windowSize WINDOWSIZE] [--dbName DBNAME] [--dst DST] [--test]
Analyze test logs.
optional arguments: -h, --help show this help message and exit
--sortByFail Sort the test cases by the number of failures.
--getDuration Get test execution time (in sec) across all tests.
--getPassRate Get test case pass rate for each build.
--getBuildQuality Get # of test pass/fail for each build.
--prioritizeTests Prioritize tests based on history. Tests that are likely to pass should be run first (they will be surprises if they fail).
--support SUPPORT Used for determining the support rate for prioritizing tests.
--checkNewTestPerf Check the performance of the newest test, using the execution time of prev 10 test runs (outlier removed). Returns the number of standard deviation away from previous runs.
--checkNewTestPerfARIMA Check the performance of the newest test using ARIMA model
--perfTrend Check the performance trend of each test.
--testDeviation Find tests with high deviations in execution time (unstable).
--testStepDeviation TESTSTEPDEVIATION Input takes in a test id. Find steps in a test with high deviations in execution time (unstable).
--testDeviationEntropy Find tests with high deviations in execution time (unstable).
--arimaModel Examine ARIMA model (p, i, q, and drift), and execution time distribution.
--examineAllHistory Use 10 ... n points to build n - 10 ARIMA models and look for outliers in all historical builds.
--windowSize WINDOWSIZE Number of most recent tests to consider when computing testDeviation
--dbName DBNAME Name of the database which stores the test results
--dst DST Output folder for storing the graphs generated by the tool.