/autobahn-testsuite

Automated WebSocket & WAMP protocol Testsuite

Primary LanguagePythonApache License 2.0Apache-2.0

Autobahn|Testsuite

The Autobahn|Testsuite provides a fully automated test suite to verify client and server implementations of The WebSocket Protocol for specification conformance and implementation robustness.

Autobahn|Testsuite also provides a couple of other tools useful for WebSocket (and WAMP) implementors and developers.

Reports

For some current reports on the test coverage of Autobahn|Python see

Test Suite Coverage

The test suite will check an implementation by doing basic WebSocket conversations, extensive protocol compliance verification and performance and limits testing.

Autobahn|Testsuite is used across the industry and contains over 500 test cases covering

  • Framing
  • Pings/Pongs
  • Reserved Bits
  • Opcodes
  • Fragmentation
  • UTF-8 Handling
  • Limits/Performance
  • Closing Handshake
  • Opening Handshake (under development)
  • WebSocket compression (permessage-deflate extension)

Other Tools

Besides the automated testsuite (aka "fuzzing" server/client), wstest also includes a number of other handy modes:

  • WebSocket echo server and client
  • WebSocket broadcast server (and client driver)
  • Testee modes to test AutobahnPython against the test suite
  • wsperf controller and master (see below for more)
  • WAMP server and client, for developing WAMP implementations
  • WebSocket Mass-Connect

How to install

The following recipe still works, but the new, recommended way is using a Docker toolchain image we provide. Please checkout this - much easier and repeatable.

The testsuite comes as a single command line tool, wstest. You will need Python 2 or PyPy (recommended).

Right now we only support Python 2 and Python 3 will not work. The testsuite is developed and tested on CPython 2 and PyPy. The latter is a high-performance Python implementation.

The recommended way to install wstest is into it's own, dedicated virtualenv.

On Debian/Ubuntu systems, you can install virtualenv like sudo apt-get install python-virtualenv.

Create a new virtualenv in your HOME and install Autobahn testsuite:

virtualenv ~/wstest
source ~/wstest/bin/activate
pip install autobahntestsuite

You will now have the wstest tool:

(wstest)oberstet@thinkpad-t430s:~$ which wstest
/home/oberstet/wstest/bin/wstest
(wstest)oberstet@thinkpad-t430s:~$ wstest -a
Autobahn 0.10.9
AutobahnTestSuite 0.7.4
(wstest)oberstet@thinkpad-t430s:~$ wstest --help
Usage: wstest [options]
Options:
  -d, --debug            Debug output [default: off].
  -a, --autobahnversion  Print version information for Autobahn and
                         AutobahnTestSuite.
  -m, --mode=            Test mode, one of: echoserver, echoclient,
                         broadcastclient, broadcastserver, fuzzingserver,
                         fuzzingclient, testeeserver, testeeclient, massconnect,
                         serializer [required]
  -t, --testset=         Run a test set from an import test spec.
  -s, --spec=            Test specification file [required in some modes].
  -o, --outfile=         Output filename for modes that generate testdata.
  -w, --wsuri=           WebSocket URI [required in some modes].
  -u, --webport=         Web port for running an embedded HTTP Web server;
                         defaults to 8080; set to 0 to disable. [optionally used
                         in some modes: fuzzingserver, echoserver,
                         broadcastserver, wsperfmaster]. [default: 8080]
  -i, --ident=           Testee client identifier [optional for client testees].
  -k, --key=             Server private key file for secure WebSocket (WSS)
                         [required in server modes for WSS].
  -c, --cert=            Server certificate file for secure WebSocket (WSS)
                         [required in server modes for WSS].
      --version          Display Twisted version and exit.
      --help             Display this help and exit.

(wstest)oberstet@thinkpad-t430s:~$

How to use

Testing WebSocket server implementations

To test a WebSocket server implementation and generate compliance test reports, first start the WebSocket server that you want to test. Here, we are using a example from Autobahn|Python:

(python279_1)oberstet@thinkpad-t430s:~/scm/crossbario/autobahn-python/examples/twisted/websocket/testee$ python testee_server.py
2015-12-21 21:31:54+0100 [-] Log opened.
2015-12-21 21:31:54+0100 [-] TesteeServerFactory starting on 9001
2015-12-21 21:31:54+0100 [-] Starting factory <__main__.TesteeServerFactory object at 0x7faf23551210>

Then, run wstest in fuzzing client mode:

cd ~
mkdir test
cd test
wstest -m fuzzingclient

The testsuite will now start a WebSocket fuzzing client connecting on TCP port 9001 to the WebSocket servers to be tested:

(wstest)oberstet@thinkpad-t430s:~/test$ wstest -m fuzzingclient
Auto-generating spec file 'fuzzingclient.json'
Loading spec from /home/oberstet/test/fuzzingclient.json
...
Autobahn Fuzzing WebSocket Client (Autobahn Version 0.7.4 / Autobahn Testsuite Version 0.10.9)
Ok, will run 521 test cases against 1 servers
...

Testing WebSocket client implementations

To test a WebSocket client implementation and generate compliance test reports, start wstest in fuzzing server mode:

cd ~
mkdir test
cd test
wstest -m fuzzingserver

The testsuite will now start a WebSocket fuzzing server listening on TCP port 9001 for WebSocket clients to be tested:

(wstest)oberstet@thinkpad-t430s:~/test$ wstest -m fuzzingserver
Auto-generating spec file 'fuzzingserver.json'
Loading spec from /home/oberstet/test/fuzzingserver.json
...
Autobahn WebSockets 0.7.4/0.10.9 Fuzzing Server (Port 9001)
Ok, will run 521 test cases for any clients connecting
...

Note: The fuzzing server mode will also start a Web server on port 8080 that renders a HTML page for browser WebSocket clients to be tested.

On first run, the tool will auto-generated a test configuration file:

(wstest)oberstet@thinkpad-t430s:~/test$ cat fuzzingserver.json

{
   "url": "ws://127.0.0.1:9001",
   "outdir": "./reports/clients",
   "cases": ["*"],
   "exclude-cases": [],
   "exclude-agent-cases": {}
}

You can tweak that file to run only some tests, e.g. "cases: ["1.*", "2.1.*"]" will run only the tests under section 1.* and subsection 2.1.*.

More Information

For more information take a look at the following information

Get in Touch

Get in touch on IRC #autobahn on chat.freenode.net or join the mailing list.