CONFIGURATION and COMPILING
---------------------------
When just checked out from git, run:
./autogen.sh
Next run:
./configure --enable-maintainer-mode
make
You need to have a C++ compiler and boost installed (and libcurl of course).
PKG_CONFIG_PATH can be used to find and use a specific installation
of libcurl, but don't forget to also set LD_LIBRARY_PATH correctly if
you installed it in a non-standard place.
RUNNING
-------
To run the test, first start the server:
./http_server
and in a different terminal (the server doesn't go to the background) run the client:
./http_client [-p port] [hostname]
The default hostname is 'localhost' and the default port is 9001.
An alternative way to run the client is using strace, for example:
strace -tt -s256 -e trace=network,select,poll -e write=4,5,6 -e read=4,5,6 -o outfile ./http_client
and then inspect outfile.
EXPLANATION
-----------
The http_server listens on port 9001 and accepts any number of connections.
Each connection is full HTTP pipeline capable (version 1.1), that is - it deals
correctly with whatever http_client is feeding it ;).
The client sends (multiple) requests (without waiting for replies) of the form:
GET / HTTP/1.1\r\n
Host: localhost:9001\r\n
Accept: */*\r\n
X-Sleep: 500\r\n
X-Request: 3\r\n
\r\n
Where the number after X-Requests starts at 0 and is incremented with 1 for
every next request (32 in total).
The server waits 'X-Sleep' milliseconds after receiving the request
and then replies with:
HTTP/1.1 200 OK\r\n
Keep-Alive: timeout=10 max=400\r\n
Content-Length: 32\r\n
Content-Type: text/html\r\n
X-Connection: 1\r\n
X-Request: 3\r\n
X-Reply: 3\r\n
\r\n
<html><body>hello</body></html>\n
Where X-Connection enumerates the socket connections (and therefore should
remain the same for the full duration of the test: we want this to be a pipeline
only using a single connection).
The returned X-Request header is the same as the one that was received
(that this is a reply for), and the X-Reply header enumerates the replies
that the server sent over this connection in the order they are generated.
It should be the case therefore that X-Request and X-Reply are always the
same number (and they are).
LIBCURL BUGS
------------
In order of importance,
- At the end of this application while a request is timing out, libcurl
starts to return from curl_multi_perform() immediately, while
curl_multi_fdset() keeps returning a bit set on the fdread set. At this
point there is activity on this socket, but libcurl doesn't read that.
The result is select() immediately returns, and get into a tight loop
of the main loop, calling curl_multi_perform() / curl_multi_fdset() / select()
10,000 times per second (until finally the 4 seconds of CURLOPT_TIMEOUT
did pass).
- The http_client application first sends a single request and waits for
a reply. This is necessary because if we start immediately with sending
six requests we end up with six connections. I think that it would be
better if:
1) libcurl had an option to pass to an easy handle that pipelining is
guaranteed to be supported - and libcurl would then ALWAYS do
pipelining on a single connection (until CURLMOPT_MAX_PIPELINE_LENGTH
is reached).
2) If libcurl would simply queue extra request until it got back headers
for the first request and thus knows if pipelining is supported in
exactly the same way as this happens now (where the application has
to add extra code in order to make sure that it doesn't add more than
one connection the first time). This seems especially important
after an error occurs on the socket: the new connection is AGAIN
'unknown' to libcurl (or so is my experience), while the applicaiton
is oblivious of this and now no longer CAN be the one who limits
the number of requests to one until (again) headers are received.
PS Both these problems have now been addressed through the addition of
CURLMOPT_PIPELINE_POLICY_FUNCTION.
CarloWood/curl_http_pipeline_tester
A test server and client to test http pipelining with libcurl.
C++