Add support for packet loss

Question

Add support for packet loss

Closed this issue a month ago · 15 comments

Some Ookla speedtest servers support a packet loss value. I'm not sure why it's only some, rather than all...

For example these server IDs should all support packet loss:

30690
40788
32950
34948
11445
48354
40788

It gets return by the Ookla CLI in both normal text output and also JSON output (note I've removed a bunch of stuff from this output to make it easier to read):

$ speedtest --accept-license --accept-gdpr -f json-pretty -s 32950
{
    "type": "result",
    "timestamp": "2023-12-27T17:33:06Z",
    "ping": {
    },
    "download": {
    },
    "upload": {
    },
    "packetLoss": 0,
    "isp": "My ISP",
    "interface": {
    },
    "server": {
        "id": 32950,
        "host": "speedtest.boxbroadband.co.uk",
        "port": 8080,
        "name": "Box Broadband Ltd",
        "location": "London",
        "country": "United Kingdom",
        "ip": "45.10.101.252"
    },
    "result": {
    }
}

Would it be possible to add packet loss support to speedtest-go? I like the fact the Ookla CLI can handle the packet loss data, but recently I switched away because it stopped being able to get anywhere close to my max connection speed but the lack of packet loss data is bugginng me.

Answer 1 · 2023-12-27T23:43:36.000Z

Hi @kura, I don't know the protocol defined by speedtest.net, so we can only implement the uplink packet loss. Is this what you need?

Answer 2 · 2023-12-27T23:46:50.000Z

And we may only be able to implement this feature on specific platforms.

Answer 3 · 2023-12-27T23:50:29.000Z

@r3inbowari I'm not sure about the speedtest.net protocol either tbh.

Any kind of packet loss measurement would be good, otherwise I guess the only option for me would be to have something running iperf and dumping measurements.

What exactly would the uplink packet loss measurement be?

I only care about it working on Linux so if you do find you can only implement it on specific platforms then I'd guess Linux is probably the most likely to be supported?

Answer 4 · 2023-12-28T00:57:57.000Z

What exactly would the uplink packet loss measurement be?

Maybe like this? I am not sure.

retransmission rate = tcpOutboundRetransBytes / (tcpOutboundRetransBytes + actuallyOutboundSize) * 100%

I don't know the protocol defined by speedtest.net, so we can only implement the uplink packet loss.

This is just the retransmission rate, which is quite different from packet loss.

speedtest-go_windows_amd64.zip

Answer 5 · 2023-12-28T10:28:40.000Z

Hmmm, yeah retrans isn't really the same as packet loss.

I assume what the official Ookla client is doing is sending UDP packets to the server and measuring packet loss (similar to the iperf approach), which might explain why the speedtest server listens on udp/5060 and/or udp/8080.

Answer 6 · 2023-12-28T15:53:53.000Z

Ok, the official Ookla client is definitely doing some UDP traffic.

# Before running Ookla speedtest
$> cat /proc/net/udp
   sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops

 2590: 00000000:14E9 00000000:0000 07 00000000:00000000 00:00000000 00000000   119        0 17860 2 0000000000000000 0

 3121: 00000000:96FC 00000000:0000 07 00000000:00000000 00:00000000 00000000   119        0 17862 2 0000000000000000 0

 5482: 3500007F:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000   101        0 19625 2 0000000000000000 0

 5540: 00000000:006F 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 17782 2 0000000000000000 0

 5752: 0100007F:0143 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 20484 2 0000000000000000 0

# During a run of Ookla speedtest
$> cat /proc/net/udp
   sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops

 2590: 00000000:14E9 00000000:0000 07 00000000:00000000 00:00000000 00000000   119        0 17860 2 0000000000000000 0

 3121: 00000000:96FC 00000000:0000 07 00000000:00000000 00:00000000 00000000   119        0 17862 2 0000000000000000 0

 5482: 3500007F:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000   101        0 19625 2 0000000000000000 0

 5540: 00000000:006F 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 17782 2 0000000000000000 0

 5752: 0100007F:0143 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 20484 2 0000000000000000 0

 6180: 00000000:E2EF 00000000:0000 07 00000000:00000000 00:00000000 00000000  1000        0 292130 2 0000000000000000 0

 7644: 00000000:E8A7 00000000:0000 07 00000000:00000000 00:00000000 00000000  1000        0 292132 2 0000000000000000 0

During the Ookla run 2 new lines appear, 1000 is the UID of my user.

 6180: 00000000:E2EF 00000000:0000 07 00000000:00000000 00:00000000 00000000  1000        0 292130 2 0000000000000000 0

 7644: 00000000:E8A7 00000000:0000 07 00000000:00000000 00:00000000 00000000  1000        0 292132 2 0000000000000000 0

I decided to capture UDP packets with tshark from a speedtest.net run and it's definitely sending UDP packets in the background. I haven't figured out exactly what each part means yet but here is a single capture:

0000   00 15 5d 71 7d e5 00 15 5d fa aa b5 08 00 45 00   ..]q}...].....E.
0010   00 5f a0 4a 40 00 40 11 76 c6 ac 15 e4 61 2d 0a   ._.J@.@.v....a-.
0020   65 fc 94 e6 1f 90 00 4b 23 da 4c 4f 53 53 20 36   e......K#.LOSS 6
0030   39 39 37 37 34 34 30 20 31 34 20 44 33 42 41 46   9977440 14 D3BAF
0040   42 34 45 2d 33 35 41 43 2d 34 39 44 37 2d 38 37   B4E-35AC-49D7-87
0050   37 39 2d 38 42 45 33 33 46 38 35 32 41 43 33 20   79-8BE33F852AC3 
0060   31 34 30 32 30 30 38 33 30 37 33 37 0a            140200830737.

69977440 value doesn't change in a single run, maybe some kind of ID?
14 this looks to be incrementing counter. It's definitely going from 1 to n
D3BAFB4E-35AC-49D7-8779-8BE33F852AC3 not sure what this is. Some kind of UUID? It doesn't change in a single run
140200830737 timestamp

Answer 7 · 2023-12-29T20:50:29.000Z

This packet looks like it reports a loss counter. and 140200830737 does not look like a timestamp?
I don't know. any more?

Answer 8 · 2023-12-29T20:54:44.000Z

You are right. That is not a timestamp, not sure why I thought it was.

I'll do another complete run and packet capture tomorrow and post a larger set of packets and see if I can figure out more of the data that is being sent.

Answer 9 · 2023-12-29T20:59:17.000Z

I think the udp protocol of speedtest may be similar to tcp.
https://gist.github.com/sdstrowes/411fca9d900a846a704f68547941eb97

~~140200830737 more likes share id.~~
eg https://www.speedtest.net/result/1568942211

Answer 10 · 2023-12-29T21:10:15.000Z

D3BAFB4E-35AC-49D7-8779-8BE33F852AC3 not sure what this is. Some kind of UUID? It doesn't change in a single run

This UUID widely used in a single test process and is randomly generated by the user client. Its purpose is to mark that the test is associated with the upload\download resource. likes this:
https://speedtest.boxbroadband.co.uk:8080/download?size=25000000&guid=6b7df3db-584b-44b9-9b04-94cbd5d14268

Answer 11 · 2023-12-30T12:11:45.000Z

Ok, so I did a PCAP capture of an entire test again. It's attached here as test.pcap.gz

The last 54-70 bytes are the payload, and they tend to alternate in size as the payload content alternates. The increase in size looks like it's the counters going in to double and then triple digits.

# 1 packet 68 bytes
LOSS 200231533 6 0 D41D9238-2B08-491B-BE8A-5AC2 A0FCA3D5 1925215 24835

# 2 packet 54 bytes
LOSS 672475500 0 70A65ECE-9199-47CA-926F-C9D84 9BB405B

# 3 packet 68 bytes
LOSS 200231533 6 1 D41D9238-2B08-491B-BE8A-5AC2 A0FCA3D5 1925215 77000

# 4 packet 54 bytes
LOSS 672475500 1 70A65ECE-9199-47CA-926F-C9D84 9BB405B

# 5 packet 68 bytes
LOSS 200231533 6 2 D41D9238-2B08-491B-BE8A-5AC2 A0FCA3D5 1925216 33921

# 6 packet 54 bytes
LOSS 672475500 2 70A65ECE-9199-47CA-926F-C9D84 9BB405B

I think you are right about it being like the TCP protocol.

I'm not sure why there are 2 UUIDs, 70A65ECE-9199-47CA-926F-C9D84 is the UUID of the test itself but I don't know what D41D9238-2B08-491B-BE8A-5AC2 references.

It also sends these packets on 2 different UDP connections, the odd numbered packets are being sent from src: 40913, dst: 8080 and the even numbered packets are src: 33527, dst: 8080

Answer 12 · 2024-01-17T14:58:27.000Z

~~unfortunately, there seems to be no clue about the intention of this udp packet at present.~~

Answer 13 · 2024-05-07T02:27:20.000Z

The latest development is that we have tracked the relevant tcp flow.

Currently known:

SpeedTest uses udp to send three consecutive chunks of data(As we saw earlier those packets). These consecutive packets are only sent once so that a counter in the server can confirm the number of drop.
Initialize the packet loss pipeline using INITPLOSS in tcp.
Use the PLOSS instruction to get packet loss information.

The first instruction during pipeline initialization should be HI (testguid), eg: HI D3BAFB4E-35AC-49D7-8779-8BE33F852AC3

Answer 14 · 2024-05-07T03:24:47.000Z

# Send the following 7 data packets in order to simulating loss of 2, 5, 6, 7.

LOSS 1778004272 0 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 18921818
LOSS 1778004272 1 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 18986184
LOSS 1778004272 3 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 19039274
LOSS 1778004272 4 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 19091435
LOSS 1778004272 8 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 19147531
LOSS 1778004272 9 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 19198868
LOSS 1778004272 10 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 19307534

[2024-05-07 **:**:**.***]# SEND ASCII/24 to ***.***.**.*** :8080 >>>
LOSS 1778004272 0 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 18921818
[2024-05-07 **:**:**.***]# RECV ASCII/24 from ***.***.**.*** :8080 <<<
RECV 0 18921818 [1 0 0]

[2024-05-07 **:**:**.***]# SEND ASCII/24 to ***.***.**.*** :8080 >>>
LOSS 1778004272 1 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 18986184
[2024-05-07 **:**:**.***]# RECV ASCII/24 from ***.***.**.*** :8080 <<<
RECV 1 18986184 [2 1 0]

[2024-05-07 **:**:**.***]# SEND ASCII/24 to ***.***.**.*** :8080 >>>
LOSS 1778004272 3 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 19039274
[2024-05-07 **:**:**.***]# RECV ASCII/24 from ***.***.**.*** :8080 <<<
RECV 3 19039274 [3 3 0]

[2024-05-07 **:**:**.***]# SEND ASCII/24 to ***.***.**.*** :8080 >>>
LOSS 1778004272 4 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 19091435
[2024-05-07 **:**:**.***]# RECV ASCII/24 from ***.***.**.*** :8080 <<<
RECV 4 19091435 [4 4 0]

[2024-05-07 **:**:**.***]# SEND  ASCII/24 to ***.***.**.*** :8080 >>>
LOSS 1778004272 8 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 19147531
[2024-05-07 **:**:**.***]# RECV ASCII/24 from ***.***.**.*** :8080 <<<
RECV 8 19147531 [5 8 0]

[2024-05-07 **:**:**.***]# SEND ASCII/24 to ***.***.**.*** :8080 >>>
LOSS 1778004272 9 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 19198868
[2024-05-07 **:**:**.***]# RECV ASCII/24 from ***.***.**.*** :8080 <<<
RECV 9 19198868 [6 9 0]

[2024-05-07 **:**:**.***]# SEND ASCII/24 to ***.***.**.*** :8080 >>>
LOSS 1778004272 10 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43 19307534
[2024-05-07 **:**:**.***]# RECV ASCII/24 from ***.***.**.*** :8080 <<<
RECV 10 19307534 [7 10 0]

# At the same time, we can observe these in the tcp pipe.

[2024-05-07 **:**:**.***]# SEND ASCII/40 >>>
HI 8CF3AB01-AA9B-45F4-B9BF-C987833E0C43
[2024-05-07 **:**:**.***]# RECV ASCII/44 from SERVER <<<
HELLO 2.11 (2.11.0) 2023-11-29.2207.3251a05

[2024-05-07 **:**:**.***]# SEND ASCII/10 >>>
INITPLOSS
[2024-05-07 **:**:**.***]# RECV ASCII/3 from SERVER <<<
OK

[2024-05-07 **:**:**.***]# SEND ASCII/6 >>>
PLOSS
[2024-05-07 **:**:**.***]# RECV ASCII/12 from SERVER <<<
PLOSS 0 0 0

[2024-05-07 **:**:**.***]# SEND ASCII/6 >>>
PLOSS
[2024-05-07 **:**:**.***]# RECV ASCII/12 from SERVER <<<
PLOSS 0 0 0

[2024-05-07 **:**:**.***]# SEND ASCII/6 >>>
PLOSS
[2024-05-07 **:**:**.***]# RECV ASCII/12 from SERVER <<<
PLOSS 2 0 1

[2024-05-07 **:**:**.***]# SEND ASCII/6 >>>
PLOSS
[2024-05-07 **:**:**.***]# RECV ASCII/12 from SERVER <<<
PLOSS 2 0 1

[2024-05-07 **:**:**.***]# SEND ASCII/6 >>>
PLOSS
[2024-05-07 **:**:**.***]# RECV ASCII/12 from SERVER <<<
PLOSS 3 0 3

[2024-05-07 **:**:**.***]# SEND ASCII/6 >>>
PLOSS
[2024-05-07 **:**:**.***]# RECV ASCII/12 from SERVER <<<
PLOSS 4 0 4

[2024-05-07 **:**:**.***]# SEND ASCII/6 >>>
PLOSS
[2024-05-07 **:**:**.***]# RECV ASCII/12 from SERVER <<<
PLOSS 5 0 8

[2024-05-07 **:**:**.***]# SEND ASCII/6 >>>
PLOSS
[2024-05-07 **:**:**.***]# RECV ASCII/12 from SERVER <<<
PLOSS 6 0 9

[2024-05-07 **:**:**.***]# SEND ASCII/6 >>>
PLOSS
[2024-05-07 **:**:**.***]# RECV ASCII/13 from SERVER <<<
PLOSS 7 0 10

Answer 15 · 2024-05-07T03:33:46.000Z

This is my conclusion, what do you think? @kura

Define PLOSS [x0] [x1] [x2]

$x_{0} \in [0, +\infty)$

$PacketLoss= [1-\frac{(x_{0}-x_{1})}{x_{2}+1}] * 100 %$

x0 The number of packets currently received by the server (mapped into a continuous space).
x1 The number of dup packets (typical value is zero).
x2 The maximum value represented by the packet currently received by the server.