Handshake fails when server selects a secondary key share

[let4be]

Hi!

I'm using ClientHello from Firefox(Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/114.0)
x25519 doesn't seem to be supported, for example this server selects x25519 as preferable curve - https://pbs.twimg.com/media/F47NEN4WUAAZwRg.jpg

verified by adding some additional error details here:
https://github.com/refraction-networking/utls/blob/master/handshake_client_tls13.go#L530

mismatch is here:

curveIDForCurve(hs.ecdheKey.Curve()) = X25519
hs.serverHello.serverShare.group = CurveP256

As a result handshake fails and I cannot connect to this website with uTLS

[gaukas]

If you were talking about supporting X25519 as a supported_group which can used in key_share, then it is already supported. Most of the popular parrots actually defaults X25519 by including it in both supported_groups and key_share.

Will you be able to share more details?

And now most of TLS servers actually prefers X25519, as it is uncommon to see HelloRetryRequest being sent back when using any of these popular parrots.

[let4be]

The problem probably stems from the way I construct client hello spec(from raw bytes, parsing an existing clienthello of a real browser)
I will provide more details on weekends

[let4be]

Hey, so as I said I'm constructing ClientHelloSpec from a raw client hello

you can try using this to reproduce with my ClientHello with something like

uConfig := &utls.Config{
    RootCAs:                     config.RootCAs,
    NextProtos:                  config.NextProtos,
    ServerName:                  config.ServerName,
    InsecureSkipVerify:          config.InsecureSkipVerify,
    DynamicRecordSizingDisabled: config.DynamicRecordSizingDisabled,
    OmitEmptyPsk: true,
}
utlsClient := utls.UClient(conn, uConfig, utls.HelloCustom)

const RAW_CLIENT_HELLO_HEX = "1603010200010001fc030330763375ed2f6ff5c540f3fff9fd30990a4df94f19b316026879c47d92fdd8ab208a0033a895c43328ed2d201f181545dee8352e7d96495f61c94206831512ea950022130113031302c02bc02fcca9cca8c02cc030c00ac009c013c014009c009d002f00350100019100000012001000000d7062732e7477696d672e636f6d00170000ff01000100000a000e000c001d00170018001901000101000b00020100002300000010000e000c02683208687474702f312e310005000501000000000022000800060403050306030033006b0069001d00206156007db796a6479460f411a0411c5e0b596ec9496726c8e726e3e895a897240017004104a08447acf76355bb67bd81e6a58a5842ac087ba8c36c75546b0ed8ad51a5b8f447b8cb8d3c211ccd69bb54fc1c8f9dd2477578cadcdde365a053832928d87e57002b00050403040303000d00140012040305030603080408050806040105010601002d00020101001c000240010015008f0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
spec := &utls.ClientHelloSpec{}

rawClientHelloBytes, err := hex.DecodeString(RAW_CLIENT_HELLO_HEX)
if err != nil {
    panic(err)
}

if err := spec.FromRaw(rawClientHelloBytes, false, true); err != nil {
    panic(fmt.Errorf("cannot translate ClientHello to ClientHelloSpec: %w", err))
}
if err := utlsClient.ApplyPreset(spec); err != nil {
    panic(fmt.Errorf("uTlsConn.ApplyPreset() error: %+v", err))
}
if err := utlsClient.Handshake(); err != nil {
    panic(fmt.Errorf("uTlsConn.Handshake() error: %+v", err))
}

This client hello works with most of the websites but it fails with https://pbs.twimg.com/media/F47NEN4WUAAZwRg.jpg
Any ideas?

[let4be]

Narrowed it down to a supported client hello spec:

utlsClient := utls.UClient(conn, uConfig, utls.HelloFirefox_105)

if err := utlsClient.Handshake(); err != nil {
    panic(fmt.Errorf("uTlsConn.Handshake() error: %+v", err))
}

uTLS client with HelloFirefox_105 cannot handshake with this particular website

[let4be]

Basically this check
https://github.com/refraction-networking/utls/blob/master/handshake_client_tls13.go#L528C2-L528C2
never passes when using utls.HelloFirefox_105 with some websites but the real-deal firefox loads without an issue

[let4be]

If I keep downgrading supported Firefox parrots, utls.HelloFirefox_56 is the first one that is able to load a given website

[let4be]

starting from HelloFirefox_63 all parrots define a KeyShareExtension with X25519, could this be a culprit? is it implemented properly in uTLS?

[gaukas]

Interesting. Since this error does not replicate on other sites (according to what you reported in this thread), I believe the problem is unlikely in KeyShareExtension or how SupportedGroups are handled in uTLS, but more likely due to a certain behavior that is unique to the target TLS server.

If possible you may want to pcap the entire TLS handshake process (possibly dump the keylog) and inspect what does server responds.

---

I am actually aware that most of modern browsers (including certain versions of Google Chrome and Mozilla Firefox) suffers from downgrading attack when the server selects a deprecated/not-advertised supported_group or cipher_suite (they may directly proceed or send a new ClientHello advertising the selected ones), while uTLS strictly forbids such behavior.

[Yopi]

I'm getting the same error on multiple other websites -- mostly different bank websites, and moving to the older version as let4be indicated also works for me

[gaukas]

Care to try pcap and see what happens on those servers? My theory is the server being too old (very likely for the banks to be in TLS1.2-early stages) and does not support X25519 as a KeyShare. Also failed to correctly implement HelloRetryRequest, or does not speak at least ONE from our advertised supported_groups.

[let4be]

I will try to take a look tomorrow, but you can also try to reproduce this on your side with that server - pbs.twimg.com

[gaukas]

After looking into this issue I found several problems.

For uTLS, when we generate and send key shares in a ClientHello message, they are saved to a *KeySharesParameters to be retrieved later, while the first non-GREASE key share is copied and saved directly in the handshake states.

utls/u_parrots.go

Lines 2433 to 2477 in fc79497

	case *KeyShareExtension:
	preferredCurveIsSet := false
	for i := range ext.KeyShares {
	curveID := ext.KeyShares[i].Group
	if isGREASEUint16(uint16(curveID)) { // just in case the user set a GREASE value instead of unGREASEd
	ext.KeyShares[i].Group = CurveID(GetBoringGREASEValue(uconn.greaseSeed, ssl_grease_group))
	continue
	}
	if len(ext.KeyShares[i].Data) > 1 {
	continue
	}
	if scheme := curveIdToCirclScheme(curveID); scheme != nil {
	pk, sk, err := generateKemKeyPair(scheme, curveID, uconn.config.rand())
	if err != nil {
	return fmt.Errorf("HRR generateKemKeyPair %s: %w",
	scheme.Name(), err)
	}
	packedPk, err := pk.MarshalBinary()
	if err != nil {
	return fmt.Errorf("HRR pack circl public key %s: %w",
	scheme.Name(), err)
	}
	uconn.HandshakeState.State13.KeySharesParams.AddKemKeypair(curveID, sk.secretKey, pk)
	ext.KeyShares[i].Data = packedPk
	if !preferredCurveIsSet {
	// only do this once for the first non-grease curve
	uconn.HandshakeState.State13.KEMKey = sk.ToPublic()
	preferredCurveIsSet = true
	}
	} else {
	ecdheKey, err := generateECDHEKey(uconn.config.rand(), curveID)
	if err != nil {
	return fmt.Errorf("unsupported Curve in KeyShareExtension: %v."+
	"To mimic it, fill the Data(key) field manually", curveID)
	}
	uconn.HandshakeState.State13.KeySharesParams.AddEcdheKeypair(curveID, ecdheKey, ecdheKey.PublicKey())
	ext.KeyShares[i].Data = ecdheKey.PublicKey().Bytes()
	if !preferredCurveIsSet {
	// only do this once for the first non-grease curve
	uconn.HandshakeState.State13.EcdheKey = ecdheKey
	preferredCurveIsSet = true
	}
	}
	}

However in (*UConn).clientHandshake, the *KeySharesParameters was mistakenly overridden by a NewKeySharesParameters().

utls/u_conn.go

Lines 566 to 571 in fc79497

	if c.vers == VersionTLS13 {
	hs13 := c.HandshakeState.toPrivate13()
	hs13.serverHello = serverHello
	hs13.hello = hello
	hs13.keySharesParams = NewKeySharesParameters()
	if !sessionIsLocked {

It caused us to lose all but the first key share (saved elsewhere).

[gaukas]

For the target host pbs.twimg.com however, they have at least 2 sets of configurations, TLS12-only and TLS13-compatible.

For the TLS12-only ones, the connection WILL not fail as the server selects x25519, the first advertised Key Share. The other group, TLS13-compatible servers will instead select secp256r1, the second, which is lost as mentioned.

This problem happens ONLY if we advertise more than one key share at the same time (Firefox's default behavior). Advertising the support of more than one key share (through supported_groups extension) does not cause any issue.

[gaukas]

Looks like I'm the one to blame. I literally could no longer recall why there was an overriding behavior -- perhaps there's no specific reason, or maybe a nil-pointer problem.

if hs13.keySharesParams == nil {
    hs13.keySharesParams = NewKeySharesParameters()
}

Hope this would work better.

[gaukas]

I will tag & release ASAP, since it is indeed an unintended breaking behavior.

[Yopi]

Thank you for the quick solution!