kmqtt-broker - Error on TLS javax.net.ssl.SSLException : Unable to parse TLS packet header

[nishankhadka09]

I have been getting error like this whenever I have tried to connect to the MQTT Broker from some client with TLS enabled. Can anyone point me in the right direction please.

javax.net.ssl.SSLException: Unable to parse TLS packet header
at com.android.org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:807)
at com.android.org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:747)
at com.android.org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:712)
at com.android.org.conscrypt.Java8EngineWrapper.unwrap(Java8EngineWrapper.java:237)
at socket.tls.TLSSocket.read0(TLSSocket.kt:136)
at socket.tls.TLSSocket.runHandshake(TLSSocket.kt:122)
at socket.tls.TLSSocket.<init>(TLSSocket.kt:28)
at socket.tls.TLSServerSocket.createSocket(TLSServerSocket.kt:78)
at socket.ServerSocket.accept(ServerSocket.kt:107)
at socket.ServerSocket.select(ServerSocket.kt:138)
at socket.ServerSocketLoop.run(ServerSocketLoop.kt:22)
at mqtt.broker.Broker.listen(Broker.kt:68)
at com.example.headphonetest.broker.MqttBrokerManager$startBroker$1.invokeSuspend(MqttBrokerManager.kt:106)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:585)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:802)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:706)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:693)
socket.SocketClosedException
at socket.tls.TLSSocket.send0-GBYM_sE(TLSSocket.kt:81)
at socket.tls.TLSSocket.read0(TLSSocket.kt:162)
at socket.tls.TLSSocket.runHandshake(TLSSocket.kt:122)
at socket.tls.TLSSocket.<init>(TLSSocket.kt:28)
at socket.tls.TLSServerSocket.createSocket(TLSServerSocket.kt:78)
at socket.ServerSocket.accept(ServerSocket.kt:107)
at socket.ServerSocket.select(ServerSocket.kt:138)
at socket.ServerSocketLoop.run(ServerSocketLoop.kt:22)
at mqtt.broker.Broker.listen(Broker.kt:68)
at com.example.headphonetest.broker.MqttBrokerManager$startBroker$1.invokeSuspend(MqttBrokerManager.kt:106)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:585)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:802)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:706)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:693)

I have a PKCS12 file (containing both the self - signed certificate and private key) stored in the android project's raw folder. The CA that signed the certificate is installed on both the MQTT client and broker device. Except for the warning of key length being 1024 -bits, ssl- checker shows the certificate as valid.

[davidepianca98]

Hello, this usually happens if you are trying to connect to the broker port where there is no TLS setup. Please check that.
Also can you please post the MQTTClient constructor without sensitive information?
Can you reproduce the problem on a public broker like test.mosquitto.org so that I can try it?
Which version of Android are you running on, are you on an emulator and if yes which device?

[nishankhadka09]

This issue is when I am using the kmqtt-broker library. I haven't tried the client library yet. I used the kmqtt-broker lib to create my own standalone broker, which works fine when there is no TLS enabled. But when I enable TLS, the broker.listen() throws the mentioned exception.
Android version - 13 & 14 (Tiramisu and UpsideDown cake)

broker constructor

 broker = Broker(
                tlsSettings = if (isTlsEnabled) TLSSettings(keyStoreFilePath = getTlsFilepath(), keyStorePassword = "P@ss")else null,
                authentication = if (authenticationNeeded) {
                    object : Authentication {
                        override fun authenticate(
                            clientId: String,
                            username: String?,
                            password: UByteArray?
                        ): Boolean {
                            return username == mqttUserName && password?.toByteArray()
                                ?.decodeToString() == mqttPassword
                        }
                    }
                } else null,
                connectionCallbacks = connectionCallbacks, bytesMetrics = bytesMetrics,
            )
            broker.listen()

private fun getTlsFilepath(): String {
       val inputStream = application.resources.openRawResource(R.raw.mqtt_test_cert)
       val file = File.createTempFile("keyStore", ".p12", application.cacheDir)
       val outputStream = FileOutputStream(file)
       inputStream.use { input ->
           outputStream.use { output ->
               input.copyTo(output)
           }
       }
       Timber.d("File path is ${file.absolutePath}")
       return file.absolutePath

   }

[davidepianca98]

Sorry I missed the broker part. I have replicated your exact setup with the same code. Then I run it on an Android 14 emulator.
I used this command to forward the port from the host PC to the emulator: .\adb.exe forward tcp:1883 tcp:1883
And then I use MQTT X to connect and it works.

Please check again and make sure that the client is setup to use TLS, because I get that same error when I try to connect with MQTT X (the client) with TLS disabled.

[nishankhadka09]

Thank you! Client wasn't set up properly, fixed it and it's working.