Microdot + ssl socket connection
Closed this issue · 30 comments
Hi,
I'm facing a problem with Microdot and an async SSL socket connection on an ESP32 4MB (firmware esp32-20230426-v1.20.0). Microdot works fine (many thanks for this great job), but it frequently drops the async SSL connections when I access the website. My code is straightforward:
SSL async connection (part of the code)
source: https://github.com/micropython/micropython/blob/0fff2e03fe07471997a6df6f92c6960cfd225dc0/extmod/uasyncio/stream.py#L74
async def connector():
ai = socket.getaddrinfo(host, 443, 0, socket.SOCK_STREAM)
ai = ai[0]
s = socket.socket(ai[0], ai[1], ai[2])
s.setblocking(False)
try:
s.connect(ai[-1])
except OSError as e:
if e.args[0] != uerrno.EINPROGRESS:
s.close()
raise
ssls = ssl.wrap_socket(s)
ss = Stream(ssls)
yield core._io_queue.queue_write(s)
ss.write(bytes(message, 'utf8')) # To send, I use it only once.
data = await ss.read(512) # This function stays in a loop.
# Website
async def server():
app = Microdot()
@app.route('/')
async def index(request):
return send_file('/index.html')
@app.route('/static/<path:path>')
async def static(request, path):
return send_file('/static/' + path)
app.run(port=80)
async def main():
connector_task = asyncio.create_task(connector())
server_task = asyncio.create_task(server())
await asyncio.gather(connector_task, server_task)
asyncio.run(main())and this error happens (not every time, but frequently) if the SSL async connection is established and i acess the website (even if i took ss.write and ss.read out of the code):
Task exception wasn't retrieved
future: coro= <generator object 'serve' at 3ffe8a80>
Traceback (most recent call last):
File "uasyncio/core.py", line 1, in run_until_complete
File "microdot_asyncio.py", line 269, in serve
File "microdot_asyncio.py", line 337, in handle_request
File "microdot_asyncio.py", line 158, in write
File "uasyncio/stream.py", line 1, in stream_awrite
File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 113] ECONNABORTED
Sorry if i miss somenthing....thanks
@BrunoESP32 MicroPython does not support SSL on their asyncio implementation. You are starting a non-encrypted server on port 80 and then attempting to connect to port 443 which is not handled by your Microdot server. How does this work at all?
It seems SSL is implemented in version 1.21 of micropython.
Shouldn't it be possible now?
@wohltat I think this is currently planned, but the 1.21 release does not have support for SSL on asyncio yet. Where did you see that it does?
Ok, i'm not sure about it and i don't understand it fully.
I just noticed that there was a lot about TLS and SSL added in 1.21 like SSLContext:
https://micropython.org/resources/micropython-ChangeLog.txt
"In the ssl module, SSLContext has been added to be more compatible with CPython."
My guess was that this was kind of a missing link or something, since i remember reading about that, but i may be totally wrong.
The idea was that the sockets work with asyncio and that they can use ssl.wrap_socket like in the following example client code:
https://www.i-programmer.info/programming/hardware/16596-esp32-in-micropython-client-sockets.html?start=2
as i read it in asyncio documentation TLS works on nonblocking sockets:
https://github.com/peterhinch/micropython-async/blob/master/v3/docs/TUTORIAL.md#76-socket-programming
"Support for TLS on nonblocking sockets is platform dependent. It works on ESP32, Pyboard D and ESP8266."
Should it be possible or do i miss something?
@wohltat non-blocking sockets is not the same as asyncio. Asyncio uses non-blocking sockets, but it has a lot of logic on top, and as far as I know TLS is not implemented yet at this level.
Carglglz said to have build a running server with microdot_async + tls (https).
micropython/micropython#8177 (comment)
micropython/micropython#8177 (comment)
I think those examples only work with CPython at the moment?, unless I'm missing something in that example
...Right now MicroPython
sslmodule has noSSLContext, and inapp.run
In micropython v1.21 there is a SSLContext so it seems a different situation now.
See micropython/micropython#11897. Once that is merged, I'll have a look and see if I need to make any changes on my side.
@wohltat @miguelgrinberg I've tested both microdot_asyncio.py and microdot_asyncio_websocket.py as part of the development process of micropython/micropython#11897 and they work great, no modifications were needed 👍🏼 , I've only added this to microdot_asyncio.py to silence a broken pipe error while doing concurrent requests testing with ab e.g.
@@ -373,12 +373,12 @@ class Microdot(BaseMicrodot):
print_exception(exc)
# print("Dispatch request now")
res = await self.dispatch_request(req)
- if res != Response.already_handled: # pragma: no branch
- await res.write(writer)
try:
+ if res != Response.already_handled: # pragma: no branch
+ await res.write(writer)
await writer.aclose()
except OSError as exc: # pragma: no cover
- if exc.errno in MUTED_SOCKET_ERRORS:
+ if abs(exc.errno) in MUTED_SOCKET_ERRORS:
pass
else:
raise
I've added abs(exc.errno) since ssl sockets returns OSError: -32 in await res.write(writer) which I'm pretty sure is a broken pipe error see https://github.com/orgs/micropython/discussions/11859.
$ ab -n 10 -c 2 -v 1 https://127.0.0.1:4444/
This is ApacheBench, Version 2.3 <$Revision: 1901567 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient).....done
Server Software:
Server Hostname: 127.0.0.1
Server Port: 4444
SSL/TLS Protocol: TLSv1.2,AES256-SHA256,4096,256
Document Path: /
Document Length: 3280 bytes
Concurrency Level: 2
Time taken for tests: 0.703 seconds
Complete requests: 10
Failed requests: 0
Total transferred: 33460 bytes
HTML transferred: 32800 bytes
Requests per second: 14.22 [#/sec] (mean)
Time per request: 140.683 [ms] (mean)
Time per request: 70.341 [ms] (mean, across all concurrent requests)
Transfer rate: 46.45 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 41 59 14.9 56 82
Processing: 34 63 16.2 67 87
Waiting: 18 39 12.6 45 55
Total: 80 123 17.2 127 136
Percentage of the requests served within a certain time (ms)
50% 127
66% 134
75% 134
80% 135
90% 136
95% 136
98% 136
99% 136
100% 136 (longest request)
That sounds great!
Just so that i can reproduce it, what exacltly did you do?
Do i understand it correctly that you did the following:
- You used the asyncio-tls branch and recompiled micropython?
- Tested it with the microdot examples?
On which platform did you try it on?
You used the asyncio-tls branch and recompiled micropython?
Tested it with the microdot examples?
Yes, that's correct
On which platform did you try it on?
on unix and esp32 port
I compiled the asyncio-tls branch and uploaded the firmware bin to an esp32.
When trying to run the echo_async_tls.py I still get the following error.
Traceback (most recent call last):
File "<stdin>", line 22, in <module>
AttributeError: 'SSLContext' object has no attribute 'load_cert_chain'
Did you do anything else?
on reboot:
MPY: soft reboot
MicroPython v1.12-4344.g7736ab832.dirty on 2023-11-12; Generic ESP32 module with ESP32
Type "help()" for more information.
>>>
The version number seems wrong, since i can do for example import mip for example.
I forgot the #MICROPY_SSL_MBEDTLS_EXTRAS Macro.
I put it in mbedtls_config.h:
#define MICROPY_SSL_MBEDTLS_EXTRAS
The version number seems wrong, since i can do for example import mip for example.
see https://github.com/orgs/micropython/discussions/12400, I think you need to update the git tags from upstream micropython
I've only added this to
microdot_asyncio.pyto silence a broken pipe error while doing concurrent requests testing withabe.g.@@ -373,12 +373,12 @@ class Microdot(BaseMicrodot): print_exception(exc) # print("Dispatch request now") res = await self.dispatch_request(req) - if res != Response.already_handled: # pragma: no branch - await res.write(writer) try: + if res != Response.already_handled: # pragma: no branch + await res.write(writer) await writer.aclose() except OSError as exc: # pragma: no cover - if exc.errno in MUTED_SOCKET_ERRORS: + if abs(exc.errno) in MUTED_SOCKET_ERRORS: pass else: raise
Doesn't work for me. Still get that negativ errors.
Starting async server on 0.0.0.0:4443...
Traceback (most recent call last):
File "microdot_asyncio.py", line 331, in handle_request
File "microdot_asyncio.py", line 72, in create
File "microdot_asyncio.py", line 113, in _safe_readline
File "asyncio/stream.py", line 1, in readline
OSError: (-30464, 'SSL - An unexpected message was received from our peer')
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3ffd96e0>
Traceback (most recent call last):
File "asyncio/core.py", line 1, in run_until_complete
File "microdot_asyncio.py", line 269, in serve
File "microdot_asyncio.py", line 340, in handle_request
File "microdot_asyncio.py", line 143, in write
File "asyncio/stream.py", line 1, in stream_awrite
File "asyncio/stream.py", line 1, in write
OSError: -30464
Traceback (most recent call last):
File "asyncio/stream.py", line 1, in _serve
OSError: [Errno 12] ENOMEM
Strange also
class Microdot(BaseMicrodot): starts at line 212 and the
the next line print_exception(exc) starts at 338 and not on 373 as i expected from the snippet.
Furthermore the line # print("Dispatch request now") does not exist, so it seems there are more changes in microdot_asyncio.py than shown above.
This is the modified version of echo_async_tls.py example that i used
import ssl
import binascii
from microdot_asyncio import Microdot, send_file
from microdot_asyncio_websocket import with_websocket
app = Microdot()
@app.route('/')
def index(request):
return send_file('tls/index.html')
@app.route('/echo')
@with_websocket
async def echo(request, ws):
while True:
data = await ws.receive()
await ws.send(data)
cert = binascii.unhexlify(
b"308205b53082039da00302010202090090195a9382cbcbef300d06092a864886f70d01010b050030"
b"71310b3009060355040613024155310c300a06035504080c03466f6f310c300a06035504070c0342"
b"617231143012060355040a0c0b4d6963726f507974686f6e31143012060355040b0c0b4d6963726f"
b"507974686f6e311a301806035504030c116d6963726f707974686f6e2e6c6f63616c301e170d3233"
b"303731353136323034395a170d3238303731333136323034395a3071310b30090603550406130241"
b"55310c300a06035504080c03466f6f310c300a06035504070c0342617231143012060355040a0c0b"
b"4d6963726f507974686f6e31143012060355040b0c0b4d6963726f507974686f6e311a3018060355"
b"04030c116d6963726f707974686f6e2e6c6f63616c30820222300d06092a864886f70d0101010500"
b"0382020f003082020a0282020100944fdb40b587af0cf7e9696c355d24a70936874e6a3bd2598166"
b"ce2495aaf9b4af01b54471f7cbf3626ae0720bf0bfd520507f79ec553c62898bfd2598385f56061b"
b"0e8f452625c82d3c83e2a0d070ab9be2db21faf88c58e4a61d62f8ff43960aa1ffdadaad41f7cb2e"
b"b337070a39f08ff9fe20c09b19926cbbc4a5154b796ff7e7ce11334e090d360c81072af08758f6cd"
b"7bad75bc7b95b6dcc801c85de81d72806ca3ce0782bfcbdffce707f9fb1572a7db0d74445dc32d5f"
b"bea12a3ab1d47edf668ebfa60ed8b51e654e76292e3894ee574ea851064956906aa8afe00e67664e"
b"110b5a6ff7db51f7944463cdd626ff2ec7886c229f4ca5985168f20f8f210972b5ff9181d4f3beb8"
b"914ec5b24a0953253b3d42ab55e98bd70cb25e7a24c603b27ec83e1ce31c90b728b47a5f606ff2a1"
b"0ff784a016894c28f7e71f51a78b0a7601bbbc8c1b132b04e567394a327a7aa4674e8e4c0bfaec4b"
b"eeccf0ed09d1660933d718a2f34ff91d79d875a73fbac07182a9531ca52bd360e2678f95ff9b4ba2"
b"1490d7456548364b2eb335c207d6e1e48ccd7d8cb43868a334c095bd9673be7403f3b69b545ee904"
b"a3f513d2b2a2dd46f06820cd394819551dd05d9b34a8a3238a521f6c1c3592f76d5ef29e181c60ee"
b"bcaf4c63098794c15d4f82e7425e75ff8f5430247ecc0e7f2983b715506012f187d54a7b6729bc61"
b"fa4d10a9f22b0203010001a350304e301d0603551d0e041604147a6d126931b58fa1c3dff3c9215f"
b"6202e61fa8da301f0603551d230418301680147a6d126931b58fa1c3dff3c9215f6202e61fa8da30"
b"0c0603551d13040530030101ff300d06092a864886f70d01010b0500038202010051b3a4834d2bf1"
b"95ac645bca61e95289eff38d80ab2ee732c6ebd1370781b66955a7583c78831f9fb7d8b236a1b5ff"
b"c9183b4e17225951e1fb2c39f7218af30ec3fd8e4f697e0d10ecd05eb14208535dc55bc1e25d8a43"
b"050670d4de3e4cb8c884e6cbb6b884320d496b354acf5258bcb0ddaefd065ee8fccbddf3a2bfa10d"
b"bfeb8ab6b2580b50f0678760599269b612f81ba1310bfcd39427fec49211769c514cdd0305081d8a"
b"11ebe705496d4dcc31ac9fab96a2d298ee4423789baffbfa0fa82ee1b5113f9cf597647a36640cad"
b"abf535205c322e16153d6ab04b0817f57d8a9a6ca2db2ab10986ae9eab343547e52c78a641868bb5"
b"e2981182fcc55d86cdc6aa8478b226318a3be72fb726dd0b90f30df810c4d6c6b5a0ecb3c6cc375b"
b"8d3d244a07d8517ad390929be7b75f679beb63d8c1028905af2383144a4ed560e45907d301846acc"
b"9dbec86bcdd7fbf8a805b59f359c8bd997f5eb7b8aea6f7a538f9663ec2c12e07d4b37650e92b783"
b"74356daee4a501eeb27fef79b472b2fcce4363a9ff4d80f96a3b47dc4c4ef380ef231d193a517071"
b"b31078fa9f9a80cfd943f7e99e4ed8548c9ea80fd845ecc2c89726be273fa8b36680d645998fd1e6"
b"2367638f4953e9af68531aedb2ee49dffaaed07a4a5b97551712058219ac6f8da71710949f761271"
b"5273a348dcce40c556bdab00a4ae3a7b23a5934ac88b7640df"
)
key = binascii.unhexlify(
b"308209280201000282020100944fdb40b587af0cf7e9696c355d24a70936874e6a3bd2598166ce24"
b"95aaf9b4af01b54471f7cbf3626ae0720bf0bfd520507f79ec553c62898bfd2598385f56061b0e8f"
b"452625c82d3c83e2a0d070ab9be2db21faf88c58e4a61d62f8ff43960aa1ffdadaad41f7cb2eb337"
b"070a39f08ff9fe20c09b19926cbbc4a5154b796ff7e7ce11334e090d360c81072af08758f6cd7bad"
b"75bc7b95b6dcc801c85de81d72806ca3ce0782bfcbdffce707f9fb1572a7db0d74445dc32d5fbea1"
b"2a3ab1d47edf668ebfa60ed8b51e654e76292e3894ee574ea851064956906aa8afe00e67664e110b"
b"5a6ff7db51f7944463cdd626ff2ec7886c229f4ca5985168f20f8f210972b5ff9181d4f3beb8914e"
b"c5b24a0953253b3d42ab55e98bd70cb25e7a24c603b27ec83e1ce31c90b728b47a5f606ff2a10ff7"
b"84a016894c28f7e71f51a78b0a7601bbbc8c1b132b04e567394a327a7aa4674e8e4c0bfaec4beecc"
b"f0ed09d1660933d718a2f34ff91d79d875a73fbac07182a9531ca52bd360e2678f95ff9b4ba21490"
b"d7456548364b2eb335c207d6e1e48ccd7d8cb43868a334c095bd9673be7403f3b69b545ee904a3f5"
b"13d2b2a2dd46f06820cd394819551dd05d9b34a8a3238a521f6c1c3592f76d5ef29e181c60eebcaf"
b"4c63098794c15d4f82e7425e75ff8f5430247ecc0e7f2983b715506012f187d54a7b6729bc61fa4d"
b"10a9f22b0203010001028202000b41080520013cc242299f0b4bfd5663aa6a4dd8206d8ba7a90f11"
b"036babfea8bc42e7eb5aae8ff656f87f3188406b7e13a6a815ab5e4867bdc236a25caba26857ac43"
b"ed9134b4d73cbf83ce759f7b7d3a25fbb4d76376dae3f6caf210ace60703a58951a51852922803d2"
b"2b91c82fdf563d85101d2d67c259a7e1e318fb922a71e85015b40beed9e6c90a1d6e1fb45586dcce"
b"ceb9c964a356ade82b6275e5c01e492a753f940852df788eab454aadc7d1dc74ddcf7dc493a3e4c9"
b"0557bbfe747e701b4b27b5c518a29dbcd8385525a1bb835e72a489096e15387e2f70b112c6bbd79e"
b"a97ae2562f7947cd2367635e25b5656a54aac7f1c892243dc135e5025a44d724884b244e8fe4abb4"
b"c67bbd2e652d5fc5942b55c24b7f642f65b9b6d37110a955c63eb4f26435be056effbd777f14db8d"
b"3d8073f7583b24656edb19911e1307101443a50717c32dbb80b6212e6f0ee43f629b1e718a958a5c"
b"fdcd99762f5bff821ac49b0e77c9d1426f8bb31142df030549330dde5cc92fa20d09744ceac6ae02"
b"fb354e9b930173e08488375f7c795b3b934c72b58a3353332d5129d56151b57a793d99868885ebd4"
b"aac11ca03e09f5b6bd9dda5322a0ab81e468839ea373ecd2b5ac4ffc99740581b35add07f83ff18e"
b"c2111555ead17783294b2330ad874bd966c1d60b44e5f379650910a8a05eb92cb7550191c13251f5"
b"0a11afa7510282010100c5a4aa380f6bdd4b4524deb44425aa7ef61039a46ad0d09e2ca2cd7fb757"
b"ff325f81eaf3a2e790afb3ffb0d71f3ffa52db1a24d3149839f03d1acfe33ef721fe310895986c5a"
b"fe88ceb82318ed540456b8aa7e07dc7b982345c4f040b1544bd2ee1e4cb0315bd8db3794ea93d705"
b"f41cc1c06badf72de36d2b4a4399846d6c851260e5044e9495be8225307edb97071bdea08c99ccfe"
b"54219f6a785db47864e03cf2851abcb62941d3efeea7cdf136d9e23845cf9ea0323b156c686c6d30"
b"1cbb5a8c7f1db23a998bf549874b2c13685b20d200d2d91be92c40480a0cca18c28f654dd644c60d"
b"e8e03824c0ff83e7cbfc44b2aa16ad537a09565ed4afbe63b8930282010100c01a5e6108420c3d2e"
b"ccd0b559e08680f47b3e7271ee4ea9bf4740cc5c418a53225778eddb716447b02d234909f8291581"
b"a45be0591952bacda55e774338962502c1d73f2d5383259aaa69f2603fde216ca9557d8b4e629888"
b"c697fec1aaf9f99ebd223c06399cc13cd21bd01e3660acc148ba841e5c89b3f8f04efac07f8072a5"
b"bacb4f5cfece528496bb35e906361efdb89a17fe4999f47508d5e48914ac651172ddc994993b4672"
b"7ec62810d6c204af4b5fd52ba4f8cb3c8720fbd469b219868e28294e60276bc2483e78d96a0edf29"
b"e237fe6f1660705d5cd3590c476e37c5d367b19bfb0a1c29ef296dfd3e9fabf5b37e1fb7357a3032"
b"c8a641b467d7090282010100bc6d55bf66ac6e69017dba38e0b38c4dc8a8055c845d9a5702b51ff8"
b"4042cbd1298f0201cf70b7d75b634d247aed92e9056c72692f3c46188d190fd35647648824154c11"
b"ea54025149cbf1e224f9b1bd4007836a594117f5a0e1b62fe72037bddc38d4e231dc9fedb79ae8dd"
b"93e5602b3e6905fff02536aaf0d7b78517e4fece0b8c872ac9040d93781e9e92832604a80462ca49"
b"234fe1c3c0695061fdd9be4aaeb08447ce5c590f2250a01629586bf3e421c424c1d576ae2fa99010"
b"b7346460165ed61de8bac782d0928e4313bd59037051e6691e85e692c2a22bbaafbe555742bca7a8"
b"1fae4933e332df317b7f3551c7e91211d6a33c38c4b85a4b46d769b3028201003884497a00a4f5d6"
b"d63af9b830fe06744ff926512345ba2ce49280f4debb858799d5e4450e4798fa2251d54cbabb20d3"
b"2bf5fff5cc20d01f173b6cc467a9713ae849c11adc29f2ae90874c6e3b74eed42494d90afb7e0f31"
b"d323a23a181e4636f345af99bb371df01805b49b11186c6ec6daafcd08e5aeb99d268e05e5b65d42"
b"dd914c194841cacfaa24726594edf7e43c3f204ea8c85c9bf806a66efb097302b514773dc41324c6"
b"400f1e1b5180ed49d58cb6600fdc143a2ecf8e9ba84d8451502de890e6771181f981a9a782475aa2"
b"bb3ecbbc76503e0530e28b676a5e6585d114b63021b4c4afae82a74cadb1cbe61a7e393ff975a942"
b"1edebb531f51618902820100214d9f1efa774b9d4e0a996442c2744560c84b133045b1af9241d60f"
b"c2f82043ac169dc9496ebb5f26b5cb8a6636c57d44e06843bf1f082be42fe5933a7ab7a6878dccf3"
b"58606a9fd6984ea525fe34f9e86f7bae33e707be0dec8fbef2deed253c822f6b812e7bd8c64bc302"
b"5c9a9e58811d30981a329f7b130148b0eb2ac62cec516942f7530963edab832bd0bacf344b183b9d"
b"ba9d54535dceff640f94d79599edf8dd0c32029950ede63f2f579b0d3c9a13c04df73fec03c4bcbe"
b"ff7ecf69ba082445673a263685475b91390963e2d42705ba89ff107e96bbb7a887daa016f282f1e6"
b"bdd7b9bb14579166f8c13be876cdef07e13c6ef08ff49d4207c7c7ff"
)
sslctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
sslctx.load_cert_chain(cert, keyfile=key)
app.run(port=4443, debug=True, ssl=sslctx)
see https://github.com/orgs/micropython/discussions/12400, I think you need to update the git tags from upstream micropython
Thanks, that helped. Version number is correct now:
MicroPython v1.22.0-preview.73.g7736ab832.dirty on 2023-11-13; Generic ESP32 module with ESP32
I tried to test the hello_async_tls.py but it also fails:
import ssl
from microdot_asyncio import Microdot
from cert import cert, key
app = Microdot()
htmldoc = '''<!DOCTYPE html>
<html>
<head>
<title>Microdot Example Page</title>
<meta charset="UTF-8">
</head>
<body>
<div>
<h1>Microdot Example Page</h1>
<p>Hello from Microdot!</p>
<p><a href="/shutdown">Click to shutdown the server</a></p>
</div>
</body>
</html>
'''
@app.route('/')
async def hello(request):
return htmldoc, 200, {'Content-Type': 'text/html'}
@app.route('/shutdown')
async def shutdown(request):
request.app.shutdown()
return 'The server is shutting down...'
sslctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
# sslctx.load_cert_chain('cert.pem', 'key.pem')
sslctx.load_cert_chain(cert, keyfile=key)
app.run(port=4443, debug=True, ssl=sslctx)> ab -n 10 -c 2 -v 1 https://192.168.178.67:4443/
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.178.67 (be patient)...SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
..done
Server Software:
Server Hostname: 192.168.178.67
Server Port: 4443
Document Path: /
Document Length: 0 bytes
Concurrency Level: 2
Time taken for tests: 1.417 seconds
Complete requests: 10
Failed requests: 0
Total transferred: 0 bytes
HTML transferred: 0 bytes
Requests per second: 7.06 [#/sec] (mean)
Time per request: 283.392 [ms] (mean)
Time per request: 141.696 [ms] (mean, across all concurrent requests)
Transfer rate: 0.00 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 17 142 362.5 20 1171
Waiting: 0 0 0.0 0 0
Total: 17 142 362.5 20 1171
Percentage of the requests served within a certain time (ms)
50% 20
66% 20
75% 21
80% 87
90% 1171
95% 1171
98% 1171
99% 1171
100% 1171 (longest request)
error in micropython console:
Starting async server on 0.0.0.0:4443...
Traceback (most recent call last):
File "asyncio/stream.py", line 1, in _serve
OSError: [Errno 12] ENOMEM
Traceback (most recent call last):
File "asyncio/stream.py", line 1, in _serve
OSError: [Errno 12] ENOMEM
At the moment a can't get it to run on the esp32.
ab -n 10 -c 2 -v 1 https://192.168.178.67:4443/
At the moment a can't get it to run on the esp32.
Performance on the esp32 is limited due to memory constrains see #34, which means only one SSL socket connection at a time is possible (unless you do some tweaks in the firmware like disabling bluetooth and enabling mbedtls dynamic buffer .e.g in sdkconfig.base
# SSL
# Use 4kiB output buffer instead of default 16kiB
# CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN=y
CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=16384
CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=16384
CONFIG_MBEDTLS_DYNAMIC_BUFFER=y
)
However it should work with ab -n 1 -v3 https://192.168.178.67:4443/ or $ curl --insecure https://192.168.178.67:4443/ (the --insecure flag is to discard a potential certificate verification error since this is a common pitfall while testing self-signed certs, and it can be tricky to get it right)
Strange also
class Microdot(BaseMicrodot): starts at line 212 and the
the next line print_exception(exc) starts at 338 and not on 373 as i expected from the snippet.
Furthermore the line # print("Dispatch request now") does not exist, so it seems there are more changes in microdot_asyncio.py than shown above.
I may have an outdated microdot copy so
I've just clone microdot repo and tested your script on unix port and it works as expected.
Can you test this on the esp32 ?https://github.com/micropython/micropython/blob/7736ab832220bbad9f4816afd9f3e0e574a0fabb/tests/net_inet/asyncio_tls_open_connection.py
Can you test this on the esp32 ?https://github.com/micropython/micropython/blob/7736ab832220bbad9f4816afd9f3e0e574a0fabb/tests/net_inet/asyncio_tls_open_connection.py
That TLS ansyc client code runs fine, here is the response :
write GET
read response
read: b'HTTP/1.1 200 OK'
close
done
The hello_async_tls.py still fails after recompiling micropython with only the changes in the micropython/ports/esp32/boards/sdkconfig.base and after also after removing bluetooth.
This is how i deactivated bluetooth:
~/micropython/ports/esp32 > idf.py menuconfig
Component config ---> Bluetooth ---> [ ] Bluetooth
After that i recompiled and flashed the esp32 again, but i still get the same error:
Starting async server on 0.0.0.0:4443...
Traceback (most recent call last):
File "asyncio/stream.py", line 1, in _serve
OSError: [Errno 12] ENOMEM
I'm not sure if everything compiled correctly.
With only changes to sdkconfig.base :
>>> import micropython
>>> micropython.mem_info()
stack: 720 out of 15360
GC: total: 64000, used: 1056, free: 62944, max new split: 53248
No. of 1-blocks: 15, 2-blocks: 5, max blk sz: 18, max free sz: 3923
without bluetooth:
>>> micropython.mem_info()
stack: 720 out of 15360
GC: total: 64000, used: 1392, free: 62608, max new split: 110592
No. of 1-blocks: 22, 2-blocks: 6, max blk sz: 18, max free sz: 3902
That TLS ansyc client code runs fine, here is the response :
👍🏼
OSError: [Errno 12] ENOMEM
Well then it seems a memory problem from mbedtls memory, what I would try next is using EC key/cert instead of RSA, see
https://github.com/JustinS-B/Mosquitto_CA_and_Certs, https://github.com/orgs/micropython/discussions/10559, micropython/micropython#5543,
TLDR: RSA keys are larger .e.g 4096 bits vs EC keys 256 bits, which makes the SSL handshake more resource intensive.
I would also try to freeze microdot in the firmware see https://docs.micropython.org/en/latest/reference/manifest.html
and compile hello_async_tls.py into bytecode with mpy-cross https://docs.micropython.org/en/latest/reference/mpyfiles.html
Thanks! That helped a lot. Its working now.
Changing the algorithm to EC did not help. Still the same error.
But freezing microdot freed enough memory so that hello_async_tls.py and echo_async_tls.py run without problems (except from the negativ OSError messages).
For documentation for those who want to do the same.
I cloned the micropython repo and put the micodot py files in the micropython/ports/esp32/modules folder. All files in that folder are automatically frozen when compiling, according to the manifest.py file.
How would one connect with micropython to a websocket as a client? To the example server mentioned above or another websocket server.
I guess this should not be too complicated since the server with websockets works.
Nervertheless I'm struggling to adapt the examples i found.
Any ideas?
@wohltat see micropython/micropython-lib#752, I'm about to add WebSocket support to that, but in the meantime checkout my fork/branch at https://github.com/Carglglz/micropython-lib/tree/aiohttp-websocket/micropython/uaiohttpclient
@wohltat see micropython/micropython-lib#752, I'm about to add WebSocket support to that, but in the meantime checkout my fork/branch at https://github.com/Carglglz/micropython-lib/tree/aiohttp-websocket/micropython/uaiohttpclient
The clientsession_ws_example.py example works but if I use URL = "wss://echo.websocket.events" then i get a core panic:
Guru Meditation Error: Core 1 panic'ed (StoreProhibited). Exception was unhandled.
Core 1 register dump:
PC : 0x4009cba9 PS : 0x00060433 A0 : 0x8009c75b A1 : 0x3ffce670
A2 : 0x1345815f A3 : 0x00000001 A4 : 0x00004000 A5 : 0xffffffff
A6 : 0xfffffffa A7 : 0x00060423 A8 : 0x3ffeb8b0 A9 : 0x0e6f968f
A10 : 0x00000054 A11 : 0x00000004 A12 : 0x3ffe4364 A13 : 0x3ffe4388
A14 : 0x3ffe43b8 A15 : 0x0000000e SAR : 0x0000001b EXCCAUSE: 0x0000001d
EXCVADDR: 0x1345816b LBEG : 0x400834ad LEND : 0x400834ba LCOUNT : 0x00000028
Backtrace: 0x4009cba6:0x3ffce670 0x4009c758:0x3ffce690 0x40082080:0x3ffce6b0 0x400820d2:0x3ffce6d0 0x4009d342:0x3ffce6f0 0x400d611e:0x3ffce710 0x400d8ef0:0x3ffce790 0x400f1059:0x3ffce7e0 0x400f0ba5:0x3ffce800 0x400f0dcd:0x3ffce840 0x400f0e46:0x3ffce870 0x400df893:0x3ffce890 0x400e6d81:0x3ffce8c0 0x400e6e45:0x3ffce8e0 0x40085405:0x3ffce900 0x400df824:0x3ffce9a0 0x400e6d81:0x3ffce9d0 0x400e6e45:0x3ffce9f0 0x40085405:0x3ffcea10 0x400df824:0x3ffceab0 0x400e6d81:0x3ffceae0 0x40085355:0x3ffceb00 0x400df824:0x3ffceba0 0x400e6d81:0x3ffcec00 0x400e6e45:0x3ffcec20 0x40085405:0x3ffcec40 0x400df824:0x3ffcece0 0x400e6d81:0x3ffced50 0x400e6d96:0x3ffced70 0x400f601e:0x3ffced90 0x400f61b8:0x3ffcee20 0x400d7bd2:0x3ffcee70
ELF file SHA256: bd33ef349408122b
Rebooting...
ets Jun 8 2016 00:22:57
rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:4728
load:0x40078000,len:14876
ho 0 tail 12 room 4
load:0x40080400,len:3368
entry 0x400805cc
MicroPython v1.22.0-preview.73.g7736ab832.dirty on 2023-11-18; Generic ESP32 module with ESP32
Type "help()" for more information.
Update:
I tried it around 30 times and It crashes in different places. Mostly during the handshake, sometimes after sending and sometimes even after receiving the echo message.
Happens with both, compiled and .py version.
There are different crash messages. Sometimes there is:
Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
Seems kind of random. Could this be connected to the negative OSError messages maybe?
The clientsession_ws_example.py example works but if I use URL = "wss://echo.websocket.events" then i get a core panic:
I can reproduce it, sometimes it works but then the core panic appears anyway. 🤷🏼♂️
So far I've tested:
- unix port
ws:andwss:works OK - esp32 port
ws:works OK andwss:core panic'ed
So something may be wrong with my aiohttpclient-websocket implementation, because echo_async_tls.py does work...
FYI I've implemented it following https://github.com/miguelgrinberg/microdot/blob/main/src/microdot_asyncio_test_client.py,
and https://github.com/danni/uwebsockets,
and there is also https://github.com/marcidy/micropython-uasyncio-websockets in case you want to test it.
If you find out something let me know 👀
@wohltat I've just decoded the backtrace and it seems related to this option
CONFIG_MBEDTLS_DYNAMIC_BUFFER=y, so you may want to disable that and see if it works, but this may limit SSL connections to only one at a time...
Great that seems to work. Since i only need one client connection anyways this is probably not a problem.
Does it mean there is only one SSL connection in total or can i use microdot tls ws server in parallel?
Does it mean there is only one SSL connection in total or can i use microdot tls ws server in parallel?
I meant only one active SSL connection at a time but
I haven't tested this with the latest firmware (there were some improvements in esp32 port related to this recently)
so just test it and let me know if that works 👍🏼
See micropython/micropython#11897. Once that is merged, I'll have a look and see if I need to make any changes on my side.
I think this can be closed now (unless you're waiting for v1.22 release).
Also @wohltat in case you're interested an aiohttp (client only) MicroPython implementation is now available at https://github.com/micropython/micropython-lib/tree/master/python-ecosys/aiohttp 👍🏼
@Carglglz Yes, this ticket is a reminder to make sure my example app works fine once 1.22 is released. Thanks!
Closing this, after verifying that TLS on MicroPython 1.22 works when using latest asyncio code. 🎉