netcan/asyncio

heap-use-after-free

Closed this issue · 4 comments

awen162 commented

环境

Ubuntu 18.04.4 LTS
gcc 11.1.0

测试内容

在echo_server中，在write之前加300ms的sleep时间。
在echo_client中，创建多个连接（测试了17个）同时发送给echo_server，并且使用wait_for进行超时控制，超时是300ms。
客户端和服务端的时间300ms，是特意设置的。

测试结果

偶尔会有成功，偶尔会有超时，偶尔会产生heap-use-after-free的coredump。重复测试多次，是会复现问题的。

请帮忙看看是哪里导致了内存释放了，还在被使用。

测试代码

server

#include <asyncio/runner.h>
#include <asyncio/start_server.h>
#include <asyncio/sleep.h>
#include <asyncio/wait_for.h>
#include <arpa/inet.h>

using namespace std::chrono;
using asyncio::Task;
using asyncio::Stream;
using asyncio::get_in_addr;
using asyncio::get_in_port;

Task<> handle_echo(Stream stream) {
    auto& sockinfo = stream.get_sock_info();
    auto sa = reinterpret_cast<const sockaddr*>(&sockinfo);
    char addr[INET6_ADDRSTRLEN] {};

    auto data = co_await stream.read(100);
    fmt::print("Received: '{}' from '{}:{}'\n", data.data(),
               inet_ntop(sockinfo.ss_family, get_in_addr(sa), addr, sizeof addr),
               get_in_port(sa));

    co_await asyncio::sleep(300ms);
    fmt::print("Send: '{}'\n", data.data());
    co_await stream.write(data);

    fmt::print("Close the connection\n");
    stream.close();
}

Task<void> amain() {
    auto server = co_await asyncio::start_server(
            handle_echo, "127.0.0.1", 8888);

    fmt::print("Serving on 127.0.0.1:8888\n");

    co_await server.serve_forever();
}

int main() {
    asyncio::run(amain());
    return 0;
}

client

#include <iostream>
#include <asyncio/open_connection.h>
#include <asyncio/runner.h>
#include <asyncio/sleep.h>
#include <asyncio/schedule_task.h>
#include <asyncio/wait_for.h>

using namespace asyncio;
using namespace std::chrono;

Task<> tcp_echo_client(std::string_view message) {
        auto stream = co_await asyncio::open_connection("127.0.0.1", 8888);

        fmt::print("Send: '{}'\n", message);
        co_await stream.write(Stream::Buffer(message.begin(), message.end() + 1));

        fmt::print("Send: ok\n");

        auto data = co_await asyncio::wait_for(stream.read(100), 300ms);

        fmt::print("Received: '{}'\n", data.data());

        fmt::print("Close the connection\n");
        stream.close();
}

int main(int argc, char** argv) {

    std::vector<ScheduledTask<Task<>>> tasks;
    for (size_t i = 0; i < 16; ++i)
    {
        tasks.emplace_back(tcp_echo_client("hello world! 1"));
    }
    
    try
    {
        asyncio::run(tcp_echo_client("hello world!"));
    }
    catch(const std::exception& e)
    {
        std::cerr << e.what() << '\n';
    }
    printf("end ....\n");

    return 0;
}

coredump的输出

./echo_client 
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world! 1'
Send: 'hello world!'
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
Send: ok
=================================================================
==26685==ERROR: AddressSanitizer: heap-use-after-free on address 0x612000002a60 at pc 0x5580ce2d05a6 bp 0x7ffc3359c220 sp 0x7ffc3359c210
WRITE of size 1 at 0x612000002a60 thread T0
    #0 0x5580ce2d05a5 in asyncio::Handle::set_state(asyncio::Handle::State) /home/ubuntu/c++/asyncio/include/asyncio/handle.h:25
    #1 0x5580ce2f1e3d in asyncio::EventLoop::run_once() /home/ubuntu/c++/asyncio/src/event_loop.cpp:62
    #2 0x5580ce2f11f7 in asyncio::EventLoop::run_until_complete() /home/ubuntu/c++/asyncio/src/event_loop.cpp:17
    #3 0x5580ce2d4fbe in decltype(auto) asyncio::run<asyncio::Task<void> >(asyncio::Task<void>&&) (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0xf3fbe)
    #4 0x5580ce2cda06 in main /home/ubuntu/c++/asyncio/test/st/echo_client.cpp:40
    #5 0x7fe6a576abf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
    #6 0x5580ce1f1d49 in _start (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0x10d49)

0x612000002a60 is located 32 bytes inside of 312-byte region [0x612000002a40,0x612000002b78)
freed by thread T0 here:
    #0 0x5580ce282f47 in operator delete(void*) (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0xa1f47)
    #1 0x5580ce2c68a1 in asyncio::Stream::read(long) [clone .actor] /home/ubuntu/c++/asyncio/include/asyncio/stream.h:46
    #2 0x5580ce2c6b29 in asyncio::Stream::read(long) [clone .destroy] /home/ubuntu/c++/asyncio/include/asyncio/stream.h:34
    #3 0x5580ce2e0ee3 in std::__n4861::coroutine_handle<asyncio::Task<std::vector<char, std::allocator<char> > >::promise_type>::destroy() const (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0xffee3)
    #4 0x5580ce2dbd41 in asyncio::Task<std::vector<char, std::allocator<char> > >::destroy() (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0xfad41)
    #5 0x5580ce2d5f81 in asyncio::Task<std::vector<char, std::allocator<char> > >::~Task() (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0xf4f81)
    #6 0x5580ce2dee4f in asyncio::detail::WaitForAwaiterRegistry<asyncio::Task<std::vector<char, std::allocator<char> > >, std::chrono::duration<long, std::ratio<1l, 1000l> > >::~WaitForAwaiterRegistry() (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0xfde4f)
    #7 0x5580ce2ce5c8 in asyncio::Task<decltype ((((declval<asyncio::detail::GetAwaiter<asyncio::Task<std::vector<char, std::allocator<char> > > >::type>)()).await_resume)())> asyncio::detail::wait_for<asyncio::Task<std::vector<char, std::allocator<char> > >, long, std::ratio<1l, 1000l> >(asyncio::NoWaitAtInitialSuspend, asyncio::Task<std::vector<char, std::allocator<char> > >&&, std::chrono::duration<long, std::ratio<1l, 1000l> >) [clone .actor] /home/ubuntu/c++/asyncio/include/asyncio/wait_for.h:101
    #8 0x5580ce2f0d93 in std::__n4861::coroutine_handle<asyncio::Task<std::vector<char, std::allocator<char> > >::promise_type>::resume() const (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0x10fd93)
    #9 0x5580ce2ef90b in asyncio::Task<std::vector<char, std::allocator<char> > >::promise_type::run() (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0x10e90b)
    #10 0x5580ce2f1eb5 in asyncio::EventLoop::run_once() /home/ubuntu/c++/asyncio/src/event_loop.cpp:63
    #11 0x5580ce2f11f7 in asyncio::EventLoop::run_until_complete() /home/ubuntu/c++/asyncio/src/event_loop.cpp:17
    #12 0x5580ce2d4fbe in decltype(auto) asyncio::run<asyncio::Task<void> >(asyncio::Task<void>&&) (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0xf3fbe)
    #13 0x5580ce2cda06 in main /home/ubuntu/c++/asyncio/test/st/echo_client.cpp:40
    #14 0x7fe6a576abf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)

previously allocated by thread T0 here:
    #0 0x5580ce282427 in operator new(unsigned long) (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0xa1427)
    #1 0x5580ce2d3590 in asyncio::Stream::read(long) (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0xf2590)
    #2 0x5580ce2ccb2e in tcp_echo_client(std::basic_string_view<char, std::char_traits<char> >) [clone .actor] /home/ubuntu/c++/asyncio/test/st/echo_client.cpp:28
    #3 0x5580ce2f0df3 in std::__n4861::coroutine_handle<asyncio::Task<void>::promise_type>::resume() const (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0x10fdf3)
    #4 0x5580ce2f0063 in asyncio::Task<void>::promise_type::run() (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0x10f063)
    #5 0x5580ce2f1eb5 in asyncio::EventLoop::run_once() /home/ubuntu/c++/asyncio/src/event_loop.cpp:63
    #6 0x5580ce2f11f7 in asyncio::EventLoop::run_until_complete() /home/ubuntu/c++/asyncio/src/event_loop.cpp:17
    #7 0x5580ce2d4fbe in decltype(auto) asyncio::run<asyncio::Task<void> >(asyncio::Task<void>&&) (/home/ubuntu/c++/asyncio/build/test/st/echo_client+0xf3fbe)
    #8 0x5580ce2cda06 in main /home/ubuntu/c++/asyncio/test/st/echo_client.cpp:40
    #9 0x7fe6a576abf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)

SUMMARY: AddressSanitizer: heap-use-after-free /home/ubuntu/c++/asyncio/include/asyncio/handle.h:25 in asyncio::Handle::set_state(asyncio::Handle::State)
Shadow bytes around the buggy address:
  0x0c247fff84f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c247fff8500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
  0x0c247fff8510: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c247fff8520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c247fff8530: 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa
=>0x0c247fff8540: fa fa fa fa fa fa fa fa fd fd fd fd[fd]fd fd fd
  0x0c247fff8550: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c247fff8560: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
  0x0c247fff8570: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c247fff8580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c247fff8590: 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==26685==ABORTING
awen162 commented

看到handle.h头文件中有几行注释，可能是跟这个问题有关。

// handle maybe destroyed, using the increasing id to track the lifetime of handle.
// don't directly using a raw pointer to track coroutine lifetime,
// because a destroyed coroutine may has the same address as a new ready coroutine has created.

现在的问题，也是handle被free了，后面还被使用到了。

netcan commented

看到handle.h头文件中有几行注释，可能是跟这个问题有关。

// handle maybe destroyed, using the increasing id to track the lifetime of handle.
// don't directly using a raw pointer to track coroutine lifetime,
// because a destroyed coroutine may has the same address as a new ready coroutine has created.

现在的问题，也是handle被free了，后面还被使用到了。

是的，vector析构后将task也给析构了。可以试试在最后，同步等待所有task结束。

asyncio::run([&] -> Task<> {
   for (auto& task: tasks) {
      co_await task;
   }
});
awen162 commented

client使用下面的代码测试了下，还是会偶发出现coredump，问题没有得到解决。
看coredump前的输出，end .... 这个没有打印出来，说明vector还是没有析构的，不知道是哪里析构了。

int main(int argc, char** argv) {
    std::vector<ScheduledTask<Task<>>> tasks;
    for (size_t i = 0; i < 17; ++i)
    {
        tasks.emplace_back(tcp_echo_client("hello world!"));
    }
    asyncio::run([&]() -> Task<> {
        for (auto& t : tasks) {
            co_await t;
        }
    }());
   printf("end ....\n");
    return 0;
}
netcan commented

client使用下面的代码测试了下，还是会偶发出现coredump，问题没有得到解决。 看coredump前的输出，end .... 这个没有打印出来，说明vector还是没有析构的，不知道是哪里析构了。

int main(int argc, char** argv) {
    std::vector<ScheduledTask<Task<>>> tasks;
    for (size_t i = 0; i < 17; ++i)
    {
        tasks.emplace_back(tcp_echo_client("hello world!"));
    }
    asyncio::run([&]() -> Task<> {
        for (auto& t : tasks) {
            co_await t;
        }
    }());
   printf("end ....\n");
    return 0;
}

已解决，看来是某个task抛异常析构的时候，没有正确的cancel掉。

  • 👍1