ntop/nDPI

Wireshark reports an error after linking the nDPI

SuperBigYB opened this issue · 10 comments

SuperBigYB commented

Describe the bug

The epan_init() function of wireshark reports an error after linking nDPI:
Dissector bug: epan/dissectors/packet-wireguard.c:416: failed assertion "DISSECTOR_ASSERT_NOT_REACHED"

nDPI Environment (please complete the following information):

  • OS name: Ubuntu.
  • OS version: 22.04
  • Architecture: amd64
  • nDPI version or commit hash: 4.8-stable.
  • nDPI compilation flags used: none.

How to reproduce the reported bug

Before linking nDPI, my simple example of using libwireshark-dev works properly But after the nDPI is linked, the epan_init() function of wireshark reports an error:
Distributor bug: epan/transmitters/packet wireguard. c: 416: failed assertion "DISSECTOR_ASSERT_NOT_RECHED"

Additional context

#include <atomic>
#include <cfile.h>
#include <epan/column.h>
#include <epan/epan.h>
#include <epan/epan_dissect.h>
#include <epan/print.h>
#include <epan/print_stream.h>
#include <epan/timestamp.h>
#include <iostream>
#include <ndpi/ndpi_api.h>
#include <pcap/pcap.h>
#include <wsutil/privileges.h>
#include <wsutil/report_message.h>
#include <wsutil/wslog.h>

struct PcapPacketHeader {
    struct {
        uint32_t tv_sec;
        uint32_t tv_usec;
    } ts;
    uint32_t caplen;
    uint32_t len;
};

static void ws_perror(const char *format, va_list ap) {
    vfprintf(stderr, format, ap);
    printf("\n");
}

static const nstime_t *raw_get_frame_ts(packet_provider_data *prov, guint32 frame_num) {
    if (prov->frames) {
        auto frame = frame_data_sequence_find(prov->frames, frame_num);
        if (frame) {
            return &frame->abs_ts;
        }
    }
    return nullptr;
}

class CWireShark {
public:
    CWireShark() {
        m_cf.snap = WTAP_MAX_PACKET_SIZE_STANDARD;
    }

    ~CWireShark() {
        cleanup();
    }

    bool init() {
        ws_log_init("ws_test", ws_perror);
        ws_log_set_level(LOG_LEVEL_NOISY);

        report_message_routines rmr{};
        rmr.vreport_failure = ws_perror;
        init_report_message("AnaTraf", &rmr);

        timestamp_set_type(TS_RELATIVE);
        timestamp_set_precision(TS_PREC_AUTO);
        timestamp_set_seconds_type(TS_SECONDS_DEFAULT);

        init_process_policies();

        wtap_init(false);

        auto ret = epan_init(nullptr, nullptr, false);
        if (!ret) {
            printf("epan_init error");
            printf("\n");
            return false;
        }

        packet_provider_funcs ppf{};
        ppf.get_frame_ts = raw_get_frame_ts;
        ppf.get_interface_description = nullptr;
        ppf.get_interface_name = nullptr;
        ppf.get_modified_block = nullptr;

        m_cf.epan = epan_new(&m_cf.provider, &ppf);

        m_cf.provider.frames = new_frame_data_sequence();

        auto *ep = epan_load_settings();
        if (!ep) {
            printf("epan_load_settings error");
            printf("\n");
            return false;
        }

        prefs_apply_all();

        build_column_format_array(&m_cf.cinfo, ep->num_cols, true);

        m_cf.cinfo.epan = m_cf.epan;

        return true;
    }

    void cleanup() {
        col_cleanup(&m_cf.cinfo);

        if (m_cf.provider.frames) {
            free_frame_data_sequence(m_cf.provider.frames);
            m_cf.provider.frames = nullptr;
        }

        if (m_cf.provider.wth) {
            wtap_close(m_cf.provider.wth);
            m_cf.provider.wth = nullptr;
        }

        if (m_cf.epan) {
            epan_free(m_cf.epan);
            m_cf.epan = nullptr;
        }

        //    epan_cleanup();

        wtap_cleanup();
    }

    epan_dissect_t *dissect_packet(const PcapPacketHeader &pcap_hdr, const uint8_t *data, uint32_t pkt_id) {
        wtap_rec wr{};
        wtap_rec_init(&wr);
        wr.rec_header.packet_header.pkt_encap = WTAP_ENCAP_ETHERNET;
        wr.rec_header.packet_header.len = pcap_hdr.len;
        wr.rec_header.packet_header.caplen = pcap_hdr.caplen;
        wr.tsprec = WTAP_TSPREC_USEC;
        wr.ts.secs = pcap_hdr.ts.tv_sec;
        wr.ts.nsecs = (int) pcap_hdr.ts.tv_usec * 1000;

        frame_data frame{};
        frame_data_init(&frame, pkt_id, &wr, 0, 0);
        frame.has_ts = true;
        frame.prev_dis_num = pkt_id - 1;

        auto edt = epan_dissect_new(m_cf.epan, true, true);

        frame_data_set_before_dissect(&frame, &m_cf.elapsed_time, &m_cf.provider.ref, m_cf.provider.prev_dis);

        frame_data_sequence_add(m_cf.provider.frames, &frame);

        frame.frame_ref_num = 1;

        auto tvb = tvb_new_real_data(data, pcap_hdr.caplen, (int) pcap_hdr.caplen);

        epan_dissect_run(edt, m_cf.cd_t, &wr, tvb, &frame, &m_cf.cinfo);

        m_cf.provider.prev_dis = frame_data_sequence_find(m_cf.provider.frames, pkt_id);

        epan_dissect_fill_in_columns(edt, true, true);

        frame_data_destroy(&frame);

        wtap_rec_cleanup(&wr);

        return edt;
    }

    capture_file m_cf{};
};

uint32_t pkt_id = 1;

CWireShark ws;

pcap_t *sniffer = nullptr;

void handle_pkt(uint8_t *arg, const pcap_pkthdr *pkthdr, const uint8_t *data) {
    PcapPacketHeader pcap_hdr{};
    pcap_hdr.ts.tv_sec = pkthdr->ts.tv_sec;
    pcap_hdr.ts.tv_usec = pkthdr->ts.tv_usec;
    pcap_hdr.caplen = pkthdr->caplen;
    pcap_hdr.len = pkthdr->len;

    auto edt = ws.dissect_packet(pcap_hdr, data, pkt_id);
    if (!edt) {
        fprintf(stderr, "dissect packet error\n");
    }

    char buf[1 << 17] = {0};

    FILE *fmem = fmemopen(buf, sizeof(buf), "w+");
    if (!fmem) {
        fprintf(stderr, "switch memory to file pointer error\n");
    }

    write_psml_preamble(&ws.m_cf.cinfo, fmem);
    write_psml_columns(edt, fmem, true);
    write_psml_finale(fmem);
    fflush(fmem);
    std::cout << buf << std::endl;
    std::cout << "--------------" << std::endl;

    fclose(fmem);

    epan_dissect_free(edt);

    pkt_id++;

    std::cout << "*************" << std::endl;
}

int main(int argc, char *argv[]) {

    std::cout << "start" << std::endl;

    ws.init();

//    auto a = ndpi_detection_get_sizeof_ndpi_flow_struct(); // if open it, will crash

    char err_buf[PCAP_ERRBUF_SIZE];

    sniffer = pcap_open_offline("/root/yb/test/tcp_sp.pcap", err_buf);
    if (!sniffer) {
        std::cout << "pcap file error" << std::endl;
        exit(EXIT_FAILURE);
    }

    pcap_loop(sniffer, -1, handle_pkt, nullptr);

    pcap_close(sniffer);

    std::cout << "end" << std::endl;
}
IvanNardi commented

I don't know Wireshark code/API so well, however:

  • could you share how to compile and run your code?
  • I don't know what you are trying to achieve, but you can use ndpiReader as extcap with un-modified Wireshark and view flow classification directly from the GUI....
SuperBigYB commented

I don't know Wireshark code/API so well, however:

  • could you share how to compile and run your code?
  • I don't know what you are trying to achieve, but you can use ndpiReader as extcap with un-modified Wireshark and view flow classification directly from the GUI....

thank you for reply, you can use cmake like:

cmake_minimum_required(VERSION 3.13)
project(untitled)

add_executable(${PROJECT_NAME} main.cpp)

# include
target_include_directories(${PROJECT_NAME} PRIVATE
        /usr/include/glib-2.0
        /usr/lib/x86_64-linux-gnu/glib-2.0/include/
        /usr/local/include/wireshark/
)

target_link_libraries(${PROJECT_NAME} pcap glib-2.0 wireshark wiretap wsutil ndpi)

after apt install libwireshark-dev to run it.

utoni commented

No crashes on my machine so far. If you're using a debian based distro, you may also try /usr/include/wireshark/ instead of /usr/local/include/wireshark/. That should be the directory where the package libwireshark-dev places it's header files.

utoni commented

It crashes from time to time, but only with -O0 (tried also: -Og, -Os, -O1, -O2, -O3):

AddressSanitizer:DEADLYSIGNAL
=================================================================
==36830==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000013 (pc 0x7f775670ba87 bp 0x614000000240 sp 0x7ffdd555f1e0 T0)
==36830==The signal is caused by a READ memory access.
==36830==Hint: address points to the zero page.
    #0 0x7f775670ba87 in write_psml_columns (/lib/x86_64-linux-gnu/libwireshark.so.16+0x230ba87)
    #1 0x564e69bc7ae4 in handle_pkt(unsigned char*, pcap_pkthdr const*, unsigned char const*) /home/toni/git/ndpi-wireshark/main.cpp:190
    #2 0x7f775c64a909  (/lib/x86_64-linux-gnu/libpcap.so.0.8+0x24909)
    #3 0x7f775c635cee in pcap_loop (/lib/x86_64-linux-gnu/libpcap.so.0.8+0xfcee)
    #4 0x564e69bc7d67 in main /home/toni/git/ndpi-wireshark/main.cpp:221
    #5 0x7f7753c46249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #6 0x7f7753c46304 in __libc_start_main_impl ../csu/libc-start.c:360
    #7 0x564e69bc74a0 in _start (/home/toni/git/ndpi-wireshark/main+0x34a0)
utoni commented

This is most likely not related to nDPI.
But to be sure: please try adding -fsanitize=address to your CFLAGS and upload any crash traces here.

SuperBigYB commented

This is most likely not related to nDPI. But to be sure: please try adding to your and upload any crash traces here.-fsanitize=address``CFLAGS

Thank you for your reply.

I modified and simplified the code according to your suggestions:
ws.c:

#include <epan/epan.h>
#include <epan/timestamp.h>
#include <ndpi/ndpi_api.h>
#include <stdio.h>
#include <wsutil/privileges.h>
#include <wsutil/report_message.h>
#include <wsutil/wslog.h>

static void ws_perror(const char *format, va_list ap) {
    vfprintf(stderr, format, ap);
}

int main() {
    printf("start\n");

    ws_log_init("ws_test", ws_perror);
    ws_log_set_level(LOG_LEVEL_NOISY);

    struct report_message_routines rmr;
    rmr.vreport_failure = ws_perror;
    init_report_message("ws_test", &rmr);

    timestamp_set_type(TS_RELATIVE);
    timestamp_set_precision(TS_PREC_AUTO);
    timestamp_set_seconds_type(TS_SECONDS_DEFAULT);

    init_process_policies();

    wtap_init(false);

    int ret = epan_init(NULL, NULL, false);
    if (ret) {
        printf("\nsuccess\n");
        exit(EXIT_SUCCESS);
    }else {
        printf("\nfail\n");
        exit(EXIT_FAILURE);
    }
}

CMakeLists.txt:

cmake_minimum_required(VERSION 3.13)
project(untitled C)

add_executable(${PROJECT_NAME} ws.c)

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=address")

# include
target_include_directories(${PROJECT_NAME} PRIVATE
        /usr/include/glib-2.0
        /usr/lib/x86_64-linux-gnu/glib-2.0/include/
        /usr/include/wireshark/
)

target_link_libraries(${PROJECT_NAME}
        glib-2.0 wireshark wiretap wsutil
#        ndpi  # if add ndpi, epan_init() will fail
)

When I link nDPI in CMakeLists.txt, Wireshark's epan_init() will fail.

IvanNardi commented

@SuperBigYB

I am able to reproduce the issue with your latest snapshot.

My 2 cents: wireshark code is linking to nDPI internal implementation of gcry_md_open() instead of the standard one in in libgcrypt.
Try to:

  • compile nDPI with --local-libgcrypt flag
  • link your code to that version of nDPI and to libgcrypt
SuperBigYB commented

@SuperBigYB

I am able to reproduce the issue with your latest snapshot.

My 2 cents: wireshark code is linking to nDPI internal implementation of gcry_md_open() instead of the standard one in in libgcrypt. Try to:

  • compile nDPI with --local-libgcrypt flag
  • link your code to that version of nDPI and to libgcrypt

@IvanNardi
thanks,
According to your answer, I used --with-local-libgcrypt to test successfully.
Could you please tell me what impact adding this flag will have on nDPI?

IvanNardi commented

@IvanNardi thanks, According to your answer, I used --with-local-libgcrypt to test successfully. Could you please tell me what impact adding this flag will have on nDPI?

Short answer: non real differences: nDPI has the same capabilities in both cases, and, AFAIK, the same performance.

nDPI needs some low-level cryptographic algorithms: it can get them or from an internal, simplified, implementation (default) or via the external libgcrypt library (--with-local-libgcrypt).
For some users, in some constraint environments, it is important to not have (another) external dependency.

BTW, libgcrypt is the same library used by Wireshark...

  • 👍1
SuperBigYB commented

谢谢，根据你的回答，我曾经测试成功。您能告诉我添加此标志将对 nDPI 产生什么影响吗？--with-local-libgcrypt

简短的回答：非实际差异：nDPI 在两种情况下具有相同的功能，并且 AFAIK 具有相同的性能。

nDPI 需要一些低级加密算法：它可以从内部简化的实现（默认）或通过外部 libgcrypt 库 （） 获取它们。对于某些用户来说，在某些约束环境中，没有（另一个）外部依赖关系是很重要的。--with-local-libgcrypt

顺便说一句，libgcrypt 与 Wireshark 使用的库相同......

@IvanNardi
Thank you very much.
Your answer perfectly solved my problem.

  • 🎉1