UART splits packets on every 0xFF byte.

Question

UART splits packets on every 0xFF byte.

Closed this issue 12 days ago · 25 comments

hitecSmartHome commented 13 days ago

Board

ESP32-Wrover-E

Device Description

PSRAM
FLASH
SPI

Hardware Configuration

Version

latest master (checkout manually)

IDE Name

VsCode

Operating System

Windows 10

Flash frequency

80

PSRAM enabled

yes

Upload speed

115200

Description

When the ESP32 receives a frame which has 0xFF in it, it will break the packet into multiple packets.
We measured with oscilloscope and there is no byte timeout, the bytes are continous.
So a single continous packet which has 3pcs 0xFF byte in it, it will be split into 3 different packets.

Sketch

#define MBUS_BAUD 115200
#define MBUS_RX 35
#define MBUS_TX 32
#define MBUS_RTS 33

#define MBUS_RX_TIMEOUT 1 // Can't set it higher if UART_MODE_RS485_HALF_DUPLEX
#define MAX_MBUS_DATA_LENGTH 256
#define MAX_RX_BUFFER_SIZE 256

void Modbus::init() {
    Serial1.setRxBufferSize(MAX_RX_BUFFER_SIZE);
    Serial1.setTxBufferSize(MAX_MBUS_DATA_LENGTH);
    Serial1.begin(MBUS_BAUD, SERIAL_8N1, MBUS_RX, MBUS_TX);
    Serial1.setPins(-1, -1, -1, MBUS_RTS);
    Serial1.setMode(UART_MODE_RS485_HALF_DUPLEX);
    Serial1.setRxTimeout(MBUS_RX_TIMEOUT);

    Serial1.onReceive(
        std::bind(&Modbus::handlePacket, this),
        PACKET_TRIGGER_ONLY_ON_TIMEOUT
    );
    Serial1.onReceiveError(
        std::bind(&Modbus::handleReceiveError, this, std::placeholders::_1)
    );
}

void Modbus::handlePacket() {
    int available = Serial1.available();

    uint8_t rawPacket[available] = {0};
    int read = Serial1.readBytes(rawPacket, available);

    utils.printRawPacket("UART packet: ", rawPacket, read);
}

void Utils::printRawPacket(const char* title, uint8_t* data, int length) {
    printf("%s: ", title);
    for (int i = 0; i < length; i++) {
        printf("0x%02x ", data[i]);
    }
    printf("\n");
}

Expected single packet:

0x04 0x03 0x40 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0x00 0x00 0x00 0x1b 0x00 0x00 0x00 0x0b 0x00 0x01 0x00 0x1b 0x00 0x00 0x00 0x35 0x00 0x00 0x00 0x1c 0x00 0x00 0x00 0x1f 0x00 0x00 0x00 0x1f 0x00 0x00 0x00 0x27 0x00 0x00 0x00 0x25 0x00 0x00 0x00 0x83 0x00 0x00 0x00 0x58 0x00 0x00 0x94 0x20 0x00 0x04 0xff 0x3d 0xff 0xff 0xeb 0x35

Received packets instead.

0x04 0x03 0x40

0xff 0xff 0xff

0xff 0x00 0x00 0x00 0x1b 0x00 0x00 0x00 0x0b 0x00 0x01 0x00 0x1b 0x00 0x00 0x00 0x35 0x00 0x00 0x00 0x1c 0x00 0x00 0x00 0x1f 0x00 0x00 0x00 0x1f 0x00 0x00 0x00 0x27 0x00 0x00 0x00 0x25 0x00 0x00 0x00 0x83 0x00 0x00 0x00 0x58 0x00 0x00 0x94 0x20 0x00 0x04

0xff 0x3d 0xff 0xff 0xeb 0x35

I have checked existing issues, online documentation and the Troubleshooting Guide

I confirm I have checked existing issues, online documentation and Troubleshooting guide.

Answer 1 · 2025-03-17T16:01:04.000Z

@hitecSmartHome - a couple questions:

What is the value of PACKET_TRIGGER_ONLY_ON_TIMEOUT?
Could you please set CORE_DEBUG_LEVEL=5 to get a Verbose Log output. (-DCORE_DEBUG_LEVEL=5 to the compiler options)
The issue sketch uses some external classes. Please make it just Arduino plain API code in order to allow us to test it.
Confirm what Arduino Core version is being used for this report.

Answer 2 · 2025-03-17T18:36:02.000Z

Sorry for not being more precise. I'm using ESP-IDF v5.3.0.240821 and I don't know the exact arduino version for now. Will have to check this out.

PACKET_TRIGGER_ONLY_ON_TIMEOUT is true. I'm waiting for packets and not bytes.

#define MBUS_BAUD 115200
#define MBUS_RX 35
#define MBUS_TX 32
#define MBUS_RTS 33

#define PACKET_TRIGGER_ONLY_ON_TIMEOUT 1
#define MBUS_RX_TIMEOUT 1 // Can't set it higher if UART_MODE_RS485_HALF_DUPLEX
#define MAX_MBUS_DATA_LENGTH 256
#define MAX_RX_BUFFER_SIZE 256

void setup(){
    Serial1.setRxBufferSize(MAX_RX_BUFFER_SIZE);
    Serial1.setTxBufferSize(MAX_MBUS_DATA_LENGTH);
    Serial1.begin(MBUS_BAUD, SERIAL_8N1, MBUS_RX, MBUS_TX);
    Serial1.setPins(-1, -1, -1, MBUS_RTS);
    Serial1.setMode(UART_MODE_RS485_HALF_DUPLEX);
    Serial1.setRxTimeout(MBUS_RX_TIMEOUT);

    Serial1.onReceive([](){
        int available = Serial1.available();

        uint8_t rawPacket[available] = {0};
        int read = Serial1.readBytes(rawPacket, available);

        printf("Raw uart data: ");
        for (int i = 0; i < length; i++) {
            printf("0x%02x ", data[i]);
        }
        printf("\n");

    },PACKET_TRIGGER_ONLY_ON_TIMEOUT);
}

If I throw any frame ( valid or invalid modbus frame it does not matter ) and if it has any 0xff it will split it into two packets. But it will split only when there are more bytes after a 0xff byte. So for example 0x00 0x02 0xff will not be a separate packet but 0x00 0x02 0xff 0x33 0x21 will be a separate packet like 0x00 0x02 and 0xff 0x33 0x21 We measured it with oscilloscope directly on the esp32's receiver pin and the data does not have any break or timeout. If you need I can take a picture of the oscilloscope's display.

Answer 3 · 2025-03-17T18:43:56.000Z

@hitecSmartHome - I have modified the original Sketch and I am able to reproduce the reported issue.

I have found out that when the UART is set using EVEN or ODD parity bit, it works correctly.
The issue seems related to the IDF UART driver of the ESP32 that for some reason gets a BREAK after a specific incoming sequence of bits.

Answer 4 · 2025-03-17T18:46:53.000Z

It seems that 0xff + some other byte will trigger a BREAK. This seems to interfere with the IDF UART Driver and it causes the package to be broken into pieces when it is read using "onReceive()".

I'll try to figure out a way to create a work around for it..

Answer 5 · 2025-03-17T18:48:48.000Z

I did not see any correlation to the sequence of bits or bytes. I observed only this 0xFF break only. But thank you for the response, I have tried out some configuration options but nothing helps so far. Thank you very much for your time!

Answer 6 · 2025-03-17T19:18:59.000Z

@hitecSmartHome - Not sure if this information may help within your project, but I also have found out that this 0xFF issue only happens with the ESP32. It doesn't happen with the ESP32-S2, S3, C3 and C6.

Therefore, I think that it may be an ESP32 internal hardware design problem that has been solved latter.
Using another SoC may solve this specific issue in a product.

Answer 7 · 2025-03-17T19:51:49.000Z

Lastly, I also verified that when MBUS_RX_TIMEOUT is 2 or higher, the issue doesn't happen.
Therefore, it seem linked to the ESP32 SoC and the settings used. It won't be fixed with software change.

It seems that there is no way to fix it, unless using ODD/EVEN parity or a RX_TIMEOUT higher than 1.

Answer 8 · 2025-03-17T19:52:54.000Z

@hitecSmartHome - Not sure if this information may help within your project, but I also have found out that this 0xFF issue only happens with the ESP32. It doesn't happen with the ESP32-S2, S3, C3 and C6.

Therefore, I think that it may be an ESP32 internal hardware design problem that has been solved latter. Using another SoC may solve this specific issue in a product.

That is bad news for us for sure.

It seems that there is no way to fix it, unless using ODD/EVEN parity or a RX_TIMEOUT higher than 1.

How can we set RX_TIMEOUT higher than 1 with UART_MODE_RS485_HALF_DUPLEX mode?

Answer 9 · 2025-03-17T20:02:10.000Z

How can we set RX_TIMEOUT higher than 1 with UART_MODE_RS485_HALF_DUPLEX mode?

E (32) uart: tout_thresh = 2 > maximum value = 1
[    35][W][HardwareSerial.cpp:196] onReceive(): OnReceive is set to Timeout only, thus FIFO Full is now 120 bytes.

Using #define MBUS_RX_TIMEOUT 2
First line is IDF error message. Seconds is Arduino Core warning message.

=======
But it seems that if I set the BaudRate to 500000, it goes fine.
Could you please check if this is correct on your end?

Answer 10 · 2025-03-17T20:06:46.000Z

I have tried to set it to 5 but I got an error message like that and it did not have any effect. I can try that again tomorrow.

Answer 11 · 2025-03-17T22:46:23.000Z

@hitecSmartHome - I have found out why Serial can't set a higher RX Timeout.
ESP32 Arduino has fixed the UART Clock Source to use REF_TICK (1MHz) when the baud rate is lower than 250,000.
This is useful for applications that need to set a lower APB frequency in order to save power.
REF_TICK is a steady UART Clock Reference.
But it has a drawback... When REF_TICK is the UART Clock Source, RX Timeout is limited to 1.

When it uses APB Clock as UART Clock Source, RX Timeout can be way higher.

ESP32 and ESP32-S2 are the affected target.
ESP32-S3, C3, C6, P4 and others are not affected because those SoC can use the XTAL (40Mhz) as UART clock source, therefore, RX Timeout can also be higher than 1.

A way to solve this issue is by setting the Baudrate to higher than 250000 and setting the RX Timeout to 2.
In that case 0xff won't break the Serial1 input package.

#define MBUS_BAUD 250001  // higher than 250,000 allows RX_TIMEOUT to goes up to 101
#define MBUS_RX 35
#define MBUS_TX 32 
#define MBUS_RTS 33

#define MBUS_RX_TIMEOUT 2 // Can't set it higher than 1 if Baud rate is 250000 or lower

Answer 12 · 2025-03-18T06:51:48.000Z

That is still bad news because we are limited to 115200 baud with modbus communication because of external slave chips.

Answer 13 · 2025-03-18T07:00:40.000Z

And why does it split at specifically 0xff? If it is a timing/clock issue it would not matter which byte it is

Answer 14 · 2025-03-18T10:44:38.000Z

If I use the uart driver directly I can set the rx timeout hovewer I want. With symbol time of 2 it works with the uart driver.

uart_config_t uart_config = {
    .baud_rate = baud,
    .data_bits = UART_DATA_8_BITS,
    .parity = UART_PARITY_DISABLE,
    .stop_bits = UART_STOP_BITS_1,
    .flow_ctrl = UART_HW_FLOWCTRL_DISABLE,
    .source_clk = UART_SCLK_DEFAULT,
};

uart_driver_install(_uartNum, HS_UART_BUF_SIZE * 2, HS_UART_BUF_SIZE * 2, HS_UART_QUEUE_SIZE, &uart_queue, 0);
uart_param_config(_uartNum, &uart_config);
uart_set_pin(_uartNum, tx, rx, rts, UART_PIN_NO_CHANGE);
uart_set_mode(_uartNum, UART_MODE_RS485_HALF_DUPLEX);
uart_set_always_rx_timeout(_uartNum, 1);
uart_set_rx_timeout(_uartNum, byteTimeout);

We will continue to use the driver directly and avoid Serial implementation. Thanks for the help!

Answer 15 · 2025-03-18T10:56:46.000Z

So basically this is my implementation so far and it seems to work. This still triggers on packet timeout and I got all the data reliably. Still testing...

void HsUart::init(int baud, uint8_t rx, uint8_t tx, uint8_t rts, byte byteTimeout){
    uart_config_t uart_config = {
        .baud_rate = baud,
        .data_bits = UART_DATA_8_BITS,
        .parity = UART_PARITY_DISABLE,
        .stop_bits = UART_STOP_BITS_1,
        .flow_ctrl = UART_HW_FLOWCTRL_DISABLE,
        .source_clk = UART_SCLK_DEFAULT,
    };
    
    uart_driver_install(_uartNum, HS_UART_BUF_SIZE * 2, HS_UART_BUF_SIZE * 2, HS_UART_QUEUE_SIZE, &uart_queue, 0);
    uart_param_config(_uartNum, &uart_config);
    uart_set_pin(_uartNum, tx, rx, rts, UART_PIN_NO_CHANGE);
    uart_set_mode(_uartNum, UART_MODE_RS485_HALF_DUPLEX);
    uart_set_always_rx_timeout(_uartNum, 1);
    uart_set_rx_timeout(_uartNum, byteTimeout);
    
    // This spawns a task with the stack in psram
    worker.offloadToPsRam(
        std::bind(&HsUart::uart_event_task, this),
        4096, 12, "HsUart"
    );
}

void HsUart::uart_event_task() {
    uart_event_t event;
    uint8_t data[HS_UART_BUF_SIZE];
    while (true) {
        if (xQueueReceive(uart_queue, (void *)&event, portMAX_DELAY)) {
            switch (event.type) {
                case UART_DATA: 
                    int len = uart_read_bytes(_uartNum, data, event.size, portMAX_DELAY);
                    handlePacket(data,len);
                    break;
                case UART_FIFO_OVF:
                    emitErr(HS_UART_FIFO_OVF_ERROR);
                    break;
                case UART_BUFFER_FULL:
                    uart_flush_input(_uartNum);
                    xQueueReset(uart_queue);
                    emitErr(HS_UART_BUFFER_FULL_ERROR);
                    break;
                case UART_FRAME_ERR:
                    emitErr(HS_UART_FRAME_ERROR);
                    break;
                default:
                    break;
            }
        }
    }
}

Answer 16 · 2025-03-18T11:32:00.000Z

Well, it works except a weird 0x00 byte which is not measurable by an oscilloscope on the rx line but it is sometimes gets in the tx buffer.

Answer 17 · 2025-03-18T11:54:01.000Z

And why does it split at specifically 0xff? If it is a timing/clock issue it would not matter which byte it is

I think that it is a combination of timming and clock as you say.
ESP32 UART is implemented as an FSM (Finite State Machine) in the silicon chip.
Using REF_TICK (1MHz) there may not be enough clocks for the FSM to detect correctly the RX Timeout when threre many "1"s (0xFF), which seems to be confused with an Idle UART Line and not 1 symbol 1111111 + stop bit.

That is why setting EVEN or ODD parity in the UART Frame also solves the issue. That parity adds "time" for the correct RX Timeout detection. Another way may be using 2 Stop bits, instead of just 1.

Answer 18 · 2025-03-18T11:56:48.000Z

Well, it works except a weird 0x00 byte which is not measurable by an oscilloscope on the rx line but it is sometimes gets in the tx buffer.

This may be some UART BREAK detection... It happens in the very begining of the UART set up.
We see it when pins are set in the IO Matrix. It sounds like the FSM is active and the time it take to set the IO Matrix causes a BREAK detection.

Answer 19 · 2025-03-18T11:59:29.000Z

Well, it works except a weird 0x00 byte which is not measurable by an oscilloscope on the rx line but it is sometimes gets in the tx buffer.

This may be some UART BREAK detection... It happens in the very begining of the UART set up. We see it when pins are set in the IO Matrix. It sounds like the FSM is active and the time it take to set the IO Matrix causes a BREAK detection.

We observe this behaviour after uart setup at inconsistent rates. Sometimes we get 2-3 single 0x00 bytes under 1 minute but sometimes we get only one in five minutes.

Answer 20 · 2025-03-18T12:10:36.000Z

So basically this is my implementation so far and it seems to work. This still triggers on packet timeout and I got all the data reliably. Still testing...

You can also modify the arduino-esp32/cores/esp32/esp32-hal-uart.c code to use APB Clock Source (.source_clk = UART_SCLK_DEFAULT) and build the whole project using Arduino as IDF Component. That would allow the project to still use Arduino Serial1 and also modify the Arduino Core as necessary.

Just modifying https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/esp32-hal-uart.c#L42 in this way shall force the ESP32 to always use APB as UART Clock Source:

//#define REF_TICK_BAUDRATE_LIMIT 250000  // this is maximum UART badrate using REF_TICK as clock
#define REF_TICK_BAUDRATE_LIMIT 0  // Force ESP32 and ESP32-S2 to always use APB as Clock Source

Answer 21 · 2025-03-18T12:16:30.000Z

Well, it works except a weird 0x00 byte which is not measurable by an oscilloscope on the rx line but it is sometimes gets in the tx buffer.

This may be some UART BREAK detection... It happens in the very begining of the UART set up. We see it when pins are set in the IO Matrix. It sounds like the FSM is active and the time it take to set the IO Matrix causes a BREAK detection.

We observe this behaviour after uart setup at inconsistent rates. Sometimes we get 2-3 single 0x00 bytes under 1 minute but sometimes we get only one in five minutes.

Do you see also any BREAK verbose log mode messages or BREAK Error in the onReceiveError()?
This may be solved with a strong external pull-up resistor (1K ohm to +VCC) in the UART RX line.

Answer 22 · 2025-03-18T13:43:08.000Z

Do you see also any BREAK verbose log mode messages or BREAK Error in the onReceiveError()?

I dont see any BREAK event associated with the 0x00 byte. We do see break events because it is triggered after every timeout for every frame.

This may be solved with a strong external pull-up resistor (1K ohm to +VCC) in the UART RX line.

We have a 2.2k ohm resistor

Answer 23 · 2025-03-18T14:52:00.000Z

@hitecSmartHome - I think that we have clarified the issue and there is a work around for the 0xFF issue.
Feel free to close the issue in case that you consider it resolved.

Answer 24 · 2025-03-18T15:16:43.000Z

Thank you for the help. We ended up using the raw idf driver and using that instead of the Serial.

Answer 25 · 2025-03-27T17:40:35.000Z

@SuGlider is it possible to make the choice of clock a parameter in menuconfig? Or at least define REF_TICK_BAUDRATE_LIMIT as:

#ifndef REF_TICK_BAUDRATE_LIMIT 
#define REF_TICK_BAUDRATE_LIMIT 250000
#endif

Although the menuconfig setting is more elegant.

EDIT: Preparing a Pull Request for this.