Extended Array Rx/Tx

[bhclowers]

This may not be a bug, but rather a question (I'm not sure how to change the category of this submission). In short, I'd like to find a way to transfer an array of values from a Teensy 4.1 to python. After investigating the code it looks like the buffer is maxed out to 254 based upon MAX_PACKET_SIZE = 0xFE.

Because I'm ultimately interested in the larger buffers that can be accomodated using the external memory on the Teensy 4.1, it would be great to have an example on how to manage the handling of these large arrays. So this begs the question, how to break the array into chunks and send it with minimal delays and dropped packets?

This clearly requires coordination on the Teensy and Python side but I'm running into my limits and would appreciate any direction that folks can provide.

Teensy Scheme

void SendADCBinary(void) {
  // use this variable to keep track of how many
  // bytes we're stuffing in the transmit buffer
  uint32_t sendSize = 0;
  
  ///////////////////////////////////////// Stuff buffer with array
  //Ideally adcbuffer has more than 10000 elements  
  sendSize = myTransfer.txObj(adcbuffer, sendSize);
  
  ///////////////////////////////////////// Send buffer
  myTransfer.sendData(sendSize);
 
}

Python Code

import numpy as np
import time
from pySerialTransfer import pySerialTransfer as txfer

link = txfer.SerialTransfer('COM10')

link.open()
time.sleep(2) # allow some time for the Arduino to completely reset

send_size = 0
npArray = np.arange(2048)
npArray.dtype = np.uint32
npList = npArray.tolist()
arrSize = link.tx_obj(npList)

[PowerBroker2]

I've never done something like that with Python in the mix, but you might get some inspiration from the following examples. These examples show how to transfer an entire text file (extended char array) between Arduinos:

TX
RX

[bhclowers]

I see the pattern here but I'm running into a couple of issues. The first is that if the array to transfer does not divide evenly by the maximum transfer size, how can you know the expected number of bytes to receive?

The other is that I can't seem to get the python size going well. I keep getting some odd errors.

#include "SerialTransfer.h"

/*
 * https://github.com/PowerBroker2/pySerialTransfer/issues/14
 * Perhaps python integers are 32?
 * https://github.com/PowerBroker2/pySerialTransfer/issues/27
 */

const int ledPin = 13;

SerialTransfer myTransfer;

//////////////////
//Array Generation
#define ARRAYMAX 10000 //1E6
EXTMEM uint32_t adcbuffer[ARRAYMAX];//Trying 32bit to see if we can do more averages with a 12 bit adc.
volatile uint32_t adcidx;
volatile bool doneflag = false;

void setup(){
 
  Serial.begin(9600);//COM9 on Windows
  SerialUSB1.begin(9600);//COM10 on Windows
//  while(!Serial&&millis()<5000){
//    
//  }
  myTransfer.begin(Serial);

  pinMode(ledPin, OUTPUT);

  //Fill up buffer
  for(uint32_t i = 0; i < ARRAYMAX; i++) {
    adcbuffer[i] = i;
  }
  SerialUSB1.println(sizeof(adcbuffer));
  delay(5000);
}


void SendADC(){

  uint32_t arraySize = sizeof(adcbuffer);
  uint16_t numPackets = arraySize / (MAX_PACKET_SIZE - 2); // Reserve one byte for current file index

  if (arraySize % MAX_PACKET_SIZE){ // Add an extra transmission if needed
    numPackets++;
  }
  
  SerialUSB1.println("Sending ADC");
  SerialUSB1.println(numPackets);
  SerialUSB1.println(sizeof(adcbuffer));
  
  for (uint16_t i=0; i<numPackets; i++) // Send all data within the array across multiple packets
  {
    uint16_t arrayIndex = i * MAX_PACKET_SIZE; // Determine the current array index
    uint8_t dataLen = MAX_PACKET_SIZE - 2;

    if ((arrayIndex + (MAX_PACKET_SIZE - 2)) > arraySize) // Determine data length for the last packet if file length is not an exact multiple of MAX_PACKET_SIZE-2
      dataLen = arraySize - arrayIndex;
    
    uint8_t sendSize = myTransfer.txObj(arrayIndex); // Stuff the current file index
    sendSize = myTransfer.txObj(adcbuffer[arrayIndex], sendSize, dataLen); // Stuff the current file data
    
    myTransfer.sendData(sendSize, 1); // Send the current file index and data
    
    delay(100);
  }  
}



void loop(){
  digitalWrite(ledPin, HIGH);   // set the LED on
  delay(1000);                  // wait for a second
  digitalWrite(ledPin, LOW);    // set the LED off
  delay(1000);                  // wait for a second

  // put your main code here, to run repeatedly:
  char ch;
  if (SerialUSB1.available()) {
    ch = SerialUSB1.read();
    if (ch == 's')  SendADC();
  }
}

This throws an error in python. Clearly, something is off and I'm not sure where to start to fix it.

recList = []
try:
    
    for i in range(159): #this is the number of packets to send. 
        ###################################################################
        # Parse response list
        ###################################################################
        tempList  = link.rx_obj(obj_type=type(recList), obj_byte_size=252, list_format='i')
        recList.append(tempList)
    
except KeyboardInterrupt:
    try:
        link.close()
    except:
        pass

except:
    import traceback
    traceback.print_exc()
    
    print(i)
    
    try:
        link.close()
    except:
        pass
print(i)
print(recList)

[PowerBroker2]

This throws an error in python

What's the full error (in tags)?

The first is that if the array to transfer does not divide evenly by the maximum transfer size, how can you know the expected number of bytes to receive?

This snippet is what handles the last packet gracefully:

if ((arrayIndex + (MAX_PACKET_SIZE - 2)) > arraySize) // Determine data length for the last packet if file length is not an exact multiple of MAX_PACKET_SIZE-2
      dataLen = arraySize - arrayIndex;

[PowerBroker2]

A couple of notes:

1. Why not send data realtime with smaller packets and let Python do the averaging?
2. Use 115200 baud or higher to get data moving faster

[PowerBroker2]

Been a while - assuming issue is resolved