Receiving duplicate data. Can only read .rx_obj() 7 times before IndexError
dunderMethods opened this issue · 10 comments
Hi, thank you for providing this crucial library!
I'm sure I have overlooked something. I appreciate your help in resolving this.
Full Arduino code at the bottom of this post.
The Ultimate Goal:
Arduino reads one sample from each of 8 sensors, into an array[8], and transmits the array over serial, 80 times/sec.
Python parses the incoming array and plots the values on a graph in real time.
The good:
I believe everything is correct and working well on the arduino side. When I use Serial.print() to print the array of sensor values instead of using SerialTransfer, the values change each time like they should - so the Arduino is not sending duplicate packets.
The Python Code: (I'm working in Jupyter on Windows10)
import time
from pySerialTransfer import pySerialTransfer as txfer
link = txfer.SerialTransfer('COM7', baud=115200)
link.open()
time.sleep(2) # allow some time for the Arduino to completely reset
def get_packet(st_obj, byte_size):
while True:
st_obj.available()
yield st_obj.rx_obj(obj_type = list,
obj_byte_size = 32,
list_format = 'l')
stream = get_packet(link, 32)I can print 7 times...
for x in range(7):
print(stream.__next__())Prints the same packet 7 times:
[-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593]
[-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593]
[-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593]
[-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593]
[-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593]
[-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593]
[-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593]
If I try to print an 8th time:
print(stream.__next__())
I get an IndexError:
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-4-0991908dffd0> in <module>
----> 1 stream.__next__()
<ipython-input-2-d12a92ebb86c> in get_packet(st_obj, byte_size)
5 def get_packet(st_obj, byte_size):
6 while True:
----> 7 st_obj.available()
8 yield st_obj.rx_obj(obj_type = list,
9 obj_byte_size = 32,
C:\Users\me\AppData\Local\Continuum\anaconda3\lib\site-packages\pySerialTransfer\pySerialTransfer.py in available(self)
475 elif self.state == find_payload:
476 if self.payIndex < self.bytesToRec:
--> 477 self.rxBuff[self.payIndex] = recChar
478 self.payIndex += 1
479
IndexError: list assignment index out of range
What am I doing wrong here?
Full Arduino Code:
#include "HX711-multi.h"
#include "SerialTransfer.h"
// Hardware Config
#define CLK1 2 // clock pin to the first group of ADCs
#define DIN1 3 // data pin to the first lca
#define DIN2 4 // data pin to the second lca
#define DIN3 5 // data pin to the third lca
#define DIN4 6 // data pin to the fourth lca
#define CLK2 8 // clock pin to the 2nd group of ADCs
#define DIN5 7 // data pin to the fifth lca
#define DIN6 10 // data pin to the sixth lca
#define DIN7 11 // data pin to the seventh lca
#define DIN8 12 // data pin to the eighth lca
// System Config
#define CHANNEL_COUNT 4
#define NUM_GROUPS 2
#define GAIN 128 // Gain level for HX711 to use ( 32, 64, 128 are your options)
// Construct scale objects
byte DINS1[CHANNEL_COUNT] = {DIN1, DIN2, DIN3, DIN4};
HX711MULTI scales1(CHANNEL_COUNT, DINS1, CLK1, GAIN);
byte DINS2[CHANNEL_COUNT] = {DIN5, DIN6, DIN7, DIN8};
HX711MULTI scales2(CHANNEL_COUNT, DINS2, CLK2, GAIN);
HX711MULTI scales[NUM_GROUPS] = {scales1, scales2}; // Add all scale instances to an array so we can iterate over them
// Program Variables
long raw[NUM_GROUPS * CHANNEL_COUNT]; // Array to hold raw samples collected from each sensor
uint16_t sendSize;
int i;
SerialTransfer myTransfer;
void setup() {
Serial.begin(115200);
myTransfer.begin(Serial);
}
void loop() {
//Read sensor values into an array
scales[0].readRaw(raw); // Read values from first 4 sensors directly into the raw array
scales[1].readRaw(raw + 4); // Read values from last 4 sensors directly into the raw array
// contents of raw looks like: {-180279, -160207, 57267, -238712, -198426, -232254, 14375, -176826}
// Transfer the contents of raw to Python via serial
sendSize = myTransfer.txObj(raw, sendSize);
myTransfer.sendData(sendSize);
// Loop around, read new values from sensors, and send new values.
}Idk if this helps, but this works for me:
Python:
import time
from pySerialTransfer import pySerialTransfer as txfer
link = txfer.SerialTransfer('COM5', baud=115200)
link.open()
time.sleep(2) # allow some time for the Arduino to completely reset
while not link.available():
pass
my_list = link.rx_obj(obj_type = list,
obj_byte_size = 4*8, # 8 integers of 4 bytes each
list_format = 'i')
print(my_list)
link.close()Arduino:
#include "SerialTransfer.h"
SerialTransfer myTransfer;
void setup()
{
Serial.begin(115200);
myTransfer.begin(Serial);
}
void loop()
{
int32_t raw[8] = {-180279, -160207, 57267, -238712, -198426, -232254, 14375, -176826};
int sendSize = myTransfer.txObj(raw);
myTransfer.sendData(sendSize);
delay(2);
}Thanks for the quick response!
I tired your code which prints a single packet. I added a for loop to try and print multiple lines and I have the same issue:
link = txfer.SerialTransfer('COM7', baud=115200)
link.open()
time.sleep(2) # allow some time for the Arduino to completely reset
for x in range(20):
while not link.available():
pass
my_list = link.rx_obj(obj_type = list,
obj_byte_size = 4*8, # 8 integers of 4 bytes each
list_format = 'i')
print(my_list)
link.close()This is the result in Jupyter:
[-179581, -160986, 60252, -239150, -196882, -240494, 14857, -176687]
[-179581, -160986, 60252, -239150, -196882, -240494, 14857, -176687]
[-179581, -160986, 60252, -239150, -196882, -240494, 14857, -176687]
[-179581, -160986, 60252, -239150, -196882, -240494, 14857, -176687]
[-179581, -160986, 60252, -239150, -196882, -240494, 14857, -176687]
[-179581, -160986, 60252, -239150, -196882, -240494, 14857, -176687]
[-179581, -160986, 60252, -239150, -196882, -240494, 14857, -176687]
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-23-050703d055f8> in <module>
3 time.sleep(2) # allow some time for the Arduino to completely reset
4 for x in range(20):
----> 5 while not link.available():
6 pass
7
C:\Users\me\AppData\Local\Continuum\anaconda3\lib\site-packages\pySerialTransfer\pySerialTransfer.py in available(self)
475 elif self.state == find_payload:
476 if self.payIndex < self.bytesToRec:
--> 477 self.rxBuff[self.payIndex] = recChar
478 self.payIndex += 1
479
IndexError: list assignment index out of range
Stepping through my code in the PyCharm Debugger I notice that rxBuff is filling with new payload values (non-repeating) with each call to stream.__next__(). However, in rx_obj() start_pos always equals 0 so it always returns the first 32bits in rxBuff which explains why it prints the same values every time.
def rx_obj(self, obj_type, obj_byte_size, start_pos=0, list_format=None, byte_format=''):
buff = bytes(self.rxBuff[start_pos:(start_pos+obj_byte_size)])
len(rxBuff) is 253 and 32*8 = 256 which explains the IndexError when payIndex > len(rxBuff) near the end of the 7th call to stream.__next__()
I haven't done the most thorough analysis on how the lib operates, but I don't see anything updating start_pos to the beginning index of the next payload in rxBuff on subsequent calls to rx_obj(). I suppose I could keep track in my program and manually pass the correct start_pos when I call rx_obj() but it does not seem like that was the design intent. Even if I did do that, I don't see anything that clears rxBuff and starts at the beginning once it is full so I would still encounter the IndexError (although I have not tried it yet).
It seems a lot of folks have got this library working correctly so I assume I am misunderstanding something. I'm still a fairly new/casual programmer so I wouldn't be surprised. But tell me If I understand correctly:
The intent is that you call new_line_of_data = link.rx_obj() every time you're ready to process/use the next payload sent from the Arduino, right? As long as the Arduino keeps streaming, you should be able to keep calling link.available() (to update the input buffer) then link.rx_obj()( to read the last unread payload from the buffer) until the Arduino stops sending data and everything should be fine, right?
C:\Users\me\AppData\Local\Continuum\anaconda3\lib\site-packages\pySerialTransfer\pySerialTransfer.py in available(self) 475 elif self.state == find_payload: 476 if self.payIndex < self.bytesToRec: --> 477 self.rxBuff[self.payIndex] = recChar 478 self.payIndex += 1 479 IndexError: list assignment index out of range
This means your payload length byte is larger than is allowed by the protocol (254 if I remember correctly).
To fix it, you'll need to change your Arduino code to this:
sendSize = myTransfer.txObj(raw);
myTransfer.sendData(sendSize);Stepping through my code in the PyCharm Debugger I notice that
rxBuffis filling with new payload values (non-repeating) with each call tostream.__next__(). However, inrx_obj()start_posalways equals 0 so it always returns the first 32bits inrxBuffwhich explains why it prints the same values every time.
start_pos is the index of the first byte to parse out of the receive buffer (rx_buff). As long as you're sending different different data, you should still see that changing data with start_pos always set to 0. See the doc string:
Description:
------------
Extract an arbitrary variable's value from the RX buffer starting at
the specified index. If object_type is list, it is assumed that the
list to be extracted has homogeneous element types where the common
element type can neither be list, dict, nor string longer than a
single char
:param obj_type: type - type of object to extract from the RX buffer
:param obj_byte_size: int - number of bytes making up extracted object
:param start_pos: int - index of RX buffer where the first byte
of the value is to be stored in
:param list_format: char - array.array format char to represent the
common list element type - 'c' for a char
list is supported
:param byte_format: str - byte order, size and alignment according to
https://docs.python.org/3/library/struct.html#struct-format-strings
:return unpacked_response: obj - object extracted from the RX buffer,
None if operation failed
To be clear, start_pos is only used when multiple data objects are sent within the same packet (i.e. a float plus a bool for instance in the same packet). It helps distinguish what parts of a given packet payload you want to parse at a given time. There are some examples on this in the Arduino library that are largely analogous to how it works on the Python side. I hope that makes sense, lol.
len(rxBuff)is 253 and 32*8 = 256 which explains the IndexError whenpayIndex>len(rxBuff)near the end of the 7th call tostream.__next__()
Ah, that's true and will cause you issues. Try to be as memory efficient as possible when sending data using any non-internet protocol such as this one. Do you really need 32 bits per value?
The intent is that you call new_line_of_data = link.rx_obj() every time you're ready to process/use the next payload sent from the Arduino, right? As long as the Arduino keeps streaming, you should be able to keep calling link.available() (to update the input buffer) then link.rx_obj()( to read the last unread payload from the buffer) until the Arduino stops sending data and everything should be fine, right?
link.available() is what actually does the "unpacking" and once a full packet has been completely "unpacked", you use link.rx_obj() to parse values out of the payload (found in link.rx_buff). That is why you have to repeatedly call link.available() until it returns True. Only then can you use link.rx_obj().
Where do I send the beers? 🍻
It works great now. Thanks! You had it on the first response but I didn't realize you took the sendSize argument out of myTransfer.txObj(raw, *sendSize*) in the Arduino code. That did the trick. It is now working with sendSize = myTransfer.txObj(raw)
Your detailed response is very helpful; it sheds a lot more light on the operation of this library. Is len(rxBuff) == "a packet" or is the buffer composed of multiple packets? To answer your question, the sensors are 24-bit ADCs so a 4 byte data type is the smallest available to contain the potential range.
Thank you for providing this library and the excellent support. I see how you have helped others as well and just want to thank you for your time and generosity. I'll be using this for all of my robust communication needs 👍
if any([venmo, paypal, zelle]):
I'll = SerialTransfer(you_some_🍻_💰)
I really appreciate it! The best payoff I get from doing open source programming as a hobby is when others take an interest and incorporate it into their own projects, but thank you!
Is len(rxBuff) == "a packet" or is the buffer composed of multiple packets?
The entire contents of the rx_buff is the entire contents of a single packet's payload (the most current packet parsed). As soon as a new packet is read, the buffer is completely overwritten with the new payload.
the sensors are 24-bit ADCs so a 4 byte data type is the smallest available to contain the potential range.
You could break the packet up into multiple packets and send each sensor value separately with unique packet IDs.
link.available() is what actually does the "unpacking" and once a full packet has been completely "unpacked", you use link.rx_obj() to parse values out of the payload (found in link.rx_buff). That is why you have to repeatedly call link.available() until it returns True. Only then can you use link.rx_obj().
The entire contents of the rx_buff is the entire contents of a single packet's payload (the most current packet parsed). As soon as a new packet is read, the buffer is completely overwritten with the new payload.
May add these to the docs. The code is clean but hearing it described in English is helpful.
You could break the packet up into multiple packets and send each sensor value separately with unique packet IDs.
Would you mind providing a link or example describing this process and the benefits?
You could break the packet up into multiple packets and send each sensor value separately with unique packet IDs.
Would you mind providing a link or example describing this process and the benefits?
Instead of sending all sensor data in one packet, send 8 packets with the data from each sensor individually. Each of those 8 packets should have different packet IDs as specified in the myTransfer.sendData() call. This way, when you receive a packet, the packet ID (using link.idByte) will tell you which of the 8 sensors you received data for and then save/graph it properly - otherwise, you wouldn't be able to tell one sensor's packet from the other.
The reason why you would do this is to send a large amount of data while circumventing the ~254 byte per packet maximum that was causing you an error earlier.
Thanks for explaining. I'll look into it.