SCPI-triggered ERROR LED
Closed this issue · 46 comments
@wizath I'm playing around writing unit tests for Booster. My current test is
import unittest
from oxart.devices.booster.driver import Booster
ip = "xxx.xxx.xxx.xxx"
class TestBooster(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.dev = Booster(ip)
def test_channel_validation(self):
for chan in [-1, 8, "test", False]:
self.dev.dev.write("CHAN:ENAB? {}\n".format(chan).encode())
print(self.dev.dev.readline().decode())
# self.assertTrue("error" in self.dev.dev.readline().decode())
def test_enable(self):
self.dev.set_enabled(0, True)
self.assertTrue(self.dev.get_enabled(0))
self.dev.set_enabled(0, False)
self.assertFalse(self.dev.get_enabled(0))
self.dev.set_enabled(0, True)
self.assertTrue(self.dev.get_enabled(0))
if __name__ == "__main__":
unittest.main()I was running various versions of this code and at some point channel 0 gave a red LED (I noticed because get_enabled always returns False). This channel had been okay for over a week before I started playing with the SCPI so I'm pretty sure this is an issue with the SCPI interface.
Logstash please
Just doing it :)
logstash
[INFO] hardware revision 3
[INFO] device boot
[INFO] SYSCLK frequency: 168000000
[INFO] PCLK1 frequency: 42000000
[INFO] PCLK2 frequency: 84000000
[INFO] ADC: OK | raw: 1980 | VrefInt 1.21 V
[INFO] PGOOD: OK
[INFO] Found mainboard EEPROM, loading values
[INFO] RF Channel at 0 detected, id = B2:13
[INFO] RF Channel at 1 detected, id = B1:CF
[INFO] RF Channel at 2 detected, id = 90:A2
[INFO] RF Channel at 3 detected, id = 69:E0
[INFO] RF Channel at 4 detected, id = 6D:76
[INFO] RF Channel at 5 detected, id = 56:81
[INFO] RF Channel at 6 detected, id = 3C:9C
[INFO] RF Channel at 7 detected, id = 39:6B
>
> i2cdetect 0
[i2c_scan] start
[i2c_scan] Found I2C device at 8
[i2c_scan] Found I2C device at E
[i2c_scan] Found I2C device at 2C
[i2c_scan] Found I2C device at 2E
[i2c_scan] Found I2C device at 2F
[i2c_scan] Found I2C device at 48
[i2c_scan] Found I2C device at 49
[i2c_scan] Found I2C device at 4A
[i2c_scan] Found I2C device at 4C
[i2c_scan] Found I2C device at 4D
[i2c_scan] Found I2C device at 50
[i2c_scan] Found I2C device at 51
[i2c_scan] Found I2C device at 52
[i2c_scan] Found I2C device at 53
[i2c_scan] Found I2C device at 54
[i2c_scan] Found I2C device at 55
[i2c_scan] Found I2C device at 56
[i2c_scan] Found I2C device at 57
[i2c_scan] Found I2C device at 70
[i2c_scan] end
> status 0
[status] e=1 s=1 r1=11 r2=11 tx=-7.253 rf=-2.776 curr=0.057 t=27.00
s␛[2J
PGOOD: 1
FAN SPEED: 30 %
AVG TEMP: 0.00 CURRENT: 0.00
CHANNELS INFO
==============================================================================
#0 #1 #2 #3 #4 #5 #6 #7
DETECTED 1 1 1 1 1 1 1 1
HWID B2:13 B1:CF 90:A2 69:E0 6D:76 56:81 3C:9C 39:6B
TXPWR [V] 0.01 0.04 0.01 0.01 0.01 0.01 0.01 0.01
RFLPWR [V] 0.01 0.01 0.01 0.01 0.01 0.03 0.03 0.02
TXPWR [dBm] -7.25 -7.30 -6.33 -6.92 -7.48 -7.06 -7.77 -6.71
RFLPWR [dBm] -2.78 -2.09 -3.54 -4.30 -4.33 -5.11 -5.82 -5.77
I30V [A] 0.057 0.053 0.054 0.056 0.059 0.053 0.054 0.032
I6V0 [A] 0.254 0.253 0.278 0.500 0.260 0.257 0.250 0.268
5V0MP [V] 4.991 4.980 4.971 4.979 4.963 4.968 4.956 4.977
ON 1 1 1 1 1 1 1 1
SON 1 1 1 1 1 1 1 1
IINT 0 0 0 0 0 0 0 0
OINT 0 0 0 0 0 0 0 0
SINT 0 0 0 0 0 0 0 0
OVC 0 0 0 0 0 0 0 0
ADC1 11 58 11 12 20 21 13 10
ADC2 12 10 13 11 12 46 42 36
DAC1 4095 4095 4095 4095 4095 4095 4095 4095
DAC2 3319 3444 3328 3259 3566 3600 3608 3451
SCALE1 83 87 87 85 85 87 86 87
OFFSET1 613 693 562 600 656 635 681 594
BIASCAL 1811 1960 1690 1700 1712 1687 1657 1756
HWIS 86.00 90.00 89.00 87.00 87.00 87.00 86.00 88.00
HWIO 679.00 744.00 614.00 649.00 695.00 729.00 770.00 679.00
LTEMP 27.00 27.00 27.25 26.00 27.00 27.25 26.00 26.00
RTEMP 26.00 27.00 27.25 26.00 28.50 27.00 26.00 27.00
==============================================================================
odd...no obvious sign of an error that I can see there, but a red LED nonetheless
annoyingly I haven't been able to reprooduce this issue on another channel.
Have you tried hammering disable/enable channel commands in a loop, on the suspicion that a switching glitch on a bus/rail is common to this and the ERROR I was seeing?
Just triggered it again with a similar variant
import unittest
import time
import random
from oxart.devices.booster.driver import Booster
ip = "xxx.xxx.xxx.xxx"
def get_channel():
return random.randint(0, 7)
class TestBooster(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.dev = Booster(ip)
def test_channel_validation(self):
for chan in [0.1, -1, 8, "test", False]:
self.dev.dev.write("CHAN:ENAB? {}\n".format(chan).encode())
print(self.dev.dev.readline().decode())
# to do: add a codepath that doesn't take "all" as a param
# self.assertTrue("error" in self.dev.dev.readline().decode()
# .lower())
def test_enable(self):
for _ in range(10):
chan = get_channel()
en = bool(random.getrandbits(1))
print(chan, en)
self.dev.set_enabled(chan, en)
time.sleep(0.1)
self.assertEqual(self.dev.get_enabled(chan), en)
if __name__ == "__main__":
# to do: write tidy up method!
unittest.main()Now channel 5 is added to the list of channels with an error.
logstash
[INFO] hardware revision 3
[INFO] device boot
[INFO] SYSCLK frequency: 168000000
[INFO] PCLK1 frequency: 42000000
[INFO] PCLK2 frequency: 84000000
[INFO] ADC: OK | raw: 1980 | VrefInt 1.21 V
[INFO] PGOOD: OK
[INFO] Found mainboard EEPROM, loading values
[INFO] RF Channel at 0 detected, id = B2:13
[INFO] RF Channel at 1 detected, id = B1:CF
[INFO] RF Channel at 2 detected, id = 90:A2
[INFO] RF Channel at 3 detected, id = 69:E0
[INFO] RF Channel at 4 detected, id = 6D:76
[INFO] RF Channel at 5 detected, id = 56:81
[INFO] RF Channel at 6 detected, id = 3C:9C
[INFO] RF Channel at 7 detected, id = 39:6B
>
> status 5
[status] e=1 s=1 r1=21 r2=48 tx=-7.057 rf=-5.110 curr=0.053 t=27.25
> start
PGOOD: 1
FAN SPEED: 30 %
AVG TEMP: 0.00 CURRENT: 0.00
CHANNELS INFO
==============================================================================
#0 #1 #2 #3 #4 #5 #6 #7
DETECTED 1 1 1 1 1 1 1 1
HWID B2:13 B1:CF 90:A2 69:E0 6D:76 56:81 3C:9C 39:6B
TXPWR [V] 0.01 0.04 0.01 0.01 0.01 0.01 0.01 0.01
>
> RFLPWR [V] 0.01 0.01 0.01 0.01 0.01 0.03 0.03 0.02
TXPWR [dBm] -7.25 -7.30 -6.33 -6.92 -7.52 -7.06 -7.77 -6.70
RFLPWR [dBm] -2.78 -2.09 -3.54 -4.30 -4.37 -5.11 -5.82 -5.76
I30V [A] 0.057 0.052 0.053 0.056 0.059 0.053 0.054 0.032
I6V0 [A] 0.255 0.253 0.278 0.500 0.260 0.257 0.250 0.268
5V0MP [V] 4.991 4.980 4.971 4.977 4.963 4.968 4.956 4.977
ON 1 1 1 1 1 1 1 1
SON 1 1 1 1 1 1 1 1
IINT 0 0 0 0 0 0 0 0
OINT 0 0 0 0 0 0 0 0
SINT 0 0 0 0 0 0 0 0
OVC 0 0 0 0 0 0 0 0
ADC1 11 59 11 12 16 21 13 11
ADC2 11 10 12 11 12 47 42 37
DAC1 4095 4095 4095 4095 4095 4095 4095 4095
DAC2 3319 3444 3328 3259 3566 3600 3608 3451
SCALE1 83 87 87 85 85 87 86 87
OFFSET1 613 693 562 600 656 635 681 594
BIASCAL 1811 1960 1690 1700 1712 1687 1657 1756
HWIS 86.00 90.00 89.00 87.00 87.00 87.00 86.00 88.00
HWIO 679.00 744.00 614.00 649.00 695.00 729.00 770.00 679.00
LTEMP 27.00 27.00 27.50 27.00 28.25 27.25 26.00 26.00
RTEMP 26.00 27.00 27.50 27.00 27.00 27.25 27.00 27.00
==============================================================================
Have you tried hammering disable/enable channel commands in a loop, on the suspicion that a switching glitch on a bus/rail is common to this and the ERROR I was seeing?
Yes, see above.
Not sure if this is a physical switching glitch or a software bug, but I guess that @wizath and @gkasprow are the best people to look at that.
@dnadlinger btw do you know a good framework for fuzzing hardware. e.g. running units tests repeatedly in random orders? I'd ideally like something a bit more flexible than putting for loops inside each test (to check for interactions between the tests)
@wizath thoughts about this issue? Let me know if there is any other data you want me to take.
Running with
Version(fw_rev='v1.4.1', fw_hash='f8dd3e9', fw_build_date=datetime.datetime(2019, 9, 11, 13, 18, 7), device_id='3436510b', hw_rev='hw rev 1.3')
Driver
import serial
import collections
import dateutil.parser
Version = collections.namedtuple(
"Version", ["fw_rev", "fw_hash", "fw_build_date", "device_id", "hw_rev"])
class Booster:
""" Booster 8-channel RF PA """
def __init__(self, device):
self.dev = serial.serial_for_url("socket://{}:5000".format(device))
self.dev.flushInput()
assert self.ping()
def get_version(self):
""" Returns the device version information as a named tuple """
self.dev.write(b"*IDN?\n")
idn = self.dev.readline().decode().strip().lower().split(',')
idn[0] = idn[0].split(" ")
if idn[0][0] != "rfpa" \
or not idn[1].startswith(" built ") \
or not idn[2].startswith(" id ") \
or not idn[3].startswith(" hw rev "):
raise Exception(
"Unrecognised device identity string: {}".format(idn))
return Version(fw_rev = idn[0][1],
fw_hash = idn[0][2],
fw_build_date = dateutil.parser.parse(idn[1][7:]),
device_id = idn[2][4:],
hw_rev = idn[3][1:])
def ping(self):
""" Returns True if we are connected to a Booster """
try:
self.get_version()
except Exception:
return False
return True
def _cmd(self, cmd, channel, arg=None):
if channel not in range(8):
raise ValueError("invalid channel number {}".format(channel))
if arg is None:
cmd = "{} {}\n".format(cmd, channel)
else:
cmd = "{} {}, {}\n".format(cmd, channel, arg)
self.dev.write(cmd.encode())
response = self.dev.readline().decode().lower().strip()
if '?' in cmd and not "error" in response:
return response
elif response == "ok":
return
raise Exception(
"Unrecognised response to '{}': '{}'".format(cmd, response))
def _query_bool(self, cmd, channel, arg=None):
resp = self._cmd(cmd, channel, arg)
if resp == "0":
return False
elif resp == "1":
return True
else:
raise Exception(
"Unrecognised response to {}: '{}'".format(cmd, resp))
def _query_float(self, cmd, channel, arg=None):
resp = self._cmd(cmd, channel, arg)
try:
return float(resp)
except ValueError:
raise Exception(
"Unrecognised response to {}: '{}'".format(cmd, resp))
def set_enabled(self, channel, enabled=True):
""" Enables/disables a channel """
cmd = "CHAN:ENAB" if enabled else "CHAN:DISAB"
self._cmd(cmd, channel)
def get_enabled(self, channel):
""" Returns True is the channel is enabled """
return self._query_bool("CHAN:ENAB?", channel)and tests
import unittest
import time
import random
from oxart.devices.booster.driver import Booster
ip = "xxx.xxx.xxx.xxx"
class TestBooster(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.dev = Booster(ip)
print("Testing on: {}".format(cls.dev.get_version()))
# def test_channel_validation(self):
# for chan in [0.1, -1, 8, "test", False]:
# self.dev.dev.write("CHAN:ENAB? {}\n".format(chan).encode())
# print(self.dev.dev.readline().decode())
# # to do: add a codepath that doesn't take "all" as a param
# # self.assertTrue("error" in self.dev.dev.readline().decode()
# # .lower())
def test_enable(self):
for _ in range(1000):
print(".")
chan = random.randint(0, 8)
en = bool(random.getrandbits(1))
if chan == 8:
if en:
self.dev.dev.write(b"CHAN:ENAB all\n")
else:
self.dev.dev.write(b"CHAN:DISAB all\n")
resp = self.dev.dev.readline().decode().strip().lower()
self.assertEqual(resp, "ok")
self.dev.dev.write(b"CHAN:ENAB? all\n")
resp = self.dev.dev.readline().decode().strip().lower()
expected = "0" if en is False else "255"
# self.assertEqual(resp, expected)
else:
self.dev.set_enabled(chan, en)
# self.assertEqual(self.dev.get_enabled(chan), en)
self.dev.get_enabled(chan)
if __name__ == "__main__":
# to do: write tidy up method!
unittest.main()I have not been able to reproduce this :(
WEIRD. Since channel was enabled, no error log, no error condition. Maybe there was a glitch at SPI bus driving the led bar? Since I can't think of anything possible. Maybe I'll lower the speed
Yes, it's odd that the channel current was still active. However, IIRC one the error LED showed up, I was no longer able to enable the channel -- couldn't get a green LED any more and chan:enab? always returned 0. So I think there was more here than a simple glitch on the LED bus.
Anyway, I can't reproduce this at all any more even after quite a while of fuzzing and trying to insert various forms of malformed commands. Odd. Maybe a race or something that was affected by another change you made?
Yep, 10k random commands later and still no ERROR LEDs.
So, on the one hand, I'm now pretty confident that this isn't a hw issue (since otherwise I wouldn't expect it to disappear so easily with fw tweaks) but on the other hand we still don't know the cause so it will probably come back to bite us later...
@wizath Am I going crazy, or have you changed the way the channel IDs are defined? I'm using the same hw as yesterday, but looking at the log (e.g. "START" on the VCP) I see different channel IDs)...
No, I think I was being silly here and accidentally had SCPI connected to one Booster and VCP connected to another. So I got the logstash from the wrong Booster, which explains why it was empty!
So, there might actually have been something useful there after all. Sadly I missed it and can't reproduce the issue :) damn
let me try to reproduce on a booster with an older fw version...
ah! After much fuzzing I have reproduced it. Let's see what's in the logs...
> logstash
[INFO] network client 10.255.6.130 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.130 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.130 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.130 connected
[INFO] network client disconnected
[INFO] network client 10.255.6.130 connected
[ERROR] Temperature error on channel 0 temp = 63.00, disabling
[INFO] network client disconnected
>
> i2cdetect 0
[i2c_scan] start
[i2c_scan] Found I2C device at 2C
[i2c_scan] Found I2C device at 2E
[i2c_scan] Found I2C device at 2F
[i2c_scan] Found I2C device at 70
[i2c_scan] end
>
> i2cerr 0
Wrong number of parameters
>
> status 0
[status] e=0 s=0 r1=18 r2=12 tx=0.000 rf=0.000 curr=0.000 t=63.00 i=0.01 ip=-nan
>
> start
␛[2J
PGOOD: 1
FAN SPEED: 20 %
AVG TEMP: 27.39 CURRENT: 63.00
CHANNELS INFO
==============================================================================
#0 #1 #2 #3 #4 #5 #6 #7
DETECTED 1 1 1 1 1 1 1 1
HWID 21:4B DE:0E A8:53 5B:EA FB:68 02:41 B5:46 1F:B8
INPWR [V] 0.01 0.21 0.41 0.62 0.8>
> 3 1.03 1.24 1.44
TXPWR [V] 0.01 0.01 0.01 0.01 0.01 0.01 0.14 0.01
RFLPWR [V] 0.01 0.08 0.01 0.06 0.20 0.01 0.00 0.01
INPWR [dBm] -nan -nan -nan -nan -nan -nan -nan -nan
TXPWR [dBm] 0.00 5.00 5.00 5.00 5.00 5.00 5.00 5.00
RFLPWR [dBm] 0.00 -5.04 -4.13 -4.86 -3.96 -2.81 -4.96 -2.84
I30V [A] 0.000 0.180 0.180 0.180 0.180 0.180 0.180 0.180
I6V0 [A] 0.000 0.164 0.164 0.164 0.164 0.164 0.164 0.164
5V0MP [V] 0.000 2.051 2.051 2.051 2.051 2.051 2.051 2.051
ON 0 1 1 1 1 1 1 1
SON 0 1 1 1 1 1 1 1
IINT 0 0 0 0 0 0 0 0
OINT 0 0 0 0 0 0 0 0
SINT 0 0 0 0 0 0 0 0
ADC1 18 18 18 24 21 18 232 17
ADC2 11 127 11 105 323 13 4 11
INTSET [dBm] 32.00 30.00 29.99 29.99 29.99 34.00 33.99 31.99
DAC1 4095 4095 4095 4095 4095 4095 4095 4095
DAC2 3297 3117 3139 3140 3237 3427 3427 3271
SCALE1 84 83 83 84 85 85 79 85
OFFSET1 505 431 567 472 571 430 635 480
BIASCAL 1149 1283 1271 1327 1473 1577 1327 1335
HWIS 82.67 86.17 84.83 85.08 86.83 86.00 83.17 85.42
HWIO 651.67 532.17 594.83 588.08 632.83 503.00 600.17 538.42
LTEMP 63.00 26.50 26.25 26.50 26.25 26.25 26.00 26.25
RTEMP 63.00 26.25 26.50 26.50 26.50 26.25 26.00 26.25
==============================================================================
stop
>
The fw version I have doesn't seem to have the i2cerr command. Any other diagnostics I can play with here.
FWIW, this has mainly appeared when I'm doing fw development rather than strict randomized testing. So it may be partially triggered by the occasional malformed instruction. Hard to tell...
The fans are slowly integrating up, which is sensible.
NB the currents for all channels that I'm not using have supposedly shot up! They were ~50mA before the error condition, now 180mA. Interestingly however the channel temperatures haven't moved yet.
Use i2cerr without any parameters
> i2cdetect 0
[i2c_scan] start
[i2c_scan] Found I2C device at 2C
[i2c_scan] Found I2C device at 2E
[i2c_scan] Found I2C device at 2F
[i2c_scan] Found I2C device at 70
[i2c_scan] end
This seems like an issue with I2C bus...
Repeated querying of the currents by the SCPI interface agrees with the VCP current. However if this is real, it seems surprising that the channel temperatures aren't rising. If it's real it's very concerning though (could explain all those dead FETs!!!). If it's not real then I have no idea what's going on
> i2cerr
#0 #1 #2 #3 #4 #5 #6 #7
I2C ERR 0 78 8 724364 20 16 26 22
Sorry, the command is there, it's just not listed in the help menu (or on the wiki)!
Ok, so the conclusion here is that we definitely have I2C errors. Any other diagnostics before I power cycle the device?
one second
i2cr <ch num> 0x4A 0x01 should read raw temperature
> i2cr 0 0x4A 0x01
[i2cw] read to 74 reg 1 = 0
>
> i2cr 0 0x4A 0x01
[i2cw] read to 74 reg 1 = 0
>
> i2cr 1 0x41 0x01
[i2cw] read to 65 reg 1 = 0
>
> i2cr 2 0x41 0x02
[i2cw] read to 65 reg 2 = 0
>
> i2cr 3 0x41 0x01
[i2cw] read to 65 reg 1 = 0
>
Can I restart now before the FETs bake?
@wizath do you think the bias current reading is correct?? The channel temperatures haven't changed at all
PGOOD: 1
FAN SPEED: 20 %
AVG TEMP: 38.41 CURRENT: 63.00
CHANNELS INFO
==============================================================================
#0 #1 #2 #3 #4 #5 #6 #7
DETECTED 1 1 1 1 1 1 1 1
HWID 21:4B DE:0E A8:53 5B:EA FB:68 02:41 B5:46 1F:B8
INPWR [V] 0.01 0.21 0.41 0.62 0.83 1.03 1.24 1.44
TXPWR [V] 0.01 0.01 0.01 0.01 0.01 0.01 0.15 0.01
RFLPWR [V] 0.01 0.08 0.01 0.06 0.20 0.01 0.00 0.01
INPWR [dBm] -nan -nan -nan -nan -nan -nan -nan -nan
TXPWR [dBm] 0.00 5.00 5.00 5.00 5.00 5.00 5.00 5.00
RFLPWR [dBm] 0.00 -4.99 -4.13 -4.86 -3.95 -2.81 -4.98 -2.84
I30V [A] 0.000 0.180 0.180 0.180 0.180 0.180 0.180 0.180
I6V0 [A] 0.000 0.164 0.164 0.164 0.164 0.164 0.164 0.164
5V0MP [V] 0.000 2.051 2.051 2.051 2.051 2.051 2.051 2.051
ON 0 1 1 1 1 1 1 1
SON 0 1 1 1 1 1 1 1
IINT 0 0 0 0 0 0 0 0
OINT 0 0 0 0 0 0 0 0
SINT 0 0 0 0 0 0 0 0
ADC1 18 18 19 24 18 17 237 17
ADC2 12 128 10 103 326 12 4 11
INTSET [dBm] 32.00 30.00 29.99 29.99 29.99 34.00 33.99 31.99
DAC1 4095 4095 4095 4095 4095 4095 4095 4095
DAC2 3297 3117 3139 3140 3237 3427 3427 3271
SCALE1 84 83 83 84 85 85 79 85
OFFSET1 505 431 567 472 571 430 635 480
BIASCAL 1149 1283 1271 1327 1473 1577 1327 1335
HWIS 82.67 86.17 84.83 85.08 86.83 86.00 83.17 85.42
HWIO 651.67 532.17 594.83 588.08 632.83 503.00 600.17 538.42
LTEMP 63.00 26.50 26.25 26.50 26.25 26.25 26.00 26.25
RTEMP 63.00 26.25 26.50 26.50 26.50 26.25 26.00 26.25
==============================================================================
anyway, let me know if there is anything else to do before I power cycle
(Side note: this is why I like HITL testing with fuzzing. About the only way of catching these rare bugs. Will post my full test code in a bit)
The fact that all channels read exactly 180mA and 164mA makes me think these values are not real, just some I2C issue.
anyway, do let me know when I can power cycle and finish other bits of testing plz
That's dead I2C bus, i'll make i2c reset command tomorrow to see if it helps. You can reboot it for now i think
Ok. I2C reset command will be good, but we need to identify the cause ultimately. At least now we know where to look...
Youve got equal currents for all channels. That happened waaay long time ago, where glitch at SCL could render whole bus inacessible
ok
Well, on the bright side, the firmware is now feature-complete and everything basically works. So, it's just the last few annoying bugs to sort out now. We've come a long way :)
@wizath this particular booster v1.3 seems to have a dead channel in a way I've never seen before. Look here #282 (comment) I6V0 is 0.5A (which I think is the hardware current limit set for that LDO). That's not good!
@dnadlinger This was the v1.3 from your lab you gave me.
We should:
- make sure that Creotech/TS check these currents as part of the manufacture/tuning process
- make sure we record them for each booster we receive (added note to the wiki)
I'll send this unit back to TS for debugging and repair.
@dnadlinger can you flash one of those firmware revisions and see if it fixes your issues. Other than the SCPI interface hanging occasionally, this revision seems to work really well...