Missing samples with RSP1a
Opened this issue · 27 comments
I'm trying to use SoapySDR with Python to capture ultrasound signals from a microphone/preamp. I have my RSP1a configured with a sample rate of 250kHz and a centre frequency of 125kHz and am using sdr.readStream(...) to grab repeated chunks of 8064 samples for onward processing. On the face of it, this appeared to be working correctly, however if I feed a test signal at 120kHz into the unit, the 5kHz sine wave beat frequency (observed via the real part of the I/Q) is showing glitches consistent with samples dropped between the blocks of 8064.
Is there a way under SoapySDR to ensure a continuous feed of data without dropping samples?
@dickh768 - tonight I took the SoapSDR Python example from here, I modified it a little to count the number of samples and the elapsed time (my version of the script is attached), and I ran it with a center frequency of 125kHz and a sample rate of 250kHz using my RSPdx.
This is the output:
[INFO] devIdx: 0
[INFO] SerNo: **********
[INFO] hwVer: 4
[INFO] rspDuoMode: 0
[INFO] tuner: 1
[INFO] rspDuoSampleFreq: 0.000000
[INFO] Using format CF32.
start: 2023-06-07 22:04:09.205703
end: 2023-06-07 22:04:49.269915
elapsed: 0:00:40.064212
nsamples: 10000384
which to me looks what one would expect: 10,000,384 / 250,000 = 40.001536.
Given the fact that you are experiencing samples dropped between the blocks, I am wondering if your code is doing some processing in between blocks, and it is not calling readStream() fast enough. At a sample rate of 250kHz, with each call of readStream() returning 1,024 samples, you have on average about 4ms between each call, otherwise your Python script is going to fall behind and start dropping samples.
Franco
Hi Franco, thanks for the reply.
I tried your code and got pretty much the same result, so as you say there does not seem to be an enormous issue. I also tried to repeat my own test and on this occasion there was no glitch (!). However I then noticed that your code is not performing a defined number of readStream operations, but loops until the number of reported samples exceeds a threshold. So I modified the script to use a 'for' loop with 100,000 operations.
The result was quite interesting for me:
Number of samples expected = 102,400,000
Total of samples reported received = 100,800,000
Time elapsed time = 6min 43.27sec
So the rate of samples received was 249956.6 which is just a bit low. This could be simple measurement error but in the elapsed time you would expect to receive 100817500 samples at 250000 sps - so there is an apparent deficit of about 1 sample in every 6 read operations.
The other important take-away for me was the variability in the number of samples reportedly returned. I had been assuming that the buffer would always be returned full and/or use the number of samples in the third parameter of the call. But it seems to be more variable than that and I need to take that into account when filling my larger working buffer.
I am currently using a separate thread to perform the readStream operations and so the scheduling of cpu cycles is determined primarily by the OS. The thread is also competing with fft processing in other threads for time on the same core. Additionally, at the moment I suspect that a large number of 'print' operations for debugging purposes is also stealing quite a lot of time.
I can probably live with losing 1 sample in 6000 but it would be nice to have a notification when something is dropped - then one could potentially insert an interpolated value.
DickH
@dickh768 - there's no guarantee that readStream() will always return the same number of samples; it depends on the size of the internal buffers and how many samples are returned by the receive callback of the SDRplay API.
This morning I changed that Python script to print out when the number of samples is not 1024, and I can see the blocks of 8064 samples you mentioned in your initial comment:
start: 2023-06-08 08:31:08.515095
2023-06-08 08:31:08.587252 7 - num samples: 896
2023-06-08 08:31:08.619854 15 - num samples: 896
2023-06-08 08:31:08.650775 23 - num samples: 896
2023-06-08 08:31:08.683760 31 - num samples: 896
2023-06-08 08:31:08.724854 39 - num samples: 896
2023-06-08 08:31:08.746652 47 - num samples: 896
2023-06-08 08:31:08.779432 55 - num samples: 896
2023-06-08 08:31:08.811581 63 - num samples: 896
2023-06-08 08:31:08.842418 71 - num samples: 896
2023-06-08 08:31:08.875296 79 - num samples: 896
2023-06-08 08:31:08.907564 87 - num samples: 896
2023-06-08 08:31:08.939485 95 - num samples: 896
end: 2023-06-08 08:31:08.971609
elapsed: 0:00:00.456514
nsamples: 100864
Since at a sample rate of 250kHz the receive callback is called with 252 samples every time, and 8064 = 252 * 32, I think what we are seeing are the buffers being rotated, but I am going to double check tonight after work.
Regarding your other question of detecting dropped samples, since the SDRplay API passes an argument to the receive callback function with the sequence number of the first sample for that callback, there's a way to detect when there is a gap in that sequence. A while ago I created a simple program to just stream the samples using the SDRplay API directly (https://github.com/fventuri/single-tuner-experiments), and you can see here how I detect when there are dropped samples: https://github.com/fventuri/single-tuner-experiments/blob/main/single_tuner_recorder.c#L673-L683
Franco
As an ex hardware engineer I really hanker for the days when I could get into the nuts and bolts of a system with oscilloscopes and logic analysers etc. With something like the RSP1 you only get to interact via several layers of abstraction - with hardware drivers, prebuilt apps like sdruno, generic tools like SDRSoapy and the associated extra driver to map onto the hardware driver etc... So I usually end up with a lot of trial and error to find out what actually works...
Back to the original issue - I think that is/was largely an issue of excessive debugging overheads delaying some of my readStream calls, so I need to be more careful with my programming. I also was not explicitly handling errors returned from readStream so I was probably occasionally failing to fill parts of my circular buffer and leaving old data to be re-used. (I did see a few errors when running the tests).
Doing some more work with your original test script, I've changed the timer to use time.perf_counter as that is reputed to be the best available for Python on Windows. I've also added a dummy readStream statement before the timing loop because the first call always seems to return a null buffer and I wanted to exclude that from the test loop. As for the size of the buffer, after a few experiments, the RSP1a seems most happy when using values of 1008, 2016, 4032, 8064 etc. - and then delivers consistent buffer lengths of returned data - I assume this is related to some hardware timing 'feature' around the DSP chip.
Having done all that, I get the curious result that the reported timing is actually shorter than it should be. It gives the impression that the sampling rate is actually higher than the 250k selected. However, since the error gets smaller as test runs get longer, it must be a spurious artefact - I can only assume there is some 'optimisation' in Python which is pre-emptively running the end timer before all the samples are returned.
In short, it now seems to be running fairly happily!
DickH
@dickh768 - since I am an electrical engineer too, I can relate to your experience with all the layers of abstraction.
Going back to the specific issue of the sample rate and the timing of the samples coming in, there's actually more to the story.
A few months ago while doing some tests for the integration of SDRplay with the Linrad application, I found an interesting and unexpected fact: the receive callback is not invoked by the SDRplay API at regular uniform intervals like one would expect; instead I found out that with a sample rate of 2Msps the receive callback is fired in rapid succession 95 times out of 96, and then the 96th time there a large delay (of about 40ms or more) between two successive invocations of the receive callback, almost as if the SDRplay stored those intermediate samples in some internal buffers, performed some operation on them (an FFT perhaps?), and then dumped all of them to client application by calling the receive callback function 96 times very quickly.
If you are interested I posted my findings to the Linrad mailing list: https://groups.google.com/g/linrad/c/3ySgbS592EY/m/Q-aMbqsIAQAJ
I haven't done more research to see if this also happens at a sample rate of 250kHz (which might explain some of the odd behavior you saw), but you can run the single_tuner_recorder program I mentioned last night (https://github.com/fventuri/single-tuner-experiments) with the '-T' flag, and it will show you the elapsed time between receive callbacks.
Franco
Thanks for the additional thoughts. I had a quick look at your github and Linrad links and found them very interesting. I'm a long-lapsed radio amateur (G3UWB) who dropped out when all the RF stuff was still analogue. My SDR experience up till now has just been based on pre-packaged solutions with the RTL sticks for general listening and decoding aircraft data, wireless data links etc. So this is my first attempt to directly drive an engineered SDR device. I was quite surprised that the Soapy interface does not have callbacks and maybe that explains some of the oddities - my other experience with video and audio streams have all used callbacks in the interface.
Given that my recent programming experience is mostly with Python and my C is a bit rusty, I'll probably persevere with the Soapy approach for a while. If my frustration crosses the threshold perhaps I'll follow your lead and try the direct API approach.
Thanks for your help
DickH
I had assumed that the problem of corrupted reception would be caused by samples dropped between each frame of data received from the RSP1a - either due to a glitch in the hardware, the API - or maybe in the Soapy adaption layer. What I am actually seeing though seems to be various faults within the frames being delivered from the RSP1a - or as they are processed by Soapy.
In all my tests at 192 ksps and a centre frequency of 96kHz, every frame has a decreasing amplitude for the final 5% of the frame (each frame is 4095 samples long at this sampling rate.) At 250ksps I don't see the decrease at the end but I have captured various strange amplitude variations in the I/Q signals and also a step change in phase of my 4kHz test signal mid-frame - corresponding to one or more missing samples.
Not having any real insight into the different software layers, I don't know whether these artefacts are intrinsic to the RSP1a or whether they are being introduced by one of the adaption layers. Can anyone point me in the right direction?
Attached image of recovered time domain sequence from 192ksps frame with 4kHz test input signal. (The glitches are also visible on the I and Q signals so they are not fft related).
@dickh768 - to rule out (or not) the SoapySDR interface layer, would you mind running the same tests building and running the single_tuner_recorder utility from the single-tuner-experients repository (https://github.com/fventuri/single-tuner-experiments)?
It allows you to record the I/Q stream from your RSP1A, and it uses just native SDRplay API calls, so it will be a good test to try to figure out where this problem comes from.
Also I would avoid having a sample rate (192ksps) that is exactly twice the centre frequency (96kHz) because of the known issues with the DC (0Hz) component.
Finally I am confused by your mention of the 250ksps sample rate - are you still running that with a centre frequency of 96kHz? If so you may have 'negative' frequencies since your sample rate is more than twice your center frequency, and I am not really sure what to expect in that case.
Franco
Thanks for the reply Franco.
Answering your last question first – when using 250ksps I also shifted the centre frequency to 125k to give myself the range 0 – 250 vs 0 to 192. My interest in 192 was partly because it is a multiple of common audio sample rates, but also because I naively assumed it might reduce cpu load and hence power – I now discover that you need to run the RSP with 3072ksps and decimation of 16 to get that rate - so the API actually uses more cpu than running at 2Msps and 8x decimation. I had also worried slightly about including 0Hz in the output but it is no big deal if I move the centre frequency up by 1kHz to avoid it.
Anyway, I had given up on the Python approach a couple of days ago because of the odd results and have just started getting some useful results with C on a Raspberry Pi. I started with your single tuner code and also the example from the SDRPlay website. I already had the API and Cubic working so I was reasonably sure things should work.
Compiling your code went fairly smoothly however I have been getting strange results with all 0’s in the Q branch. For testing I just edited your command lines. The console outputs were:
./single_tuner_recorder -r 6000000 -i 1620 -b 1536 -l 3 -f 162550000 -o noaa-6M-SAMPLERATE.iq16
SerNo=223804C199 hwVer=255 tuner=0x01
SR=6000000 LO=162550000 BW=1536 If=1620 Dec=1 IFagc=0 IFgain=40 LNAgain=3
DCenable=1 IQenable=1 dcCal=3 speedUp=0 trackTime=1 refreshRateTime=2048
streaming for 10 seconds
total_samples=19475568 actual_sample_rate=1954210 rounded_sample_rate_kHz=1954
I_range=[-580,610] Q_range=[-630,573]
./single_tuner_recorder -r 2000000 -i 1620 -b 1536 -l 3 -f 162550000 -o noaa-6M-SAMPLERATE.iq16
SerNo=223804C199 hwVer=255 tuner=0x01
SR=2000000 LO=162550000 BW=1536 If=1620 Dec=1 IFagc=0 IFgain=40 LNAgain=3
DCenable=1 IQenable=1 dcCal=3 speedUp=0 trackTime=1 refreshRateTime=2048
streaming for 10 seconds
total_samples=19936224 actual_sample_rate=1998404 rounded_sample_rate_kHz=1998
I_range=[-483,720] Q_range=[0,0]
./single_tuner_recorder -r 2000000 -i 1620 -b 1536 -l 3 -d 8 -f 162550000 -o noaa-6M-SAMPLERATE.iq16
SerNo=223804C199 hwVer=255 tuner=0x01
SR=2000000 LO=162550000 BW=1536 If=1620 Dec=8 IFagc=0 IFgain=40 LNAgain=3
DCenable=1 IQenable=1 dcCal=3 speedUp=0 trackTime=1 refreshRateTime=2048
streaming for 10 seconds
total_samples=2492028 actual_sample_rate=250030 rounded_sample_rate_kHz=250
I_range=[-58,616] Q_range=[0,0]
./single_tuner_recorder -r 3072000 -i 1620 -b 1536 -l 3 -d 16 -f 162550000 -o noaa-6M-SAMPLERATE.iq16
SerNo=223804C199 hwVer=255 tuner=0x01
SR=3072000 LO=162550000 BW=1536 If=1620 Dec=16 IFagc=0 IFgain=40 LNAgain=3
DCenable=1 IQenable=1 dcCal=3 speedUp=0 trackTime=1 refreshRateTime=2048
streaming for 10 seconds
total_samples=1911294 actual_sample_rate=192021 rounded_sample_rate_kHz=192
I_range=[-47,631] Q_range=[0,0]
As you can see, the Q-range is 0,0 on three of the runs.
Rather than try and debug your code I decided to try putting together some minimal code just for the RSP1a, stripping out a lot of stuff which I don’t need. So far I have got it basically running although I also have problems with the I/Q output which doesn’t seem to make sense. A bit of debugging is obviously required.
If you have any suggestions as to why we are not getting valid figures in the Q branch on your code, I’m happy to test things out.
Best Regards
Dick H
From: Franco Venturi [mailto:notifications@github.com]
Sent: 25 July 2023 14:11
To: pothosware/SoapySDRPlay3 SoapySDRPlay3@noreply.github.com
Cc: dickh768 dick.hall@btinternet.com; Mention mention@noreply.github.com
Subject: Re: [pothosware/SoapySDRPlay3] Missing samples with RSP1a (Issue #70)
@dickh768 - to rule out (or not) the SoapySDR interface layer, would you mind running the same tests building and running the single_tuner_recorder utility from the single-tuner-experients repository (https://github.com/fventuri/single-tuner-experiments)?
It allows you to record the I/Q stream from your RSP1A, and it uses just native SDRplay API calls, so it will be a good test to try to figure out where this problem comes from.
Also I would avoid having a sample rate (192ksps) that is exactly twice the centre frequency (96kHz) because of the known issues with the DC (0Hz) component.
Finally I am confused by your mention of the 250ksps sample rate - are you still running that with a centre frequency of 96kHz? If so you may have 'negative' frequencies since your sample rate is more than twice your center frequency, and I am not really sure what to expect in that case.
Franco
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned. Message ID: pothosware/SoapySDRPlay3/issues/70/1649819644@github.com
@dickh768 you are using Low-IF mode (If=1620) so therefore in the API the only sample rate that delivers full I/Q samples is 6 MHz (final sample rate will be 2 MHz after de-rotation and down conversion). Any other sample rate is single ended, hence why Q is 0
You can use Zero IF (If=0) and then you can use any arbitrary sample rate between 2 and 10 MHz and this will also deliver full I/Q samples (but be wary of the DC spike in the center of the spectrum)
You can also use decimation with either IF mode to reduce the final sample rate if you need lower than 2 MHz - I'm not quite sure if you are using SoapySDR or your own code, but I'm pretty sure the SoapySDRPlay library just took in a sample rate and set all the other properties for you.
Andy
@dickh768 As Andy wrote, with the SDRplay API a non-zero IF (Low-IF or LIF) is only possible for a very specific set of combinations of (sample rate before decimation, IF, IF bandwidth) that are listed on page 26 of the SDRplay API Specification Guide (https://www.sdrplay.com/docs/SDRplay_API_Specification_v3.07.pdf):
(fsHz == 8192000) && (bwType == sdrplay_api_BW_1_536) && (ifType == sdrplay_api_IF_2_048)
(fsHz == 8000000) && (bwType == sdrplay_api_BW_1_536) && (ifType == sdrplay_api_IF_2_048)
(fsHz == 8000000) && (bwType == sdrplay_api_BW_5_000) && (ifType == sdrplay_api_IF_2_048)
(fsHz == 2000000) && (bwType <= sdrplay_api_BW_0_300) && (ifType == sdrplay_api_IF_0_450)
(fsHz == 2000000) && (bwType == sdrplay_api_BW_0_600) && (ifType == sdrplay_api_IF_0_450)
(fsHz == 6000000) && (bwType <= sdrplay_api_BW_1_536) && (ifType == sdrplay_api_IF_1_620)
For all the other values of sample rate and IF bandwidth, you have to set IF=0 (Zero-IF).
Franco
Thanks Andy, Franco for the pointers. The results now look very good.
With SR = 3072000 and Dec= 16 etc.
./single_tuner_recorder -r 3072000 -i 0 -b 200 -l 3 -d 16 -f 97000 -o bw200-SAMPLERATE.iq16
SerNo=223804C199 hwVer=255 tuner=0x01
SR=3072000 LO=97000 BW=200 If=0 Dec=16 IFagc=0 IFgain=40 LNAgain=3
DCenable=1 IQenable=1 dcCal=3 speedUp=0 trackTime=1 refreshRateTime=2048
streaming for 10 seconds
total_samples=1917279 actual_sample_rate=192214 rounded_sample_rate_kHz=192
I_range=[-2674,2748] Q_range=[-2668,2692]
The IQ file is at https://www.dropbox.com/scl/fi/grvcdm79e1gouoqtwkak0/bw200-192.iq16?rlkey=vt266axwblxd63fjixmalyfrz&dl=0
Converting to a .wav file and feeding into HDSDR I can then see a pretty clean spectrum with a spike 3kHz from the LH end (I’ve offset the centre to 97kHz). The results are pretty similar for 250kHz (2Msps & Dec = 8).
So it seems to me that the RSP and API are working as expected when accessing it directly through C – once you know the allowed combinations of parameters :-) . Which therefore points towards a probable issue within SoapySDR – at least for that combination of SR & Decimation.
NB: I also need to do a bit more work to check for phase discontinuities in the 3kHz recovered signal - just need to get my head around fftw.
As a further update on my investigations. I have plotted the average received sample rate over a period of ~ 30 seconds, with the RSP1a configured to sample at 3072Msps and 16x decimation = 192ksps.
This shows that the delivered samples are actually arriving at ~ 195ksps and then every 2 seconds there is a pause to align the mean rate to 192000. I am hoping that the actual sample rate is a constant 192ksps. In any case an application needs a buffer of > 6400 samples to avoid running out of data.
DH
@dickh768 - thanks for your analysis.
How did you exactly compute the 'average received sample rate'?
I just ran the single_tuner_recorder program with the -T flag to measure the time difference between successive rx_callbacks selecting a sample rate of 3.072MHz decimated by 16, the same way you did:
single_tuner_recorder -T -r 3072000 -d 16 -x 3 -b 200 -f 97e3
since at this sample rate of 192kHz each rx_callback receives 63 samples, I would have expected a time difference of 63 / 192k about 328us - however the output from the command above shows that every 96th callback the time difference is much greater (about 31ms; the last column is the time difference in ns):
96 63 30170784
192 63 30490210
288 63 31292853
384 63 29902210
480 63 30948309
576 63 29897036
672 63 32857799
768 63 29095010
864 63 31171074
I am not really sure about the reason of this behavior, where 95 out of 96 times there's virtually no time elapsed between two rx callbacks while on the 96th time the time difference is very large.
Perhaps there's some buffering that occurs inside the SDRplay API (since the code in single_tuner_recorder calls the SDRplay API functions directly), or some other DSP operation that would explain this 'stop-and-go' behavior.
Franco
Hi Franco, my application is currently configured to so that the callback routine fills a series of buffers (4095 samples for the test). I then use a queue to tell the main thread when a new buffer has been filled. My timings were therefore taken from the point at which each buffer was full - at that point I wrote the time to a file (using gettimeofday() . I also wrote the start time to the file (from within the 'reset' block) which I used to calculate the long term frequency.
The actual timing figure recorded for the start time is rather strange - and I have had this before - I assume it is because the gettimeofday statement is in a different block of code and is being executed out-of-order in some way - and hence does not give a true start time. I'm guessing that an incorrect start time is the most likely reason for the graph showing the frequency dropping asymptotically over time - as the error becomes less relevant - although I suppose it could be a genuine artefact when the API is first started.
I was prompted to collect this data because the buffer index numbers in my queue sometimes appeared to be out of sequence (with 4095 sample buffers). Clearly with the timing jumps of the order of 6400 samples I will need to consider larger buffers.
Also, seeing the magnitude of the timing steps in these direct API tests, I suspect that the choice of 4095 as the block size in SoapySDR for the 192kHz rate is too small to smooth out these glitches, and is probably why I had various issues with the Python approach.
DickH
- And the frequency shown is (total samples received)/(total time elapsed)
Update - by going back to the original data, and by ignoring the nominal start timestamp, I can see that the frequency curve is genuinely a 'feature' of the RSP itself rather than an erroneous start timestamp.
So putting my system designer's hat on, this looks to me like the response of a phase locked loop (in software) which is trying to respond to a delay in getting the callbacks working but has already started receiving samples from the ADC. It therefore has to run fast for a period until the excess samples have been used up.
To confirm your findings I wrote the attached C++ program based on SoapySDR C++ example (https://github.com/pothosware/SoapySDR/wiki/Cpp_API_Example).
I chose a buffer size of 4096 I/Q samples, which means that at a sample rate of 192 ksps, it should take about 4096 / 192k = 21.3 ms for each buffer to be ready. I used clock_gettime(CLOCK_REALTIME, ...) to get the timings for readStream(), and I added a print whenever the time is too fast (i.e. two consecutive readStream() in less than 15ms or too slow (i.e. two consecutive readStream() in more than 28ms), and this is an example of what I see here:
too slow - count = 2, time_diff = 38595585 ns
too fast - count = 3, time_diff = 6072681 ns
too slow - count = 5, time_diff = 30166037 ns
too fast - count = 6, time_diff = 3985452 ns
too slow - count = 7, time_diff = 31270063 ns
too slow - count = 8, time_diff = 28763640 ns
too fast - count = 9, time_diff = 5681552 ns
too slow - count = 10, time_diff = 29528005 ns
too slow - count = 11, time_diff = 28946036 ns
too fast - count = 12, time_diff = 5692327 ns
too slow - count = 13, time_diff = 30075115 ns
too slow - count = 14, time_diff = 28801604 ns
too fast - count = 15, time_diff = 6900704 ns
Franco
I tried compiling your code but got a library error which I can't immediately resolve. I found some forum comments suggesting that this can occur with newer versions of gcc .
make (in directory: /home/dickh/c_progs/Francocpp)
cc SoapySDRExample.o -lSoapySDR -o SoapySDRExample
/usr/bin/ld: SoapySDRExample.o: undefined reference to symbol '_ZTVNSt6thread6_StateE@@GLIBCXX_3.4.22'
/usr/bin/ld: /lib/arm-linux-gnueabihf/libstdc++.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make: *** [: SoapySDRExample] Error 1
Compilation failed.
Any suggestions?
In the meantime I looked at the distribution of my buffer-fill times:
It works on Linux Fedora 38 with gcc 13.2.1; think in your case you might need to add -lpthread, i.e. try changing the first line in the Makefile to:
LDLIBS=-lSoapySDR -lpthread
Franco
The -lpthread suggestion unfortunately didn't solve the problem. However I found this explanation at https://linuxpip.org/how-to-fix-dso-missing-from-command-line/ . The workaround is to run:
export LDFLAGS="-Wl,--copy-dt-needed-entries"
prior to running make.
So, running your program I get:
Found device #0: driver = sdrplay
label = SDRplay Dev0 RSP1A 223804C199
serial = 223804C199
[INFO] devIdx: 0
[INFO] SerNo: 223804C199
[INFO] hwVer: 255
[INFO] rspDuoMode: 0
[INFO] tuner: 1
[INFO] rspDuoSampleFreq: 0.000000
Rx antennas: RX,
Rx Gains: IFGR, RFGR,
Rx freq ranges: [1000 Hz -> 2e+09 Hz],
[INFO] Using format CF32.
too fast - count = 3, time_diff = 8906555 ns
too fast - count = 6, time_diff = 11485184 ns
too slow - count = 7, time_diff = 36249395 ns
too fast - count = 9, time_diff = 12534604 ns
too slow - count = 13, time_diff = 31429323 ns
too fast - count = 16, time_diff = 13222048 ns
too fast - count = 18, time_diff = 12175023 ns
too slow - count = 19, time_diff = 34515605 ns
too fast - count = 21, time_diff = 12241116 ns
too fast - count = 24, time_diff = 9637332 ns
too slow - count = 25, time_diff = 29400118 ns
too fast - count = 30, time_diff = 13196423 ns
too fast - count = 34, time_diff = 12467730 ns
too fast - count = 37, time_diff = 12659135 ns
too fast - count = 40, time_diff = 12390855 ns
too fast - count = 43, time_diff = 13445067 ns
too fast - count = 46, time_diff = 12966477 ns
too fast - count = 49, time_diff = 13076684 ns
too slow - count = 50, time_diff = 32478119 ns
too fast - count = 52, time_diff = 13208246 ns
too fast - count = 53, time_diff = 14645215 ns
too fast - count = 55, time_diff = 12952154 ns
too fast - count = 58, time_diff = 13289755 ns
too fast - count = 61, time_diff = 12726479 ns
too fast - count = 65, time_diff = 10054620 ns
too fast - count = 68, time_diff = 12724708 ns
too slow - count = 69, time_diff = 33461445 ns
too fast - count = 71, time_diff = 11966795 ns
too fast - count = 74, time_diff = 12255023 ns
too fast - count = 77, time_diff = 13437776 ns
too slow - count = 79, time_diff = 33203791 ns
too fast - count = 80, time_diff = 9545093 ns
too fast - count = 83, time_diff = 14598237 ns
too fast - count = 89, time_diff = 12525489 ns
too fast - count = 92, time_diff = 13324078 ns
too fast - count = 96, time_diff = 12516167 ns
too fast - count = 99, time_diff = 12605594 ns
too fast - count = 102, time_diff = 12831009 ns
too fast - count = 105, time_diff = 12996581 ns
too fast - count = 108, time_diff = 13246058 ns
too fast - count = 111, time_diff = 13082413 ns
too fast - count = 114, time_diff = 12733978 ns
too fast - count = 117, time_diff = 12741165 ns
too slow - count = 118, time_diff = 32188589 ns
too fast - count = 120, time_diff = 13528192 ns
too fast - count = 123, time_diff = 13272256 ns
too fast - count = 126, time_diff = 10516754 ns
too fast - count = 130, time_diff = 9890247 ns
too slow - count = 131, time_diff = 36364030 ns
too fast - count = 133, time_diff = 9994674 ns
too slow - count = 134, time_diff = 28349447 ns
too fast - count = 136, time_diff = 12314919 ns
too fast - count = 139, time_diff = 11976639 ns
too fast - count = 142, time_diff = 12935071 ns
too fast - count = 145, time_diff = 12715176 ns
too slow - count = 147, time_diff = 34845811 ns
too fast - count = 148, time_diff = 10350400 ns
too fast - count = 151, time_diff = 9863268 ns
too slow - count = 152, time_diff = 38171883 ns
too fast - count = 154, time_diff = 14602194 ns
too fast - count = 157, time_diff = 9523478 ns
too slow - count = 160, time_diff = 35961324 ns
too fast - count = 161, time_diff = 9482386 ns
too slow - count = 163, time_diff = 36598299 ns
too fast - count = 164, time_diff = 9523323 ns
too fast - count = 167, time_diff = 9548947 ns
too slow - count = 169, time_diff = 36196583 ns
too fast - count = 170, time_diff = 9487124 ns
too fast - count = 173, time_diff = 9590561 ns
too fast - count = 176, time_diff = 9513948 ns
too slow - count = 177, time_diff = 36344811 ns
too fast - count = 179, time_diff = 10220974 ns
too slow - count = 181, time_diff = 36572778 ns
too fast - count = 182, time_diff = 9617697 ns
too slow - count = 184, time_diff = 28138095 ns
too fast - count = 185, time_diff = 14024855 ns
HTH
While playing with the best buffering strategy, I have noticed an additional issue in the context of the Raspberry Pi. As each buffer fills up, I notify the main thread to begin processing - primarily a series of short, overlapped FFTs. If I now look at the amplitude of my test signal in the FFT outputs, you can see a periodic fall in amplitude corresponding to the filling of each buffer and the start of the processing in the main thread.
Since all Linux threads in an application, by default, all run on the same core, it looks like the extra load from the FFTs is disrupting the callback process.
In the various earlier tests, the total number of callbacks registered seems to be correct, so I think in this case, contention for processor time is causing some callbacks to be cut short - before all samples have been copied. This effect will be exacerbated by the unevenness we have seen in the callback periods.
I'm now looking at moving the processing onto another core - in the short term probably using sockets but I'll see how I get on.
(For reference, if I disable the FFT's and just save all samples to a file, the signal looks very clean.)
[ BTW I now have a busy period with visitors etc so I won't be doing much tech stuff for 2 or 3 weeks ]
@dickh768 - thanks for the update and your experiments; I too am busy with work and a couple of other parallel SDR projects, so I apologize if I can answer you only every few days.
Your observation regarding the scheduling of processes and threads on the Raspberry Pi reminded me of another comment on the gr-sdrplay3 repository that I also maintain (
fventuri/gr-sdrplay3#27 (comment)) - in their case the scenario was with trying to stream from two RSPduo's, however I found it interesting that they saw some odd behavior with their Raspberry Pi while they report that on their laptop (presumably a x86_64 architecture) there are no problems.
Also SoapySDR offers another streaming interface called 'Direct buffer access API' (https://github.com/pothosware/SoapySDR/wiki/DriverGuide#direct-buffer-access-api), and I think the SoapySDRPlay3 driver implements it too.
I haven't used it (and it looks like it is only available from C/C++, and there are no Python bindings for it), but perhaps it could be something worth looking into.
Franco
I find a little time away from the bench is often useful to gain new perspectives on a problem.
My main new discovery is that the RSP1a takes around 50mS from the start signal before it delivers stable data. Prior to that, the output contains all sorts of amplitude glitches and frequency errors etc. Therefore many of my earlier tests must be regarded as invalid because they used the first few frames of data.
As an example, I went back to a simple Python SoapySDR script on the PC and looked at the phase of my 4kHz test signal over time as seen by the RSP1. I simply recorded around 100 x 4095 sample frames (at 192kHz) to a disk file and then did an fft on successive blocks of 384 I/Q samples. This gives 0.5kHz frequency bins - so bin 8 contained the amplitude and phase of my test signal.
Plotting the phase, there is significant chaos for the first 50mS and then the phase signal becomes nice and clean - indicating that no samples are being dropped (over the 2s period at least). The evident regular drift is caused by a frequency error of around 10Hz in setting my test oscillator.
A second unexpected oddity concerns the spectrum obtained using the API via C. On the Raspberry Pi and simply forwarding the I/Q samples to a data file, I can get a very clean spectrum with a sampling rate of 2000kHz and decimation of 8x. This agrees very well with the spectrum seen on the PC with SDRUno. With most other sample rates, significant semi-random spurii of 8 – 15dB appear across the spectrum (eg with 16x decimation and 3000 or 3072ksps. )
As a cross-check I thought I would try with CubicSDR also on the RPi – and observed the same problem. At 2000ksps/8x the noisefloor is clean but at virtually all other sample rates the noisefloor is polluted by unwanted spurii. Simply recording I/Q samples using Python + SoapySDR avoids this problem so it is not intrinsic to the Rpi, but presumably to some configuration setting.
For my purposes, it looks like I can get the Rpi to work OK by just doing the sampling on one core and then using a named pipe to send the samples to a separate app for processing. The buffer in the pipe should take up the slack from the bursty delivery of samples. I will need to see whether I need to force the two apps onto different cores or whether I can rely on the scheduler to make the right choices.