Data loss while recording

[burnumburnum]

Hey Bastian,

nice library. I'm using it since it seems to be the only python library that supports 24-bit souncards (in contrast to pyaudio and sounddevice that return just additional zeros).

My problem may be related to #137. I want to continuously record data (more or less 24/7). Regularly, I need to process and/or store data. But whenever doing something except reading, I loose data. It seems that the internal buffer is very short. Is there a way to increase the buffered data? I tested different values of numframes (also None) and blocksize without success.

In this example, I record chunks of 0.1 seconds and sleep for 0.001 seconds after reading. The time recorded and the time passed drift appart.

import soundcard as sc
import time

mic24bit = sc.get_microphone('my24bitMicName')

t_recorded = 0.0

with mic24bit.recorder(samplerate=48000, channels=1) as mic:
    t0 = time.time()
    while True:
        data_tmp = mic.record(numframes=4800)
        t_recorded += len(data_tmp) / 48000
        t_passed = time.time() - t0
        
        print(len(data_tmp), t_recorded, t_passed, t_passed - t_recorded)

        # do something - simulated by sleeping a very short time
        time.sleep(0.001)

Thank you for your help.

[Chum4k3r]

Hi burnumburnum Sorry I'm not bastibe, but I do solved this problem several times using both sounddevice and soundcard. The thing is that the process that runs your audio I/O must run separate from the process that runs DSP or disk I/O. This means that your audio recording loop should pipe the data arrays to some queue, and in another process, the DSP or disk I/O consumes from this queue and does its thing. This allows your audio process to run continuously with really low latency, while your other heavy cpu tasks runs on their own heartbeat, and unbounding both tasks from each other. Also, just as a tip, you should use ´time.perf_counter()´ to measure execution times, rather than ´time.time()´ Hope it helps, JV Em qua, 1 de fev de 2023 12:02, burnumburnum ***@***.***> escreveu:

…

Hey Bastian, nice library. I'm using it since it seems to be the only python library that supports 24-bit souncards (in contrast to pyaudio and sounddevice that return just additional zeros). My problem may be related to #137 <#137>. I want to continuously record data (more or less 24/7). Regularly, I need to process and/or store data. But whenever doing something except reading, I loose data. It seems that the internal buffer is very short. Is there a way to increase the buffered data? I tested different values of numframes (also None) and blocksize without success. In this example, I record chunks of 0.1 seconds and sleep for 0.001 seconds after reading. The time recorded and the time passed drift appart. import soundcard as sc import time mic24bit = sc.get_microphone('my24bitMicName') t_recorded = 0.0 with mic24bit.recorder(samplerate=48000, channels=1) as mic: t0 = time.time() while True: data_tmp = mic.record(numframes=4800) t_recorded += len(data_tmp) / 48000 t_passed = time.time() - t0 print(len(data_tmp), t_recorded, t_passed, t_passed - t_recorded) # do something - simulated by sleeping a very short time time.sleep(0.001) Thank you for your help. — Reply to this email directly, view it on GitHub <#165>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJPYUKGHUFJYNIB2QO5ONMLWVJ3J7ANCNFSM6AAAAAAUN2XKK4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[bastibe]

There are various avenues for addressing this:

• Bump up the blocksize in the recorder (not record!), which will tell the OS to allocate larger buffers for you. Whether your OS will actually follow that instruction is up to the audio driver. On Windows, it is often ignored.
• Move all soundcard code into its own thread. Real time does not wait for anybody, regrettably, so we need to make sure it absolutely runs regularly. Windows in particular is not resilient at all against missed reads. If you do file-IO in particular, this needs to happen in its own thread or -if necessary- process. That said, typical OS block sizes are on the order of 10ms, so a one-ms wait should never be an issue.
• Reduce system load. Windows audio in particular tends to get hung up when too much IO/CPU is consumed by other processes. Back in the CD era, it was almost impossible to play an entire audio CD without hiccups in software. But WASAPI did actually improve things.
• Be aware that audio clocks do drift significantly. A second of drift per hour is not unheard-of. If the drift you are seeing is small, it might just be clock drift, instead of data loss.
• On Linux, data loss should be handled by the OS, and never occur. On macOS, data loss should be catastrophic immediately, and crash loudly. On Windows, try the latest git version, which will issue a warning when a data discontinuity is discovered.

What operating system are you using? I'm assuming Windows, right?

[burnumburnum]

Thank you both for your answers. So far, I'm using sounddevice on a Windows machine. Unfortunately, I cannot switch to Linux. Currently, the process is reading, compressing and writing data to pipe and file without significant data loss (in the order of 0.1s per day!). The main data processing is already in separated processes. I hoped to avoid changing the existing code much.

I tried again to play around with blocksizes and it helped. I was pretty sure that I put it in the reorder, not record, but maybe I mixed it up yesterday. With a blocksize of 4800 I could increase the sleep to more than 0.01 s without (significant) time loss. However, without specifying the blocksize, the time drift was more than one second per minute even without the additional sleep.

By the way, increasing the process priority in windows for crucial processes can also help. I used this once for reading from some sensor device. I'm not sure if this can be done thread-specific as well, though:

import win32api
import win32process
import win32con
pid = win32api.GetCurrentProcessId()
handle = win32api.OpenProcess(win32con.PROCESS_ALL_ACCESS, True, pid)
win32process.SetPriorityClass(handle, win32process.ABOVE_NORMAL_PRIORITY_CLASS)

To the solution in summary:

• own thread just for reading and piping data, separate thread for data processing
• increase blocksize in recorder, not record
• increase process priority if desired

Thanks! I think I will get it work now.

[bastibe]

Thank you for your report! Good luck!