AudioWorklet should have a 'oversample' and a 'bufferSize' property

[Hazmatron7]

I was recently thinking about some ideas with web audio and I came to my head that,
could it be possible to add a oversample and bufferSize property to AudioWorklet?
This is could be a very useful feature for when making custom nodes.
While the oversample property does make the bufferSize property seem useless since oversampling is basically (bufferSize * n or basically 128 * n), I'm suggesting that:

• the oversample property:

  • is used for oversampling the selected bufferSize that is used for processing to the best it can achieve.
  • can be used to achieve high quality
  • can be set with the same values as WaveShaperNode.prorotype.oversample.

• the bufferSize property:

  • can be set to a preferred buffer size. This could be good for performance, even though it is on worker thread.
  • can only be set on the AudioWorkletOptions and the can be only a read only property after initialization.
  • value is applied to all inputs and outputs of the processing block.
  • either can accept values limited to what the ScriptProcessorNode uses (a.k.a 128 - 16384). Or to a wider range base on performance and support of the worker thread and or hardware (a.k.a 128 - ?).
  • default value should be 128.
  • could be achieved with ringbuffer since the audioWorklet process only 128 frames per block. Take a look at this code of a ring buffer used in p5.js-sound library. I have tested it and works well and very handy. I definitely suggest to take a look at the way it's been used.

I also realized that this could also affect the custom parameters especially with automation rates. My suggestion to work around this is that for the parameters with a k-rate that if the oversample is not set to 'none' then the value is multiplied and for when the buffer size is greater than 128 frames the parameter produces values similar to a a-rate parameter but has 1 value for each block (a.k.a per 128 frames) from the total buffer size.

For example: say the buffer size is 512. 512 divided by 128 equal 4 blocks per 128 frames. Since there a 4 blocks in 512 the k-rate parameter should result in 4 different (or the same depending on the value prop in the main thread), values for each 128 frames per block out of 512 frames.

And for the a-rate parameters, the value will result the same method for the oversample property. In other words the value is multipled if the oversample property is not set to 'none'. Besides that buffer size shouldn't matter since an a-rate parameter does not produce 1 value per 128 frames If you know what I mean.

Also (I'm not sure about this part since it's on a worker thread but), since oversampling and different buffer sizes can cause a little bit of latency (I think), Could it be possible to report the latency in ms as a property in both the main and worker thread?

Also this should also work with respect of the AudioContext sampleRate as well.
Similar to the step process that WaveShaperNode does that must follow when using the oversample property according to the spec.

here's an example of what it could look like:

//main thread:
let worklet = new AudioWorkletNode(context, name, {
  bufferSize: 256,
  oversample: "none",
  //etc...
});

worklet.oversample;  //setter and getter. returns "none"
worklet.bufferSize; //read only! returns 256
worklet.latency; //read only!  returns a number in ms the latency of the processor from input to output


//worker thread:
class MyProcessor extends AudioWorkletProcessor {
   
   process(inputs, outputs, params) {
      this.latency; //read only!  returns a number in ms the latency of the processor from input to output
      this.oversampleType; // read only! number - returns 0 for 'none' and a number for the oversample factor. i.e returns 2 if oversample = "2x"
      this.bufferSize; //read only! number - returns the original selected buffer size from the AudioWorkletNodeOptions.
     
     //this code here is used to get the result of the buffer size for when it's oversampled:
     let oversampledSize = this.bufferSize * this.oversampleType;

     //params:

     //for k-rate parameters: 
     //since the buffer size is 256. 256 / 128 is 2 blocks. So there will be 2 values for each 128 frames for k-rate parameters. 
     //params["myParam"].length === 2
     let a = params["myParam"][0]; //returns the value for the first 128 frames
     let b = params["myParam"][1]; //returns the value for the next 128 frames

     //for a-rate parameters:
     //the buffer size doesn't matter. It's basically 'params["myParam"].length === bufferSize (256 in this example)
     let c = params["myParam"][0]; //returns the first value out of 256 values
     

   //for when either a k-rate or an a-rate parameter with oversample property is not set to "none",
   //the value is multiplied otherwise it's the original value
   //To achieve the original value, just divide it by the oversample amount:
    let originalValue = params["myParam"][0] / this.oversampleType;
   
   }
}

The goal of this feature is to achieve high quality processing and better performance in the worker thread with the audio worklet.
I hope i made sense and it's not too complicated. Also please correct if I'm wrong as well.
Do you think this is cool idea?
And could this be possible or not?
Thanks :)

[rtoy]

IIUC, you want to specify an oversample property that takes the inputs and oversamples them and makes that available to the input to the worklet node instead of the original input data If so, that puts an even greater burden on the worklet to process the data in time. Of course, since the worklet can have arbitrary code in it, you could do the oversampling as part of the node itself. (Of course that adds even more processing to the node, but somebody has to do the oversampling somewhere and that will impact how much time there is available to do the processing anyway.)

For the buffer size, would #13 work for your use-case? The input to the worklet will now have the length that is specified for the context. It won't be selectable per node, but maybe that's ok for your use-case.

[Hazmatron7]

I see what you mean about the oversample part where you mentioned about how much processing it can cause, but to be honest with you I didn't fully think it through (if you know what I mean). Well I did it's just I wasn't sure how it could work in usage wise. But I get what you mean though.

And for the bufferSize, I guess it could fit that case after looking at the issue #13. It also would make sense base on what #13 says because I use Ableton Live and If you got Options > Prefeences in the audio tab, there is a section called latency and there is a number input called 'output buffer size' and that sets the quantum size for processing in live and to the audio hardware.

Fun fact: Did you Ableton live uses the same audio system as Web Audio API. The both use float 32 bit systems.

To be honest with you, I suggested this idea because it would be great to bring back some features from the old school ScriptProcessorNode into the audio worklet and also since the WaveShaperNode has a sort of similar custom processing method, It could be possible to add a oversample for the audio worklet since it does custom processing on 128 frames per block to achieve a high quality.

Hope what I said make sense.
forgive me for my bad spelling and or my bad grammar. :)

[rtoy]

I would expect most systems to use 32-bit float because fixed point systems are really hard to get all the scaling correct without lots of care.

I'm not really sure (too long ago), but I think WaveShaperNode has oversampling because people complained that, depending on the curve, the result was too coarse and caused funny glitches.

I think it's unlikely that we'd allow oversampling for a Worklet, but #13 will work for your use case, once it gets written up in the spec and implemented by browsers.

[rtoy]

Teleconf: After some discussion today in the weekly teleconf, we've decided not to add these. Oversampling is best done by the AudioWorklet because it can make the right tradeoffs in the oversampling factor and oversampling quality. These would be hard to specify and get right if browsers did it for you.

Also, we think the user-selectable rendersize will satisfy the bufferSize property you wanted, since it's basically the same thing, except it applies to all audio nodes.

Thanks for your suggestion. Please file new issues if you discover things that you'd like to have that WebAudio doesn't already provide.

[Hazmatron7]

I see your point and I can understand why the oversampling is not going to be added. And for the buffer size. After what you said and from issue #13 says I think that user-electable a good idea and would definitely cover what i suggested after thinking about it.

Thank you for taking this idea in.
If I have any other issues or cool Ideas to suggest for WebAudio will definitely file a new issue to this.
Again Thank you very much 😄 👍
Can't wait to see the user-selectable buffer size idea when its implemented