ryanheise/just_audio

Pitch adjustment

ryanheise opened this issue · 56 comments

ryanheise commented

Is your feature request related to a problem? Please describe.

Pitch adjustment would be useful in some apps (e.g. changing pitch of voice or music).

Describe the solution you'd like

player.setPitch(1.0); // 1.0 being normal pitch

Describe alternatives you've considered

There is no other way to achieve this other than plugin support.

Additional context

This is easily implemented on Android, iOS might be more problematic (post below if you are able to share useful resources/documentation)

nt4f04uNd commented

i think you should add it to android and then some time later implement it on iOS

ryanheise commented

True.

I should probably mention that all new features from this point will only be developed on the null-safe branch (i.e. the 0.7.* releases). For now, this will require the beta channel of the Flutter SDK, although stable is coming soon.

ryanheise commented

Note to self. This API might integrate well with AVQueuePlayer:

https://developer.apple.com/documentation/avfoundation/avaudiomixinputparameters/1387042-audiotimepitchalgorithm?language=objc

(Though I'm not particularly happy with the quality of iOS's pitch/time stretching algorithms, if the above works, I may as well use it.)

leidig54 commented

This would be another massive winner for me. Any eta? As with the equalizer functionality, happy to offer a bounty to bump the priority.

ryanheise commented

Hi @leidig54

Thanks again for offering to support the project!

I will ordinarily prioritise features based on the number of thumbs up and their difficulty, although I also tend to prioritise requests from GitHub sponsors too (otherwise, you're welcome to discuss by email - the address can be found in the git log).

The next two big features I plan to work on are the visualizer and equalizer, but I may throw the pitch shifting feature into that group, too, since there may be some overlap in the section of code involved on the iOS side.

ryanheise commented

It definitely has overlap with what's happening on the visualizer branch and I want to ensure I get the overall design right in how these features interact, so I'm looking into this now.

ryanheise commented

As I originally suspected, this feature would best be handled by a move to AVAudioEngine on iOS.

Although there might be ways to get this to work with AVQueuePlayer, maybe it is finally time to make the switch to AVAudioEngine, given that a lot of the popular feature requests at the moment involve audio processing. This would also allow complete control over audio interruptions since AVQueuePlayer is hardcoded to take over this responsibility.

There are some existing libraries built over the top of AVAudioEngine that could make things easier, which I will have a look at first. For example, AudioKit seems to provide everything we need in terms of audio processing, and there are others (and there's the option of just using the AVAudioEngine API directly), but I will just need to evaluate these options based on how well they support the complete just_audio feature set.

I'd estimate the rewrite of existing features at maybe 1-to-2 months at a minimum, if worked on full time, after which I would expect pitch shifting to be an extra couple of days on top of that.

Given that, I think I may go ahead and release the feature for Android first.

  • ❤️5
nt4f04uNd commented

i really love the direction you are taking in working on your audio plugins, and how you implement features. these are by far the most mature and will-maintained flutter plugins

  • 👍6
  • 🚀1
ryanheise commented

The Android implementation is now available for testing on the pitch branch.

  • 👍3
ryanheise commented

For the first time with this project I'm starting to feel the need for a dev branch (we have a number of features that will need to be tested together before being stable) - I'll do some restructuring.

afkcodes commented

@ryanheise completely agreed with you here, just a suggestion we can also update the readme for the new features in branch. Also eady to support the project, for the good work you are doing.

  • 👍2
ryanheise commented

dev is now created. It contains the new pitch feature (Android only so far) and also #300 (buffer configuration options). My immediate goal is to bunch together the features that will involve changes to the platform interface so that I can do a single release (because the release process for changes to the platform interface is usually time consuming.) I will probably also include in this bunch the "skip silence" feature (just Android for now). Once that's all done, I can focus on the iOS implementation.

  • ❤️2
ryanheise commented

(Copying this comment from another issue to get broader interest)

Is this supported on iOS currently?

The waveform visualizer is implemented on iOS but not pitch. You can track the pitch feature here: #329

There is a big question at this point whether to continue with the current AVQueuePlayer-based implementation or switch to an AVAudioEngine-based implementation. For pitch scaling, I really want to take advantage of AVAudioEngine's built-in features, but that requires a rewrite of the iOS side - see #334 and this is a MUCH bigger project.

I would really like to see an AVAudioEngine-based solution see the light of day, but it will probably not happen if I work on it alone. If anyone would like to help, maybe we can pull it off with some solid open source teamwork. One of the attractive solutions is to use AudioKit which is a library built on top of AVAudioEngine which also provides access to pitch adjustment AND provides a ready-made API for a visualizer and equalizer. That is, it provides us with everything we need - BUT it is written in Swift and so that involves a language change and it means we may need to deal with complaints that old projects don't compile (we'd need to provide extra instructions on how to update their projects to be Swift-compatible).

Would anyone like to help me with this? (Please reply on #334)

smakarov commented

Does it possible to use BASS or Superpowered(best for pitch shifting) or you would like to implement this by yourself?

ryanheise commented

@smakarov this should be relatively straightforward to do once the AVAudioEngine / AudioKit implementation is done. See:

https://audiokit.io/playgrounds/Playback/Time%20Stretching%20and%20Pitch%20Shifting/

So the main effort will go into #334 itself.

akashrajkn commented

Is it also possible to get the pitch of recorded audio?

BTW, this plugin very helpful, thanks for developing it :)

Deleted user commented

Hi,
is there any update for this feature?

ryanheise commented

@gkmuhannad this is already implemented on Android (on the dev branch), iOS will take quite a while longer. I will publish a release including the Android implementation as soon as I update the unit tests.

Deleted user commented
ryanheise commented

The Android implementation is now published in release 0.8.0.

geekmuhannad commented

@ryanheise
Hello bro
is there any way to save the audio after applying the pitch ?

ryanheise commented

is there any way to save the audio after applying the pitch ?

This plugin is focused on playing audio, not recording or editing it. Those would radically change the way this plugin is designed. But if you can make a good case for including those features, you can make a separate feature request. I think the direction I'm going on iOS may be compatible with this, but the Android side would need a rethink because that's based on ExoPlayer which is also purely focused on playing audio and not recording or editing it.

Deleted user commented
ryanheise commented

Maybe editing is possible with ffmpeg but the license isn't ideal for mobile app distribution (maybe you could host a server to do the ffmpeg processing.)

demberto commented

@gkmuhannad I am wondering why nobody mentioned rubber band library, its under GPL tho, but its quite good at what it does and is used in pro audio apps and supports android and ios both. If you are still unsatisfied with that, your only bet is the industry standard zplane (4Tune, Elastique, or Retune) based on what you exactly need. It doesn't mention android support and is proprietary. But its the best, if you want it

ryanheise commented

@demberto thanks for contributing another possible solution, although AudioKit is probably the way to go since it doesn't have the same licensing issues that the GPL has, and AudioKit also solves a large number of other requested features, including the visualiser.

  • 👍1
demberto commented

Always happy to help

SarthakZunroof commented

I would like to get change in frequency or change in pitch of an external sound(through microphone) and than animate my visualizer according to the change.If possible give me the solution.

ryanheise commented

@SarthakZunroof please find another plugin that deals with the microphone and make a feature request with that project. It is a bit off topic for the present issue so I will mark both your comment and mine as off topic.

kekko7072 commented

Any update on this topic? I need a package to manage pitch and tune, any suggestion?

ryanheise commented

There is some progress being made in #334 on an AVAudioEngine-based implementation. This should make it much easier to implement this functionality on iOS.

Some notes for implementation, these are the classes to look at to adjust the speed and pitch of audio:

  • AVAudioUnitVarispeed
  • AVAudioUnitTimePitch
watrindade commented

@ryanheise I'm having trouble trying to use it in the version 0.9.20. Is there a way to raise a song pitch just one half step? I mean, a song in C goes to C#?

Thank you the amazing work you have been doing to the community. We very much appreaciate it.

  • 👍1
ryanheise commented

I guess you want to know the formula to map a semitone into a pitch adjustment scale. I don't actually know, because the underlying native API just says this scale is "the factor by which pitch will be shifted". An answer to this question will depend on the answer to the same question for native app developers, so I would suggest Googling for that. And if you don't find anything, I would suggest doing a little bit of experimentation and mathematics. E.g. what does a factor of 2x give you? Is it just doubling the frequency? If so, that's a starting point for your calculation.

akshdeep-singh commented

@ryanheise the pitch adjustment doesn't work on android when using just_audio_background? It shows UnimplementedError, but it worked for me before using background plugin.

ryanheise commented

Thanks for reporting, @akshdeep-singh . Actually there are several methods not implemented by just_audio_background:

androidEqualizerBandSetGain
androidEqualizerGetParameters
androidLoudnessEnhancerSetTargetGain
audioEffectSetEnabled
setCanUseNetworkResourcesForLiveStreamingWhilePaused
setPitch
setPreferredPeakBitRate
setSkipSilence

I'll implement these at my next opportunity.

Jeff09github commented

any update for web platform?

ryanheise commented

It's one of the next items on my list, although my main computer has been down so not yet.

ryanheise commented

Sorry I meant to say that fixing setPitch in just_audio_background was my next item. For web, it is not a priority, although contributions are welcome (it will require a different web implementation based on web audio).

@akshdeep-singh I have fixed the unimplemented methods in just_audio_background now. Can you try out the fix in the fix/unimplemented_methods branch and let me know if it is working for you?

ryanheise commented

Regarding fix/unimplemented_methods, I have only tested setPitch although the other methods in the above list should also work now (audio effects, skip silence, etc.)

akshdeep-singh commented

@ryanheise I have verified pitch change, it's working for me. Thanks for your support.

aamironline commented

Thank you very much for creating such a great library. I would like to know the current status of this issue. Could anyone please update? Thanks again.

ryanheise commented

I promise that my comments above truthfully reflect the status, as does the README page.

alexgrusu commented

I can see setPitch is called in setPlatform.

I'm running an app on an iOS simulator and I'm encountering this issue

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(-1008, resource unavailable, null, null)

It seems the issue occurs due to the setPitch call.

Is there a way to avoid the setPitch call?

ryanheise commented

Since the platform support is not equal, you can write code like this:

if (Platform.isAndroid) {
    await _player.setPitch(pitch);
}

If you're also cross compiling to web, then you can do this:

if (!kIsWeb && Platform.isAndroid) {
    await _player.setPitch(pitch);
}
alexgrusu commented

Since the platform support is not equal, you can write code like this:

if (Platform.isAndroid) {
    await _player.setPitch(pitch);
}

If you're also cross compiling to web, then you can do this:

if (!kIsWeb && Platform.isAndroid) {
    await _player.setPitch(pitch);
}

I'm not calling setPitch in my code. setPitch is called by setPaltform.

ryanheise commented

There is the following code in setPlatform:

        try {
          await platform.setPitch(SetPitchRequest(pitch: pitch));
        } catch (e) {
          // setPitch not supported on this platform.
        }

So here the exception is handled. Are you sure that this is the call site that is causing the issue? Maybe you're actually using a different plugin such as just_audio_background? Can you reproduce this with one of the official example project, and if so, can you tell me which one and how to reproduce it?

alexgrusu commented

There is the following code in setPlatform:

        try {
          await platform.setPitch(SetPitchRequest(pitch: pitch));
        } catch (e) {
          // setPitch not supported on this platform.
        }

So here the exception is handled. Are you sure that this is the call site that is causing the issue? Maybe you're actually using a different plugin such as just_audio_background? Can you reproduce this with one of the official example project, and if so, can you tell me which one and how to reproduce it?

I'm using just_audio only at the moment.

I tried the official example and I modified the line where the audio source is set to

await _player.setUrl();

and I'm encountering the same error

flutter: Error loading audio source: (-1008) resource unavailable

There might be an issue on my side because the actual URL doesn't contain the file extension and I added the extension to the URL (I mean in the Flutter app)

Is there a way to specify the file extension?

LE:
because the URL is something like this: https://erxample/path/file

I added m4a to the end https://erxample/path/file.m4a. I know on iOS the extension is required

ryanheise commented

@alexgrusu Would you mind opening a new bug report? I don't think this issue is specifically related to the setPitch implementation, and if there is a bug somewhere in the setPlatform code, then I would like to investigate that separately from this issue about the setPitch feature.

alexgrusu commented

@alexgrusu Would you mind opening a new bug report? I don't think this issue is specifically related to the setPitch implementation, and if there is a bug somewhere in the setPlatform code, then I would like to investigate that separately from this issue about the setPitch feature.

Sounds good, I will open a different bug.

Might be my approach wrong? I mean, setting the m4a extension manually (the extension is not provided by the URL), could lead to an exception? Is there a way of setting the extension of the file when the link doesn't contain the extension?

araa12 commented

any update of pitch change on iOS?

ryanheise commented

This is something that would require an AVAudioEngine-based implementation. You could take a look at #784 as a starting point, and consider contributing to it to add pitch adjustment.

GayakEngineer commented

@ryanheise Hello. I could not find any documentation on setPitch(). What do i have to pass as argument to setPitch() ? What would be the factor, in order to shift pitch by one semitone.

In android flutter app, just audioplayer_name.setPitch(pitch); is enough, am i right?

ryanheise commented

@GayakEngineer

Hello. I could not find any documentation on setPitch().

It should be there on https://pub.dev/packages/just_audio under the link "API Documentation". Find the class (i.e. AudioPlayer), then find the thing inside the class (i.e. setPitch). The README page gives the supported platforms (as mentioned above, only Android).

In android flutter app, just audioplayer_name.setPitch(pitch); is enough, am i right?

That same comment above confirms this is correct.

GayakEngineer commented

@GayakEngineer

Hello. I could not find any documentation on setPitch().

It should be there on https://pub.dev/packages/just_audio under the link "API Documentation". Find the class (i.e. AudioPlayer), then find the thing inside the class (i.e. setPitch). The README page gives the supported platforms (as mentioned above, only Android).

In android flutter app, just audioplayer_name.setPitch(pitch); is enough, am i right?

That same comment above confirms this is correct.

Thank you for such a prompt reply. You have done great service to flutter community.

GayakEngineer commented

@ryanheise any update on setPitch for iOS?

ryanheise commented

The iOS side depends on rewriting using AVAudioEngine (see #334 for further details)