/distributed-source-separation

Intelligent Sample Management and Processing

Primary LanguageTypeScriptGNU General Public License v3.0GPL-3.0

NeuraLib - Intelligent Sample Management and Processing

GUI Screenshot

What is it?

  • NeuraLib is a distributed sample management and processing platform
  • Leverages multiple state-of-the-art neural networks for audio processing
    • Source Separation (extract vocals, bass, drums and other using Demucs)
    • Vocal Transcription (using OpenAI Whisper)
    • Audio to Midi conversion (convert audio to midi, for further use in your DAW with Basic-Pitch)
  • Audio file library management (build up your library with music, stems and samples, then easily export it for further use)
  • Sample slicing (slice extracted stems further into individual samples e.g. drum hits, vocal chops, synth hits etc.)
  • Streamed playback (stream large audio files without needing to download the whole file)
  • Built in a scalable way using task-queues and workers

Why does it exist?

For music professionals, audio engineers and hobbyist:

  • Sampling has been a big part in music production for decades
  • To explore and deconstruct musical pieces
    • Remix tracks easily by separating music into its individual parts
    • Expand your sample library by slicing stems further into samples
    • Understand the meaning of a song by transcribing vocals
  • A central place for managing music and samples

For developers:

  • A playground for all things audio
    • web-audio
    • music information retrieval tasks
    • neural networks for audio
    • neural networks deployment / usage in an actual application
    • distributed systems dealing with audio processing
  • Offer easy to extend platform for experimentation with neural networks
    • create an endpoint, a task queue and a worker to easily add additional processing tasks
  • Learn to deal with long-running tasks
  • Experiment with Server-Sent Events

Features

  • Music / Sample Collection
    • Per user music/sample management
    • Upload/Download music
    • Manage uploaded library, extracted stems and samples
    • Export library or invidivual samples for local usage, e.g. DAW
    • Stream audio, extracted stems and samples directly from object storage
  • Sample Slicing
    • Users can slice extracted stems further into individual samples (drum hits, vocal chops, synth hits etc.)
    • Samples are automatically added to library and attached to parent audio
  • Source Separation
    • Separate uploaded music into individual stems (vocals, drums, bass, other)
    • Separation is done asynchronously
    • Using Demucs v4
  • Audio to Midi Conversion
    • Convert any audio file to midi
    • Conversion is done asynchronously
    • Using Basic Pitch (Spotify)
  • Audio to Text (Vocals)
    • Extract lyrics / text from vocals
    • Extraction is done asynchronously
    • Using OpenAI Whisper or one of the several other open-source models

System Components and Architecture

Components

Architecture Overview

Architecture Overview

Authentication/Authoriziation

This application uses Auth0 as an identity provider and general authentication/authorization platform. See AUTHENTICATION for further information about the used authorization code flow.

How to get started

Check out the USAGE section for everything you need to get started.

Update History

As this isn't really a product ready for use there are no official changelogs. See commit history for recent changes.

Links

  • Demo Video TBD.