Open Audio Search is a search engine for audio. It can index RSS feeds and use automatic speech recognition to extract text from audio. The feeds and transcripts are then indexed in a search engine, empowering users to use full-text search on the indexed transcripts and listen to the audio files while jumping right to search result snippets.
With Open Audio Search, we want to make the archives of community media, radio stations, podcasts searchable and discoverable, through open source tech and APIs.
Open Audio Search is still in development. No API stability guarantees yet. Don't run this for anything serious or on public instances at the moment.
- Core backend written in Rust, providing a REST API and managing the indexing pipeline
- Document database using CouchDB
- Full-text search using Elasticsearch Community Edition
- Web user interface using React
- Task queue with tasks written in Python (using Celery and Redis)
- Automatic Speech Recognition using Vosk toolkit (Kaldi under the hood)
Requirements: Docker and Docker Compose. You'll need a quite recent version of both.
We provide nightly
Docker images which are built and pushed to Docker Hub after each commit to the main
branch. We will provide stable images once we reach our first beta release.
For a quick start, copy docker-compose.nightly.yml
to an empty directory and start it with Docker Compose. This will download the latest images for the backend and worker and start them together with CouchDB, Elasticsearch and Redis.
mkdir openaudiosearch && cd openaudiosearch
wget https://raw.githubusercontent.com/openaudiosearch/openaudiosearch/main/docker-compose.nightly.yml
mv docker-compose.nightly.yml docker-compose.yml
docker-compose up
For the speech recognition to work you will need to download the models first. Run this command once, it will download the models into the ./data/oas
volume:
docker-compose exec worker python download_models.py
This project includes a Dockerfile to build docker images for the backend and worker. It also includes a docker-compose.yml
file to easily launch OAS together with CouchDB, Elasticsearch and Redis.
Then, the following commands will build the Docker image from source and start it together with all required services.
git clone https://github.com/openaudiosearch/openaudiosearch
cd openaudiosearch
docker-compose build
docker-compose up
It takes a little while for Elasticsearch to start up. Then, the OAS user interface and API are available at http://localhost:8080
.
Elasticsearch wants quite a lot of free disc space. If the threshold is not met, it refuses to do anything. Run the script at oas_worker/scripts/elastic-disable-threshold.sh
to disable the disc threshold (does not persist across Elasticsearch restarts):
docker-compose exec worker bash scripts/elastic-disable-threshold.sh
OAS is configured through environment variables or command line arguments. The following table lists all environment variables. Some apply to both the core and the worker, and some only to either.
default: password
applies to: core
Default password for admin
user.
default: http://admin:password@localhost:8080/api/v1
applies to: worker
HTTP URL of the OAS core API, including a valid username and password.
default: ./data/oas
applies to: worker
Storage path for models, cached files and other assets.
default: redis://localhost:6379/0
applies to: both
URL to Redis server
default: http://localhost:9200/oas
applies to: core
URL to Elasticsearch server and index prefix
default: http://admin:password@localhost:5984/oas
applies to: core
URL to CouchDB server and database prefix
default: 0.0.0.0
applies to: core
Interface for the HTTP server to bind to
default: 8080
applies to: core
Port for HTTP server to listen on
applies to: core
If set to a HTTP URL, all requests for the web UI are proxied to this address
default: 1
applies to: worker
Sets the number of worker processes/threads.
To run OAS locally for developing or testing you should install the following requirements beforehand:
- For the core: Rust, which is most easily installed with Rustup. You also will need a C compiler and OpenSSL development headers. On Debian based systems, run
apt install gcc libssl-dev pkg-config
. - For the worker: Python 3 and poetry. Also requires ffmpeg.
- For the frontend: Node.js and npm or yarn.
Clone this repository
git clone https://github.com/openaudiosearch/openaudiosearch
Start CouchDB, Elastisearch and Redis via Docker
docker-compose -f docker-compose.dev.yml up
Run the frontend in development mode
cd frontend
yarn
yarn start
Build an run the core
Compile and run the Rust core, while setting an environment variable to proxy the web UI from a local development server (see below):
cargo run -- --dev run
The --dev
argument enables debug logging (RUST_LOG=oas=debug
) and sets FRONTEND_PROXY=http://localhost:4000
To build and test in release mode, use
cargo run --release -- run
Run the worker
cd oas_worker
poetry install
./start-worker.sh
Now open http://localhost:8080 in a web browser. The UI is proxied to the live-reloading UI development server which runs at http://localhost:4000. The OAS API is served at http://localhost:8080/api/v1
. REST API docs are automatically generated and served at http://localhost:8080/swagger-ui.
Have a look at the development guide.
Open Audio Search is licensed under the AGPL v3.
Documentation is still sparse. Docs are located in the docs folder and rendered to HTML on the documentation site. We also host API docs for the Rust core. The REST API is documented from within Open Audio Search (TODO: Host and link to REST API docs).
All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome. Please open issues or talk to us on our Discord server. We want to welcome anyone and commit to creating an inclusive environment.
Development discussions currently take place here on GitHub or on our Discord chat.
Open Audio Search is a project by arso collective and cba and supported by Prototype Fund and netidee.