/VoxNovel

VoxNovel: generate audiobooks giving each character a different voice actor.

Primary LanguageJupyter NotebookMIT LicenseMIT

VoxNovel

Voxnovel_Logo

📋 Overview

VoxNovel is an innovative program that leverages the capabilities of booknlp to analyze literature, attribute quotations to specific characters, and generate a tailored audiobook where each character has a distinct voice through coqui tts. This not only provides an immersive audiobook experience but also brings each character to life with a unique voice, making the listening experience much more engaging.

🗣️ Included TTS Models

All Coqui TTS models-(Tacotron, Tacotron2, Glow-TTS, Speedy-Speech, Align-TTS, FastPitch, FastSpeech, FastSpeech2, SC-GlowTTS, Capacitron, OverFlow, Neural HMM TTS, Delightful TTS, ⓍTTS, VITS, 🐸 YourTTS, 🐢 Tortoise, 🐶 Bark), and STYLETTS2.

🌍🎙️ Accents you can give each character with the default cloning model (XTTS) - They also allow them to speak these languages, but the quotation attribution won't correctly identify for anything thats not English. English (en), Spanish (es), French (fr), German (de), Italian (it), Portuguese (pt), Polish (pl), Turkish (tr), Russian (ru), Dutch (nl), Czech (cs), Arabic (ar), Chinese (zh-cn), Japanese (ja), Hungarian (hu), Korean (ko)

outputs as a m4b with all book metadata and chapters, example output file in a audiobook player app

Example_of_output_in_audiobook_program

(as well as a folder of individual mp4 chatper files with ebook image embedded in them if you want that)

🔊 DEMOS

High Quality XTTS V2 Demos

Guardians_of_Ga.Hoole_10._Lasky._Kathryn_-_Coming_Of_Hoole_chapter_4.mp4
🔊🎉 More Demo Audio files :)

High Quality Tortoise Demos

272463996-c4f8dfdf-c5bd-4771-ab1a-6131c22a67b2.mp4

Super fast audio Balacoon Demos

271878548-53b694a9-5ddd-4174-82bc-07aff22d2330.mp4
271876316-530e8781-c77c-4424-89cd-a02df363b0bf.mp4

**Super High Quality testing with fine tuned models **

AUDIO_5811.mp4

You can fine tune your own Xtts models with around 6+ minutes of audio for free with this colab https://colab.research.google.com/drive/1GiI4_X724M8q2W-zZ-jXo7cWTV7RfaH-

🤖 Headless VoxNovel Google Colab

Explore and run the interactive version of the Headless VoxNovel project directly on Google Colab! Get started here.

GUI

gui_1_select_file

GUI Part 1 (BookNLP Processor) Info/Features -"Process File" button: Click and it'll ask you to select a ebook file.

gui_2_finetune

GUI Part 2 (Coqui TTS GUI) Info/Features
  • Select TTS Model Dropdown: This selects the TTS model that will be used for voice cloning.
  • Include fast Voice Models Checkbox: (Fast generate at cost of audio quality) Click this to be able to see every other model and singular voices supported by Coqui TTS.
    • It will update the "Select TTS Model" Dropdown for voice cloning models to also include (List of values to be added).
    • It will update the Dropdown for voices to select for each character to also include (List of values to be added).
  • Make all audio generate with Narrator voice Checkbox: This will make every character's audio be generated with the voice you have selected for the Narrator when you click the "Generate audio" button.
  • Clone new voice Button: Click this to add a new voice you can clone (make sure you have a reference audio file on hand).
  • Add Fine-tuned Xtts model to voice actor Button: If you have a folder containing all the parameters of a fine-tuned Xtts model of a specific voice, then you can click this to make that voice actor clone with that fine-tuned Xtts model, to provide much better voice cloning results.
  • Character voices Dropdowns: These are the dropdowns for selecting the Voice Actor (and the Accent of each character if using XTTS).
    • (1): The Voice actors available to select from for this character. (Default value is audio selected based on inferred gender of character being: "F, M, Other").
      • When you select a voice It will play the audio sample of that voice, if it's a fast voice model voice and a refrence audio does not exist, then it will generate one to play.
    • (2): The Accents available to select from for this character. (Optional, Default is English).
  • Chapter Delimiter Field: Will change the default chapter delimiter (The string that's used to identify chapters).
  • Silence Duration in milliseconds (ms) Field: This will change the amount of milliseconds in between each combined chunk of audio.
  • Select TTS Language Dropdown: This will let you select the default Accent used for every character which has not had the Accent manually selected for.
  • Loading bar: Will give an approximate amount of time left. (Estimate, you probably won't see accurate predictions until it's been running for 5 min).
  • Annotated book preview Block: This will show the entirety of the book with each character's lines color-coded.
    • You can click on a line while the audiobook is being generated to hear what that generated line sounds like. But only if the line has already had audio generated for it; if not, it'll play nothing.
  • Load Book Button: Clicking this will reload the color-coded annotated book view, it will just randomize the selected colors for each character's lines.
  • Generate Audio Button: Will start generating the full audiobook.
  • Select random voices Button (Will only be visible if the "include fast Voice Models" checkbox is checked): Will Select an auto-gender-inferred fast model voice for every character except for the narrator's voice.

gui_3_run

GUI Part 3 (Book Viewer) Info/Features -It's hard to explain its more of a playground if you mess around with it then you should get how it works. But it can be used to fine tune the audiobook -Close out of the window when your done with it.

📦 SetUp Install

Run in this order:

🤖 Headless VoxNovel Google Colab

Explore and run the interactive version of the Headless VoxNovel project directly on Google Colab! Get started here.

🐳 Docker (Sound not working in gui yet)
🐳 Headless Docker
Docker headless m1 🍏Mac

  1. cd ~

  2. git clone https://github.com/DrewThomasson/VoxNovel.git

  3. sudo docker run -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:headless_m1_latest

Headless Docker 🐧 Linux/Intel 🍏Mac

For Headless Docker on only cpu

  1. cd ~

  2. git clone https://github.com/DrewThomasson/VoxNovel.git

  3. sudo docker run -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest_headless

For headless docker with gpu speedup if you have a nvida gpu

  1. cd ~

  2. git clone https://github.com/DrewThomasson/VoxNovel.git

  3. sudo docker run --gpus all -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest_headless

Headless Docker 🖥️ Windows

  1. cd %USERPROFILE%

  2. git clone https://github.com/DrewThomasson/VoxNovel.git

  3. docker run -v "%USERPROFILE%/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest_headless

🐳 GUI Docker (Sound not working in gui yet)
🐧 Linux Docker 1. `cd ~`
  1. git clone https://github.com/DrewThomasson/VoxNovel.git
  2. sudo docker run --gpus all -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v /dev/snd:/dev/snd --device /dev/snd -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest
Here's the formatted text, unchanged in wording:

Certainly! Here is the full guide with the commands in an easy-to-copy format, without any changes to your original wording:

🍏 Mac Docker

Setting Up GUI Applications with Docker on macOS

This guide provides instructions on how to run a Docker container with a graphical user interface on macOS using XQuartz for X11 forwarding and volume mounting.

Install XQuartz

  1. Download and install XQuartz from XQuartz website.
  2. Open XQuartz.
  3. Go to XQuartz -> Preferences.
  4. In the Security tab, enable Allow connections from network clients.
  5. Restart XQuartz to apply these settings.

Configure and Run the Docker Container

Allow Docker to Connect to XQuartz

Open a terminal and run the following command to allow connections from your local machine to XQuartz:

xhost + $(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}')

Start the Docker Container

Run the following command to start your Docker container. This command configures the GUI to display on your host and mounts the necessary directories:

cd ~
git clone https://github.com/DrewThomasson/VoxNovel.git
docker run -e DISPLAY=$(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}'):0 \
           -v /tmp/.X11-unix:/tmp/.X11-unix \
           -v "/Users/$(whoami)/VoxNovel:/VoxNovel" \
           athomasson2/voxnovel:latest

Notes

  • XQuartz Configuration: Ensure that XQuartz is configured to allow network clients before attempting to connect.
  • Directory Existence: Verify that the directory /Users/$(whoami)/VoxNovel exists on your Mac. If not, create it or adjust the volume mount path in the Docker command as needed.
  • Firewall and Security: If you face connectivity issues, check any firewall settings and security preferences that might block the connections.
🪟 Windows Docker 1. `choco install vcxsrv` - First install VcXsrv and configur it to allow connections
how to setup VcXsrv After installing VcXsrv, it typically launches automatically. You can confirm it's running by checking for its icon in the system tray, usually located near the clock in the taskbar. It may also start automatically when you log in to your system.

To ensure it's configured to allow connections from Docker containers, follow these steps:

  1. Right-click on the VcXsrv icon in the system tray.
  2. Select "XLaunch" to open the configuration wizard.
  3. In the configuration wizard, select "Multiple windows" and proceed to the next step.
  4. Choose your preferred settings for display number and screen.
  5. In the "Extra settings" window, make sure to check the box labeled "Disable access control" to allow connections from Docker containers.
  6. Complete the configuration by clicking "Finish" and then "Save configuration" when prompted.

With these settings, VcXsrv should be running and configured to allow connections from Docker containers. You can now proceed with running your Docker commands requiring GUI support.

2. `cd $HOME` 3. `git clone https://github.com/DrewThomasson/VoxNovel.git` 4. `docker run -e DISPLAY=host.docker.internal:0 -v "/Users/$(whoami)/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest

`

🐧 Linux ## Single command Ubuntu install:(Do not use if you already have miniconda installed.)
  1. yes | wget -O - https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/Ubuntu-install.sh | bash -This will also create a desktop shortcut to run Voxnovel as well.

or manual install:

  1. sudo apt-get install calibre
  2. sudo apt-get install ffmpeg
  3. conda create --name VoxNovel python=3.10
  4. conda activate VoxNovel
  5. git clone https://github.com/DrewThomasson/VoxNovel.git
  6. cd VoxNovel
  7. pip install bs4
  8. pip install styletts2
  9. pip install tts==0.21.3
  10. pip install booknlp
  11. pip install -r Ubuntu_requirements.txt
  12. python -m spacy download en_core_web_sm
For non Latin-based Languages TTS support (Optional)

Install Mecab for (Non Latin-based Languages tts support)(Optional):

  • Ubuntu: sudo apt-get install -y mecab libmecab-dev mecab-ipadic-utf8

(For non Latin-based Languages tts support)(Optional)
python -m unidic download

pip install mecab mecab-python3 unidic
Steam Deck) (x86_64 Arch Linux)
  1. sudo -v && wget -nv -O- https://download.calibre-ebook.com/linux-installer.sh | sudo sh /dev/stdin
  2. also download it from the discovery store or flatpac I did both on my steam deck
  3. mkdir -p ~/miniconda3
  4. wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
  5. bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
  6. rm -rf ~/miniconda3/miniconda.sh
  7. ~/miniconda3/bin/conda init bash
  8. ~/miniconda3/bin/conda init zsh
  9. restart the terminal(close out and open a new window)
  10. conda create --name VoxNovel python=3.10
  11. conda activate VoxNovel
  12. git clone https://github.com/DrewThomasson/VoxNovel.git
  13. cd VoxNovel
  14. sudo pacman -S espeak-ng(make sure you have pacman fully working, there should be stuff online for the steam deck) guide on getting pacman wokring on steam deck(https://www.reddit.com/r/SteamDeck/comments/t8al0i/install_arch_packages_on_your_steam_deck/) you might have to reset the keys if something goes wrong with that: Resetting all the keys

Remove or reset all the keys installed in your system by removing the /etc/pacman.d/gnupg directory (as root) and by rerunning pacman-key --init followed by pacman-key --populate to re-add the default keys. 15. pip install styletts2 16. pip install tts==0.21.3 17. pip install booknlp 18. pip install -r SteamDeck_requirements.txt 19. pip3 install spacy 20. python3 -m spacy download en_core_web_sm 21. sudo pacman -S calibre 22. mv ~/miniconda3/envs/VoxNovel/lib/libstdc++.so.6 ~/miniconda3/envs/tts/lib/libstdc++.so.6.bak

  1. pip install mechanize
  2. pip install bs4
  3. pip install css_parser
Intel mac
  1. brew install calibre
  2. brew install ffmpeg
  3. conda create --name VoxNovel python=3.10
  4. conda activate VoxNovel
  5. git clone https://github.com/DrewThomasson/VoxNovel.git
  6. cd VoxNovel
  7. pip install styletts2
  8. pip install tts==0.21.3
  9. pip install booknlp 9.pip install -r MAC-requirements.txt
  10. pip install spacy 11.python -m spacy download en_core_web_sm
For non Latin-based Languages TTS support (Optional)

Install Mecab for (Non Latin-based Languages tts support)(Optional):

  • macOS: brew install mecab, brew install mecab-ipadic

(For non Latin-based Languages tts support)(Optional)
python -m unidic download

pip install mecab mecab-python3 unidic
Apple Silicon Mac (Tested on 2020 M1 pro 8gb ram)
  1. brew install calibre(You might have to also install it manually from their site if this doesn't work)
  2. brew install ffmpeg
  3. conda create --name VoxNovel python=3.10
  4. conda activate VoxNovel
  5. git clone https://github.com/DrewThomasson/VoxNovel.git
  6. cd VoxNovel
  7. pip install tensorflow-macos (Also optional pip install tensorflow-metal But so far I haven't gotten gpu speedup yet
  8. pip install styletts2
  9. pip install tts==0.21.3
  10. pip install --no-dependencies booknlp
  11. pip install transformers==4.30.0
  12. pip install tensorflow
  13. pip install -r MAC-requirements.txt
  14. pip install ebooklib bs4 epub2txt pygame moviepy spacy
  15. python -m spacy download en_core_web_sm
For non Latin-based Languages TTS support (Optional)

Install Mecab for (Non Latin-based Languages tts support)(Optional):

  • macOS: brew install mecab, brew install mecab-ipadic (For non Latin-based Languages tts support)(Optional)
    python -m unidic download
pip install mecab mecab-python3 unidic
Windows 11 Because of BookNLP Windows issues, all of this will be run in WSL (don't worry, it's still easy).
Easy Windows 11 install

  1. In your PowerShell, paste wsl --install to install WSL.

  2. In your PowerShell, paste wsl --update --pre-release to update WSL to a pre-release version, right now the prerelease has all the sound and such working.

  3. After you set the username and passsudo apt install nvidia-cuda-toolkitword, you'll now be able to open WSL as an application. This will allow you to run Ubuntu software. You'll now want to open WSL if you closed it and paste these commands.

  4. run this command in the Ubuntu wsl for a single command install yes | wget -O - https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/Ubuntu-install.sh | bash

  5. Congrates! VoxNovel should run in WSL now!

InDepth Windows 11 install 1. In your PowerShell, paste `wsl --install` to install WSL. 2. In your PowerShell, paste `wsl --update --pre-release` to update WSL to a pre-release version, right now the prerelease has all the sound and such working.

  1. After you set the username and password, you'll now be able to open WSL as an application. This will allow you to run Ubuntu software. You'll now want to open WSL if you closed it and paste these commands.

    (This will install Calibre for ebook conversion) cd ~ sudo apt update sudo apt upgrade sudo apt install calibre sudo apt install ffmpeg

    (This will install Miniconda) Mini conda code was taken from here:https://docs.conda.io/projects/miniconda/en/latest/ mkdir -p ~/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 rm -rf ~/miniconda3/miniconda.sh

    (This will make sure Miniconda is initialized) ~/miniconda3/bin/conda init bash ~/miniconda3/bin/conda init zsh

  2. Close out of your current WSL window and reopen it to be able to use Miniconda.

    conda create --name VoxNovel python=3.10 conda activate VoxNovel cd ~ git clone https://github.com/DrewThomasson/VoxNovel.git cd VoxNovel pip install styletts2 pip install tts==0.21.3 pip install booknlp pip install -r Ubuntu_requirements.txt pip install spacy python -m spacy download en_core_web_sm sudo apt install espeak-ng

You can access the files on your WSL Ubuntu in Windows File Explorer by putting this into the address: \\wsl.localhost\Ubuntu\home\

For non Latin-based Languages TTS support (Optional)

Install Mecab for (Non Latin-based Languages tts support)(Optional):

  • sudo apt-get install -y mecab libmecab-dev mecab-ipadic-utf8

(For non Latin-based Languages tts support)(Optional)
python -m unidic download

pip install mecab mecab-python3 unidic

🚀 To Run the program

python gui_run.py

Running with Low VRAM (4 GB)

🚀 Or to run headless

python headless_voxnovel

Modifications

  • Turns out once you set the device it stays like that for the full program.
  • So, I've split the program into two Python programs: one CPU and one GPU. I've tested this on my (4GB VRAM GPU) and this solution works. at least on my end I really hope it works on your end. 🙏

To run the fix I've made tailor made for a low Vram GPU situation:

To run the provided scripts on your system, follow these steps in order:

  1. Book Processing (CPU Only):

    • Script: 1CPU_Book_processing.py
    • This script handles the task of only processing the book using BookNLP, specifically forcing it to run on the CPU.
    • Run with python 1CPU_Book_processing.py

  2. Audio Generation (GPU Only):

    • Script: 2GPU_Audio_generation.py
    • This script is dedicated to only generating audio with the GPU and should be run after completing the book processing with 1CPU_Book_processing.py.
    • Run with python 2GPU_Audio_generation.py

Performance Results

Upon running a mini test with an epub file using the above setup, the following performance metrics were observed:

Performance Results

Testing on done with the mini epub file located in the Example_working_files.zip

Task Configuration Time (Seconds)
Book Processing GPU only (GeForce GTX 980), 4GB VRAM, 32GB RAM, Intel i7-8700K 2.922
Audio Generation GPU only (GeForce GTX 980), 4GB VRAM, 32GB RAM, Intel i7-8700K 128.48
Book Processing CPU only, 32GB RAM, Intel i7-8700K 4.964
Audio Generation CPU only, 32GB RAM, Intel i7-8700K 391.4227
To Run the auto program (Don't use temporarily on hold) This means all you do is select the book and all the voices will be auto assigned and generated for you.

python auto_noGui_run.py

📚 Supported ebook File Types:

.epub, .pdf, .mobi, .txt, .html, .rtf, .chm, .lit, .pdb, .fb2, .odt, .cbr, .cbz, .prc, .lrf, .pml, .snb, .cbc, .rb, and .tcr,

  • (Best results are from using epub or mobi for auto chapter detection)

Folders

📂 Folders used by the program

/Final_combined_output_audio: This is where all of your chapter audio files will be put in order of chapter num

/output_audiobooks: This is where all of your m4b audiobook files will be stored

/Working_files: Holds all of the working files used by the program while activly running.

  • /Working_files/temp_ebook: Holds all of the individual extracted chapter txt files from the ebook.

/tortoise: Holds all the sample voice files

GUI functions

GUI Part 1 (BookNLP Processor) -"Process File" button: Click and it'll ask you to select a ebook file.
GUI Part 2 (Coqui TTS GUI)
  • Select TTS Model Dropdown: This selects the TTS model that will be used for voice cloning.
  • Include fast Voice Models Checkbox: (Fast generate at cost of audio quality) Click this to be able to see every other model and singular voices supported by Coqui TTS.
    • It will update the "Select TTS Model" Dropdown for voice cloning models to also include (List of values to be added).
    • It will update the Dropdown for voices to select for each character to also include (List of values to be added).
  • Make all audio generate with Narrator voice Checkbox: This will make every character's audio be generated with the voice you have selected for the Narrator when you click the "Generate audio" button.
  • Clone new voice Button: Click this to add a new voice you can clone (make sure you have a reference audio file on hand).
  • Add Fine-tuned Xtts model to voice actor Button: If you have a folder containing all the parameters of a fine-tuned Xtts model of a specific voice, then you can click this to make that voice actor clone with that fine-tuned Xtts model, to provide much better voice cloning results.
  • Character voices Dropdowns: These are the dropdowns for selecting the Voice Actor (and the Accent of each character if using XTTS).
    • (1): The Voice actors available to select from for this character. (Default value is audio selected based on inferred gender of character being: "F, M, Other").
      • When you select a voice It will play the audio sample of that voice, if it's a fast voice model voice and a refrence audio does not exist, then it will generate one to play.
    • (2): The Accents available to select from for this character. (Optional, Default is English).
  • Chapter Delimiter Field: Will change the default chapter delimiter (The string that's used to identify chapters).
  • Silence Duration in milliseconds (ms) Field: This will change the amount of milliseconds in between each combined chunk of audio.
  • Select TTS Language Dropdown: This will let you select the default Accent used for every character which has not had the Accent manually selected for.
  • Loading bar: Will give an approximate amount of time left. (Estimate, you probably won't see accurate predictions until it's been running for 5 min).
  • Annotated book preview Block: This will show the entirety of the book with each character's lines color-coded.
    • You can click on a line while the audiobook is being generated to hear what that generated line sounds like. But only if the line has already had audio generated for it; if not, it'll play nothing.
  • Load Book Button: Clicking this will reload the color-coded annotated book view, it will just randomize the selected colors for each character's lines.
  • Generate Audio Button: Will start generating the full audiobook.
  • Select random voices Button (Will only be visible if the "include fast Voice Models" checkbox is checked): Will Select an auto-gender-inferred fast model voice for every character except for the narrator's voice.
GUI Part 3 (Book Viewer) -It's hard to explain its more of a playground if you mess around with it then you should get how it works. But it can be used to fine tune the audiobook -Close out of the window when your done with it.

🌟 Features

  • Free and entirely locally run
  • Supports all ebook file formats by using calibre
  • Can run on CPU or CUDA GPU
  • Autoselects a starting estimated voice by pronouns per character
  • Supports all models in Coqui TTS and all voices in the models
  • Easily create a new voice actor in seconds through voice cloning in GUI
  • Can play audio by clicking on the text in the book viewer in GUI
  • Ability to regenerate specific lines if they came out weird
  • Ability to add custome fine tuned models for specific voice with the click of a button in GUI
  • Outputs a single file output as m4b to include all the metadata(chapters book image ect)
  • Supports STYLETTS2 as a model you can select from for voice cloning for WICKED FAST speed (even on cpu)
  • Includes 26 default voices for cloning
  • Low vram option(Details in readme)
  • Gui Docker image (Graphical interface-no sound in gui yet)
  • Headless version of docker image
  • Headless version of VoxNovel
  • A Google Colab using the headless version

🔜 Incoming Planned Features

  • Ability to change the character for a line if incorrectly attributed by booknlp
  • Make it so that all the included voices and models already have their premade own demo voices
  • Make it so that the demo audio for the cloned voices is not their reference audio but what their voices sound like generated
  • Using whisper transcriptions to cut hallucinations out of generated audio
  • Incorporating local model to generate sound effects when a book discribes a location or sound effect

🙏 Special thanks to:

-@sidharthrajaram (for his Styletts2 pip install he created, I couldn't of added styletts2 without him. :) ) (https://github.com/sidharthrajaram/StyleTTS2)