My Assistant

A Customizable, Voice-Assistant Desktop.

Your support is very much appreciated! star on GitHub

📹 Video Demo:

Project.Demo.mp4

📜 Table of Contents

About the Project
Source Code Directory Structure
Usage
Getting Started
Prerequisites
Package Manager
Environment Requirements
Run for Development
FAQ
Future Work
Community
Contributors
License

🎉 About the Project

MA stands for My Assistant. The aim of this project is to develop a programmable voice assistant desktop application that provides users with a highly customizable and extensible interface to interact with their devices. Our goal is to provide a voice assistant that can be tailored to the needs and preferences of individual users, and that is optimized for desktop and laptop computers.
To achieve this aim, we have set the following objectives:

Develop a voice assistant that is highly customizable and extensible, allowing users to add their own commands and actions based on their needs and preferences.
Optimize the voice assistant for desktop and laptop computers, providing users with a convenient and intuitive interface to interact with their devices.
Provide users with a seamless and integrated experience by enabling the voice assistant to interact with other tools and applications on the desktop.
Ensure the voice assistant is secure and respects user privacy by implementing robust data privacy and security measures.

📷 Screenshots

🎨 Color Reference

🎤 What is Programmable Voice Assistant

Build a customizable virtual voice assistant. The assistant should provide users with more control and flexibility over its features and functionality rather than limiting them to preset options. Allow users to define their own automation scenarios and workflows. Users should be able to craft new commands tailored to their unique needs and preferences. Include traditional voice assistant features. In addition to offering more customization options, the assistant should support all the features of a traditional Voice assistance

🔍 Why Programmable Voice Assistant

Customization: Customize the assistant's behavior and capabilities to suit individual needs and workflows.
Flexibility: Design complex automation scenarios and workflows beyond predefined actions.
Extensibility: Integrate with external services and APIs to enhance functionality.
Open-source Community: Benefit from community-contributed resources for expanded capabilities.
Privacy and Security: Host locally for data control and end-to-end encryption.
Learning and Exploration: Gain insights into AI and voice-based interaction systems through hands-on experience.

✍️ History Of Voice Assistants

The idea of voice assistants has been around for decades, with the first voice recognition system being introduced in the 1950s by Bell Laboratories. However, it was not until the late 1990s that voice assistants began to gain popularity with the introduction of IBM’s “Via Voice” and “Dragon NaturallySpeaking” software. These early systems were limited by their inability to recognize natural language, their high cost, and the need for specialized hardware. In recent years, voice assistants have become increasingly prevalent due to the widespread adoption of smartphones and the emergence of smart speakers. Apple’s Siri, Amazon’s Alexa, and Google Assistant are some of the most popular voice assistants today. These assistants allow users to interact with their devices using natural language, perform tasks such as setting reminders, playing music, and controlling smart home devices.

❎ The Problem with Traditional Voice Assistants

Fixed, Limited Automation Options:
- Traditional voice assistants provide a predefined set of automation options that are often limited and generic in nature. These assistants offer a restricted range of actions or tasks that can be performed, limiting their usefulness in addressing diverse user needs. Users are confined to the predefined set of commands and actions, without the ability to tailor or expand the assistant's capabilities to match their specific requirements.
Lack of Customization:
- Another drawback of traditional voice assistants is the lack of customization options. Users have limited control over modifying or enhancing the assistant's features to align with their preferences and unique needs. The inability to personalize or customize the assistant's behavior hinders its ability to adapt to individual users' workflows or specific requirements, limiting its overall utility.

✅ The solution with my assistant

Our voice assistant addresses the limitations of limited customization found in traditional voice assistants by providing users with extensive customization and personalization options. The key features of our solution include:

User-Crafted Automation Scenarios:
- Our voice assistant empowers users to create their own automation scenarios and complex workflows, tailored to their specific needs. Users have the flexibility to define custom commands and actions, enabling them to automate repetitive tasks and streamline their workflows effectively.
Easy-to-Use Interface:
- We offer an intuitive and user-friendly interface that simplifies the process of creating custom commands. Users can easily set up simple phrases or triggers that activate the desired automation, without the need for advanced technical knowledge.
Commands Library:
- To further enhance customization options, our voice assistant includes a comprehensive Commands library. Users can access a collection of pre-built automation commands created by both other users and our core team. This allows users to reuse existing commands, leverage community-contributed automations, and easily expand the capabilities of their voice assistant.

⭐ Features

Account Creation and Login:
- Users can create an account securely to access personalized features, command management, and interaction with the application.
- The system allows users to log in with their credentials, maintaining user authentication throughout the session.
Create New Commands:
- The system allows users to log in with their credentials, maintaining user authentication throughout the session.
- Metadata information includes command name, description, parameters, patterns, script, script type, dependency file, and command icon.
- Uploaded files (script, dependency, icon) are validated, saved, and linked to the command.
Edit Existing Commands:
- Users can edit existing commands by modifying their metadata or uploading new files.
- The system updates the command accordingly, including retraining the user model or regenerating the executable.
Delete Commands:
- Users can easily delete their commands, and the system handles necessary cleanup tasks, such as removing the executable file and updating the user model.
Command Approval Workflow:
- Users can submit their commands for approval by an admin to make them available in the marketplace.
- The admin reviews and approves/rejects the command, updating its visibility accordingly.
- Users receive appropriate feedback regarding the approval status over awesome notifications service.
My Command Table:
- Users can view a table displaying all commands they own, with options to edit and delete each command.
- The table is visually organized and user-friendly, supporting sorting and filtering options.
Marketplace Command Installation:
- Users can seamlessly install commands from the marketplace.
- The command is added to the user's installed commands list, and the corresponding executable file is downloaded.
- The system handles all necessary tasks, such as updating the user model and installing dependencies.
- Users receive appropriate feedback regarding the installation status over awesome notifications service.
Uninstall Installed Commands:
- Users can uninstall commands they no longer need, and the system handles confirmation and cleanup tasks.
- The command is removed from the user's installed commands list, and the corresponding executable file is deleted.
- The system handles all necessary tasks, such as updating the user model and removing dependencies.
- Users receive appropriate feedback regarding the uninstallation status over awesome notifications service.

🔐 How My Assistant Works

User Interaction: Users interact with your voice assistant through a desktop app with a user-friendly interface.
Voice Input: Users can record voice commands using the app's microphone feature or enter text commands if they prefer.
Speech-to-Text (STT) Conversion: The recorded voice commands are sent to the Speech-to-Text engine, which converts the audio input into text.
Natural Language Processing (NLP): The text input is processed by the Natural Language Processing (NLP) module, powered by the Rasa framework. The NLP module extracts intent and entities from the user's input, understanding the user's request.
Command Mapping: The NLP module maps the user's intent to specific commands available in the system, determining the appropriate action to be taken.
Command Execution: Based on the command mapping, the system executes the corresponding action or task, such as opening an application, performing a specific operation, or retrieving information.
Text-to-Speech (TTS) Conversion: Upon completing the requested task, the response is sent to the Text-to-Speech engine, converting the text into an audible response.
Response Playback: The voice assistant plays back the response to the user, providing real-time feedback on the executed action.
Customization and Personalization: Your voice assistant stands out by allowing users to create, edit, and manage their own commands, adding a high level of customization and personalization to the user experience.
Integration with Marketplace: The app features a marketplace where users can browse and install commands created by others, extending the assistant's capabilities through community-contributed resources.
Approval Workflow: Users can submit their custom commands for admin approval. The admin reviews and approves or rejects the command, updating its visibility accordingly.
Data Security and Privacy: Your voice assistant prioritizes data security and privacy. The application is self-hosted, ensuring user data remains on the user's device, and end-to-end encryption is applied for secure interactions.

Workflow:

The core components of the system are the Desktop App, which serves as the user-facing interface, and the API, which acts as the central component handling communication between various components and external services. The NLP Manager is responsible for natural language processing, while the Executable Builder generates executable files for the commands. The system also integrates with Google's Speech-to-Text and Text-to-Speech APIs for voice-based interactions.

🛠️Tech Stack and Tools

The tools used in this project.

	Tool	Description
	ElectronJS	A framework for building cross-platform desktop applications using web technologies.
	Angular	Platform for building dynamic web applications.
	SASS	CSS preprocessor for creating scalable and maintainable styles.
	PrimeNG	UI component library to enhance the visual and interactive aspects of the application.
	Angular Material	UI component library that follows Google's Material Design guidelines.
	Rasa Framework	Framework for natural language processing to understand user commands and interactions.
	YAML	YAML library used for automating the training process.
	Webhook	A way for two applications to communicate with each other by sending HTTP requests.
	Multitenancy	A way to allow multiple users to share the same application without interfering with each other.
	Model Linguistic Feature	A feature that is used to represent the linguistic content of a text.
	Google Calendar API	API for interacting with Google Calendar to schedule events.
	Google Cloud	Cloud platform used for hosting and deploying the application.
	Redis	In-memory data store used for caching and performance optimization.
	Django	Web framework used for the backend server and database management.
	Async Channels	A library that allows you to create asynchronous communication channels in Django.
	PostgreSQL	Relational database management system used for data storage.
	Daphne	ASGI server used to deploy Django applications.

🪜 Source Code Directory Structure

A quick look at the top-level files and directories:

.
├── electronApp
│   ├── build
│   ├── CommandManger
│   ├── DB
│   │   ├── models
│   │   └── queries
│   ├── scriptRunner
│   ├── stt
│   ├── textToScript
│   │   └── models
│   ├── tray
│   └── tts
├── src
│   ├── app
│   │   ├── auth
│   │   │   ├── _helper
│   │   │   ├── interface
│   │   │   ├── pipes
│   │   │   │   └── only-one-error
│   │   │   ├── register-component
│   │   │   ├── services
│   │   │   │   ├── auth-service
│   │   │   │   └── not-match-validation
│   │   │   └── user-card
│   │   ├── core
│   │   │   └── services
│   │   │       ├── electron
│   │   │       └── notification
│   │   ├── recorder
│   │   │   ├── components
│   │   │   │   ├── audio-visualizer
│   │   │   │   ├── chat
│   │   │   │   ├── home-page
│   │   │   │   └── mic
│   │   │   └── services
│   │   │       ├── rasa
│   │   │       │   └── rasa.socket
│   │   │       ├── stt
│   │   │       └── tts
│   │   ├── scripts-table
│   │   │   ├── components
│   │   │   │   ├── command-management
│   │   │   │   │   ├── abstract-commands
│   │   │   │   │   ├── create-command-form
│   │   │   │   │   │   └── parameter-field
│   │   │   │   │   ├── edit-command-form
│   │   │   │   │   ├── installed-commands
│   │   │   │   │   │   └── installed-commands-service
│   │   │   │   │   └── my-commands
│   │   │   │   │       └── my-command-service
│   │   │   │   └── marketplace-component
│   │   │   │       ├── card-preview
│   │   │   │       └── command-card
│   │   │   ├── interfaces
│   │   │   └── services
│   │   ├── shared
│   │   │   ├── components
│   │   │   │   ├── google-token
│   │   │   │   ├── loader
│   │   │   │   ├── modal
│   │   │   │   ├── notifications
│   │   │   │   │   ├── interfaces
│   │   │   │   │   ├── notification-card
│   │   │   │   │   └── notification-list
│   │   │   │   └── sidebar
│   │   │   ├── directives
│   │   │   │   └── webview
│   │   │   └── snackbar-service
│   │   └── tray
│   ├── assets
│   │   ├── fonts
│   │   ├── i18n
│   │   └── icons
│   └── environments
├── stt
│   └── temp
└── test-files

1. electronApp: contains all the source code for the electron app.
1. src: contains all the source code for the angular app.
  - app: contains all the source code for the app.
    - auth: contains all the source code for the authentication module.
    - core: contains all the source code for the core module.
    - recorder: contains all the source code for the recorder module and components {audio-visualizer, chat, home-page, mic}.
    - scripts-table: contains all the source code for the scripts-table module and components {command-management, marketplace-component}.
    - shared: contains all the source code for the shared module and components {google-token, loader, modal, notifications, sidebar}.
    - tray: contains all the source code for the tray module.
  - assets: contains all the assets (ex. images, fonts...)
  - environments: contains all the environment variables.
1. stt: contains all the source code for the speech-to-text module.
1. test-files: contains all the test files.