Placks - Connecting Hands, Breaking Barriers

Project Overview

Placks is an innovative mobile application designed to break communication barriers between the hearing and non-hearing communities. Developed using SwiftUI, Placks leverages advanced AI and Core ML to recognize and transcribe sign language into text and speech. The application aims to provide real-time translation of various sign languages, making communication smoother and more accessible across different languages and regions.

Features

Sign Language Recognition: Placks utilizes Core ML to recognize sign language gestures and transcribe them into text or speech. This feature is currently under development.
Multilingual Support: The application aims to support multiple sign languages, including but not limited to LSF, LSFB, German, English, Chinese, and Indian sign languages.
Intelligent Translation: A large language model (LLM) is being integrated to transform sign language word sequences into coherent sentences.
Dictionary and Learning Tool: Users will have access to a dictionary with videos and transcribed words in sign language, which is currently in beta.
Voice Recognition: The app will eventually feature voice recognition to transcribe conversations between multiple people, making it easier for sign language users to participate in discussions.
Accessible Locations Map: Placks will include a map to help users find accessible locations, counters, and stores.
Messaging: A fast messaging system is in beta to facilitate quick interactions.

The Journey Behind Placks

Three years ago, I embarked on a new challenge: to create a mobile application. Initially, I considered developing a plant recognition app, but the market was already saturated. A pivotal moment came when I witnessed a frustrating interaction at an SNCF station. A person who was deaf struggled to purchase a ticket using sign language, with the conversation taking over ten minutes through text messages on a mobile phone. This experience highlighted the urgent need for a better solution to bridge communication gaps.

Inspired by this event, I began exploring the possibilities of AI to recognize sign language and translate it into text and speech in real-time. After two years of research and learning, I finally started developing Placks. My goal is to create an easy-to-use application that is accessible to everyone, helping to break down communication barriers between the hearing and non-hearing communities.

Risks and Challenges

Developing Placks comes with its share of challenges:

🕦 Time: The project is estimated to take 2 to 3 years to complete. Training the AI to recognize each sign language could take longer than anticipated, particularly if reliable results are to be achieved.
💶 Cost: Funding is a critical factor. While I plan to contribute personal resources, the total cost of the project, especially the AI development for each sign language, could be substantial.
📈 AI Accuracy: The most significant challenge lies in developing an AI that is both precise and reliable. There is a risk that the AI might not meet user expectations or may struggle with regional dialects and variations in sign language.

Future Benefits

Once completed, Placks will offer numerous advantages to both hearing and non-hearing users:

Enhanced Communication: The application will make it easier and faster to communicate between those who know sign language and those who do not. This will be invaluable in social interactions, medical appointments, travel, and more.
Global Reach: With support for multiple sign languages, users will be able to communicate across different regions without needing to learn a new sign language.
Public Integration: Placks is designed to be integrated with devices like Arduino or Raspberry Pi, paired with a camera, to assist in public places like metro stations or stores, making these environments more accessible to non-hearing individuals.
Ease of Use: The application will feature a simple, intuitive interface with clear instructions, making it accessible to everyone, including those who rely on assistive technologies like screen readers.

Placks represents a significant step forward in bridging the communication gap between the hearing and non-hearing worlds, offering a practical, innovative solution to a longstanding challenge.

Use Cases

Placks can be employed in various real-life scenarios to facilitate communication for deaf or hard-of-hearing individuals. Here are some examples:

Everyday Situations: Deaf or hard-of-hearing individuals can use Placks to interact with cashiers at supermarkets, servers in restaurants, hotel receptionists, and more. The app translates sign language into text or speech, making everyday interactions smoother and more accessible.
Professional Environments: In the workplace, employees who are deaf or hard-of-hearing can use Placks to communicate effectively with colleagues and supervisors. This ensures that they can participate fully in meetings, discussions, and other professional activities.
Educational Institutions: Students who are deaf or hard-of-hearing can use Placks to follow along with lectures, communicate with teachers, and interact with fellow students. The app helps bridge communication gaps, ensuring that these students have equal access to educational resources.
Public Services: Placks can be a valuable tool for deaf or hard-of-hearing individuals when dealing with public services. Whether at a hospital, government office, or social services agency, the app assists in communicating with service agents, making these interactions less frustrating and more efficient.
Noisy Environments or Time-Sensitive Scenarios: In loud environments or when time is limited, such as at airport counters, train stations, or other fast-paced settings, using sign language recognition might be impractical. In these cases, Placks' messaging feature takes over, providing a quick, simple, and effective way to communicate without relying on sign language recognition.
Large Meetings or Conferences: During large meetings or conferences, Placks can help a deaf or hard-of-hearing participant follow conversations by using voice recognition and real-time transcription. The app not only transcribes what is being said but also identifies who is speaking, ensuring that users can stay engaged and informed throughout the discussion.