AI-Powered Inclusive Communication Challenge

Communication barriers between the Deaf community and hearing individuals remain a significant issue despite advances in technology. While some machine translation systems for spoken languages have achieved remarkable success, similar progress for sign languages has been limited. Existing sign language translation tools often rely on static, rule-based systems or limited datasets, leading to inaccuracies and a lack of contextual understanding.

Moreover, the current tools lack integration with broader accessibility features such as real-time captioning, text-to-speech, and voice modulation, leaving many gaps in accessibility for the broader differently-abled population. These inadequacies result in unreliable translations, minimal adaptability to the complexities of sign language, and limited practical applications in real-world scenarios.

MADA Center is therefore looking for solution providers that can help to develop an AI-powered accessibility solution for language translation and communication for sign language and persons with communication difficulties.

This ideal solution should therefore aim to develop a cutting-edge Machine Translation system for Sign Language and Persons with Communication Difficulties, powered by Large Language Models (LLMs), complemented by additional AI-driven tools to support communication for individuals with disabilities. The innovation lies not only in the translation capabilities but also in the integration of complementary features like text-to-speech, speech-to-text systems, real-time captioning, and voice modulation to enhance accessibility for a broader audience.
For reference, we have proven in a scientific publication that is possible to build such model published here: https://link.springer.com/article/10.1007/s44282-024-00113-0

Central to the project is the creation of an advanced expert user interface (UI) designed to evaluate and refine LLM translation outcomes. This interface will enable experts to assess translation quality, adjust prompts, and make corrections, thereby improving accuracy, contextual relevance, and usability. Moreover, the project will provide an API to ensure compatibility with various applications, fostering seamless integration across platforms.

Several solutions have been attempted in the realm of sign language translation, but they often fall short due to the following reasons:

• Rule-Based Systems: Traditional systems based on predefined grammatical rules and vocabulary lack the flexibility to adapt to the nuances of sign language, such as regional variations or contextual meanings. These systems often fail to capture the full depth and complexity of sign language communication.
• Data-Driven Approaches: Efforts to use machine learning models trained on small, static datasets have proven inadequate. Limited annotated datasets restrict the models’ ability to generalize, resulting in poor accuracy and limited scalability.
• Wearable Device-Based Solutions: Some approaches use gloves or other wearable devices to capture signs, but these solutions are intrusive, often uncomfortable for users, and fail to accommodate the natural expressiveness of sign language.
• Lack of Expert Involvement: Existing systems often lack a mechanism for domain experts to actively refine outputs, resulting in stagnant performance and limited contextual relevance.

• Hardware-Dependent Solutions: Solutions requiring specialized hardware, such as wearable devices or camera setups with complex requirements, are not aligned with the project's goal of creating a scalable, software-driven approach.
• Non-Adaptive Models: Approaches that do not incorporate mechanisms for iterative improvement through expert feedback, such as rigid rule-based systems, are excluded as they fail to adapt to the complexities and nuances of sign language.
• Generalized Accessibility Tools: While broad accessibility solutions are valuable, this project specifically focuses on enhancing sign language translation. Generic tools without specific sign language components or integration will not be pursued.