Inspiration
Personal experience from a public-speaking class and the desire to continue practicing related communication skills.
What it does
The user can choose between two modes: Normal Mode and Improv Mode.
The former consists of 4 parts: setup, delivery, Q&A, and a reflection. The user briefly describes a speech they have already prepared and the audience they are addressing. Then, the app records the user as they speak and present their speech. When the user ends the speech, or time runs out, a voice agent asks them related questions about their speech's topic, and the user must answer in a collected manner (similar to a debate or a conference where the audience may ask questions).
The next mode is similar, but as the name suggests, it is on a topic the user is unprepared for (impromptu). Either a trending topic is scraped from the web, or a built-in classic topic is chosen. The user gets a brief list of speaking notes and some time to prepare before giving their speech (delivery is about 2 minutes).
In both modes, there is a reflection step where an AI agent analyzes the user's speech content, vocal delivery, facial expressions (including eye contact), and hand gestures. The analysis is relayed to the user, and they are given feedback based on the collected data (from the webcam and mic).
How we built it
For the app's frontend, we used TypeScript + React + Next.js. Visual speech data (like hand gestures and eye contact) is processed using OpenCV and open-source models for analyzing frame data. Audio data (speech-to-text), and the agent's voice, is done through ElevenLabs. All the collected data is then sent to Gemini for final analysis and feedback (for the reflection).
The web scraper for trending topics uses BrightData and Gemini.
We used Neon-Postgres for our database (storing a history of practiced speeches).
Challenges we ran into
Some challenges included accurately analyzing video data for gestures and facial expressions, tuning Gemini to deliver proper analysis/feedback, and designing a clean frontend.
Accomplishments that we're proud of
We're proud of how the final app looks, and of the agent's (mostly) reasonable responses and analysis of user speeches.
What we learned
We learned a lot about app design, computer vision, and putting various data sources together for an AI agent to analyze.
What's next for Rhetor
We probably want to start identifying and fixing major bugs, reducing response times for loading data/models, and improving the accuracy of the AI agent to make the app a more reliable tool for practicing speeches.
Built With
- elevenlabs
- gemini
- mantra
- neon
- next
- opencv
- react
- tailwind
- typescript