🤖 Doraemon AI — Video Assistant

Watch Less. Understand More.

Doraemon AI is an AI-powered video assistant that helps users listen, transcribe, understand, summarize and chat with video/audio content.

The application accepts a YouTube URL or uploaded audio/video file, extracts the audio, generates a transcript using Whisper, performs AI-based analysis, stores knowledge using RAG/vector search, and allows users to ask questions about the processed content.

✨ Features

🎬 YouTube video analysis
📁 Upload audio/video files
🎙️ Automatic speech-to-text transcription
🧠 AI-powered video understanding
📝 Automatic summarization
✅ Action item extraction
🔍 Decision extraction
❓ Open question detection
📚 RAG-based knowledge memory
💬 Chat with your video
🔎 Searchable transcript
🌐 Multiple language options
🎨 Modern Doraemon-inspired Streamlit UI
📊 Analysis statistics and processing stages

🏗️ System Architecture

                    ┌──────────────────────┐
                    │       USER           │
                    │ YouTube / File       │
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │   Streamlit UI       │
                    │      app.py          │
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │     main.py          │
                    │   AI Pipeline        │
                    └──────────┬───────────┘
                               │
              ┌────────────────┼────────────────┐
              ▼                ▼                ▼
       ┌────────────┐   ┌─────────────┐   ┌─────────────┐
       │ Extractor  │   │  Whisper    │   │ Summarizer  │
       │            │   │ Transcriber │   │ AI Analysis │
       └────────────┘   └─────────────┘   └─────────────┘
                               │
                               ▼
                       ┌──────────────┐
                       │ RAG Engine   │
                       │ Vector Store │
                       └──────┬───────┘
                              │
                              ▼
                       ┌──────────────┐
                       │ AI Chat      │
                       │ Ask Anything │
                       └──────────────┘

🛠️ Tech Stack

TechnologyPurposePythonBackend & AI pipelineStreamlitWeb UIWhisperAudio transcriptionyt-dlpYouTube media extractionFFmpegAudio/video processingRAGContext-aware question answeringChromaDBVector storageMistral / LLMAI analysispython-dotenvEnvironment configuration

📂 Project Structure

VIDEO AGENT/
│
├── assets/
│   └── doraemon.png
│
├── core/
│   ├── extractor.py
│   ├── rag_engine.py
│   ├── summarizer.py
│   ├── transcriber.py
│   └── vector_store.py
│
├── downloads/
│   └── chunks/
│
├── utils/
│   └── audio_processor.py
│
├── vector_db/
│   └── chroma.sqlite3
│
├── .env
├── app.py
├── main.py
├── requirements.txt
└── README.md

🚀 Installation

1. Clone the repository

git clone <YOUR_GITHUB_REPOSITORY_URL>
cd VIDEO-AGENT

2. Create virtual environment

Windows:

python -m venv .venv

Activate it:

.venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

If python-dotenv is missing:

pip install python-dotenv

4. Install FFmpeg

FFmpeg is required for audio/video processing.

After installation, verify:

ffmpeg -version

Make sure FFmpeg is available in your system PATH.

🔐 Environment Variables

Create a .env file in the project root.

Example:

MISTRAL_API_KEY=your_api_key_here

Add any other API keys required by your backend modules.

Never upload your .env file or API keys to GitHub.

Add this to .gitignore:

.env
.venv/
__pycache__/
vector_db/
downloads/
*.pyc

▶️ Run the Application

Start the Streamlit application:

python -m streamlit run app.py

The application will open in your browser.

Usually:

http://localhost:8501

🎬 How to Use

Step 1 — Select Source

Choose either:

YouTube URL

or

Upload File

Step 2 — Provide Content

For YouTube:

https://www.youtube.com/watch?v=VIDEO_ID

For upload:

MP4 / MOV / MKV / WEBM
MP3 / WAV / M4A

Step 3 — Select Language

Choose the appropriate language or use:

Auto-detect

Step 4 — Analyze

Click:

Analyze with Doraemon AI

The pipeline processes:

Input
  ↓
Audio Extraction
  ↓
Whisper Transcription
  ↓
AI Analysis
  ↓
RAG Memory

🧠 AI Analysis

After processing, Doraemon AI can provide:

📝 Summary

A concise explanation of the video content.

✅ Action Items

Important tasks or recommended actions identified from the content.

◆ Decisions

Important decisions discussed in the video.

❓ Open Questions

Questions that remain unanswered or require further discussion.

📜 Transcript

The complete generated transcript with search functionality.

💬 AI Chat

Users can ask questions such as:

What are the main points discussed?

What decisions were made?

Give me the important action items.

Explain the video in simple language.

What did the speaker say about AI?

Summarize the video in 5 points.

The RAG system uses the processed content to provide context-aware answers.

🔄 Processing Pipeline

YouTube / Upload
       │
       ▼
Media Extraction
       │
       ▼
Audio Processing
       │
       ▼
Whisper Transcription
       │
       ▼
Transcript
       │
       ▼
AI Analysis
       │
       ├── Summary
       ├── Action Items
       ├── Decisions
       └── Questions
       │
       ▼
Text Chunking
       │
       ▼
Embeddings
       │
       ▼
Vector Database
       │
       ▼
RAG Retrieval
       │
       ▼
Doraemon AI Chat

📊 Dashboard

The application provides an interactive dashboard showing:

WORDS
ACTION ITEMS
DECISIONS
OPEN QUESTIONS

It also displays the current AI processing stage:

INPUT
   ↓
WHISPER
   ↓
AI ANALYSIS
   ↓
RAG MEMORY

🎨 UI

The application uses a dark futuristic AI interface with:

Doraemon AI branding
Animated HUD elements
Cyan / blue / violet gradients
Processing indicators
AI status cards
Analysis dashboard
Transcript viewer
RAG chat interface

The main Doraemon image is stored at:

assets/doraemon.png

🧩 Backend Modules

`main.py`

Main orchestration layer responsible for running the complete processing pipeline.

`core/extractor.py`

Handles video/audio extraction and media processing.

`core/transcriber.py`

Handles speech-to-text transcription using Whisper.

`core/summarizer.py`

Handles AI-based summarization and content analysis.

`core/rag_engine.py`

Handles retrieval-augmented generation and question answering.

`core/vector_store.py`

Handles vector database operations and document retrieval.

`utils/audio_processor.py`

Handles audio preprocessing and related utilities.

`app.py`

Provides the Streamlit frontend and connects the UI to the backend.

🛡️ Error Handling

The application handles common problems such as:

Missing YouTube URL
Missing uploaded file
Invalid media input
Backend import errors
Missing dependencies
Transcription errors
RAG errors
Missing environment variables

📌 Future Improvements

🎤 Real-time voice conversation
🗣️ Voice responses from Doraemon AI
🌍 Improved multilingual support
🎥 Automatic video chapter generation
📑 Export summary as PDF
📄 Export transcript as TXT/PDF
🔐 User authentication
☁️ Cloud deployment
📱 Mobile responsive interface
🧠 Long-term user memory
🎞️ Automatic highlight generation
📊 Advanced video analytics

🔮 Vision

Doraemon AI aims to become a personal AI video knowledge assistant that allows users to consume long-form content faster.

Instead of watching an entire video, users can simply ask:

"Doraemon, what do I need to know from this video?"

And the system extracts the important information automatically.

👨‍💻 Project

Doraemon AI — Video Assistant

Watch Less. Understand More.

Built using Python, Streamlit, Whisper, AI analysis and RAG.

⭐ If you like this project

Give the repository a ⭐ on GitHub and feel free to contribute.