complete LLM summarization pipeline

.gitignore

@@ -5,4 +5,5 @@ __pycache__/
 instance/
 *.log
 chroma_db
-rag-system/test/loade.py
+rag-system/test/loade.py
+multimodal-engine/data/extracted_frames

README.md

@@ -11,7 +11,10 @@ A structured, end-to-end AI engineering roadmap covering three production-focuse
 ├── multimodal-engine/   # Phase 2 — AI Reel Generator (Multimodal)
 ├── ai-saftey-audit/     # Phase 3 — Safety & Bias Evaluation Toolkit
 ├── requirements.txt     # Shared top-level dependencies
+├── .env
+├── .gitignore
 └── README.md
+
 ```
 
 ---
@@ -24,14 +27,15 @@ A structured, end-to-end AI engineering roadmap covering three production-focuse
 
 A production-grade Retrieval-Augmented Generation pipeline. Upload any PDF and have a grounded, citation-backed conversation with its contents. Features a multi-LLM fallback chain (Ollama → HuggingFace → OpenAI → Google Gemini), persistent ChromaDB vector storage, sentence-aware chunking, and a full Streamlit chat UI.
 
-**Status:** Fully functional with Streamlit UI, CLI runner, and diagnostic test suite.
+🔗 **Live Demo:** [ragsystem-chatpadf.streamlit.app](https://ragsystem-chatpadf.streamlit.app/)
+
+**Status:** Fully functional — live deployed on Streamlit Cloud with UI, CLI runner, and diagnostic test suite.
 
 ---
 
 ### Phase 2 — Multimodal Engine 🔧 In Progress
 
 **`/multimodal-engine`**
-
 An AI engine designed to process, analyze, and generate content across multiple data types — text, images, and audio. The core objective is building automated pipelines that link transcription, summarization, and video processing into a single workflow.
 
 **Planned Deliverable:** AI Reel Generator — takes long-form video/audio, transcribes it, identifies key moments, and outputs a short-form highlight reel.
@@ -209,11 +213,13 @@ python rag-system/test/testdb_load.py
 
 ## 🖼️ Screenshots
 
-> _Screenshots will be added as each phase is completed._
->
-> **RAG System** — Streamlit chat UI with PDF upload, Q&A, and citation panel  
-> **Multimodal Engine** — Video input → transcript → reel output workflow  
-> **AI Safety Audit** — Red-team test results and bias audit report
+**RAG System** — Live at [ragsystem-chatpadf.streamlit.app](https://ragsystem-chatpadf.streamlit.app/)
+
+| Dashboard | Document Ingestion | Chat + Citations |
+|---|---|---|
+| ![Dashboard](rag-system/assets/image.png) | ![Ingestion](rag-system/assets/image_pdfup.png) | ![Chat](rag-system/assets/image_chat.png) |
+
+> **Multimodal Engine** and **AI Safety Audit** screenshots will be added as each phase ships.
 
 ---
 
@@ -228,10 +234,10 @@ python rag-system/test/testdb_load.py
 - [x] Distance-filtered retriever
 - [x] Multi-LLM fallback chain (Ollama / HF / OpenAI / Gemini)
 - [x] Streamlit chat UI with citations
-- [ ] Conversation memory (chat history in prompt)
-- [ ] Streaming LLM responses
-- [ ] Multi-document support
-- [ ] Confidence gate (block hallucination on off-topic queries)
+- [x] Conversation memory (chat history in prompt)
+- [x] Streaming LLM responses
+- [x] Multi-document support
+- [x] Confidence gate (block hallucination on off-topic queries)
 - [ ] Cross-encoder reranking
 - [ ] FastAPI backend
 - [ ] Docker deployment

multimodal-engine/app/video_processor.py

@@ -1,6 +1,7 @@
-import subprocess, os, glob
+import subprocess, os, glob, base64
 from google import genai
 from dotenv import load_dotenv
+from openai import OpenAI
 
 load_dotenv()
 
@@ -36,6 +37,46 @@ def extract_keyframe(video_path, keyframe_output_dir, interval_seconds=5):
         print(f"ffmpeg keyfrmae extrction fsiled")
         raise e
     
+def encode_image_to_base64(image_path):
+
+    with open(image_path, "rb") as image_file:
+        return base64.b64encode(image_file.read()).decode('utf-8')
+    
+def run_openai(frame_path, prompt):
+    print("[Fallback] Routing Visual analysis to Openai")
+
+    try:
+        openai_clinet = OpenAI()
+
+        content_payload = [{"type": "text", "text": prompt}]
+
+        for path in frame_path:
+            base64_image = encode_image_to_base64(path)
+
+            content_payload.append(
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": f"data:image/jpeg;base64,{base64_image}"
+                    }
+                }
+            ) 
+
+            print("Processing Openai Vission Inference")
+
+            response = openai_clinet.chat.completions.create(
+                model="gpt-4o-mini",
+                messages=[{"role": "user", "content": content_payload}],
+                max_tokens=1000
+            )
+
+            print("Openai Vission Inference Completed")
+
+            return response.choices[0].message.content
+    except Exception as e:
+        print(f"🚨 Fallback Exception: OpenAI engine also failed. Reason: {e}")
+        return None
+    
 
 def analyze_scene_with_gemini(frame_dir):
     print(f"[2] Initializing visaula analyzing")
@@ -76,9 +117,9 @@ def analyze_scene_with_gemini(frame_dir):
             client.files.delete(name=cloud_file.name)
             
         return response.text
-    except Exception as e:
-        print(f"Vission Infrenec as {e}")
-        return None
+    except Exception as gemin_error:
+        print(f"🚨 Fallback Exception: Gemini engine also failed. Reason: {gemin_error}")
+        return run_openai(frame_path, prompt)
 
 
 if __name__ == "__main__":

multimodal-engine/app/workflow_engine.py

@@ -0,0 +1,106 @@
+import os, sys
+from dotenv import load_dotenv
+from google import genai
+from openai import OpenAI
+
+from audio_processor import extract_audio_from_video, transcribe_audio, save_transcript_todisk
+from video_processor import extract_keyframe, analyze_scene_with_gemini
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+load_dotenv()
+
+
+def generate_production_blog(audio_transcript, visual_breakdown):
+
+    print(f"[3]: Orchastrating final content Synthesis via gemin")
+
+    client = genai.Client()
+
+    prompt = f"""
+    You are an expert technical content writer and software documentation engineer.
+    
+    I am providing you with two distinct data inputs extracted from a video tutorial:
+    1. RAW AUDIO TRANSCRIPT:
+    ---
+    {audio_transcript}
+    ---
+    
+    2. CHRONOLOGICAL VISUAL BREAKDOWN:
+    ---
+    {visual_breakdown}
+    ---
+    
+    TASK:
+    Synthesize these two inputs into a comprehensive, high-quality, step-by-step Technical Blog Post 
+    written in Markdown. 
+    
+    STRUCTURE RULES:
+    - Add a catchy title at the top (#).
+    - Write a short introduction explaining what software is being demonstrated.
+    - Break the content down into logical step-by-step sections using clear headings (##).
+    - Blend the visual timeline actions smoothly with the spoken words so it reads like a cohesive tutorial.
+    - Highlight specific keyboard shortcuts, timestamps, or interface menus mentioned on screen using code blocks or bold text.
+    - End with a summary conclusion.
+    
+    Do not add conversational commentary—return ONLY the markdown content.
+    """
+
+    try:
+        response = client.models.generate_content(
+            model="gemini-2.5-flash",
+            contents=[prompt]
+        )
+
+        return response.text
+    except Exception as e:
+        print(f"Error during content synthesis: {str(e)}")
+        return None
+
+def run_integrated_pipeline(video_path):
+
+    print(f"🚀 -----Starting Integrated Multimodal Content Generation Pipeline-----------")
+
+    audio_output_path = "../data/extracted_audio.mp3"
+    frames_dir = "../data/extracted_frames"
+    blogs_output_path = "../output/how_multimodals_work_blog.md"
+
+    if not os.path.exists("../output"):
+        os.makedirs("../output")
+
+    # phase 1 extracted the audio
+    extract_audio_from_video(video_path, audio_output_path)
+    audio_transcript = transcribe_audio(audio_output_path)
+
+    if not audio_transcript:
+        print("Pipeline stopped: audio transcribe layer Faileed")
+        return None
+    
+    #phase 2: Visual stream trace
+
+    extract_keyframe(video_path, frames_dir, interval_seconds=5) 
+    visual_breakdown = analyze_scene_with_gemini(frames_dir)
+
+    if not visual_breakdown:
+        print("Pipeline stopped: visual analysis layer Faileed")
+        return None
+    
+    #phase 3 final Blog synthesis
+
+    final_blog_content = generate_production_blog(audio_transcript, visual_breakdown)
+
+    if final_blog_content:
+        with open(blogs_output_path, "w", encoding="utf-8") as f:
+            f.write(final_blog_content)
+        print(f"\n System Success Complete multimodal blog post generated at {blogs_output_path}")
+    else:
+        print("Pipeline stopped: final content synthesis layer Faileed")
+
+if __name__ == "__main__":
+    TARGET_VIDEO = "../data/sample.mp4"
+
+    if not os.path.exists(TARGET_VIDEO):
+        print(f"⚠️ Verification Guard: Please confirm your testing file is ready at: {TARGET_VIDEO}")
+    else:
+        run_integrated_pipeline(TARGET_VIDEO)
+    
+    

multimodal-engine/data/extracted_audio.mp3

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7564824e6396f97dbfa377439e5c83386c0bf75e3375668d4deb330e72e59c5a
-size 919779
+oid sha256:86c6d4d4a32f4a4c439fcc0f1b76528be7f4d5964274f1194be842eb303156d4
+size 1845861

multimodal-engine/data/sample.mp4

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5ba6afa7d5249164d3726b5a05d30d8140f8c37b5a9dc8cd4bee4ee808aa8d14
-size 3086518
+oid sha256:2fb6398207b9a54a33b4ec03c172fc699dd194d25d2e089b9efbfa8b2174478e
+size 15769615

multimodal-engine/data/transcript.txt

@@ -1,13 +0,0 @@
-Inkscribe works on Mac or Windows. And while this demo is on the Mac version, the software is pretty much identical on both platforms.
-
-Using Inkscribe couldn't be simpler. Launch the application, drag an audio or video file onto Inkscribe's media window, click play, and start typing.
-
-Hello. I'm Mr. Ray. It's that easy.
-
-You can type anything you want in Inkscribe's text area. You can also insert time codes anywhere you want to mark when things happen.
-
-And later, you can click on the time codes to jump to that exact point in your video.
-
-Inkscribe does not do the typing for you, but its wide selection of custom keyboard shortcuts makes it super easy to type notes and transcripts without your hands ever leaving the keyboard.
-
-You can learn more about Inkscribe and find additional tutorials on Inkscribe's many features at inkscribe.com. Thanks for watching.

multimodal-engine/output/how_multimodals_work_blog.md

@@ -0,0 +1,44 @@
+# Understanding Generative AI: A Comprehensive Guide
+
+Generative Artificial Intelligence is rapidly transforming how we create and interact with digital content. This guide, drawing insights from a recent tutorial, provides a step-by-step breakdown of what generative AI is, how it works, its capabilities, potential pitfalls, and how to effectively utilize it.
+
+## 1. Defining Generative AI and the Power of Prompts
+
+The journey into generative AI begins with its core definition.
+At **0:00**, the video opens on a laptop screen featuring a chat window, prominently displaying the question `What is generative AI?` alongside a blinking cursor. This immediate visual sets the stage for understanding that generative AI creates new content using artificial intelligence. This content can range from texts and images to videos, music, and voices. The key to directing this creation lies in what's known as a **prompt**—a descriptive input provided in a chat dialogue box, guiding the AI on what you want it to generate.
+
+## 2. Unveiling Generative AI's Creative Capabilities
+
+As the video progresses to **0:05**, a friendly robot appears, and by **0:10**, it's seen with arms spread wide as a deluge of colorful items—representing diverse information and data like brain scans, molecules, books, and digital screens—erupts from a box. This vivid imagery illustrates the expansive capabilities of generative AI tools. They can swiftly provide answers to complex questions, summarize intricate information, and generate a wide array of ideas. Depending on their application, these tools can produce short stories, detailed paintings, functional pieces of code, or even intricate musical compositions.
+
+## 3. The Data Foundation and AI's Learning Process
+
+Moving to **0:25**, visuals of stacked books and a floating document, followed by a person interacting with a large tablet displaying various digital interfaces at **0:15** and **0:20**, metaphorically introduce the interaction process. Then, at **0:35**, a person observes a line of diverse digital displays, including documents, portraits, and abstract shapes. This leads into **0:40**, where a pink funnel is shown, into which web pages with cat images, spreadsheets, charts, and books are tumbling downwards.
+
+This sequence highlights that the foundation for all AI creation is built upon vast quantities of data. The AI system processes and accesses this data, as visually represented by the funnel, to identify intricate patterns and similarities. At **0:45**, these digital assets emerge from the funnel as a complex, interconnected network of colored nodes, resembling a neural network. This visual signifies the AI's internal processing, culminating in the creation of new content.
+
+## 4. The Impressive Quality of AI-Generated Content
+
+At **0:50**, the background shifts to green, revealing a tablet screen displaying a surreal, Salvador Dalí-esque landscape. This visual, immediately followed by an elegantly dressed man comparing an ornate framed painting of the same landscape to its digital counterpart on a tablet at **0:55**, underscores a crucial characteristic of AI-generated output: it is new, often remarkably impressive, and frequently challenging to distinguish from content created by humans.
+
+## 5. Exploring Diverse Outputs, Including Deepfakes
+
+The generative power of AI extends to various media. At **1:00**, a video player shows a paused scene of simple pyramids, which by **1:05**, has been augmented with two knights on horseback in the foreground. This demonstrates AI's ability to generate or modify video content. Further, at **1:10**, a web browser displays a "NEWS" page filled with eclectic and unusual articles and images, from flying saucers to a yellow lion emerging from a garage, showcasing the breadth of diverse content generation.
+
+However, this capability also introduces potential misuses. Generative AI can be utilized to produce images or videos that appear real but are fabricated, a phenomenon commonly known as **deepfakes**.
+
+## 6. Understanding AI's Limitations: Hallucinations and Quality Factors
+
+The tutorial then shifts focus to the challenges and limitations of generative AI. At **1:15**, a person interacts with a stylized robot, and by **1:20**, their speech bubbles show differing perspectives (a globe for the human, a flat Earth for the robot), illustrating potential discrepancies. The video explains that AI-generated texts can be difficult to recognize as machine-made. Even more critically, AI can produce answers that sound plausible and correct but are factually incorrect—a phenomenon termed **hallucinating**.
+
+At **1:25**, a man with a thoughtful expression has a complex thought bubble above his head, suggesting the intricate processes involved in AI's responses. The quality of generative AI's output is directly tied to two critical factors: the quality of the data it was trained on and, significantly, the quality of the prompts provided by the user. This is visually reinforced at **1:30**, as the man types on a laptop, with a glowing input field above it.
+
+## 7. Guiding AI for Effective Utilization and Human Responsibility
+
+The tutorial emphasizes the importance of user interaction. At **1:35**, an initial projection of a robotic head with somewhat unsettling red eyes appears above the laptop, which then transforms into a friendly, smiling robot with simple, round eyes by **1:40**. This visual transformation symbolizes the impact of effective interaction. To harness generative AI's full potential, users must learn to guide these tools with **meaningful prompts** and employ them thoughtfully.
+
+Generative AI offers immense potential, acting as a powerful writing or learning partner. The video concludes with compelling visuals of collaboration: at **1:45**, a diverse group of people and robots stand together with speech bubbles, and at **1:50**, they are seated around a table, actively working side-by-side. The core message is clear: while AI is adept at handling the "hard work" of content generation, humans bear the ultimate responsibility for verifying the facts and ensuring accuracy. The final "AI Campus" logo at **1:55** reinforces this educational focus.
+
+## Conclusion
+
+Generative AI is a powerful and versatile technology capable of creating diverse, high-quality content. Understanding its mechanisms, from data processing to prompt-based interaction, is crucial for effective utilization. While it offers immense potential as a creative and analytical partner, users must also be aware of its limitations, such as the potential for deepfakes and hallucinations. By learning to craft meaningful prompts and taking responsibility for factual verification, we can thoughtfully leverage generative AI to augment human capabilities across various fields.

rag-system/README.md

@@ -1,7 +1,14 @@
 # ⚙️ Enterprise RAG System — Chat with PDF
 
+[![Live Demo](https://img.shields.io/badge/🚀%20Live%20Demo-Streamlit%20Cloud-FF4B4B?style=for-the-badge&logo=streamlit)](https://ragsystem-chatpadf.streamlit.app/)
+[![Python](https://img.shields.io/badge/Python-3.10+-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://python.org)
+[![LangChain](https://img.shields.io/badge/LangChain-Framework-1C3C3C?style=for-the-badge)](https://langchain.com)
+[![ChromaDB](https://img.shields.io/badge/ChromaDB-Vector%20Store-orange?style=for-the-badge)](https://trychroma.com)
+
 A production-grade **Retrieval-Augmented Generation (RAG)** pipeline that allows users to upload multiple PDFs and hold grounded, conversational, citation-backed interactions with their contents. Built with a highly modular architecture, a real-time stream validation engine, a multi-LLM resilient fallback chain, and an interactive Streamlit UI dashboard.
 
+> 🔗 **Try it live:** [ragsystem-chatpadf.streamlit.app](https://ragsystem-chatpadf.streamlit.app/)
+
 ![Main App Dashboard](assets/image.png)
 
 ---
@@ -85,6 +92,16 @@ Upload multiple PDFs ➔ ask questions ➔ get real-time streaming answers compl
 - **Dynamic Retrieval Confidence Scoring** — Translates database Euclidean distance calculations into a user-friendly percentage rating on the dashboard interface
 
 
+---
+
+## 🎬 Demo
+
+📺 **Video Walkthrough** — Full pipeline demonstration: PDF upload → chunking → retrieval → streaming answer with citations.
+
+> ▶️ Watch the demo video to see the multi-LLM fallback chain, real-time streaming, and citation panel in action.
+
+🔗 **Live App:** [ragsystem-chatpadf.streamlit.app](https://ragsystem-chatpadf.streamlit.app/)
+
 ---
 
 ## 🖼️ User Interface Captures