storyboard-ai README

The project's README from github.com/yogendra-yatnalkar/storyboard-ai. As one markdown file: README.md.

Storyboard AI

[!NOTE] Latest Release: SAM3 object segmentation is now completely optional, allowing you to run the entire pipeline directly with just your Gemini API key (running whiteboard animations in single-pass mode).

An intelligent agentic pipeline that automates the creation of high-quality, fully narrated whiteboard animation videos from a simple text prompt.

Overview

Storyboard AI is a complete end-to-end framework. It takes in a high-level topic or context and handles everything: researching the topic, writing a compelling narrative script, planning the visual storyboard, generating custom whiteboard-style artwork, animating the drawing process, synthesizing voiceover narration, and burning perfectly timed subtitles.

It operates autonomously using an agentic approach, meaning the Director Agent breaks down the user request into manageable scenes, delegates tasks to specialized sub-agents/tools, and finally stitches everything back together.

Demo Video: What is Adhik Maas and its relation with Shalivahan Shaka (HINDI) (strictly 4 scenes)

[!IMPORTANT] The entire demo video below was generated automatically from a single input prompt/instruction: the title itself.

https://github.com/user-attachments/assets/433e86bc-7ad4-433b-8b09-117e1f3af9e9

Major Pipeline Steps Executed:

  1. Web-search (not deepresearch): Performed web-grounded research to gather facts about Adhik Maas and Shalivahan Shaka.
  2. Grounded Image Generation: Generated custom whiteboard illustrations utilizing internet-grounded reference images for scene visual accuracy.
  3. Whiteboard Animation + Veo: Segmented objects and calculated vector sketch contours using SAM 3 for custom drawing animation, alongside Veo video generation to stitch dynamic segments.
  4. Gemini Native Audio TTS: Synthesized high-quality narration audio in Hindi natively.

Core Features

⚡ Key Advantages

⚙️ Setup & Configuration

1. Environment Configuration (.env)

Create a .env file in the genai-pipeline folder (or copy genai-pipeline/.env.example) and configure the following variables:

# Google API Key for LLM, TTS, Image Gen, and Veo Video Gen
GOOGLE_API_KEY="your-google-api-key-here"

# Hugging Face Access Token (Optional: Only required to download SAM 3 model weights if hosting SAM3 yourself)
# HF_API_KEY="your-huggingface-token-here"

# Set to TRUE if using Vertex AI, or FALSE to use Google Developer API (default)
GOOGLE_GENAI_USE_VERTEXAI=FALSE

2. SAM 3 Model Hosting (FastAPI & GCP Cloud Run - Optional)

The whiteboard drawing sequence generator can utilize instance segmentation for advanced multi-pass drawing. We host a self-contained FastAPI server that wraps the Segment Anything Model 3 (SAM 3).

3. Pipeline Configuration Settings (config.py)

Core configuration parameters are set in genai-pipeline/config.py:

4. Python Environment & Dependencies

Set up your Python environment (Conda environment recommended) and install the verified dependencies:

🚀 How to Run & View Outputs

Step 1: Install Python Dependencies

Depending on the component you are running, install the appropriate requirements:

Step 2: Run the Pipeline CLI

# Navigate to the core agent directory
cd genai-pipeline

# Run the interactive pipeline script
python pipeline.py

Step 3: Interactive CLI Setup

The CLI will guide you through:

  1. Context/Prompt: Enter the main topic for your video (e.g., "The History of Space Travel").
  2. Research Mode: Choose between [1] Deep Research, [2] Web Search (Fast), or [3] None.
  3. Reference Images: Enable or disable internet search for visual references (Y/n).
  4. Fast Mode: Enable parallel image/audio generation for all scenes to save time (Y/n).
  5. Narration Language: Enter the target language for the script (e.g., hindi or english).
  6. Veo Video: Enable or disable Veo AI video generation (y/N).

Step 4: Locate Outputs

All intermediate assets and final outputs are saved under the genai-pipeline/output/run_<timestamp>/ folder:

🏷️ Release v1.0.0

This release marks the official launch of Storyboard AI v1.0.0! With this launch, the agentic whiteboard generation pipeline is fully operational with complete end-to-end automation, multimodal asset generation (images, speech pacing, video stitching), and optional GPU-accelerated SAM 3 segments hosting on GCP Cloud Run.

🗺️ Roadmap & Upcoming Features

We are actively developing new features to expand compatibility and ease of deployment:

✅ Completed Features

storyboard-ai (yogendra-yatnalkar/storyboard-ai)

Searching Database…

175 ·GitHub
Storyboard-ai
What is this?Storyboard AI is an agentic pipeline that turns a single text prompt into a fully narrated whiteboard animation video: it researches the topic, writes a script, plans scenes, generates whiteboard-style artwork, animates the drawing, synthesizes voiceover, and burns subtitles.
Is this a library I import or a standalone pipeline I run?It is an end-to-end Python pipeline run from a CLI, driven by a Director Agent that breaks the request into scenes and delegates to specialized sub-agents/tools, not a library you import into another app.
How do I get the pipeline running?Create a .env file in the genai-pipeline folder (copy .env.example) with your Gemini API key, install the root requirements.txt, then run the pipeline CLI which interactively asks for the topic, research mode, reference images, language and whether to use Veo video.
Do I need a Gemini API key before trying this?Yes, a Gemini API key is the only hard requirement; the Director Agent, image generation, and TTS all run on the Gemini stack, and the pipeline can run entirely in single-pass mode with just that key.
What happens if I skip the SAM3 segmentation server?Without a SAMAPIURL pointing to a hosted SAM3 server, the pipeline simply skips object segmentation and draws the whiteboard animation in a simpler single-pass mode instead of the advanced multi-pass drawing.
How does it keep video generation costs low?Instead of generating long AI video clips, it stretches a small number of static line-art sketch sequences to match the narration length, e.g. only 32 seconds of raw visuals are stretched into a 2 minute 30 second final video.
What languages can the narration be in?The Director, script narration, and subtitles can be generated dynamically in multiple languages such as Hindi, English, and Spanish, selected interactively when running the CLI.
What does the optional Veo integration add?Veo video generation can be enabled to insert rich AI-generated video segments into specific planned scenes instead of relying only on the hand-drawn whiteboard animation.
What are similar projects?CodeVideoModel, storycast, dramaclaw, Pixelle-Video, huobao-drama, Omni-Interaction-Agent, WanGP, generative-models
Which topics is it in?gcp, subtitles, speech-synthesis, text-to-video, object-detection, agentic, fastapi, tts, video-processing, multimodal, video-generation, agent-orchestration
Creator
yogendra-yatnalkar
Size
20 MB
Stars
175
Updated
CommitJul 01Releasev1.01Jul 01
Age
3y
Commits · 90 Days