VideoLingo README

The project's README from github.com/Huanshere/VideoLingo. As one markdown file: README.md.

VideoLingo Logo

Connect the World, Frame by Frame

Huanshere%2FVideoLingo | Trendshift

English｜简体中文｜繁體中文｜日本語｜Español｜Русский｜Français

🌟 Overview (Try VL Now!)

VideoLingo is an all-in-one video translation, localization, and dubbing tool aimed at generating Netflix-quality subtitles. It eliminates stiff machine translations and multi-line subtitles while adding high-quality dubbing, enabling global knowledge sharing across language barriers.

Key features:

Difference from similar projects: Single-line subtitles only, superior translation quality, seamless dubbing experience

🎥 Demo

Dual Subtitles

https://github.com/user-attachments/assets/a5c3d8d1-2b29-4ba9-b0d0-25896829d951

Cosy2 Voice Clone

https://github.com/user-attachments/assets/e065fe4c-3694-477f-b4d6-316917df7c0a

GPT-SoVITS with my voice

https://github.com/user-attachments/assets/47d965b2-b4ab-4a0b-9d08-b49a7bf3508c

Language Support

Input Language Support(more to come):

🇺🇸 English 🤩 | 🇷🇺 Russian 😊 | 🇫🇷 French 🤩 | 🇩🇪 German 🤩 | 🇮🇹 Italian 🤩 | 🇪🇸 Spanish 🤩 | 🇯🇵 Japanese 😐 | 🇨🇳 Chinese* 😊

*Chinese uses a separate punctuation-enhanced whisper model, for now...

Translation supports all languages, while dubbing language depends on the chosen TTS method.

Installation

Meet any problem? Chat with our free online AI agent here to help you.

Note: For Windows users with NVIDIA GPU, follow these steps before installation:

  1. Install CUDA Toolkit 12.6
  2. Install CUDNN 9.3.0
  3. Add C:\Program Files\NVIDIA\CUDNN\v9.3\bin\12.6 to your system PATH
  4. Restart your computer

Note: FFmpeg is required. Please install it via package managers:

  • Windows: choco install ffmpeg (via Chocolatey)
  • macOS: brew install ffmpeg (via Homebrew)
  • Linux: sudo apt install ffmpeg (Debian/Ubuntu)

uv automatically downloads Python 3.10 and creates an isolated environment — no need to install Python or Anaconda yourself.

  1. Clone the repository
git clone https://github.com/Huanshere/VideoLingo.git
cd VideoLingo
  1. One-command setup (installs uv + Python 3.10 + all dependencies)
python setup_env.py
  1. Start the application
.venv\Scripts\streamlit run st.py        # Windows
.venv/bin/streamlit run st.py            # macOS / Linux

Or double-click OneKeyStart.bat on Windows.

Option B: Using Conda

⚠️ Not recommended. This method will not be maintained going forward. Please use uv (Option A) above.

Click to expand Conda installation steps
  1. Clone the repository
git clone https://github.com/Huanshere/VideoLingo.git
cd VideoLingo
  1. Install dependencies (requires python=3.10)
conda create -n videolingo python=3.10.0 -y
conda activate videolingo
python install.py
  1. Start the application
streamlit run st.py

Docker

Alternatively, you can use Docker (requires CUDA 12.4 and NVIDIA Driver version >550), see Docker docs:

docker build -t videolingo .
docker run -d -p 8501:8501 --gpus all videolingo

APIs

VideoLingo supports OpenAI-Like API format and various TTS interfaces:

Note: VideoLingo works with 302.ai - one API key for all services (LLM, WhisperX, TTS). Or run locally with Ollama and Edge-TTS for free, no API needed!

For detailed installation, API configuration, and batch mode instructions, please refer to the documentation: English | 中文

Current Limitations

  1. WhisperX transcription performance may be affected by video background noise, as it uses wav2vac model for alignment. For videos with loud background music, please enable Voice Separation Enhancement. Additionally, subtitles ending with numbers or special characters may be truncated early due to wav2vac's inability to map numeric characters (e.g., "1") to their spoken form ("one").

  2. Using weaker models can lead to errors during processes due to strict JSON format requirements for responses (tried my best to prompt llm😊). If this error occurs, please delete the output folder and retry with a different LLM, otherwise repeated execution will read the previous erroneous response causing the same error.

  3. The dubbing feature may not be 100% perfect due to differences in speech rates and intonation between languages, as well as the impact of the translation step. However, this project has implemented extensive engineering processing for speech rates to ensure the best possible dubbing results.

  4. Multilingual video transcription recognition will only retain the main language. This is because whisperX uses a specialized model for a single language when forcibly aligning word-level subtitles, and will delete unrecognized languages.

  5. For now, cannot dub multiple characters separately, as whisperX's speaker distinction capability is not sufficiently reliable.

📄 License

This project is licensed under the Apache 2.0 License. Special thanks to the following open source projects for their contributions:

whisperX, yt-dlp, json_repair, BELLE

📬 Contact Me

⭐ Star History

Star History Chart

If you find VideoLingo helpful, please give me a ⭐️!

VideoLingo (Huanshere/VideoLingo)

Searching Database…

18.8K ·GitHub
VideoLingo
这是什么东西?VideoLingo 是一款一站式 AI 工具,用于视频翻译、本地化和配音。它能生成 Netflix 品质的单行字幕,还可以加上高质量配音,全程一键自动完成。
它是在线服务,还是要装在我自己的电脑上运行?在本地运行:它会在浏览器里启动一个 Streamlit 界面,整个流程都从这里操作,也可以改用 Docker 方式。
我怎么把它跑起来?用系统的包管理器安装 FFmpeg,克隆仓库,然后运行一条命令的 uv 安装——它会帮你装好 Python 3.10 和所有依赖。在 Windows 上也可以直接双击 OneKeyStart.bat。
AI 部分需要我配置 API 密钥吗,能不能免费使用?通常你需要配置一个兼容 OpenAI 的大语言模型(默认是通过 OpenRouter 使用 DeepSeek V4 Flash)、WhisperX 和一个 TTS 语音合成服务——302.ai 用一个密钥就能覆盖这三项。想免费使用的话,可以在本地跑 Ollama,再搭配不需要任何 API 密钥的 Edge-TTS。
它还有哪些不足?背景噪音会影响语音识别的准确度,以数字结尾的字幕可能被截断,较弱的大语言模型可能无法通过严格的 JSON 校验步骤——这时你可以删掉输出文件夹,换一个模型重试。配音不是百分之百完美,多语言音轨只保留主语言,也不支持按角色分配不同声音。
它支持哪些系统?Windows、macOS 和 Linux 都支持,每个系统都有各自的安装步骤,另外还有一条需要 NVIDIA 环境的 Docker 路线。界面本身是一个本地运行的 Streamlit 网页应用。
它用什么许可证?有什么限制吗?Apache 2.0,这是一个宽松的许可证——你可以自由使用和修改,包括商用。
我需要 NVIDIA 显卡吗?在本地用 WhisperX 做语音识别时显卡才有帮助,Docker 方式需要 CUDA 12.4 和 550 以上的 NVIDIA 驱动;Windows 的 NVIDIA 用户应先安装 CUDA Toolkit 12.6 和 CUDNN 9.3.0。你也可以把语音识别交给 302.ai 的 API,而不在本地跑。
视频交过去之后,实际会发生什么?它会用 yt-dlp 下载视频,用 WhisperX 生成词级的字幕,再通过 NLP 和 AI 把字幕切分成自然的段落,然后用自定义或 AI 生成的术语表进行翻译,外加一轮三步的 Translate-Reflect-Adaptation 翻译流程,最后导出符合 Netflix 标准的单行字幕——还可以在上面加配音。
它支持哪些语言?输入支持英语、俄语、法语、德语、意大利语、西班牙语、日语和中文,之后还会增加更多。翻译支持所有语言,配音则取决于你选择的 TTS 方式。
配音是怎么做的?能克隆我自己的声音吗?配音可以通过 GPT-SoVITS、Azure、OpenAI、fish-tts、siliconflow 或 edge-tts 实现,customtts.py 还能让你接入自己的 TTS。演示里用作者本人的声音,通过 Cosy2 和 GPT-SoVITS 展示了声音克隆。
和其他字幕工具相比,为什么选它?作者从三点把自己和其他同类项目区分开:只做符合 Netflix 标准的单行字幕(不做多行)、翻译质量更好、配音体验更顺畅。
What are similar projects?pyvideotrans, voice-pro, Index-Translate, OpenCreator, VoiceStudio, autoclip, seamless_communication, VidBee
Which topics is it in?dubbing, yt-dlp, subtitles, voice-cloning, whisper, localization, speech-to-text, tts, text-to-speech, video-processing, translation, ffmpeg
Creator
Huanshere
Size
15.9 MB
Stars
18.8K
Updated
CommitSep 30Releasev3.1.1Sep 27
Age
2y
Commits · 90 Days