← Back to Articles

Building an Open-Source AI Video Editor: Whisper, LLMs, and FFmpeg

By Ali Afzal Yousuf · November 2026 · 4 min read

Content creators and developers producing short-form video for TikTok, YouTube Shorts, and Instagram Reels frequently pay $20–$40/month for proprietary video SaaS tools. These platforms promise automated jump cuts, animated captions, and zoom transitions.

Under the hood, almost all of these tools wrap three open technologies: an audio transcription model (Whisper), an LLM for editorial planning, and a media processing utility (FFmpeg).

To eliminate the subscription tax and give creators total ownership over their video pipeline, we built and open-sourced reelcut.

The Architecture: Decoupling Planning from Rendering

Trying to make an AI model manipulate video bytes directly is slow and prone to hallucination. Instead, reelcut enforces strict separation of concerns:

The reelcut Pipeline:
1. faster-whisper: Generates word-level timestamps locally with zero API cost.
2. LLM Director: Generates a validated plan.json (cuts, punchline stamps, zooms, animated captions).
3. FFmpeg: Deterministically executes cuts, burns highlighted captions, and masters audio to −14 LUFS.

Why plan.json is a Game Changer

Because the AI only outputs a clean JSON schema, the process is fully transparent and editable. If a name is misspelled in a caption or a cut starts 0.5s too early, you don't need to re-run the entire model. You open plan.json, make the edit, and re-render in seconds.

Quick Start

pip install "reelcut[all] @ git+https://github.com/3liAf/reelcut"

Execute an automated edit using your choice of model (Claude, OpenAI, Gemini, or local Ollama):

# Automated full pass
reelcut make raw_clip.mp4

# Or inspect the edit plan before rendering:
reelcut plan raw_clip.mp4
# (tweak clip.reelcut/plan.json)
reelcut render raw_clip.mp4

Open Source

The complete codebase, test suite, and examples are available under the MIT license on GitHub:

→ github.com/3liAf/reelcut