How an AI video clipper works: FFmpeg, Whisper, local LLMs
11 steps · about 4 h 10 min · Intermediate
- Changing Aspect Ratio: Crop, Fit, or Reframe Video (article, TotalMedia, 10 min)
Start with the core idea: what aspect ratio is and how 16:9 becomes 9:16 by cropping, fitting or reframing.
- FFmpeg Tutorial for Beginners: Mastering the Command Line (video, YouTube, 25 min)
FFmpeg does the cutting, cropping and rendering in most clippers, so learn its basic commands first.
- A Beginner's Cookbook for FFmpeg (repo, GitHub (talwrii), 30 min)
Short, runnable recipes for trimming, resizing and extracting frames let you practise what the video showed.
- ffmpeg Documentation (reference, ffmpeg.org, 30 min)
The official reference explains streamcopy, transcoding and filtergraphs, so you can read any clipper's FFmpeg calls.
- yt-dlp: A feature-rich command-line audio/video downloader (repo, GitHub (yt-dlp), 20 min)
See how tools fetch source media and captions from a link, and why FFmpeg is a recommended dependency.
- openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision (repo, GitHub (OpenAI), 30 min)
Learn how speech-to-text produces the timestamped transcript that drives captions and highlight picking.
- whisper.cpp: Port of OpenAI's Whisper model in C/C++ (repo, GitHub (ggml-org), 20 min)
Shows how transcription can run fast on a local machine, and how voice-activity detection trims silence.
- Quickstart - Ollama (doc, Ollama docs, 20 min)
Run a language model locally, the same kind of component clippers use to score which moments are worth clipping.
- neonclipping architecture diagram (iamajmeena/neonclipping) (article, GitDiagram, 10 min)
Now see how the pieces fit: source fetching, moment curation, layouts, captions and rendering as one pipeline.
- Release NeonClipping v1.0.0 (repo, GitHub (iamajmeena), 10 min)
A concrete list of highlight signals (loudness jumps, speech cues, motion, local AI score) to compare with what you learned.
- iamajmeena/neonclipping (repo, GitHub (iamajmeena), 45 min)
Finish by reading the real open-source code and README, including the chat-spike detection for live streams.