How an AI video clipper works: FFmpeg, Whisper, local LLMs

11 steps · about 4 h 10 min · Intermediate

  1. Changing Aspect Ratio: Crop, Fit, or Reframe Video (article, TotalMedia, 10 min)

    Start with the core idea: what aspect ratio is and how 16:9 becomes 9:16 by cropping, fitting or reframing.

  2. FFmpeg Tutorial for Beginners: Mastering the Command Line (video, YouTube, 25 min)

    FFmpeg does the cutting, cropping and rendering in most clippers, so learn its basic commands first.

  3. A Beginner's Cookbook for FFmpeg (repo, GitHub (talwrii), 30 min)

    Short, runnable recipes for trimming, resizing and extracting frames let you practise what the video showed.

  4. ffmpeg Documentation (reference, ffmpeg.org, 30 min)

    The official reference explains streamcopy, transcoding and filtergraphs, so you can read any clipper's FFmpeg calls.

  5. yt-dlp: A feature-rich command-line audio/video downloader (repo, GitHub (yt-dlp), 20 min)

    See how tools fetch source media and captions from a link, and why FFmpeg is a recommended dependency.

  6. openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision (repo, GitHub (OpenAI), 30 min)

    Learn how speech-to-text produces the timestamped transcript that drives captions and highlight picking.

  7. whisper.cpp: Port of OpenAI's Whisper model in C/C++ (repo, GitHub (ggml-org), 20 min)

    Shows how transcription can run fast on a local machine, and how voice-activity detection trims silence.

  8. Quickstart - Ollama (doc, Ollama docs, 20 min)

    Run a language model locally, the same kind of component clippers use to score which moments are worth clipping.

  9. neonclipping architecture diagram (iamajmeena/neonclipping) (article, GitDiagram, 10 min)

    Now see how the pieces fit: source fetching, moment curation, layouts, captions and rendering as one pipeline.

  10. Release NeonClipping v1.0.0 (repo, GitHub (iamajmeena), 10 min)

    A concrete list of highlight signals (loudness jumps, speech cues, motion, local AI score) to compare with what you learned.

  11. iamajmeena/neonclipping (repo, GitHub (iamajmeena), 45 min)

    Finish by reading the real open-source code and README, including the chat-spike detection for live streams.