Turns long, unedited phone footage into finished 1080x1920 Instagram reels — no NLE, no timeline, no manual keyframes. Every animated element is driven by something measured out of the source audio, so nothing on screen can drift out of sync with what you hear.
Built by visy-ani.
Two pipelines share one canvas and one layout module:
| source | what it does | |
|---|---|---|
| practice reel | long 16:9 guitar practice | finds the good takes, reads the strumming pattern, animates it |
| talking-head reel | 9:16 letterboxed export | reframes, grades, and captions to word-level speech timings |
Two problems that a normal editor is bad at.
Finding the take. An hour of practice footage has maybe six minutes worth
keeping. There is no speech to cut on, so the only signal is the instrument:
where it is sounding, how densely notes land, how steady the timing is.
analyze.py measures all three and hands back a ranked table.
Motion that stays locked. Hand-keyframed graphics drift. Here the beat grid, the strum cells, the waveform pulse and the caption highlights are all read out of the audio itself, so a graphic cannot slip against the take — it is the take.
- Python ≥ 3.12 and uv
ffmpeg+ffprobeonPATH- macOS — type is rendered with system fonts (
Avenir Next,Arial Bold,Apple Color Emoji). On Linux, repointFONTinmanifest_reel.pyandFONT_BOLD/FONT_REGinlayout.py. - Apple silicon is assumed for the
h264_videotoolboxfinal encode.--draftuses x264 and works anywhere.
git clone https://github.com/visy-ani/video-editor.git
cd video-editor
uv syncDeps: librosa (analysis), numpy, pillow (all type and graphics),
soundfile, auto-editor (optional dead-air pass).
scripts/
layout.py shared 9:16 geometry — the one source of truth for the canvas
analyze.py find and rank the takes in a long recording
strum.py fit a grid to the strums, read the pattern off it
motion.py render the pattern as a transparent PNG sequence
textcard.py render a static text overlay as one full-canvas PNG
reel.py cut a take and composite it into the banded reel
manifest_reel.py the talking-head renderer (its own compositor)
script_data.py example script module for the talking-head reel
script_local.py your own script module, if you make one (gitignored)
raw/ source recordings (gitignored)
work/ derived scratch (gitignored)
out/ renders + captions (gitignored)
reels/ cuts cleared for posting (gitignored)
The four content folders ship with only a README. Clone this and they start
empty — put footage in raw/, everything else is generated.
uv run python scripts/analyze.py raw/practice.mp4Extracts mono audio, gates on RMS against a noise floor measured from the recording's own 10th percentile, bridges short gaps so one take is not split by a breath, then scores each span:
score = (steadiness * 0.5 + min(density/4, 1) * 0.3) * length_fit
steadiness is 1 / (1 + std(IOI)/mean(IOI)) over note attacks — a run full
of restarts and hesitation spreads its inter-onset intervals wide and scores
low. Writes work/takes.json and prints a ranked table.
The ranking is a starting point, not a verdict. It catches hesitant playing; it cannot hear a wrong note. Pick the take yourself.
Knobs: --floor (dB gate, default = noise floor + 12), --gap (bridge
pauses shorter than this), --min (discard takes shorter than this).
uv run python scripts/strum.py raw/practice.mp4 --start 457.92 --dur 30.14Beat trackers impose a grid and the playing drifts against it. This searches
eighth-note period and phase for the fit that best explains where the strums
actually landed — maximising circular concentration R = |mean(e^{iθ})|
over onset phases — then reads the pattern off slot occupancy. A slot counts
as part of the pattern if it is struck in most bars.
Down/up comes from slot parity, not timbre: continuous strumming is a pendulum, so the hand travels down on the beat and up on the "&". That is a sounder inference than the spectral-brightness guess this used to make.
eighth 0.2905s -> 103.3 BPM R=0.261 median quantise err 54 ms
105 strums over ~13 bars
slot 1 & 2 & 3 & 4 &
dir D U D U D U D U
bars% 108 85 100 85 100 108 123 100
pattern: DUDUDUDU
Writes work/strum.json — period, BPM, offset, per-slot pattern, and every
strum tagged with its slot.
uv run python scripts/motion.py --strum work/strum.json \
--eyebrow "DAY 1 - MY FIRST SONG" --title "GIVE ME SOME SUNSHINE" \
--footer "@yourhandle"Writes work/mg/%05d.png — a transparent PNG per frame at 30fps.
Cells are lit by the measured strum times, not by the fitted grid. That distinction is the whole point: the playing sits a median 54ms off the grid, so a grid-driven highlight would visibly slip while an onset-driven one lands exactly on what you hear. Each cell holds its glow for 240ms and decays.
uv run python scripts/reel.py raw/practice.mp4 --start 457.92 --dur 30.14 \
--mg work/mg -o out/reel_01.mp4Or skip the motion pass entirely and use a static card:
uv run python scripts/reel.py raw/practice.mp4 --start 132 --dur 38 \
--top "DAY 12" --bottom "Am F C G" --sub "92 BPM" -o out/reel_01.mp4--draft swaps VideoToolbox for x264 while you are checking a cut.
Audio gets a highpass at 78Hz plus a gentle low shelf to take about 12dB off
room rumble, 30ms fades at both ends so cuts do not pop, and loudness
normalisation to about −14 LUFS. Pass --denoise none to skip the cleanup or
--no-loudnorm to keep the original levels.
A different source and a genuinely different renderer.
The source is a 1080x1920 export whose picture is only a 1080x591 letterboxed band at y=669 — the rest is dead black carrying burned-in text. Cropping to that band throws both away and gets the whole canvas back, so the reel is rebuilt from scratch: a graded rounded card on a designed background, captions cut to word-level speech timings, and a goal card that visibly breaks where the story turns.
Word timings drive every caption, so the transcript needs them:
uv run python -c "import mlx_whisper, json; \
json.dump(mlx_whisper.transcribe('work/speech.wav', \
path_or_hf_repo='mlx-community/whisper-large-v3-mlx', \
language='hi', word_timestamps=True), open('work/tx_hi.json','w'))"Ask whisper for the language actually spoken. If it comes back in a script you do not want to caption in — Hindi returns Devanagari — transliterate in the script module rather than re-cutting the transcript, so the word indices stay put.
That index-for-index alignment is the load-bearing invariant: entry i
in WORDS is transcript word i, which is what lets every caption inherit
a measured timestamp instead of a guessed one. Fix transcription errors in
place too. Never add or drop an entry — everything after it lands on the
wrong syllable. manifest_reel.py checks this on startup and fails with a
readable message rather than captioning the wrong word.
Everything specific to one video lives in a script module; the renderer carries no copy of its own. Start from the committed example:
cp scripts/script_data.py scripts/script_local.py # gitignoredmanifest_reel.py imports script_local if it exists and falls back to
script_data if it does not, so your own copy never has to enter the repo.
A module supplies:
SOURCE, DURATION |
which file, how much of it |
BAND, ANCHOR, ZOOM |
the letterboxed picture, the face, the push |
WORDS, CARDS, EMPHASIS |
the caption script and how it is grouped |
BEATS |
the sound-off story spine |
GOAL, GOAL_IN/BREAK/OUT |
the vision-board card and when it turns |
OUTRO, OUTRO_T, OUTRO_EMOJI |
what takes the top band at the end |
The background glow pulses on loudness, one value per output frame:
uv run python -c "import numpy as np, librosa; \
y, sr = librosa.load('work/speech.wav', sr=22050, mono=True); \
e = librosa.feature.rms(y=y, hop_length=sr//30)[0]; \
np.save('work/env.npy', (e/max(float(e.max()),1e-9)).astype(np.float32))"manifest_reel.py reads env[frame] and maps it to 0.55 + 0.45*env.
uv run python scripts/manifest_reel.py -o out/manifest_reel.mp4--draft (x264 instead of VideoToolbox) and --dur N together make a check
pass about four times faster. A full pass is ~100s.
layout.py owns the 1080x1920 geometry and the -vf chain that gets a source
onto it. A 16:9 source scaled to 1080 wide is 1080x608, leaving 1312px of
canvas. The video band sits 560px down, so the lower band is the bigger
type zone and its dead bottom edge falls under Instagram's caption and buttons.
1080 x 1920
+----------------+
| 560 top band | eyebrow / title / rule / subtitle
+----------------+
| video 1080x608
+----------------+
| 752 bottom | strum cells, count row, strum counter, handle
+----------------+
band_geometry() returns top / bottom_start / bottom_h and every
renderer lays out against those, so moving the band moves all of them at once.
Portrait sources fall through to a plain height-scale and fill the frame, so
the same pipeline handles both orientations.
manifest_reel.py holds no words, no timings and no framing of its own. It
imports script_local if that module exists and falls back to the committed
script_data example if it does not, then reads everything about the video
off it — source, duration, band geometry, zoom curve, captions, beats, goal
card, outro.
Two things fall out of that. A new reel is a new script module and nothing else. And a personal script stays out of the repo without the renderer needing to know it is missing.
The one invariant is the index alignment between WORDS and the transcript,
so it is checked at startup rather than left to fail as a caption landing on
the wrong syllable.
This ffmpeg build has neither libfreetype nor libass, so drawtext,
subtitles and ass are all unavailable. Type is rendered by PIL into a
full 1080x1920 RGBA PNG and composited with a plain overlay=0:0, which
sidesteps all three. It also means overlays are always full-canvas layers at
origin — no offset arithmetic anywhere.
Don't reach for drawtext. It will fail.
Apple Color Emoji only ships fixed strikes, so manifest_reel.py renders
emoji at the one large strike the face carries (160px) and downsamples with
Lanczos.
reel.py — filter graph. ffmpeg does everything: scale, pad, overlay the
pre-rendered PNG sequence, grade, encode. Python only computes geometry and
builds the command. Fast, and the frames never enter the process.
manifest_reel.py — frame streaming. ffmpeg decodes and grades on the way
in and encodes on the way out; everything between is numpy. It streams raw
rgb24 from a decoder subprocess into a compositor and out to an encoder
subprocess.
That costs speed, and it buys two things a filter chain cannot give:
- A float crop box.
Image.resize(box=...)resamples subpixel, so the zoom push never stair-steps the wayzoompandoes. Zoom is one continuous eased curve, each segment starting where the last ended, so it never resets mid-sentence. It caps at 1.22 — about 2.1x on the band's 497px — past which sensor noise starts to show. - Per-frame state. Captions highlight the live word by index, the goal card tracks its own break animation, drifting sparks carry phase.
The frame is assembled bottom-up each tick: gradient + vignette background, radial glow scaled by the audio envelope, sparks (drawn behind the card so it occludes them naturally), the graded video card through a rounded mask, then the overlay stack, then film grain from a 10-frame pre-rolled cycle.
draw_beat yields to the goal card, and both yield to the outro. The goal
card is the graphic doing real work — it eases in on the number the speaker
names, fills its bar to 7.5%, and cracks to red exactly where he says it is
not happening.
Every timed element traces back to a measurement:
| element | driven by |
|---|---|
| take selection | RMS gate + onset density + IOI spread |
| strum cells | onset times, slotted against a fitted grid |
| strum counter | running count of onsets ≤ t |
| caption highlight | whisper word start/end |
| background pulse | per-frame RMS envelope |
| story beats | hand-set, but aligned to sentence ends |
The band crop must be even. The picture is 591 rows, but the grade runs in yuv420p and ffmpeg silently emits 590. Reading 591 rows' worth of bytes per frame then desynchronises every later frame by one row, and the picture visibly tears further down the card as the drift accumulates. The crop takes 590 and the renderer raises if frame 0 is not the size it expects.
No beat grid is drawn, deliberately. strum.py measures period and phase
fine, but the strums sit a median 54ms off that grid. An animated grid would
visibly slip against the playing, so the graphics are driven by onsets
instead.
R is a confidence number, read it. The circular concentration from
strum.py says how well a single periodic grid explains the onsets. A low
R means tempo drifted or the playing is not in a steady eighth-note
pattern — the printed pattern is then not worth putting on screen.
Don't reach for a spectral denoiser. afftdn was tried and removed: even
at nr=4 it ate the pick attack and halved the onsets analyze.py and
strum.py depend on. The cleanup here is a highpass and a low shelf, which
leave transients alone. The same chain is applied before analysis in
strum.py so the measurement and the render hear the same audio.
Motion frames are the expensive artefact. reel.py --mg work/mg
composites what motion.py already wrote rather than re-rendering, so keep
work/mg/ until the cut is final.
auto-editor is included and cuts on loudness rather than speech, so the
guitar reads as signal:
uv run auto-editor raw/practice.mp4 --edit audio:threshold=0.03 --margin 0.3sIt also exports Final Cut Pro XML and DaVinci Resolve timelines if you ever want to finish a cut by hand.
Written by visy-ani.
Built on librosa, Pillow, NumPy, ffmpeg and auto-editor.