Skip to content

About

Turns long phone footage into finished 1080x1920 reels. Every animated element is driven by measurements taken from the source audio, so nothing on screen drifts.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

video-editor

Turns long, unedited phone footage into finished 1080x1920 Instagram reels — no NLE, no timeline, no manual keyframes. Every animated element is driven by something measured out of the source audio, so nothing on screen can drift out of sync with what you hear.

Built by visy-ani.

Two pipelines share one canvas and one layout module:

source what it does
practice reel long 16:9 guitar practice finds the good takes, reads the strumming pattern, animates it
talking-head reel 9:16 letterboxed export reframes, grades, and captions to word-level speech timings

Why it exists

Two problems that a normal editor is bad at.

Finding the take. An hour of practice footage has maybe six minutes worth keeping. There is no speech to cut on, so the only signal is the instrument: where it is sounding, how densely notes land, how steady the timing is. analyze.py measures all three and hands back a ranked table.

Motion that stays locked. Hand-keyframed graphics drift. Here the beat grid, the strum cells, the waveform pulse and the caption highlights are all read out of the audio itself, so a graphic cannot slip against the take — it is the take.


Requirements

  • Python ≥ 3.12 and uv
  • ffmpeg + ffprobe on PATH
  • macOS — type is rendered with system fonts (Avenir Next, Arial Bold, Apple Color Emoji). On Linux, repoint FONT in manifest_reel.py and FONT_BOLD/FONT_REG in layout.py.
  • Apple silicon is assumed for the h264_videotoolbox final encode. --draft uses x264 and works anywhere.
git clone https://github.com/visy-ani/video-editor.git
cd video-editor
uv sync

Deps: librosa (analysis), numpy, pillow (all type and graphics), soundfile, auto-editor (optional dead-air pass).


Repository layout

scripts/
  layout.py         shared 9:16 geometry — the one source of truth for the canvas
  analyze.py        find and rank the takes in a long recording
  strum.py          fit a grid to the strums, read the pattern off it
  motion.py         render the pattern as a transparent PNG sequence
  textcard.py       render a static text overlay as one full-canvas PNG
  reel.py           cut a take and composite it into the banded reel
  manifest_reel.py  the talking-head renderer (its own compositor)
  script_data.py    example script module for the talking-head reel
  script_local.py   your own script module, if you make one   (gitignored)

raw/    source recordings        (gitignored)
work/   derived scratch          (gitignored)
out/    renders + captions       (gitignored)
reels/  cuts cleared for posting (gitignored)

The four content folders ship with only a README. Clone this and they start empty — put footage in raw/, everything else is generated.


Pipeline A — the practice reel

1. Find the takes

uv run python scripts/analyze.py raw/practice.mp4

Extracts mono audio, gates on RMS against a noise floor measured from the recording's own 10th percentile, bridges short gaps so one take is not split by a breath, then scores each span:

score = (steadiness * 0.5 + min(density/4, 1) * 0.3) * length_fit

steadiness is 1 / (1 + std(IOI)/mean(IOI)) over note attacks — a run full of restarts and hesitation spreads its inter-onset intervals wide and scores low. Writes work/takes.json and prints a ranked table.

The ranking is a starting point, not a verdict. It catches hesitant playing; it cannot hear a wrong note. Pick the take yourself.

Knobs: --floor (dB gate, default = noise floor + 12), --gap (bridge pauses shorter than this), --min (discard takes shorter than this).

2. Read the strumming pattern

uv run python scripts/strum.py raw/practice.mp4 --start 457.92 --dur 30.14

Beat trackers impose a grid and the playing drifts against it. This searches eighth-note period and phase for the fit that best explains where the strums actually landed — maximising circular concentration R = |mean(e^{iθ})| over onset phases — then reads the pattern off slot occupancy. A slot counts as part of the pattern if it is struck in most bars.

Down/up comes from slot parity, not timbre: continuous strumming is a pendulum, so the hand travels down on the beat and up on the "&". That is a sounder inference than the spectral-brightness guess this used to make.

eighth 0.2905s -> 103.3 BPM   R=0.261   median quantise err 54 ms
105 strums over ~13 bars

  slot      1     &     2     &     3     &     4     &
  dir       D     U     D     U     D     U     D     U
  bars%   108    85   100    85   100   108   123   100

  pattern: DUDUDUDU

Writes work/strum.json — period, BPM, offset, per-slot pattern, and every strum tagged with its slot.

3. Render the motion graphic

uv run python scripts/motion.py --strum work/strum.json \
    --eyebrow "DAY 1 - MY FIRST SONG" --title "GIVE ME SOME SUNSHINE" \
    --footer "@yourhandle"

Writes work/mg/%05d.png — a transparent PNG per frame at 30fps.

Cells are lit by the measured strum times, not by the fitted grid. That distinction is the whole point: the playing sits a median 54ms off the grid, so a grid-driven highlight would visibly slip while an onset-driven one lands exactly on what you hear. Each cell holds its glow for 240ms and decays.

4. Composite

uv run python scripts/reel.py raw/practice.mp4 --start 457.92 --dur 30.14 \
    --mg work/mg -o out/reel_01.mp4

Or skip the motion pass entirely and use a static card:

uv run python scripts/reel.py raw/practice.mp4 --start 132 --dur 38 \
    --top "DAY 12" --bottom "Am  F  C  G" --sub "92 BPM" -o out/reel_01.mp4

--draft swaps VideoToolbox for x264 while you are checking a cut.

Audio gets a highpass at 78Hz plus a gentle low shelf to take about 12dB off room rumble, 30ms fades at both ends so cuts do not pop, and loudness normalisation to about −14 LUFS. Pass --denoise none to skip the cleanup or --no-loudnorm to keep the original levels.


Pipeline B — the talking-head reel

A different source and a genuinely different renderer.

The source is a 1080x1920 export whose picture is only a 1080x591 letterboxed band at y=669 — the rest is dead black carrying burned-in text. Cropping to that band throws both away and gets the whole canvas back, so the reel is rebuilt from scratch: a graded rounded card on a designed background, captions cut to word-level speech timings, and a goal card that visibly breaks where the story turns.

1. Transcribe

Word timings drive every caption, so the transcript needs them:

uv run python -c "import mlx_whisper, json; \
    json.dump(mlx_whisper.transcribe('work/speech.wav', \
    path_or_hf_repo='mlx-community/whisper-large-v3-mlx', \
    language='hi', word_timestamps=True), open('work/tx_hi.json','w'))"

Ask whisper for the language actually spoken. If it comes back in a script you do not want to caption in — Hindi returns Devanagari — transliterate in the script module rather than re-cutting the transcript, so the word indices stay put.

That index-for-index alignment is the load-bearing invariant: entry i in WORDS is transcript word i, which is what lets every caption inherit a measured timestamp instead of a guessed one. Fix transcription errors in place too. Never add or drop an entry — everything after it lands on the wrong syllable. manifest_reel.py checks this on startup and fails with a readable message rather than captioning the wrong word.

2. Write the script module

Everything specific to one video lives in a script module; the renderer carries no copy of its own. Start from the committed example:

cp scripts/script_data.py scripts/script_local.py    # gitignored

manifest_reel.py imports script_local if it exists and falls back to script_data if it does not, so your own copy never has to enter the repo. A module supplies:

SOURCE, DURATION which file, how much of it
BAND, ANCHOR, ZOOM the letterboxed picture, the face, the push
WORDS, CARDS, EMPHASIS the caption script and how it is grouped
BEATS the sound-off story spine
GOAL, GOAL_IN/BREAK/OUT the vision-board card and when it turns
OUTRO, OUTRO_T, OUTRO_EMOJI what takes the top band at the end

3. Build the audio-reactive envelope

The background glow pulses on loudness, one value per output frame:

uv run python -c "import numpy as np, librosa; \
    y, sr = librosa.load('work/speech.wav', sr=22050, mono=True); \
    e = librosa.feature.rms(y=y, hop_length=sr//30)[0]; \
    np.save('work/env.npy', (e/max(float(e.max()),1e-9)).astype(np.float32))"

manifest_reel.py reads env[frame] and maps it to 0.55 + 0.45*env.

4. Render

uv run python scripts/manifest_reel.py -o out/manifest_reel.mp4

--draft (x264 instead of VideoToolbox) and --dur N together make a check pass about four times faster. A full pass is ~100s.


Architecture

One canvas, one layout module

layout.py owns the 1080x1920 geometry and the -vf chain that gets a source onto it. A 16:9 source scaled to 1080 wide is 1080x608, leaving 1312px of canvas. The video band sits 560px down, so the lower band is the bigger type zone and its dead bottom edge falls under Instagram's caption and buttons.

1080 x 1920
+----------------+
|  560  top band |  eyebrow / title / rule / subtitle
+----------------+
|  video 1080x608
+----------------+
|  752  bottom   |  strum cells, count row, strum counter, handle
+----------------+

band_geometry() returns top / bottom_start / bottom_h and every renderer lays out against those, so moving the band moves all of them at once. Portrait sources fall through to a plain height-scale and fill the frame, so the same pipeline handles both orientations.

Content lives in a script module, never in the renderer

manifest_reel.py holds no words, no timings and no framing of its own. It imports script_local if that module exists and falls back to the committed script_data example if it does not, then reads everything about the video off it — source, duration, band geometry, zoom curve, captions, beats, goal card, outro.

Two things fall out of that. A new reel is a new script module and nothing else. And a personal script stays out of the repo without the renderer needing to know it is missing.

The one invariant is the index alignment between WORDS and the transcript, so it is checked at startup rather than left to fail as a caption landing on the wrong syllable.

All type is PIL, never ffmpeg

This ffmpeg build has neither libfreetype nor libass, so drawtext, subtitles and ass are all unavailable. Type is rendered by PIL into a full 1080x1920 RGBA PNG and composited with a plain overlay=0:0, which sidesteps all three. It also means overlays are always full-canvas layers at origin — no offset arithmetic anywhere.

Don't reach for drawtext. It will fail.

Apple Color Emoji only ships fixed strikes, so manifest_reel.py renders emoji at the one large strike the face carries (160px) and downsamples with Lanczos.

Two composition strategies

reel.py — filter graph. ffmpeg does everything: scale, pad, overlay the pre-rendered PNG sequence, grade, encode. Python only computes geometry and builds the command. Fast, and the frames never enter the process.

manifest_reel.py — frame streaming. ffmpeg decodes and grades on the way in and encodes on the way out; everything between is numpy. It streams raw rgb24 from a decoder subprocess into a compositor and out to an encoder subprocess.

That costs speed, and it buys two things a filter chain cannot give:

  • A float crop box. Image.resize(box=...) resamples subpixel, so the zoom push never stair-steps the way zoompan does. Zoom is one continuous eased curve, each segment starting where the last ended, so it never resets mid-sentence. It caps at 1.22 — about 2.1x on the band's 497px — past which sensor noise starts to show.
  • Per-frame state. Captions highlight the live word by index, the goal card tracks its own break animation, drifting sparks carry phase.

The frame is assembled bottom-up each tick: gradient + vignette background, radial glow scaled by the audio envelope, sparks (drawn behind the card so it occludes them naturally), the graded video card through a rounded mask, then the overlay stack, then film grain from a 10-frame pre-rolled cycle.

Only one thing owns the top band

draw_beat yields to the goal card, and both yield to the outro. The goal card is the graphic doing real work — it eases in on the number the speaker names, fills its bar to 7.5%, and cracks to red exactly where he says it is not happening.

Measured, not keyframed

Every timed element traces back to a measurement:

element driven by
take selection RMS gate + onset density + IOI spread
strum cells onset times, slotted against a fitted grid
strum counter running count of onsets ≤ t
caption highlight whisper word start/end
background pulse per-frame RMS envelope
story beats hand-set, but aligned to sentence ends

Gotchas

The band crop must be even. The picture is 591 rows, but the grade runs in yuv420p and ffmpeg silently emits 590. Reading 591 rows' worth of bytes per frame then desynchronises every later frame by one row, and the picture visibly tears further down the card as the drift accumulates. The crop takes 590 and the renderer raises if frame 0 is not the size it expects.

No beat grid is drawn, deliberately. strum.py measures period and phase fine, but the strums sit a median 54ms off that grid. An animated grid would visibly slip against the playing, so the graphics are driven by onsets instead.

R is a confidence number, read it. The circular concentration from strum.py says how well a single periodic grid explains the onsets. A low R means tempo drifted or the playing is not in a steady eighth-note pattern — the printed pattern is then not worth putting on screen.

Don't reach for a spectral denoiser. afftdn was tried and removed: even at nr=4 it ate the pick attack and halved the onsets analyze.py and strum.py depend on. The cleanup here is a highpass and a low shelf, which leave transients alone. The same chain is applied before analysis in strum.py so the measurement and the render hear the same audio.

Motion frames are the expensive artefact. reel.py --mg work/mg composites what motion.py already wrote rather than re-rendering, so keep work/mg/ until the cut is final.


Optional: strip dead air

auto-editor is included and cuts on loudness rather than speech, so the guitar reads as signal:

uv run auto-editor raw/practice.mp4 --edit audio:threshold=0.03 --margin 0.3s

It also exports Final Cut Pro XML and DaVinci Resolve timelines if you ever want to finish a cut by hand.


Credits

Written by visy-ani.

Built on librosa, Pillow, NumPy, ffmpeg and auto-editor.

About

Turns long phone footage into finished 1080x1920 reels. Every animated element is driven by measurements taken from the source audio, so nothing on screen drifts.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages