On this page

Digital Human Koubo talking-head production and Remotion 4 automated editing workflow

Document type
Field manual
Updated
Reading time
11 min

Configure andyhuo520/digital-human-koubo across its 0-to-9 talking-head production pipeline, 5 Remotion 4 editing styles, 48 packaging components, and finance-explainer skill, including cost estimation, voice unification, and word alignment.

Source
Document details
andyhuo520/digital-human-koubo
License
MIT

Convert a topic prompt, script, or existing audio file into a finished talking-head video with word-aligned captions, animated charts, and three-ratio covers by chaining Renoise MCP, whisper.cpp, and Remotion 4.

Pipeline stages
10End-to-end loop from stage 0 to stage 9
Remotion 4 editing styles
5Covering 16:9 and 9:16 templates
Packaging components
48Plus 1 inline Highlight wrapper

01Pipeline architecture, 5 styles, and license scope

System purpose

andyhuo520/digital-human-koubo, released by Berryxia (andyhuo520) under the MIT license, packages Claude Code skills that convert a topic prompt, full script, or existing audio file into a captioned, mixed, and cover-packaged AI talking-head video. The repository contains 2 skills: skills/digital-human-koubo drives talking-head generation and Remotion 4 editing, while skills/finance-explainer runs a chart-only whiteboard animation pipeline for long-form finance videos (when Style A includes a presenter corner frame, the main skill calls templates/lin-169).

Execution runs from stage 0 through stage 9 across 10 steps: stage 0 checks local dependencies and the Renoise MCP account; stage 1 diagnoses the script and recommends 2 of the 5 templates; stage 2 writes the segmented script.json; stage 3 estimates credit usage and generates clips in batches after user confirmation; stage 4 downloads clips over 6 concurrent HTTP Range requests and verifies them with whisper-cli back-listening and 4 extracted frames; stage 5 unifies multi-segment audio with 1/3-octave spectral matching; stage 6 generates the word-level timeline tl.json; stage 7 applies a Remotion 4 template and 48 packaging components; stage 8 renders the final cut and normalizes loudness; stage 9 produces 16:9, 9:16, and 3:4 covers and prepends a 0.5-second cover clip to the opening frame.

Style codeAspect ratio and durationTemplate pathVisual characteristics and scheduling
Style A (Finance explainer motion)16:9, 3–20 minutestemplates/lin-169Dark blue grid background, hand-drawn whiteboard charts, yellow highlight cards, and a rounded presenter box; includes 13 scene types compiled from plan.json to scenes.json via lin_scenes.py
Style B (Kinetic-caption talking head)9:16, 30–120 secondstemplates/huazi-916Full-screen presenter footage with pop-up kinetic typography, picture-in-picture screen recordings, and bilingual subtitles; configures intensity in src/edit.json across restrained, standard, and lively
Style C (Product launch video)16:9, 30–90 secondstemplates/reference/launch-169Dark backdrop, high-contrast title cards, 3D tilted product windows, and rounded presenter cut-ins for feature or product launches
Style D (Hands-on tutorial)16:9, 3–8 minutestemplates/reference/tutorial-169Persistent left-side step list, zoom focus on target UI buttons, top chapter progress bar, and lower-right presenter window
Style E (Multi-host news mashup)16:9, 3–7 minutestemplates/reference/multihost-169Alternating human and multi-character digital presenters (such as the orange cat manager and Shiba Inu news anchor in persona.json) with two-layer nameplates and quote bubbles
Digital Human Koubo 10-stage production pipeline

  1. Preflight and generation layer (stage 0 to stage 4)

    • doctor.sh and Renoise get_account preflight
    • brief.json style diagnosis and script.json segmentation
    • build_tasks.py quote, rangedl.py download, and whisper-cli 4-frame QA

  2. Acoustic and timeline layer (stage 5 to stage 6)

    • voicefx/unify.py 1/3-octave spectral matching
    • Sibilance spread compressed from 13.2 dB to 5.9 dB
    • align.py punctuation sentence splitting and word-level tl.json

  3. Editing and delivery layer (stage 7 to stage 9)

    • 5 Remotion 4 templates and 48 packaging components
    • check_edit.py time-anchor validation and -14/-16 LUFS render
    • make_cover.py three-ratio covers and cover_first.py 0.5-second opening frame

Stage 0 to stage 4 generate presenter segments through Renoise MCP; stage 5 to stage 9 run entirely locally and support free editing trials with existing footage.

While README.md, the skills/digital-human-koubo/SKILL.md front matter, and references/packaging.md state 48 packaging components, lines 105 and 123 of SKILL.md retain an older count of 47 components and a 47-cell overview from before TextBehind was added; inspecting PACK in templates/packaging/src/kit/registry.tsx shows 49 exported keys, where Highlight acts as an inline text highlighter wrapper (returning null when invoked as a standalone layer) and 48 components mount directly in the pack array.

02Skill mounting, Renoise MCP, and doctor.sh check

Installation and verification

  • Not completed:

    Install ffmpeg, Node.js 18 or later, and Python 3 (with numpy, Pillow, playwright, and Chromium) on the local machine.

  • Not completed:

    Install whisper-cli (whisper.cpp) and download ggml-medium-q5_0.bin into ~/models/ or ~/.cache/whisper/ for segment back-listening and word-level alignment.

  • Not completed:

    Register a Renoise account and configure the MCP server to generate new digital-human clips or reference portraits; testing stage 5 through stage 9 with existing local video consumes 0 credits.

  • Not completed:

    Run on macOS when using the TextBehind depth-typography component so scripts/personmask.swift can compile against the system Apple Vision person-segmentation framework.

This report inspected source files, scripts, and documentation only; it did not invoke Renoise MCP for billed generation tasks or execute local Remotion video renders.

Route One: Mount the Claude Code skills and connect Renoise MCP

bash
git clone https://github.com/andyhuo520/digital-human-koubo.git
cd digital-human-koubo
mkdir -p ~/.claude/skills
ln -s "$(pwd)/skills/digital-human-koubo" ~/.claude/skills/digital-human-koubo
ln -s "$(pwd)/skills/finance-explainer" ~/.claude/skills/finance-explainer
claude mcp add renoise -- npx -y @renoise-ai/mcp-server

Route Two: Run doctor.sh and install missing local dependencies

scripts/doctor.sh is a read-only check script that inspects the local environment and prints OS-specific installation commands without modifying the system:

bash
bash skills/digital-human-koubo/scripts/doctor.sh

If doctor.sh exits with code 1, install the missing dependencies and the ggml-medium-q5_0.bin model on macOS:

bash
brew install ffmpeg node whisper-cpp
python3 -m pip install numpy pillow playwright
python3 -m playwright install chromium
mkdir -p ~/models
curl -L -o ~/models/ggml-medium-q5_0.bin https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium-q5_0.bin

0348 packaging components and automation scripts

Core capabilities

Stage 3 task builder and quoting

python3 scripts/build_tasks.py

Assemble prompts and calculate Renoise credit cost

Reads persona.json and script.json to emit renoise_tasks.json; Grok Video 1.5 costs 17 credits/s at 720p and 10 credits/s at 480p, capped at 15 seconds per segment.

Stage 4 concurrent download and dual QA

rangedl.py / assemble.py / patch_words.py

6-range download, whisper back-check, and 4-frame inspection

Bypasses single-connection ~300 KB/s throttling; extracts frames at 0.5s, 35%, 70%, and end -0.3s into review_grid.jpg, while patch_words.py patches 1 to 2 seconds of misread audio in place.

Stage 5 cross-segment voice unification

python3 scripts/voicefx/unify.py

Eliminate timbre and room-tone shifts across segments

Applies 120 Hz to 10 kHz 1/3-octave spectral matching (up to ±6 dB), 4.5 to 9 kHz dynamic de-essing, and shared early reflections, reducing sibilance spread from 13.2 dB to 5.9 dB.

Stage 6 word-level timestamp alignment

python3 scripts/align.py

Produce a word-aligned timeline with under 50 ms error

Runs whisper-cli -ml 1 to obtain per-word start and end timestamps, enforces sentence splits from script.json punctuation, and outputs tl.json for Remotion.

Stage 7 cross-template packaging kit

PackLayer (templates/packaging/src/kit/registry.tsx)

Mount 48 motion components via "segment:word" anchors

Covers 8 hook components, 15 evidence/data components, 8 annotation/person components, and 17 pacing/interaction components; defaults to at + dur + 3.5 seconds when until is omitted.

Stage 9 three-ratio covers and first-frame splice

make_cover.py / cover_first.py

Render 16:9, 9:16, and 3:4 covers and prepend to video

Renders HTML covers with Playwright aligned to face coordinates from face_center.swift, then encodes a 0.5-second opening segment via cover_first.py.

The 48 components documented in references/packaging.md and templates/packaging/src/kit/registry.tsx fall into 4 groups:

Component groupCountRepresentative components and behaviorTiming and layout rules
Openers and hooks8TeaserColdOpen (flashes 3 to 5 highlight moments), ColdOpenFreeze, SearchTyping (1.6 times push-in), NumberSpike, SlamText, WordStack, TornStrip, TextBehind (depth typography behind the presenter)TextBehind pairs with cutout_video.py transparent WebM masks, defaulting to y=0.36 in landscape and y=0.28 in portrait
Evidence and data15EvidenceCard, DocFocus (dims non-target sentences by 35%), SourceDoc, ZoomFocus, Counter, Odometer, GlassPanel, SpecCard, LineDraw, BarCompare, ReviewCards, ArchiveCards, MapRoute, CompareWipe, PaperCollageFull-screen components cover the presenter video while keeping the voice track active to mask lip-sync artifacts
Annotations and people8HandDraw, CalloutPill, LeaderLabel, TypedNameTag, LowerThird, NamePlate, SourceTag, FreezeFrameAlso exports key 49, Highlight, for inline text highlighting
Pacing, interaction, and structure17PunchInHost (hard-cuts 1.0/1.3 or 1.0/1.18/1.32 framing on sentence boundaries), KeywordStack, RetentionCaption, TiltLine (-12 degree quote), Aside, Danmaku, ReactionPop (59 bundled Noto animated emoji), MemeWindow, PollPop, GlitchInterstitial, HostFloaters, ProgressChapters, ProgressTracker, ChapterCard, RecapBoard, Countdown, GhostWordIn 9:16 portrait mode, omitted coordinates automatically fall back to VERT safe defaults (1080×1920)

04Credit control, depth-text framing, and voice cloning

Operating principles

Generate segment 1 alone before submitting batches of 10

After quoting via estimate_cost in stage 3 and receiving an explicit yes from the user, generate segment 1 first and inspect its extracted frames and audio; submit the remaining segments via create_tasks in batches of at most 10 only after verifying framing, tone, and lip sync.

Split 720p and 480p by shot type to save 41% in credits

Mandarin delivery defaults to roughly 5.4 characters/s, keeping each segment within 65 Chinese characters or 32 English words (leaving 0.6 seconds of tail padding). Set full-screen segments (full, vlog, depth) to 720p (17 credits/s) and picture-in-picture or B-roll-covered segments (pip) to 480p (10 credits/s).

Use landscape or medium framing for TextBehind segments

TextBehind requires headroom and visible background on both sides of the presenter (set shot=depth in script.json). Landscape medium shots where the presenter occupies about 28% of the frame work best; portrait selfie close-ups (89% subject coverage) occlude the text, and cutout_video.py warns when subject coverage exceeds 60%.

Pair custom voice clones with Wan 3.0 reference mode

Grok Video accepts image and text prompts only, not external audio files. When cloning a custom voice with Xiaomi MiMo (mimo-v2.5-tts-voiceclone using 12 to 20 seconds of dry audio), drive full-screen lip sync with Wan 3.0 reference mode (reference image plus reference audio at roughly 6 credits/s for 480p).

For foreign-language delivery, persona.json records whisper back-check accuracy rates from month 10 of 2026: 100% for English, 100% for Japanese, 100% for French, 100% for Spanish, 99% for Arabic, and 94% for Korean. Translate lines into the target language inside script.json before building prompts rather than relying on live model translation.

05End-to-end 9:16 kinetic-caption talking-head workflow

Worked example

The following sequence follows skills/digital-human-koubo/SKILL.md to initialize projects/demo/ and run Style B (huazi-916) from task generation, download verification, voice unification, and word alignment through Remotion rendering and first-frame cover splicing.

1. Build tasks, download segments, and run dual QA

bash
mkdir -p projects/demo/raw
python3 skills/digital-human-koubo/scripts/build_tasks.py \
  skills/digital-human-koubo/persona.json \
  projects/demo/script.json \
  projects/demo/renoise_tasks.json
python3 skills/digital-human-koubo/scripts/rangedl.py "<cdn_mp4_url>" projects/demo/raw/s01.mp4
python3 skills/digital-human-koubo/scripts/assemble.py \
  projects/demo/raw \
  projects/demo/script.json \
  projects/demo/check

2. Unify cross-segment audio and align word timestamps

bash
python3 skills/digital-human-koubo/scripts/voicefx/unify.py \
  projects/demo/raw \
  --out projects/demo/unified
python3 skills/digital-human-koubo/scripts/align.py \
  projects/demo/unified/full.wav \
  projects/demo/script.json \
  projects/demo/tl.json

3. Initialize the Remotion project, validate anchors, and render covers

bash
bash skills/digital-human-koubo/scripts/new_edit.sh huazi-916 projects/demo/edit
cp projects/demo/tl.json projects/demo/edit/src/tl.json
python3 skills/digital-human-koubo/scripts/check_edit.py projects/demo/edit/src
cd projects/demo/edit && npx remotion render src/index.ts Main ../main_body.mp4 && cd ../../..
python3 skills/digital-human-koubo/scripts/make_cover.py \
  --frame projects/demo/check/s01_f2.jpg \
  --title "AI Video Pipeline" \
  --sub "Renoise + Remotion 4" \
  --out projects/demo/covers
python3 skills/digital-human-koubo/scripts/cover_first.py \
  projects/demo/covers/cover_9x16.jpg \
  projects/demo/main_body.mp4 \
  projects/demo/final_9x16.mp4

06Likeness rights, third-party licenses, and OS boundaries

Limitations and caveats

LimitationTechnical boundary and compliance requirement
Remotion commercial license thresholdEvery editing template depends on Remotion; per THIRD_PARTY.md, Remotion is free for individuals and companies of up to 3 people, while companies above 3 people require a Remotion company license
Animated emoji and map attributionThe 59 Google Noto animated emoji in templates/*/public/emoji/*.json use CC BY 4.0 and require crediting Animated emoji: Google Noto; world-atlas map data comes from Natural Earth and must be checked against local border regulations before formal publication
Apple Vision matting requires macOSscripts/cutout_video.py and scripts/personmask.swift rely on the macOS Apple Vision framework and cannot compile or run TextBehind person segmentation on Windows or Linux
Renoise 7-day material expirationUploaded Renoise reference assets expire after 7 days; when reference_material in persona.json expires or accounts change, re-run upload_files on the local reference image to obtain a fresh ID
Voice cloning consent and AI disclosureXiaomi MiMo voice cloning must only be used on voices you own or have explicit permission to clone (API key stored in ~/.config/ai-teardown/mimo.env); published videos should retain the AI presenter disclosure defined in persona.json

07Faceless whiteboard mode and official references

Next steps

  • To produce faceless 8–15 minute finance explainers driven entirely by hand-drawn whiteboard charts and number flippers, use the companion skills/finance-explainer skill to compile plan.json into the 13 scene types in templates/lin-169 with lin_sfx.py synthesized sound effects.

  • Before configuring the pack array, inspect the component gallery under skills/digital-human-koubo/assets/packaging/ or run npx remotion render src/index.ts Gallery inside templates/packaging to render the full 48-component showcase video.

  • To expand full-body or half-body pose prompts for digital-human reference portraits, combine this workflow with the AI Pose Gallery ZH Guide.

Source pinned at f661a6c047bb74ea306ef58aeb1d461a52c01871. Official reference files:

References