Scope
Openverse searches existing openly licensed assets, and MMAudio generates sound from video or text. earshot helps agents analyze audio, create short effects, and check sound against picture timing. Arrange the tracks in editing software, then listen to assess the result.
| Production need | First tool to try | Material for the next step |
|---|---|---|
| Recorded ambience and recognizable object sounds | Openverse | Source file, creator, license, and original page |
| Candidate soundscapes for silent short shots | MMAudio | FLAC audio and an MP4 with generated audio |
| Effect selection, loudness, and sound-to-picture timing checks | earshot | Measurements, candidate rankings, and timeline reports |
| Precise transitions, clicks, and notification sounds | earshot procedural effects | WAV effects and timing data |
Openly licensed assets
Rank 01/01
WordPress/openverseOpenverse
Search openly licensed images and audio
Audio search supports categories such as sound_effect and music, plus license filters. The platform aggregates third-party metadata. Check the license on the original page before use.
- Asset search
- Sound effects
- License checks
Select Audio on the home page and search for a sound and material, such as rain, footsteps on gravel, or a wooden door. After previewing a result, open its original hosting page to check the downloadable file, creator, and applicable license. Save the license URL and download date.
The official audio API endpoint, https://api.openverse.org/v1/audio/, supports q, category, license, and license_type. This example follows the official interface specification. The request returned 504 during research, so no asset results were available to verify.
curl --fail --max-time 30 --get 'https://api.openverse.org/v1/audio/' \
--data-urlencode 'q=rain' \
--data-urlencode 'category=sound_effect' \
--data-urlencode 'license=cc0' \
--data-urlencode 'page_size=3'Responses can include fields such as foreign_landing_url, creator, license, license_url, and attribution. Search filters cannot replace checking the original source. The platform terms also state that Openverse does not verify the licensing status of assets.
Video-conditioned sound generation
Rank 01/01
hkchengrex/MMAudioMMAudio
Generate synchronized sound from picture and text
Supports video with text or text alone. The default large_44k_v2 model outputs FLAC and MP4. Official model weights use CC BY-NC 4.0 and permit non-commercial use only.
- Video-to-audio
- Research model
- Non-commercial weights
The official installation has been tested only on Ubuntu and requires Python 3.9+, PyTorch 2.5.1+, and matching torchvision and torchaudio packages. Choose a PyTorch package for your GPU and driver first. The CUDA 11.8 installation command below is the official example.
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 --upgrade
git clone https://github.com/hkchengrex/MMAudio.git
cd MMAudio
pip install -e .Save a short shot as input.mp4, then run:
python demo.py --duration=8 --video=input.mp4 \
--prompt "Footsteps on gravel, light wind" \
--negative_prompt "speech, music" \
--seed 42 --output ./outputThe first run downloads the required models, and the default output and training duration is 8 seconds. The authors report about 6 GB of GPU memory for half-precision inference, although actual requirements vary by environment. The script saves the audio and composited video to ./output; omit --video for text-to-audio generation.
Even with --negative_prompt, the model can generate unintelligible speech, unwanted music, or sounds that do not match unfamiliar objects. Large deviations from the training duration can also reduce quality. Start with one short shot and listen to each section against the events on screen.
Agent audio analysis and short effects
Rank 01/01
Leonxlnx/earshotearshot
Give agents audio measurements and timing evidence
Provides a CLI, Python API, and MCP. It can describe, compare, and locate audio events, analyze video soundtracks, and synthesize short effects procedurally. kit render outputs effects and timing data; a complete mix still needs separate arrangement.
- CPU
- Audio analysis
- Procedural effects
Requirements are Python 3.10+, with ffmpeg and ffprobe on PATH. The official workflow runs on CPU without a GPU. Use a separate environment from MMAudio.
git clone https://github.com/Leonxlnx/earshot
cd earshot
python -m venv .venv
.venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv/bin/python -m pip install -e ".[mcp,llm,dev]"
.venv/bin/python -m earshot doctor --loadThese paths use Linux/macOS syntax. The analysis models download about 1.1 GB on first use, but measure, sfx, and kit do not load them. The core local workflow needs no paid API key; optional remote Gemini critique requires a key and explicit consent to upload.
| Command | Purpose | Interpretation limit |
|---|---|---|
measure | Loudness, peaks, and other measurements | Numbers cannot judge emotion or aesthetics |
match, rank, find | Compare candidates or locate recorded passages | Event labels are less reliable in mixes |
timeline, sheet | Timeline and visual checks | Check against the actual events on screen |
critique | List corrections based on a reference video style | Built-in scoring rules do not suit every video |
sfx, kit | Synthesize short effects and export effect kits with timing data | Does not automatically mix an entire video |
Place candidate.wav, film.mp4, and soundtrack.wav in the earshot directory, then run these checks. These filenames stand for assets you prepare. No audio processing was performed for this guide.
.venv/bin/python -m earshot measure candidate.wav
.venv/bin/python -m earshot match candidate.wav "a single crisp trackpad click" --alts "a gunshot" "a balloon burst"
.venv/bin/python -m earshot timeline soundtrack.wav --video film.mp4
.venv/bin/python -m earshot sheet soundtrack.wav -o sheet.png --video film.mp4
.venv/bin/python -m earshot critique soundtrack.wav --video film.mp4Start with a built-in procedural effect for a transition sound:
.venv/bin/python -m earshot sfx render whoosh_fast/crisp -o whoosh.wav --dur 0.8 --apex 0.32
.venv/bin/python -m earshot sfx check whoosh.wav
.venv/bin/python -m earshot kit init --preset launch -o audio/kit.json
.venv/bin/python -m earshot kit render audio/kit.json --out audio/kitkit render outputs effect files and kit.lock.json, including measured onset, peak, and end positions. Place these files on the editing timeline. For a complete scripted workflow, read the project's real-sound workflow.
Short-film sound design example
The example is an 8-second AI-generated shot: a person walks along a gravel path, the picture cuts to a product close-up, and a title appears. The events and deliverables below are a production example, not completed test results.
Picture and event list
- Lock shot duration
- Mark footsteps and transitions
- Separate dialogue and music
Assets and candidate generation
- Openverse search and license records
- MMAudio non-commercial candidate soundscapes
- earshot short effects
Mixing and checks
- Arrange tracks in editing software
- earshot measurements and timing checks
- Listen and export
Select candidate sounds before mixing. You do not need to use every tool.
Lock the picture edit and record when feet touch the ground, transitions occur, and the title appears. Preserve the original dialogue and music tracks.
Search Openverse for footsteps on gravel and ambience. Record each source, license, and download date. Select passages that match the pace of the picture.
For a non-commercial test, use MMAudio to generate candidate sound for the same shot. Fix the prompt and seed, check event synchronization, then compare a result from another seed.
Use earshot to synthesize a title cue or transition sound. Match its onset or peak to the picture. Put the sounds you need on separate tracks in the editor.
Export
soundtrack.wavand run earshot timeline and measurement checks. Listen on headphones and ordinary speakers. Remove stray speech, distortion, cut-off tails, and overpowering transitions.Save the mix, asset list, generation parameters, and listening decisions. After compositing the video, watch the complete export and check that original tracks were not layered twice.
Acceptance checks and common failures
| Symptom | Check | Correction |
|---|---|---|
| Footsteps occur after contact with the ground | Compare picture timing with audio onset | Move the clip or replace the candidate |
| Unexplained speech or background music | Listen to generated results on separate tracks | Regenerate or use verified recordings |
| An effect scores well but sounds harsh | Compare spectrum, peaks, and loudness-matched playback | Lower gain, adjust frequency bands, or replace the effect |
| Every transition is too strong | Play the complete video continuously | Reduce repeated effects and adjust track levels |
| License records are still missing after mixing | Compare the asset list with original pages | Complete the evidence or replace the asset |
- Not completed:
The sound for each main event has a position on the timeline, without obvious delay or repetition.
- Not completed:
There is no unwanted speech, popping, distortion, or cut-off tail.
- Not completed:
Dialogue is intelligible. Music and effects do not obscure the main information.
- Not completed:
Loudness and peaks meet the delivery platform's specifications. You have listened to the final encoded file.
- Not completed:
You have recorded usage rights for every recording, model, and derived effect.
Tool selection
Commercial delivery
Licensed assets and manual mixing
Find candidate assets with Openverse and verify each item's rights, then use earshot for selection and checks. MMAudio's non-commercial weights do not cover ordinary commercial client production.
Non-commercial research
Compare MMAudio candidates with recordings
Compare generated soundscapes and licensed recordings for the same short shot. Record your choice using event timing, measurements, and listening results.
Motion design
earshot short effects and timing data
Create cues and transitions with sfx or kit, then arrange them on the timeline. Check peak positions against the actual picture instead of relying only on model scores.