Sound Prompter

Sound Prompter lets you generate sound effects (optionally with music) that match your composition. It generates .wav files in realtime (e.g. ~10s generation time for a 10s clip on a recent NVIDIA GPU). Click to play the example:

Prompt: “person cutting a fruit”

Prompt: “spaceship and laser beam”

Requirements

Windows only currently

NVIDIA GPU with at least 10 to 12GB VRAM

~10GB free RAM when invoking the plugin

Windows Download

Workflow

Apply the effect to the layer you want to create the SFX for. Can be a subcomp or an adjustment layer to let multiple layers participate in the SFX creation.

Click “Add Text Input” to add a mask and link it in the “Positive Prompt” field. Then you can change the name of the mask to modify the prompt. Add another Text Input and link in “Negative Prompt” to supress certain keywords (e.g. music).

Click “Generate Audio”. On the first invocation this might take up to a minute to initialize as the AI model needs to be loaded and the video frames have to be rendered. Once this is done you can iterate on it with different prompts and settings much faster.

Once the computation is done the created audio file (.wav) is added to your project and timeline above the effect layer (in the unregistered plugin version the file will contain some gaps and beep sounds).

Parameters

Text Prompt

Textual description of the SFX.

Negative Text Prompt

Supresses those keywords during generation. Can be used to avoid the generation of background music by including the term “music”. If the generated sound sounds harsh or otherwise shrill, add “shrill”.

Seed

Modify the Seed value to get different variations.

Prompt Influence

The higher this value the stronger the text and video guidance will influence the generated sounds.

Steps

The number of iterations the model runs. Lower numbers run faster but might produce worse results.

Model

Choose between XL and XXL model. The XXL model needs to be installed manually (from the component manager) and has higher runtime requirements.

Backend & Performance

Hardware Acceleration (on Silicon Macs and CUDA machines only)

Run calculations on the GPU. This will give massive speedups compared to CPU mode.

Lower Precision

Compute with reduced precision if possible. This can save up to half of the memory and give you some speedups at the cost of sometimes slightly reduced quality.

Model Offloading

Enabling this will make sure only the AI model parts which are needed for computation are kept on the GPU. This might lower VRAM usage under some settings at the cost of moving AI models in and out of GPU memory. Options are:

No offloading: Keep all models on the GPUs VRAM.
CPU: Move unneeded models to the RAM and back if needed. This will occupy RAM.
Full Unload: Completely unload models if not needed. This saves both VRAM and RAM but might be much slower, as the models have to be loaded again for every request.

Samples (not available for all settings)

The number of ai samples to calculate. This will improve the models accuracy.

Parallel (only available if Samples > 2)

This will render all samples at the same time (faster), if disabled computation might be slower but require less VRAM.

Computation Tiles (not available for all settings)

Split the computation into several tiles. This can help if you run out of memory.