Synthesizing

Synthesizing is a website for making long pieces of music with AI, with fine-grained control along the timeline. A track is built from ten-second chunks, each with its own prompt and settings, and every chunk carries on from the one before it, so the music can change as it goes and still sound like one piece.

It's built on MusicGen, the music model Meta released as part of AudioCraft. As well as being good, MusicGen was trained on licensed music, so we could avoid the copyright questions hanging over other music models.

The way of keeping MusicGen going came from Endless music with MusicGen. Synthesizing turned it into a site where anyone could build a track, and that needed a queue of chunks waiting to be generated, GPU workers in Python to take them, and a few cloud providers along the way. Piotr Nierobisz and I built it together.

The front end

The track editor

A track is a list of chunks, one row for every ten seconds. Each row shows where it starts, its waveform, the prompt it was made from, its chaos and seed, and buttons to play it or delete it. Play at the top plays the whole track, one chunk after another.

Only the first and the last chunk can be deleted. Every chunk is made from the one before it, so taking one out of the middle would leave a join that was never generated.

Adding a chunk

Under the list there's a box for the prompt and an Add Chunk button. Each chunk costs one credit, taken when it's added. A prompt needs at least six words, because MusicGen does much better with a description than with a couple of words (more on that in Superprompting).

Rather than MusicGen's sampling settings, the form has one slider, Chaos, from 0 to 100. The server turns it into a temperature and a top-k:

if ($chaos !== null) {
    if ( $chaos < 50 ) {
        $proportion = $chaos / 50;  //  How far through the 0..50 range are we
        $chunkInfo->temperature = $proportion;  //  0 to 1 as chaos goes from 0 to 50
        $chunkInfo->topK = floor( 1 + 99 * $proportion );  //  1 to 100
        $chunkInfo->topP = 0;  //  0 for now
    }
    else {
        $proportion = ($chaos-50) / 50;  //  How far through the 50..100 range are we
        $chunkInfo->temperature = 1 + 9 * $proportion;
        $chunkInfo->topK = 100 + 150 * $proportion;
        $chunkInfo->topP = 0;
    }
}

Top-k is how many of the most likely next tokens MusicGen chooses between, and the temperature is how evenly it chooses. The bottom half of the slider goes from always taking the most likely token up to MusicGen's normal temperature, and the top half goes from there to ten times as random, with the choice widening to 250 tokens:

In the screenshot, the first chunk was made at 75 and the second at 8.

Track starters

A new track can also be started from a preset. Each one adds four chunks at once with four prompts written to follow on from each other, so one instrument develops over 40 seconds:

SynthesizingProjectUi::PresetButton(
    "Piano 40 seconds",
    [
        "Gentle piano arpeggios flow like water, with soft sustained chords creating a peaceful morning atmosphere.",
        "Solo piano performance with flowing melodic lines and light staccato notes, medium tempo.",
        "Dramatic solo piano piece with thunderous bass notes and rapid ascending scales.",
        "Virtuosic piano composition featuring sweeping arpeggios across all octaves, passionate dynamics."
    ]
);

There are 18 of them, from piano, guitar and drums to the oboe, French horn, accordion and banjo.

Waiting for a chunk

Adding a chunk doesn't wait for it to be made. The browser polls the status of the first chunk that isn't ready: every 10 seconds while it's queued, showing its place in the queue, and every 3 seconds while it's being generated, showing how far through it is.

const status = Synthesizing.Data.ChunkStatus(json.success)
if ("notStarted" in status) {
    ...
}
else if ("ready" in status) {
    onReady(firstNotReadyChunkIndex)
}
else if ("queuePosition" in status) {
    onQueuePosition(firstNotReadyChunkIndex, status.queuePosition)
    timeoutId = setTimeout(fetchChunkStatus, 10000); // Wait for 10 seconds
}
else if ("generatedPercent" in status) {
    onGeneratedPercent(firstNotReadyChunkIndex, status.generatedPercent)
    timeoutId = setTimeout(fetchChunkStatus, 3000); // Wait for 3 seconds
}

When it's ready, the browser fetches the chunk's audio and waveform picture, and starts checking on the next one. The audio is a 32kHz stereo WAV, which the browser copies into a Web Audio buffer for playing.

Stitching the chunks together

Continuing from the previous chunk

The first chunk of a track is generated from its prompt alone. Every chunk after that is a continuation: the job carries the whole of the previous chunk, and MusicGen carries on from the end of it under the new chunk's prompt. How that works inside MusicGen is in Endless music with MusicGen: the audio is turned into tokens, and the model predicts what comes after them.

The previous chunk is the most recent finished one before it in the same project:

SELECT * FROM synthesizing_project_chunk
WHERE SynthesizingProjectChunkID < :chunkId
AND SynthesizingProjectChunkAddedTimestamp <= :timestamp
AND SynthesizingProjectChunkProjectID = :projectId
AND SynthesizingProjectChunkStatus = :ready
ORDER BY SynthesizingProjectChunkID DESC LIMIT 1

What generate_continuation returns starts with the prompt audio, so the worker asks for 10 seconds plus the length of the prompt, and slices the prompt off the front:

continuationSampleCount = data_tensor.shape[-1]
audioPromptDurationSeconds = continuationSampleCount / fileSampleRate
totalDurationSeconds = DURATION_SECONDS + audioPromptDurationSeconds

self.model.set_generation_params(
    True,
    top_k,
    top_p,
    temperature,
    totalDurationSeconds,
    cfg_coef,
    False
)

generatedSamples = self.model.generate_continuation(
    prompt=data_tensor,
    descriptions=[prompt],
    prompt_sample_rate=fileSampleRate,
    progress=True,
    progress_callback=progress_callback,
).cpu()[:, :, continuationSampleCount:]

Because the prompt is the music that came before, a new prompt changes the direction rather than cutting to something else: the new chunk starts from where the music was and moves towards the new description.

It also means a track can only be made one chunk at a time, since each chunk needs the one before it to be finished. That shapes the queue, below.

Avoiding clicks

Cutting off the prompt leaves a join between the last sample of one chunk and the first sample of the next. The model was continuing from that point, but the two samples don't quite meet, and a sudden step in the waveform is heard as a click. So the worker eases the first six samples of the new chunk from the last sample of the previous one:

def smooth_transition(chunk, last_sample):
    num_samples = min(6, chunk.shape[1])  # Use up to 6 samples, or fewer if chunk is smaller
    fade_in = np.linspace(0, 1, num_samples)
    fade_out = np.linspace(1, 0, num_samples)

    for channel in range(chunk.shape[0]):
        chunk[channel, :num_samples] = (
            last_sample[channel] * fade_out +
            chunk[channel, :num_samples] * fade_in
        )
    return chunk

Six samples at 32kHz is less than 0.2 milliseconds, far too short to hear as a fade, but enough to take the step out of the waveform.

Superprompting

MusicGen was trained on music with long, detailed descriptions, and it does much better with a sentence or two about the instruments, the mood and the tempo than with a few words. But a few words is what most people type. So before MusicGen sees it, a language model rewrites the prompt into the kind of description MusicGen was trained on. That's superprompting.

It does a second job at the same time. People often describe music by naming a band or a musician, and the language model replaces any name with a description of that artist's music instead.

The instruction

The instruction to the language model is a template kept in the database and edited in the admin panel, so it could be improved without a release. When a chunk is added, the user's prompt goes into the template, and the result is saved as a superprompt job attached to the chunk:

//  Superprompting
$superpromptId = null;
try {
    if ($chunkInfo->superprompt) {
        $defaultPrompt = SynthesizingAiDatabase::GetDefaultInstruction();
        $instruction = SynthesizingAiSuperprompt::Replace($defaultPrompt->instruction, $chunkInfo->prompt);
        $superpromptId = SynthesizingAiSuperpromptJobDatabase::Add($defaultPrompt->model, $instruction);
    }

    $chunkId = SynthesizingProjectChunkDatabase::Add($memberId, $projectId, $chunkInfo, $superpromptId);

Replace puts the prompt where the template says {{USER_PROMPT}}. The template holds both messages for the language model, the system message and the user's, separated by |||, and the worker splits them apart. The rewritten prompt is saved with the chunk, along with the provider and model that wrote it.

Running the language model

At first we ran the language model ourselves, with Ollama in a Docker container, the way AI models in Docker containers runs models, using Llama 3 and Mixtral. Then it moved to Modal, next to MusicGen, as Llama 3.1 8B Instruct on an A10G. In the end the main one was Llama 3.1 70B Instruct Turbo on Together.ai, a much bigger model that we didn't have to host, with the Modal one as the fallback:

try:
    print("[SynthesizingModalWorker.onJob] Generating superprompt with together")
    [systemMsg, instructionMsg] = prompt.split("|||")
    messages = [
        {"role": "system", "content": systemMsg},
        {"role": "user", "content": instructionMsg},
    ]
    response = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
        messages=messages,
        temperature=0.7,
        top_p=0.7,
        top_k=50,
        repetition_penalty=1,
        stop=["<|eot_id|>","<|eom_id|>"],
    )
    superpromptComputed = response.choices[0].message.content
    provider = "together"
    model = "Llama-3.1-70B-Instruct-Turbo"
except Exception as e:
    print("[SynthesizingModalWorker.onJob] Error generating superprompt, switching to modal", e)
    superpromptComputed = self.superprompt_worker.compute_superprompt.remote(
        prompt,
        model
    )
    provider = "modal"
    model = "Llama-3.1-8B-Instruct-Turbo"

The rewritten prompt replaces the user's in the job, and MusicGen is given that.

The architecture

The site is PHP and MySQL, and the GPUs are somewhere else. The PHP side never calls them: Python workers ask the site for work. That keeps the site simple, and a worker can run anywhere that can reach the site, whether that's a machine on the desk, a rented GPU server or a small program handing the work on to Modal.

The database

A track is a project, and its chunks are rows in synthesizing_project_chunk:

The other tables are the projects, the superprompt instructions and jobs, and two from the shared Worker module: worker, one row for each machine that has asked for work, and worker_request, one row for each job handed out. Every change to a project sends the version number the browser has, and the server refuses it if the project has moved on, so two tabs open on the same track can't get it into a mess.

The queue

There's no queue table. A chunk that's not started and has no worker is in the queue, and its QueuedTimestamp is its place in it.

The chunks of one track have to be made in order, so the next chunk to work on is the first waiting chunk of a track that has nothing generating, taken from whichever track has been waiting longest:

WITH MinChunkIDs AS (
    SELECT SynthesizingProjectID, MIN(SynthesizingProjectChunkID) as chunk_id
    FROM synthesizing_project
             JOIN synthesizing_project_chunk as spc_outer ON SynthesizingProjectChunkProjectID = SynthesizingProjectID
    WHERE SynthesizingProjectChunkStatus = :notStarted
      AND NOT EXISTS (
        SELECT 1
        FROM synthesizing_project_chunk as spc_inner
        WHERE spc_inner.SynthesizingProjectChunkProjectID = spc_outer.SynthesizingProjectChunkProjectID
          AND spc_inner.SynthesizingProjectChunkStatus = :generating
    )
    GROUP BY SynthesizingProjectID
)
SELECT MinChunkIDs.chunk_id
FROM MinChunkIDs
         JOIN synthesizing_project_chunk as spc_outer ON MinChunkIDs.SynthesizingProjectID = spc_outer.SynthesizingProjectChunkProjectID
    AND MinChunkIDs.chunk_id = spc_outer.SynthesizingProjectChunkID
ORDER BY spc_outer.SynthesizingProjectChunkQueuedTimestamp ASC

So really it's a queue of tracks. Two people's tracks are generated side by side on two workers, but a second worker can't speed up one track. The place in the queue shown in the track editor comes from the same ranking.

Choosing a chunk and claiming it are two separate queries, so two workers asking at the same moment can choose the same chunk. The claim only works if the chunk still has no worker, and if nothing was updated, another worker got there first and this one chooses again:

while (true) {

    //  Get the next queued chunk ID if there is one
    $chunkId = SynthesizingProjectChunkDatabase::GetNextQueuedChunkID();

    //  Mark chunk with this worker if no worker is assigned
    //  Another worker might have just grabbed this same chunk ID, and
    //  this ensures that only one of the workers gets the chunk
    $row = (object)[
        'SynthesizingProjectChunkWorkerID' => $workerId,
        'SynthesizingProjectChunkStatus' => SynthesizingProjectChunkStatusEnum::GENERATING,
    ];
    $where = (object)[
        'SynthesizingProjectChunkID' => $chunkId,
        'SynthesizingProjectChunkWorkerID' => 0,
        'SynthesizingProjectChunkStatus' => SynthesizingProjectChunkStatusEnum::NOT_STARTED,
    ];
    $rowsAffected = Database::UpdateFields('synthesizing_project_chunk', $where, $row);

    //  If the above statement didn't update anything that means that the
    //  worker ID was no longer 0 on that chunk so another worker managed to
    //  grab it between SQL queries
    if ($rowsAffected > 0) {
        return $chunkId;
    }
}

When GetNextQueuedChunkID finds nothing, it throws, and the worker is told there's no job.

This works well for one table. We also made a generic version of it for other queues, but that needed extra columns adding to every table that took part in a queue.

The worker API

The workers come from a Worker module that other sites share. A site registers a worker type with a password, a timeout and functions for each step, and the module provides four endpoints:

The id and code identify one job handed out. The code is random, so a broken or misbehaving worker can't report on someone else's job just by sending a different id. The instance id names the machine, so we could tell the workers apart.

A worker asks for a job:

{
    "typeName": "synthesizing_chunk",
    "authPassword": "...",
    "instanceId": "4be1c09f2d7a3e51"
}

and gets one back, with the chunk's settings, the previous chunk as a base64 WAV, and the superprompt instruction if there is one:

{
    "success": {
        "id": 51237,
        "code": "9e3f0c...",
        "job": {
            "chunkInfo": {
                "topK": 175,
                "topP": 0,
                "temperature": 5.5,
                "prompt": "jazzy funky upbeat underground drum and bass with a hint of orchestral violins and some piano",
                "cfgCoef": 3,
                "seed": 0
            },
            "continuation": "UklGRiQAFABXQVZFZm10IBAAAAABAAIAAH0AAAD0AQAEABAAZGF0YQ...",
            "superprompt": {
                "prompt": null,
                "model": null
            }
        }
    }
}

While it works, it reports how far it's got:

{
    "typeName": "synthesizing_chunk",
    "id": 51237,
    "code": "9e3f0c...",
    "progress": 40,
    "instanceId": "4be1c09f2d7a3e51"
}

and when it's done, it sends the result: the WAV and the waveform picture as base64, the rewritten prompt, and which provider and model rewrote it:

{
    "typeName": "synthesizing_chunk",
    "authPassword": "...",
    "id": 51237,
    "code": "9e3f0c...",
    "result": {
        "audio": "UklGRiSAEwBXQVZFZm10IBAAAAABAAIAAH0AAAD0AQAEABAAZGF0YQ...",
        "waveform": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAAy...",
        "superprompt": "",
        "provider": "",
        "model": ""
    },
    "instanceId": "4be1c09f2d7a3e51"
}

Here's the whole conversation for one chunk, with the browser polling alongside:

Timeouts and failures

Every call to /api/worker/next first checks on the jobs already handed out. Any job that hasn't been heard from for 60 seconds is given up, and its chunk goes back to the end of the queue. So nothing else has to watch for workers that have died: the next worker to ask for work clears up after them.

A worker that was only slow finds out on its next progress report, which gets an error back because the job isn't its any more, and it drops the job. A worker that fails sends an error, and the chunk goes back in the queue straight away. There's no limit on how many times a chunk is tried.

The Python worker

The worker side is a small Python package, and WorkerService is a loop around those four calls:

def run(self):
    print("[WorkerService.run] Starting worker service")

    while True:
        try:
            print("[WorkerService.run] Fetching next job")
            workerJob = self.next()

            if workerJob:
                print("[WorkerService.run] Job found")
                self.job(workerJob)
            else:
                print("[WorkerService.run] No job found")
                time.sleep(2)

        except Exception as e:
            msg = getattr(e, 'message', repr(e))
            stack_trace = traceback.format_exc()

            print("[WorkerService.run] Could not contact server - ", msg, stack_trace)
            time.sleep(5)

It asks again every 2 seconds when there's nothing to do, and waits 5 seconds if it can't reach the site. A job is handed to the worker type's own function along with three callbacks for success, error and progress, and progress reports are limited to one every 5 seconds. The checks on the data going back and forth are a Python version of the ones in Simple typechecking goes a long way.

On a development machine the worker runs in a Docker container, built and started with the Docker module for Node.js:

const Docker = require('Docker');

console.log(`Starting synthesizing worker...`);
const serviceName = `synthesizing.worker`;
const workerPath = __dirname+`/worker`;
const dockerfileContent = [
    `FROM python:3.9`,
    `WORKDIR /app`,
    `RUN apt-get update && apt-get install -y git nano wget ffmpeg portaudio19-dev rubberband-cli`,
    `RUN pip install torch==2.0.1`,
    `WORKDIR /app/synthesizing`,
    `COPY . .`,
    `RUN pip install -r requirements.txt`,
    ...
    `RUN python -c "from audiocraft.models import MusicGen; MusicGen.get_pretrained('large', device='cpu')"`,
    `CMD ["python", "start.py"]`
].join(`\n`);

const containerId = Docker.Service.Start( serviceName, dockerfileContent, workerPath, [])

The providers

Generating a chunk needs a GPU with MusicGen in its memory. We tried several ways of getting one, in this order.

Our own servers

The first versions ran MusicGen in Docker on our own machines, the way AI models in Docker containers describes. At first a Node.js server kept the queue in memory and handed each request to one of a fixed list of GPU services over WebSockets, and then came the PHP site and the Python worker described above. A fixed set of machines doesn't grow with the queue, though.

Replicate

Replicate runs a model packaged as a container, and you call it by name from its client library. We packaged the worker as a private model and called it with Replicate's Node client from a Node version of the worker. It got as far as generating, but we moved on before the results were connected back to the site.

Runpod Serverless

Runpod's serverless endpoints call a function in your container for each request, registered with runpod.serverless.start({"handler": predict}). Our handler loaded MusicGen inside each request, so every chunk paid for loading the model before it could start. We moved on to Runpod's pods within a week.

Runpod pods

A Runpod pod is a GPU server running your container. This is how Synthesizing first ran in production. Each pod runs the Python worker, which asks the site for work like any other, and the site starts and stops pods through a small Node service wrapped around runpodctl, Runpod's command line tool. We built runpodctl from Piotr Nierobisz's fork of it, which could attach a network volume to a new pod.

Not every GPU is free all the time, so the site asks for the GPUs it can use in order of preference, and takes the first one it can get:

const CLOUD_API_GPU_TYPES = [
    'NVIDIA GeForce RTX 4090',
    'NVIDIA RTX 4000 Ada Generation',
    'NVIDIA RTX A4000',
    'NVIDIA RTX A4500',
    'NVIDIA RTX A5000',
    'NVIDIA RTX A40',
    'NVIDIA RTX A6000'
];

foreach ($gpuTypes as $gpuType) {
    foreach ($networkVolumeIds as $networkVolumeId) {
        try {
            return CloudGpuRunpodDatabase::Add(
                $args,
                $communityCloud,
                $containerDiskSize,
                ...
                $networkVolumeId
            );
        }
        catch(Throwable $ex) {
            Log::Exception("Cloud", "Error adding runpod server", $ex);
            echo "Error: {$ex->getMessage()}";
        }
    }
}

throw new CloudGpuRunpodAddError("Error adding runpod server");

Prices changed, so we kept an eye on them. These were the prices in the admin panel's list of GPUs:

Starting a GPU server

The hard part with GPU servers is how long they take to start. Before a new server can generate anything, it has to boot, get the software image and the GPU drivers onto it, get the model onto the machine, and load the model into the GPU's memory. Only then can it start serving requests.

We worked on making that as fast as we could. Apart from keeping servers running, which is a balance between cost and speed, the biggest improvement was keeping the model as close as possible to the servers:

On a pod the network volume is mounted at /workspace, so the worker's start-up script installs its packages from there without going to the internet, and loads the model from there too:

"pip install --verbose --no-index --find-links /workspace/python_modules -r requirements.txt \n" \
"python start.py --gpu True" > start.sh

self.model =  MusicGen.get_pretrained('/workspace/musicgen-stereo-large', device=device)

A network volume lives in one datacentre, and a pod can only use it if it's in the same one. Not every datacentre has every GPU, so we kept a copy of the model in two datacentres, and the loop above tries each GPU in both before giving up.

The scaler works with an average of 280 seconds from asking for a pod to the worker serving, against 45 seconds to generate a chunk. A new pod takes as long to arrive as six chunks take to make, which is why the scaler, below, has to look at the whole queue.

Modal

Modal runs Python from your own program on its GPUs. You write a class in your own code and decorate it with the image it needs, the GPU and any volumes. modal deploy sends it to Modal, which builds the image on its side. After that, calling a method with .remote() runs it in a container on one of their GPUs, starting one if none is free.

baseImage = (
    modal.Image
    .debian_slim(python_version="3.9")
    .pip_install_from_requirements("requirements-musicgen.txt", "/workspace/python_modules")
)
volume = modal.Volume.from_name("workspace", create_if_missing=False)
MODEL_DIR = "/workspace"
...
app = modal.App(MODAL_APP_NAME)
sys.path.append("/workspace")
from audiocraft.models import MusicGen

@app.cls(
    image=baseImage,
    volumes={MODEL_DIR: volume},
    gpu="A10G",
    container_idle_timeout=10
)
class SynthesizingMusicgenWorker:
    def __init__(self):
        self.model = MusicGen.get_pretrained(MUSICGEN_WEIGHTS_PATH, device="cuda")
        self.last_progress = 0

    @modal.method()
    def compute_chunk(
        self,
        worker_job_json: dict,
        instance_id: str
    ) -> dict:

The model loads once when a container starts, and stays in the GPU's memory for every chunk that container makes. The weights are on a Modal volume laid out the same way as the Runpod one, so the model is again next to the GPUs. container_idle_timeout=10 stops a container 10 seconds after its last chunk, which keeps the cost down, at the price of starting a container again after any pause.

The job's settings are passed to the method, not the class. Our first version passed them to the class instead, and since Modal keeps a separate set of containers for every different set of class arguments, a warm container with the model loaded was almost never used again.

The Python worker that asks the site for work is the same WorkerService, now running next to the site. It looks the deployed classes up by name, and hands each chunk to Modal:

musicgen_worker_reference = modal.Cls.lookup(MUSICGEN_APP_NAME, MUSICGEN_CLASS_NAME)
musicgen_worker = musicgen_worker_reference()
...
result = self.musicgen_worker.compute_chunk.remote(
    worker_job_json,
    instance_id
)

The code reaches Modal when the worker's own image is built:

FROM python:3.9
RUN apt-get update && apt-get install -y nano
WORKDIR /app/synthesizing
COPY . .
RUN pip install modal requests audiocraft transformers together
...
WORKDIR /app/synthesizing/Modal
RUN modal deploy SynthesizingMusicgenWorker.py
RUN modal deploy SynthesizingSuperpromptWorker.py
WORKDIR /app/synthesizing
CMD ["python", "start.py"]

The progress reports come straight from the Modal container to the site, every 10 per cent, using the job's id and code.

Our first test was on an L4, with the model loaded from the volume, and three runs at the default settings took:

We tried an A100 and settled on the A10G.

Scaling

With Modal, scaling is Modal's job: containers start when calls arrive and stop when they're idle. With Runpod it was ours. Once a minute a cron job compares the pods that are running with the number of tracks that have chunks waiting, and aims for one pod for each, since one track can only use one at a time:

static function ScalingCron(): void {
    $runpodInstances = SynthesizingDatabase::CountRunningWorkers();
    $projectQueueSize = SynthesizingProjectDatabase::CountQueue();
    $chunkQueueSize = SynthesizingProjectChunkDatabase::CountQueue();
    $futureProjectCount = SynthesizingProjectDatabase::CountQueueWhereChunkSize(0);
    $serversStarted = 0;
    $serversStopped = 0;

    if ($runpodInstances > $projectQueueSize) {
        $serversStopped = SynthesizingDatabase::ScaleDown($runpodInstances, $projectQueueSize);
    }
    else if ($runpodInstances < $projectQueueSize) {
        $serversStarted = SynthesizingDatabase::ScaleUp($runpodInstances, $futureProjectCount);
    }
    else {
        echo "No scaling needed";
    }
    ...
}

Scaling up starts at most 4 pods a minute. Scaling down removes half of the spare pods, at most 10 at a time, and only pods with no job. A pod is only stopped in the last 10 minutes of each hour it has been running, so an hour that's been paid for isn't given back early:

$secondsSinceHour = ($now - $firstSeenTimestamp) % 3600;
$justShyOfHour = 50*60;

$numberOfJobsAssigned = SynthesizingDatabase::CountAssignedJobs($instance->name);
if ($numberOfJobsAssigned > 0) {
    echo "Skipping server $instanceRef as it has $numberOfJobsAssigned jobs assigned";
    continue;
}

if ($secondsSinceHour >= $justShyOfHour) {
    echo "Deleting server $instanceRef";
    CloudGpuRunpodDatabase::Delete($instanceRef);
    CloudGpuRunpodDatabase::MarkDeleted($instanceRef);
    $removedCounter++;
}

As a backstop, every pod is started with SELF_DESTRUCT=1, and its worker stops itself after 300 empty answers in a row, about 10 minutes with nothing to do:

# 30 requests/minute
# If no chunks for more than 10minutes then self destruct
if self.self_destruct and self.no_data_count > 300:
    print("[WorkerService.next] Self destructing...")
    os._exit(0)

Every start and stop emails the admin, and every run logs the size of the queue and what it did. The settings:

With a minimum of 0, the first chunk after a quiet spell waits for a pod to start. The first version tried to get ahead of that, using the 280 seconds a pod takes to start and the 45 seconds a chunk takes: "By the time new instance of runpod worker is created, 7 chunks across 3 projects will be processed". It only started pods when more than three tracks each had more than seven chunks waiting, and the simpler rule above replaced it. The pods that are still starting aren't counted as running, so while they start, each minute can ask for up to 4 more.

All of this is the balance between cost and speed that Modal's idle timeout handles in one setting: keep GPUs running and pay for them while they wait, or stop them and pay the start-up time when the next chunk arrives.

Additional features

Continuing your own music

Members could start a track from their own music instead of a prompt. They drop in an audio file of up to 4 hours, and choose where to continue from, or leave it at the end. The browser takes the 10 seconds before that point, resamples it to 32kHz and turns it into a WAV, all without uploading the whole file:

async function processFile(audioBuffer, processedFileName, specifiedTime) {
    const duration = audioBuffer.duration;
    let start, end;
    if (!isNaN(specifiedTime) && specifiedTime !== 0) {
        end = Math.min(specifiedTime, duration);
        start = Math.max(0, end - 10);
    } else {
        end = duration;
        start = Math.max(0, end - 10);
    }
    const segment = extractSegment(audioBuffer, start, end);
    const resampledBuffer = await resampleBuffer(segment, 32000);
    const wavBlob = bufferToWav(resampledBuffer);
    addFileToForm(wavBlob, processedFileName);
    updateFileIndicator(processedFileName);
}

Those 10 seconds become the first chunk of the track, and the chunks after it continue from them like any others.

Advanced mode

Advanced mode swaps the chaos slider for MusicGen's own settings: the seed, temperature, top-p, top-k and the classifier-free guidance coefficient, starting from MusicGen's defaults of temperature 1.0, top-k 250 and guidance 3.0. It also lets you turn superprompting off, so you can prompt MusicGen directly with exactly the words you want.

Bulk generation

To learn what MusicGen responds to, the admin panel can make tracks in bulk:

That's how we tried out sets of different combinations of prompts, and the same prompt with many different seeds.

Seeds

Setting a seed seeds Python's, NumPy's and PyTorch's random numbers before generating:

if seed != 0:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)

We found that even with the same seed and the same prompt, MusicGen wouldn't produce exactly the same result twice, so there was a source of randomness inside the MusicGen library that was outside our control.