How Modal works

Modal runs Python on its own GPUs, called from your own Python program. We moved Synthesizing's MusicGen and language model workers onto it in August 2024. Its Python interface is clever: one file describes the machine the code needs and is also the code that runs on it. That same cleverness caught us out a few times, and underneath it all is a balance every GPU service has to strike, between paying to keep a GPU warm and making someone wait for one to start.

One file, two places

Everything about a Modal app is written in Python, in the same file as the code it runs: the image the code needs, the GPU, the volumes and how long containers stay up. This was our first test, which made one chunk of music with MusicGen on an L4:

import modal

baseImage = (
    modal.Image.debian_slim(python_version="3.9")
    .pip_install_from_requirements("requirements.txt", "/workspace/python_modules")
    .pip_install("audiocraft")
)
volume = modal.Volume.from_name("workspace", create_if_missing=False)
MODEL_DIR = "/workspace"
app = modal.App("example-synthesizing")

@app.function(
  image=baseImage,
  volumes={MODEL_DIR: volume},
  gpu="L4",
)
def main():
  from audiocraft.models import MusicGen
  ...
  model = MusicGen.get_pretrained("/workspace/musicgen-stereo-large", device="cuda")
  model.set_generation_params(True, 250, 1, 1.0, 10, 3.0, False)
  generatedSamples = model.generate(["Delicate symphony of the night"], progress=True).cpu()

The image

modal.Image is built up one step at a time, starting from a slim Debian with Python 3.9 and adding packages. Nothing is built on your own machine. The steps are sent to Modal, which builds the image on its side and keeps each layer, so changing the last step only rebuilds that step.

The decorators

@app.function and @app.cls record the function or class on the app, with the settings it needs, and hand back an object that stands in for it. Calling .remote() on that object sends the call to Modal and waits for the result, and .local() runs it where you are.

Inside the container

When a container starts, Modal imports the same file inside it. The decorators run again, and this time Modal finds the function or class by its name and runs it. The file that describes the deployment is also the code that's deployed. The volume is mounted at /workspace, so the model loads from Modal's own storage next to the GPU instead of being downloaded.

Deploying and calling by name

modal run runs a file once. modal deploy keeps it on Modal under the app's name, ready to be called from any program with the account's credentials. Synthesizing's dispatcher ran modal deploy while its own Docker image was being built, and then looked the class up by name when it started:

musicgen_worker_reference = modal.Cls.lookup(MUSICGEN_APP_NAME, MUSICGEN_CLASS_NAME)
musicgen_worker = musicgen_worker_reference()
...
result = self.musicgen_worker.compute_chunk.remote(
    worker_job_json,
    instance_id
)

The arguments and the result are pickled and sent through Modal, so they have to be things Python can pickle. We sent the job as a dictionary, with the previous chunk inside it as a base64 WAV file, and got back a dictionary with the new chunk and its waveform picture, both in base64.

Where it caught us out

Imports run on both sides

Because the file runs in both places, everything at the top of it runs in both places too. The first test imported MusicGen inside the function, so only the container needed it. The production worker loaded the model in the class's constructor, and imported MusicGen at the top of the file:

app = modal.App(MODAL_APP_NAME)
sys.path.append("/workspace")
from audiocraft.models import MusicGen

@app.cls(
    image=baseImage,
    volumes={MODEL_DIR: volume},
    gpu="A10G",
    container_idle_timeout=10
)
class SynthesizingMusicgenWorker:
    def __init__(self):
        self.model = MusicGen.get_pretrained(MUSICGEN_WEIGHTS_PATH, device="cuda")

That import also runs when modal deploy reads the file on our side, so the dispatcher's own image had to install AudioCraft and Transformers, and PyTorch with them, just so the file could be read. None of them ever ran a model there:

RUN pip install modal requests audiocraft transformers together
...
WORKDIR /app/synthesizing/Modal
RUN modal deploy SynthesizingMusicgenWorker.py
RUN modal deploy SynthesizingSuperpromptWorker.py

Modal has a way round this, with image.imports():, which only runs the imports inside it when the file is loaded in the container. We didn't use it.

Class arguments make separate containers

The first production version passed each chunk's settings to the class:

@app.cls(
    image=baseImage,
    volumes={MODEL_DIR: volume},
    gpu="A100",
)
class SynthesizingWorker:
    def __init__(
            self,
            top_k: int,
            top_p: int,
            temperature: float,
            duration: int,
            cfg_coef: float,
            seed: int,
            prompt: str,
            instance_id: str,
            worker_id: int,
            worker_code: str):
        self.model = MusicGen.get_pretrained("/workspace/musicgen-stereo-large", device="cuda")
        ...

Modal treats each different set of class arguments as a different object, with containers of its own. Every chunk has its own prompt and job id, so every chunk asked for a new container, which loaded the model from the start, and a warm container that already had the model was never used again. Moving the settings into the method's arguments meant one set of containers for every chunk, and the A100 gave way to the cheaper A10G.

Progress goes around Modal

.remote() waits and then returns the result, and nothing comes back while it runs. To show progress on the site, the container sends it there itself, every 10 per cent, with the job's id and code:

if progress % 10 == 0 and progress != self.last_progress:
    payload = {
        "typeName": WORKER_TYPE,
        "id": worker_id,
        "code": worker_code,
        "progress": progress,
        "instanceId":  instance_id
    }
    requests.post(SERVER_URL+"/api/worker/progress", json=payload)
    self.last_progress = progress

So the containers need to reach the site, and the site has to accept progress for a job from a machine that isn't the one that took the job.

Keeping containers warm

A new container has to start and then load the model before it can make anything. For MusicGen that means reading the stereo large model's weights from the volume and onto the GPU in the constructor. Once that's done, the container keeps the model for every chunk it makes, which on the L4 took about 52 seconds each. When the calls stop, the container waits for its idle timeout and then stops:

Modal bills by the second for the whole time a container is up, starting, loading, working or idle. An A10G costs about $1.10 an hour, so one kept warm around the clock costs about $26 a day, or about $800 a month, before it makes anything. These are the timeouts we used:

The dispatcher asks the site for work every 2 seconds when it has none, so while chunks are waiting, the next one reaches Modal well within 10 seconds and finds the container still warm. After a pause the container has gone, and the next chunk waits for a new one to start and load the model.

That's the balance, and it changes with scale. With few users, most requests come after a pause, so either you pay to keep a GPU warm through the quiet times, or the first person after each pause waits for a container to start. A short timeout keeps the bill down and puts the wait on the user. With many users, containers are rarely idle, the idle time after each burst is a small part of the bill, and only a sudden rise in requests meets a cold start. Modal's keep_warm setting keeps a minimum number of containers running for exactly that, and the larger the service, the more of those warm containers are paid for by work rather than by waiting.

What has changed since

Modal's interface has changed since we used it, though the ideas are the same: