Edit 2024-05-16: This has been updated to reflect recent updates to our ComfyUI example, which you should check out for an easier introduction to running ComfyUI on Modal.

ComfyUI is a popular no-code, visual editor for building complex image generation workflows. Because of its ease of use and customizability, the community of ComfyUI users has built an impressive collection of workflows in a relatively short period of time, such as:

• Sketch to image
• Image upscaling
• Inpainting i.e. filling in an image based on a prompt

Prototyping with ComfyUI is fun and easy, but there isn’t a lot of guidance today on how to “productionize” your workflow, or serve it as part of a larger application. In this blog post, I’m going to show you how you can use Modal to manage your ComfyUI development process from prototype to production as a scalable API endpoint.

Example: Serve ComfyUI inpainting as an API endpoint

This diagram illustrates how you can turn your own ComfyUI workflow into a production-ready endpoint:

1. Spin up a ComfyUI development instance
2. Export your workflow as JSON
3. Eject from JSON into a Python-based ComfyUI workflow
4. Serve your workflow as a web endpoint

Read on for more detail in each step, or watch this walkthrough video:

1) Spin up a ComfyUI development instance

Run our ComfyUI example to spin up your own ComfyUI development instance where you can build your workflow.

This spins up a container running ComfyUI that you can access at a url like https://<your-workspace-name>--example-comfy-ui-web-dev.modal.run

You can further customize this ComfyUI instance by adding custom checkpoints and plugins to a model.json in the examples directory. This will rebuild the image and you may need to refresh your browser.

When you are done with your session, remember to Ctrl+C out of the container to spin down the instance so you aren’t charged for inactivity.

2) Export your workflow as JSON

After you’ve designed your workflow in the UI, the first step to productionizing it is to export the workflow as an API-compatible JSON object.

To do so, first click the gear icon in the top right of the menu box:

Then, check Enable Dev mode Options:

Now you should see a Save (API Format) option in your menu:

Copy the contents of that file into the workflow_api.json file within the comfyui folder in our examples repo.

3) Eject out of JSON into Python

Our ComfyUI example shows you how to quickly take this workflow JSON and serve it as an API. However, this approach has some drawbacks:

• You need to spin up a separate ComfyUI server to receive requests which introduces inference latency
• You have to parameterize a complex JSON object to handle user input

As opposed to using the JSON in an API request, you can also call the ComfyUI node objects directly from the ComfyUI repo. For example, a node defined like this in JSON:

# workflow_api.json

"2": {
  "inputs": {
    "ckpt_name": "512-inpainting-ema.ckpt"
  },
  "class_type": "CheckpointLoaderSimple",
  "_meta": {
    "title": "Load Checkpoint"
  }
}

Maps to this in Python:

# _generated_workflow_api.py
from nodes import (
    ...
    CheckpointLoaderSimple,
    ...
)

checkpointloadersimple = CheckpointLoaderSimple()
checkpointloadersimple_2 = checkpointloadersimple.load_checkpoint(
    ckpt_name="512-inpainting-ema.ckpt"
)

4) Serve your workflow as a Modal endpoint

This gist shows how to convert a ComfyUI JSON into a Python script and serve that script behind a Modal endpoint. It consists of three steps:

• Running get_python_workflow that creates a generated Python script called _generated_workflow_api.py based on your ComfyUI JSON
• Refactoring _generated_workflow_api.py into a function called run_python_workflow
• Standing up a Modal web endpoint that accepts input parameters and runs run_python_workflow

First, we do some Modal scaffolding:

# workflow_api.py

from comfyapp import image # import the image created in the main ComfyUI example

app = App(name="example-comfy-python-api")

# Volume where we will store generated images
vol_name = "comfyui-images"
vol = Volume.from_name(vol_name, create_if_missing=True)

Next, we copy over the relevant portions of _generated_workflow_api.py , namely the main() function which we’ll adapt into a new function called run_python_workflow :

# workflow_api.py
from .comfy_ui import image # import the image created in the main ComfyUI example

app = App(name="example-comfy-python-api")

# Volume where we will store generated images
vol_name = "comfyui-images"
vol = Volume.from_name(vol_name, create_if_missing=True)

# kind of a silly helper function, but the generated `main` calls it so we keep it
def get_value_at_index(obj: Union[Sequence, Mapping], index: int) -> Any:
    ...

def run_python_workflow(item: Dict):
    # In the generated version, these are in the global scope, but for Modal we move into the function scope
    import torch
    from nodes import (
        CheckpointLoaderSimple,
        CLIPTextEncode,
        KSampler,
        LoadImage,
        SaveImage,
        VAEDecode,
        VAEEncodeForInpaint,
    )

    with torch.inference_mode():
        loadimage = LoadImage()
        loadimage_1 = loadimage.load_image(image=image_name)

        checkpointloadersimple = CheckpointLoaderSimple()
        checkpointloadersimple_2 = checkpointloadersimple.load_checkpoint(
            ckpt_name="512-inpainting-ema.ckpt"
        )

        ...
        saveimage_8 = saveimage.save_images(
            filename_prefix="ComfyUI",
            images=get_value_at_index(vaedecode_7, 0),
        )

        return saveimage_8

Next, we stand up a web endpoint:

# Serves the python workflow behind a web endpoint
# Generated images are written to a Volume
@app.function(image=image, gpu="any", volumes={"/data": vol})
@web_endpoint(method="POST")
def serve_workflow(item: Dict):
    saved_image = run_python_workflow(item)
    images = saved_image["ui"]["images"]

    # Commits the ComfyUI image output directory to the Volume
    for i in images:
        filename = "output/" + i["filename"]
        with open(f'/data/{i["filename"]}', "wb") as f:
            f.write(pathlib.Path(filename).read_bytes())
        vol.commit()

    return HTMLResponse(f"<html>Image saved at volume {vol_name}! </html>")

Lastly, we handle the inputs prompt and image in our run_python_workflow function:

# Downloads an image url into the ComfyUI input/ directory
def download_image(url, save_path='/root/input/'):
    import requests
    try:
        response = requests.get(url)
        response.raise_for_status()
        pathlib.Path(save_path + url.split('/')[-1]).write_bytes(response.content)
        print(f"{url} image successfully downloaded")

    except requests.exceptions.RequestException as e:
        print(f"Error downloading {url} image: {e}")


def run_python_workflow(item: Dict):
    ...

    download_image(item['image'])
    with torch.inference_mode():
        loadimage = LoadImage()
        # point to the downloaded input image
        loadimage_1 = loadimage.load_image(image=item["image"].split('/')[-1])

        ...

        cliptextencode = CLIPTextEncode()
        # insert the prompt here
        cliptextencode_3 = cliptextencode.encode(
            text=f"closeup photograph of a {item['prompt']} in the yosemite national park mountains nature",
            clip=get_value_at_index(checkpointloadersimple_2, 1),
        )
        ...

Now you can stand up the web endpoint by running:

> modal serve workflow_api.py

Query the endpoint:

> curl 'https://<your_workspace>--example-comfy-python-api-serve-workflow-dev.modal.run' \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"prompt": "white heron", "image": "https://raw.githubusercontent.com/comfyanonymous/ComfyUI_examples/master/inpaint/yosemite_inpaint_example.png"}'

Image saved at volume comfyui-images!

And download the image from the volume:

> modal volume get comfyui-images "**"
Wrote 1567275 bytes to ComfyUI_00001_.png

Conclusion

With Modal, ComfyUI users can define a standard development instance in code and then migrate to a Python-based workflow when ready to scale. The result is a pipeline that can leverage Modal’s autoscaling to meet spikes in demand and spin down to save cost during periods of inactivity.

With this framework, developers can effectively combine ComfyUI’s ease of use with Modal’s developer experience, giving them the best of both worlds.

Acknowledgements

Thanks to the Fable team for their guidance, and check out how they’re building on top of Modal to modernize creative software at fable.app. And special thanks to comfyanonymous (whoever you may be…) for building the stable diffusion platform that everyone’s talking about!