ComfyUI is a popular no-code, visual editor for building complex image generation workflows. Because of its ease of use and customizability, the community of ComfyUI users has built an impressive collection of workflows in a relatively short period of time, such as:

• Sketch to image
• Image upscaling
• Inpainting i.e. filling in an image based on a prompt

Prototyping with ComfyUI is fun and easy, but there isn’t a lot of guidance today on how to “productionize” your workflow, or serve it as part of a larger application. In this blog post, I’m going to show you how you can use Modal to manage your ComfyUI development process from prototype to production as a scalable API endpoint.

Example: Serve ComfyUI inpainting as an API endpoint

Check out our ComfyUI API example, which stands up a web endpoint that takes a prompt and input image via http request, runs the ComfyUI inpainting example, and writes the generated image to a volume.

To run this example, first sign up for an account and clone our examples repo. To stand up the web endpoint, run:

> cd 06_gpu_and_ml && modal serve comfyui.workflow_api

Then, pass a prompt and input image via HTTP. This triggers a ComfyUI workflow run and saves the generated image to a volume:

> curl 'https://<your-workspace-name>--example-comfy-python-api-serve-workflow-dev.modal.run' \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"prompt": "white heron", "image": "https://raw.githubusercontent.com/comfyanonymous/ComfyUI_examples/master/inpaint/yosemite_inpaint_example.png"}'

Image saved at volume comfyui-images!

This diagram illustrates how you can turn your own ComfyUI workflow into a production-ready endpoint:

1. Spin up a ComfyUI development instance
2. Export your workflow as JSON
3. Run our helper function to eject from JSON into a Python-based ComfyUI workflow
4. Serve your workflow as a web endpoint

Read on for more detail in each step, or watch this walkthrough video:

1) Spin up a ComfyUI development instance

Run our ComfyUI example to spin up your own ComfyUI development instance where you can build your workflow:

> modal serve 06_gpu_and_ml/comfyui/comfy_ui.py

This spins up a container running ComfyUI that you can access at a url like https://<your-workspace-name>--example-comfy-ui-web-dev.modal.run

You can further customize this ComfyUI instance by adding custom checkpoints to CHECKPOINTS and custom plugins to PLUGINS in the comfy_ui.py script. This will rebuild the image and you may need to refresh your browser.

When you are done with your session, remember to Ctrl+C out of the container to spin down the instance so you aren’t charged for inactivity.

2) Export your workflow as JSON

After you’ve designed your workflow in the UI, the first step to productionizing it is to export the workflow as an API-compatible JSON object.

To do so, first click the gear icon in the top right of the menu box:

Then, check Enable Dev mode Options:

Now you should see a Save (API Format) option in your menu:

Copy the contents of that file into the workflow_api.json file within the comfyui folder in our examples repo.

3) Eject out of JSON into Python

This JSON can be used directly in an API request to a running ComfyUI instance. However, this approach has some drawbacks:

• You need to spin up a separate ComfyUI server to receive requests
• You have to parameterize a complex JSON object to handle user input

We recommend using this get_python_workflow helper function that calls a ComfyUI-to-Python extension to turn this JSON into a Python script called _generated_workflow_api.py.

> cd 06_gpu_and_ml && modal run comfyui.comfy_api::get_python_workflow
# saves Python script to _generated_workflow_api.py in the comfyui directory

This script is not immediately runnable out of the box, and we’ll show how to reference it in step 4. At a high level, the script makes the connection between a JSON-defined node and its corresponding Python object. For example, a node defined like this in JSON:

# workflow_api.json

"2": {
  "inputs": {
    "ckpt_name": "512-inpainting-ema.ckpt"
  },
  "class_type": "CheckpointLoaderSimple",
  "_meta": {
    "title": "Load Checkpoint"
  }
}

Maps to this in Python:

# _generated_workflow_api.py
from nodes import (
    ...
    CheckpointLoaderSimple,
    ...
)

checkpointloadersimple = CheckpointLoaderSimple()
checkpointloadersimple_2 = checkpointloadersimple.load_checkpoint(
    ckpt_name="512-inpainting-ema.ckpt"
)

We are now effectively treating ComfyUI as an importable Python package. We no longer need to spin up a separate ComfyUI server that is always listening for requests, and the workflow is much easier to parameterize.

4) Serve your workflow as a Modal endpoint

Our goal is to stand up a web endpoint that accepts inputs as POST requests, runs a ComfyUI workflow, and writes the generated images to a Volume.

First, we create a new file called worfklow_api.py with the usual Modal scaffolding:

# workflow_api.py

from .comfy_ui import image # import the image created in the main ComfyUI example

app = App(name="example-comfy-python-api")

# Volume where we will store generated images
vol_name = "comfyui-images"
vol = Volume.from_name(vol_name, create_if_missing=True)

Next, we copy over the relevant portions of _generated_workflow_api.py , namely the main() function which we’ll adapt into a new function called run_python_workflow :

# workflow_api.py
from .comfy_ui import image # import the image created in the main ComfyUI example

app = App(name="example-comfy-python-api")

# Volume where we will store generated images
vol_name = "comfyui-images"
vol = Volume.from_name(vol_name, create_if_missing=True)

# kind of a silly helper function, but the generated `main` calls it so we keep it
def get_value_at_index(obj: Union[Sequence, Mapping], index: int) -> Any:
    ...

def run_python_workflow(item: Dict):
    # In the generated version, these are in the global scope, but for Modal we move into the function scope
    import torch
    from nodes import (
        CheckpointLoaderSimple,
        CLIPTextEncode,
        KSampler,
        LoadImage,
        SaveImage,
        VAEDecode,
        VAEEncodeForInpaint,
    )

    with torch.inference_mode():
        loadimage = LoadImage()
        loadimage_1 = loadimage.load_image(image=image_name)

        checkpointloadersimple = CheckpointLoaderSimple()
        checkpointloadersimple_2 = checkpointloadersimple.load_checkpoint(
            ckpt_name="512-inpainting-ema.ckpt"
        )

        ...
        saveimage_8 = saveimage.save_images(
            filename_prefix="ComfyUI",
            images=get_value_at_index(vaedecode_7, 0),
        )

        return saveimage_8

Next, we stand up a web endpoint:

# Serves the python workflow behind a web endpoint
# Generated images are written to a Volume
@app.function(image=image, gpu="any", volumes={"/data": vol})
@web_endpoint(method="POST")
def serve_workflow(item: Dict):
    saved_image = run_python_workflow(item)
    images = saved_image["ui"]["images"]

    # Commits the ComfyUI image output directory to the Volume
    for i in images:
        filename = "output/" + i["filename"]
        with open(f'/data/{i["filename"]}', "wb") as f:
            f.write(pathlib.Path(filename).read_bytes())
        vol.commit()

    return HTMLResponse(f"<html>Image saved at volume {vol_name}! </html>")

Lastly, we handle the inputs prompt and image in our run_python_workflow function:

# Downlaods an image url into the ComfyUI input/ directory
def download_image(url, save_path='/root/input/'):
    import requests
    try:
        response = requests.get(url)
        response.raise_for_status()
        pathlib.Path(save_path + url.split('/')[-1]).write_bytes(response.content)
        print(f"{url} image successfully downloaded")

    except Exception as e:
        print(f"Error downloading {url} image: {e}")


def run_python_workflow(item: Dict):
    ...

    download_image(item['image'])
    with torch.inference_mode():
        loadimage = LoadImage()
        # point to the downloaded input image
        loadimage_1 = loadimage.load_image(image=item["image"].split('/')[-1])

        ...

        cliptextencode = CLIPTextEncode()
        # insert the prompt here
        cliptextencode_3 = cliptextencode.encode(
            text=f"closeup photograph of a {item['prompt']} in the yosemite national park mountains nature",
            clip=get_value_at_index(checkpointloadersimple_2, 1),
        )
        ...

Now you can stand up the web endpoint by running:

> cd 06_gpu_and_ml && modal serve comfyui.workflow_api

Query the endpoint:

> curl 'https://<your_workspace>--example-comfy-python-api-serve-workflow-dev.modal.run' \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"prompt": "white heron", "image": "https://raw.githubusercontent.com/comfyanonymous/ComfyUI_examples/master/inpaint/yosemite_inpaint_example.png"}'

Image saved at volume comfyui-images!

And download the image from the volume:

> modal volume get comfyui-images "**"
Wrote 1567275 bytes to ComfyUI_00001_.png

The full example is here.

Conclusion

With Modal, ComfyUI users can define a standard development instance in code and then migrate to a Python-based workflow when ready to scale. The result is a pipeline that can leverage Modal’s autoscaling to meet spikes in demand and spin down to save cost during periods of inactivity.

With this framework, developers can effectively combine ComfyUI’s ease of use with Modal’s developer experience, giving them the best of both worlds.

Acknowledgements

Thanks to the Fable team for their guidance, and check out how they’re building on top of Modal to modernize creative software at fable.app. And special thanks to comfyanonymous (whoever you may be…) for building the stable diffusion platform that everyone’s talking about!