How to convert a ComfyUI workflow to Python code

Tutorials

April 2, 2024•5 minute read

Data Engineer

Edit 2024-08-26: Our latest recommended solution for productionizing a ComfyUI workflow is detailed in this example. As a result, this post has been largely re-written to focus on the specific use case of converting a ComfyUI JSON workflow to Python.

ComfyUI workflow diagram

ComfyUI is a popular no-code, visual editor for building complex image generation workflows. While ComfyUI started as a prototyping / experimental playground for Stable Diffusion, increasingly more users are using it to deploy image generation pipelines in production.

Modal is a great solution for this and our ComfyUI example walks you through the step-by-step process of serving your ComfyUI workflow behind an API endpoint. This is our recommended solution for productionizing ComfyUI in most cases.

However, some users prefer defining and iterating on their ComfyUI workflows in Python. For example, if you’re doing some complex user prompt handling in your workflow, Python is arguably easier to work with than handling the raw workflow JSON object. In this blog post, we’ll show you how to convert your ComfyUI workflow to executable Python code as an alternative design to serving a workflow in production.

Export your workflow as JSON

The native representation of a ComfyUI workflow is in JSON. First, we need to extract that representation from the UI.

Click the gear icon in the top right of the menu box:

ComfyUI menu box with gear circled

Check Enable Dev mode Options:

ComfyUI enable dev options selection

Click Save (API Format) option in your menu:

ComfyUI menu with api json cirlced

Save the file as workflow_api.json.

Convert JSON to Python

We’ll use the ComfyUI to Python Extension to convert the JSON from the previous step to Python code. This tool requires ComfyUI to be installed on the host machine, so we’ll use Modal to install ComfyUI in a container and then run the tool in that container.

Save the following script to a file called comfypython.py in the same directory as the workflow_api.json you created in the previous step.

import pathlib

import modal

comfyui_python_remote_path = pathlib.Path("/root/comfy/ComfyUI/ComfyUI-to-Python-Extension")
image = (  # build up a Modal Image to run ComfyUI, step by step
    modal.Image.debian_slim(  # start from basic Linux with Python
        python_version="3.11"
    )
    .apt_install("git")  # install git to clone ComfyUI
    .pip_install("comfy-cli==1.0.33")  # install comfy-cli
    .run_commands(  # use comfy-cli to install the ComfyUI repo and its dependencies
        "comfy --skip-prompt install --nvidia"
    )
    .run_commands(  # download all models and custom nodes required in your workflow
        "comfy --skip-prompt model download --url https://huggingface.co/stabilityai/stable-diffusion-2-inpainting/resolve/main/512-inpainting-ema.safetensors --relative-path models/checkpoints"
    )
    .run_commands(
        # As of this writing, main is broken, so using this fork's fix
        "git clone -b fix/custom_module_init https://github.com/albertpurnama/ComfyUI-to-Python-Extension/ /root/comfy/ComfyUI/ComfyUI-to-Python-Extension",
        f"cd {comfyui_python_remote_path} && pip install -r requirements.txt",
    )
)
app = modal.App(name="comfy-python", image=image)


@app.function(
    gpu="A10G",
    mounts=[
        modal.Mount.from_local_file(
            pathlib.Path(__file__).parent / "workflow_api.json",
            comfyui_python_remote_path / "workflow_api.json",
        )
    ],
)
def comfyui_to_python():
    """
    Put the workflow json you want to convert into the same directory as this script
    """
    import subprocess

    result = subprocess.run(["python", "comfyui_to_python.py"], cwd=comfyui_python_remote_path)
    if result.returncode != 0:
        raise RuntimeError(f"Exited unexpectedly with code {result.returncode}")
    else:
        try:
            return (comfyui_python_remote_path / "workflow_api.py").read_text()
        except FileNotFoundError:
            print("Error: File workflow_api.py not found.")


@app.local_entrypoint()
def fetch_comyfui_to_python():
    """
    Write the generated python to _generated_workflow_api.py in the same directory
    """
    (pathlib.Path(__file__).parent / "_generated_workflow_api.py").write_text(comfyui_to_python.remote())

At a high level, this script will convert a JSON node representation:

"2": {
  "inputs": {
    "ckpt_name": "512-inpainting-ema.ckpt"
  },
  "class_type": "CheckpointLoaderSimple",
  "_meta": {
    "title": "Load Checkpoint"
  }
}

Into a Python object:

from nodes import (
    ...
    CheckpointLoaderSimple,
    ...
)

checkpointloadersimple = CheckpointLoaderSimple()
checkpointloadersimple_2 = checkpointloadersimple.load_checkpoint(
    ckpt_name="512-inpainting-ema.ckpt"
)

Run modal run comfypython::fetch_comfyui_to_python to convert workflow_api.json into a Python file called _generated_workflow_api.py in your local directory.

Run the Python workflow

Now we can run this generated code and fetch the generated images. Add the following to the comfypython.py script we created in the previous step:

@app.function(
    gpu="A10G",
    # Mount the generated workflow Python code
    mounts=[
        modal.Mount.from_local_file(
            pathlib.Path(__file__).parent / "_generated_workflow_api.py",
            comfyui_python_remote_path / "_generated_workflow_api.py",
        )
    ],
)
def run_comfyui_python():
    import subprocess

    result = subprocess.run(["python", "_generated_workflow_api.py"], cwd=comfyui_python_remote_path)
    if result.returncode != 0:
        raise RuntimeError(f"Exited unexpectedly with code {result.returncode}")
    else:
        # Iterate through the output directory and return the generated images
        output_dir = comfyui_python_remote_path.parent / "output"
        image_bytes = []
        for file in output_dir.iterdir():
            if file.name.endswith(".png"):
                with open(file, "rb") as f:
                    image_bytes.append(f.read())
        return image_bytes


@app.local_entrypoint()
def fetch_images():
    image_bytes = run_comfyui_python.remote()
    # Write image bytes to local files
    for i, val in enumerate(image_bytes):
        pathlib.Path(f"image_{i}.png").write_bytes(val)

Run modal run comfypython.py::fetch_images to run the Python workflow and write the generated images to your local directory.

Conclusion

Some of our users have had success using this approach to establish the foundation of a Python-based ComfyUI workflow, from which they can continue to iterate. For example:

Add command-line arguments to _generated_workflow_api.py to handle prompts
Turn run_comfyui_python into a web endpoint that can receive requests via API

Generally speaking though, we don’t recommend this approach anymore to productionize your ComfyUI pipeline because the extra step required to convert from JSON to Python isn’t worth the marginal ergonomic beneftis of coding in Python. This is especially true if your workflow contains a lot of models or custom nodes. Your best bet is to follow our ComfyUI example which directly serves your ComfyUI workflow JSON.

Export your workflow as JSON

Convert JSON to Python

Run the Python workflow

Conclusion

Ship your first app in minutes.