Edit 2024-08-26: Our latest recommended solution for productionizing a ComfyUI workflow is detailed in this example. As a result, this post has been largely re-written to focus on the specific use case of converting a ComfyUI JSON workflow to Python.
ComfyUI is a popular no-code, visual editor for building complex image generation workflows. While ComfyUI started as a prototyping / experimental playground for Stable Diffusion, increasingly more users are using it to deploy image generation pipelines in production.
Modal is a great solution for this and our ComfyUI example walks you through the step-by-step process of serving your ComfyUI workflow behind an API endpoint. This is our recommended solution for productionizing ComfyUI in most cases.
However, some users prefer defining and iterating on their ComfyUI workflows in Python. For example, if you’re doing some complex user prompt handling in your workflow, Python is arguably easier to work with than handling the raw workflow JSON object. In this blog post, we’ll show you how to convert your ComfyUI workflow to executable Python code as an alternative design to serving a workflow in production.
Export your workflow as JSON
The native representation of a ComfyUI workflow is in JSON. First, we need to extract that representation from the UI.
- Click the gear icon in the top right of the menu box:
- Check Enable Dev mode Options:
- Click Save (API Format) option in your menu:
Save the file as workflow_api.json.
Convert JSON to Python
We’ll use the ComfyUI to Python Extension to convert the JSON from the previous step to Python code. This tool requires ComfyUI to be installed on the host machine, so we’ll use Modal to install ComfyUI in a container and then run the tool in that container.
Save the following script to a file called comfypython.py in the same directory as the workflow_api.json you created in the previous step.
import pathlib
import modal
comfyui_python_remote_path = pathlib.Path("/root/comfy/ComfyUI/ComfyUI-to-Python-Extension")
image = ( # build up a Modal Image to run ComfyUI, step by step
modal.Image.debian_slim( # start from basic Linux with Python
python_version="3.11"
)
.apt_install("git") # install git to clone ComfyUI
.pip_install("comfy-cli==1.0.33") # install comfy-cli
.run_commands( # use comfy-cli to install the ComfyUI repo and its dependencies
"comfy --skip-prompt install --nvidia"
)
.run_commands( # download all models and custom nodes required in your workflow
"comfy --skip-prompt model download --url https://huggingface.co/stabilityai/stable-diffusion-2-inpainting/resolve/main/512-inpainting-ema.safetensors --relative-path models/checkpoints"
)
.run_commands(
# As of this writing, main is broken, so using this fork's fix
"git clone -b fix/custom_module_init https://github.com/albertpurnama/ComfyUI-to-Python-Extension/ /root/comfy/ComfyUI/ComfyUI-to-Python-Extension",
f"cd {comfyui_python_remote_path} && pip install -r requirements.txt",
)
)
app = modal.App(name="comfy-python", image=image)
@app.function(
gpu="A10G",
mounts=[
modal.Mount.from_local_file(
pathlib.Path(__file__).parent / "workflow_api.json",
comfyui_python_remote_path / "workflow_api.json",
)
],
)
def comfyui_to_python():
"""
Put the workflow json you want to convert into the same directory as this script
"""
import subprocess
result = subprocess.run(["python", "comfyui_to_python.py"], cwd=comfyui_python_remote_path)
if result.returncode != 0:
raise RuntimeError(f"Exited unexpectedly with code {result.returncode}")
else:
try:
return (comfyui_python_remote_path / "workflow_api.py").read_text()
except FileNotFoundError:
print("Error: File workflow_api.py not found.")
@app.local_entrypoint()
def fetch_comyfui_to_python():
"""
Write the generated python to _generated_workflow_api.py in the same directory
"""
(pathlib.Path(__file__).parent / "_generated_workflow_api.py").write_text(comfyui_to_python.remote())At a high level, this script will convert a JSON node representation:
"2": {
"inputs": {
"ckpt_name": "512-inpainting-ema.ckpt"
},
"class_type": "CheckpointLoaderSimple",
"_meta": {
"title": "Load Checkpoint"
}
}Into a Python object:
from nodes import (
...
CheckpointLoaderSimple,
...
)
checkpointloadersimple = CheckpointLoaderSimple()
checkpointloadersimple_2 = checkpointloadersimple.load_checkpoint(
ckpt_name="512-inpainting-ema.ckpt"
)Run modal run comfypython::fetch_comfyui_to_python to convert workflow_api.json into a Python file called _generated_workflow_api.py in your local directory.
Run the Python workflow
Now we can run this generated code and fetch the generated images. Add the following to the comfypython.py script we created in the previous step:
@app.function(
gpu="A10G",
# Mount the generated workflow Python code
mounts=[
modal.Mount.from_local_file(
pathlib.Path(__file__).parent / "_generated_workflow_api.py",
comfyui_python_remote_path / "_generated_workflow_api.py",
)
],
)
def run_comfyui_python():
import subprocess
result = subprocess.run(["python", "_generated_workflow_api.py"], cwd=comfyui_python_remote_path)
if result.returncode != 0:
raise RuntimeError(f"Exited unexpectedly with code {result.returncode}")
else:
# Iterate through the output directory and return the generated images
output_dir = comfyui_python_remote_path.parent / "output"
image_bytes = []
for file in output_dir.iterdir():
if file.name.endswith(".png"):
with open(file, "rb") as f:
image_bytes.append(f.read())
return image_bytes
@app.local_entrypoint()
def fetch_images():
image_bytes = run_comfyui_python.remote()
# Write image bytes to local files
for i, val in enumerate(image_bytes):
pathlib.Path(f"image_{i}.png").write_bytes(val)Run modal run comfypython.py::fetch_images to run the Python workflow and write the generated images to your local directory.
Conclusion
Some of our users have had success using this approach to establish the foundation of a Python-based ComfyUI workflow, from which they can continue to iterate. For example:
- Add command-line arguments to
_generated_workflow_api.pyto handle prompts - Turn
run_comfyui_pythoninto a web endpoint that can receive requests via API
Generally speaking though, we don’t recommend this approach anymore to productionize your ComfyUI pipeline because the extra step required to convert from JSON to Python isn’t worth the marginal ergonomic beneftis of coding in Python. This is especially true if your workflow contains a lot of models or custom nodes. Your best bet is to follow our ComfyUI example which directly serves your ComfyUI workflow JSON.
