Skip to content

Stable Diffusion

One way to use Stable Diffusion is to have a frontend GUI. This note shows how to setup a frontend called ComfyUI and use Stable Diffusion.

ComfyUI Setup

Clone the ComfyUI Repository
git clone https://github.com/comfyanonymous/ComfyUI
Set Local Python Version
pyenv local 3.12.10
Create Virtual Environment
uv venv
Install Stable pytorch
uv pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu128
Install Dependencies
uv pip install -r requirements.txt
Start the Server
uv run main.py

The UI should be accessible at http://127.0.0.1:8188.

Note

  • Python 3.13 is supported but using 3.12 is recommended because some custom nodes and their dependencies might not support it yet.
  • Put your SD checkpoints (the huge ckpt/safetensors files) in: models/checkpoints
  • Put your VAE in: models/vae

Stable Diffusion

The ComfyUI does not include any models out of the box. We're going to use SDXL (Stable Diffusion XL).

  1. Download the Stable Diffusion XL Base 1.0 model from Hugging Face. It's the file named sd_xl_base_1.0.safetensors. Place in the models/checkpoints directory.

  2. Download the Stable Diffusion XL Refiner 1.0 refiner from Huggine Face. It's the file named sd_xl_refiner_1.0.safetensors. Place in the models/checkpoints directory.

  3. Download the Stable Diffusion XL VAE VAE from Hugging Face. It's the file named sdxl_vae.safetensors. Place in the models/vae directory.

If you reload the UI and running the default sample template for image generation, you'll see the Stable Diffusion model and refiner available in the Load Checkpoint node.

CLIP Text Encode Node

Used for prompt (both positive and negative)

Empty Latent Image Node

Starting point for Stable Diffusion. Can set things like image size and how many images. This node represents an empty starting image.

KSampler Node

Property Description
control_after_generate Whether the image generated is same or random.
steps How long it bakes the image. Go between 30 and 40.
cfg How close it follows prompt (between 7 and 10 is good)

ComfyUI Manager

To add more nodes easiest way is to use ComfyUI Manager tool. Shutdown ComfyUI and follow these steps.

Clone the ComfyUI-Manager into the custom_nodes Directory
cd custom_nodes
git clone https://github.com/ltdrdata/ComfyUI-Manager comfyui-manager
Install ComfyUI-Manager Dependencies
uv pip install -r custom_nodes/comfyui-manager/requirements.txt
Install pip into Virtual Environment
source .venv/bin/activate
python -m ensurepip
deactivate

Restart the ComfyUI and you should see a Manager button appear.

Example Workflow with OpenPose

See YouTube - Run SDXL Locally With ComfyUI (2024 Stable Diffusion Guide) by Matt Wolfe

References