Skip to content

Diffusers Library

Build flexible media generation workflows with modular 🧨 Diffusers pipelines.

The Diffusers Library is for creators who want more control over diffusion execution. Instead of running a fixed, monolithic "generate image" step, you break the diffusion process into individual, connectable stages — so you can inspect, reuse, and customize each part of the pipeline.

Experimental — under active development

APIs, node interfaces, workflow templates, and library structure may change at any time without notice or migration support. Pin to a specific commit if you need stability, and expect breakage when updating.

Installation

In the editor, open Manage → Library Management, click Add Library, and paste:

https://github.com/griptape-ai/griptape-nodes-library-diffusers

Use the modal's Advanced Options to pin a branch, tag, or commit. Or via the CLI:

gtn libraries download https://github.com/griptape-ai/griptape-nodes-library-diffusers

See the Libraries guide for general install, update, and troubleshooting help.

Model downloads

All models are downloaded locally to the Hugging Face cache the first time they're used (default: ~/.cache/huggingface/hub on Linux/macOS, %USERPROFILE%\.cache\huggingface\hub on Windows). To store them elsewhere, set the HF_HOME environment variable before launching the engine. See Hugging Face Models for account and token setup.

How it works

A typical flow looks like this:

  1. Build a pipeline once with the Pipeline Builder, and reuse it across multiple generations.
  2. Create or load latents (noise, empty, encoded image/video, or a saved tensor).
  3. Run diffusion with Generate Media Latents to produce new latents — optionally multiple times for multi-stage or rediffusion workflows.
  4. Transform latents with math, masked compositing, or upsampling between stages.
  5. Decode the final latents back into images or video with Decode Media Latent.

Because every stage is a node, you can branch, chain, and reorder steps — enabling patterns like multi-stage refinement, ControlNet stacking, latent composition, first/last-frame video conditioning, and latent upscaling that aren't possible with a single end-to-end generate node.

Supported models

Models are selected on the Pipeline Builder via a provider dropdown. Currently supported:

  • Flux and Flux2 (including Flux2-Klein)
  • Stable Diffusion XL
  • Qwen-Image (and Qwen-Edit)
  • Z-Image
  • LTX (video)
  • LTX-2.x (text/image/video-to-video, with image and video conditioning, IC-LoRA, and HDR IC-LoRA for linear HDR output)
  • WAN (text-to-video and image-to-video)

Models are loaded from Hugging Face repositories in Diffusers format (single-file .safetensors checkpoints are not loaded directly — use a Hugging Face repo ID). Multiple LoRAs can be attached to a pipeline via the builder.

Node reference

Node groups mirror the categories in the node picker.

Pipeline

Create

Processing

Transform

Conditioning

Encode / Decode

IO

Live previews

Enable live image previews to stream intermediate decoded images during generation. Useful for monitoring long runs, at the cost of inference speed.

  1. Open Settings → Library Settings in the editor.
  2. Scroll to the Modular Diffusion Library section.
  3. Toggle Enable Image Preview Intermediates on.

Performance and memory

The Pipeline Builder caches the loaded pipeline in memory and reuses it across runs, only rebuilding when the configuration changes.

For lower-VRAM setups, the builder exposes a Memory Optimization Strategy selector:

  • Automatic — Griptape picks reasonable defaults for the chosen model.
  • Manual — you control each knob individually:
    • Quantization mode: fp8, int8, or int4 (via optimum-quanto / bitsandbytes) — shrinks transformer weights at the cost of some quality.
    • CPU offload strategy: Model (whole submodules) or Sequential (per-layer) — moves weights to CPU when idle to free VRAM, at the cost of inference speed.
    • Attention slicing — runs attention in smaller chunks; cheap memory win, small speed hit.
    • VAE slicing — decodes the latent in batches of 1; helps with large batch sizes.
    • Transformer layerwise casting — keeps the transformer in a lower precision and upcasts per layer during compute.

Enable only what you need — each option trades some speed for memory. The Pipeline Builder includes per-parameter help badges with more detail.

Using alongside the Advanced Media Library

The Diffusers Library and the Advanced Media Library share several upstream dependencies (e.g. diffusers, transformers, torch) and each maintains its own in-memory pipeline cache. If both libraries are loaded and used in the same session, you may hold two full copies of overlapping model weights in VRAM/RAM at once, leading to out-of-memory errors. Prefer using one library per session, or reduce memory pressure via the optimization knobs above.

Workflow templates included

  • Text2Image
  • MultistageText2Image
  • LoRAText2Image
  • ControlnetText2Image
  • Image2Image
  • FirstAndLastFrameImage2Video
  • LTX23-HDR-Text2Video-Upsample-Two-Stage

Support

Found a bug or have a feature request? Open an issue.