vnassets/TECH_SPEC.md

VNAsset — Visual Novel Asset Pipeline CLI

Problem

ComfyUI's node-graph execution is too slow for batch visual novel asset production:

• Each run pays Python dispatch overhead across ~15 nodes
• Models may unload/reload between runs due to VRAM management
• The two-step flow (generate base sprite → edit variants) requires manual intervention and separate workflow runs
• No batch orientation: each sprite variant is a full cold or warm restart

Hardware Target

AMD Ryzen AI Max 395 Pro (Strix Halo)

• Integrated RDNA 3.5 GPU, unified memory architecture
• 128 GB shared RAM → VRAM is effectively unlimited (up to ~96 GB allocatable to GPU)
• ROCm/HIP backend
• Bottleneck is GPU compute throughput, not memory capacity
• Implications: model offloading is pointless, everything stays resident; optimizations should target fewer FLOPs, not fewer bytes

Core Design Principles

1. Models stay hot. All models loaded once at startup, held in VRAM for the session lifetime.
2. No node graph. Direct PyTorch forward calls. No serialization, no dispatch loop, no intermediate tensor copies between "nodes."
3. Generate-then-edit as first-class pipeline. The tool knows that output of generation feeds into editing. Optional human-review gate between stages.
4. Batch-native. Accept lists of prompts/variants and process them in one warm session.
5. CLI-first. Single binary, YAML/JSON config files, stdin/stdout where useful.
6. AMD-first. ROCm is the primary backend. No CUDA-only paths. torch.compile where ROCm supports it.

Pipeline Architecture

┌─────────────────────────────────────────────────────────────┐
│                      VNAsset Session                         │
│                                                              │
│  ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐ │
│  │ SDXL     │   │ Qwen     │   │ Qwen VL  │   │ VAE(s)   │ │
│  │ UNet     │   │ Edit UNet│   │ 7B TE    │   │          │ │
│  │ (~3.5GB) │   │ (~5GB)   │   │ (~14GB)  │   │ (~1GB)   │ │
│  └──────────┘   └──────────┘   └──────────┘   └──────────┘ │
│       │               │               │              │       │
│       ▼               │               │              │       │
│  ┌─────────┐          │               │              │       │
│  │ Generate│          │               │              │       │
│  │ Phase   │──────────┼───────────────┼──────────────┤       │
│  │         │  base.png │              │              │       │
│  └─────────┘          │               │              │       │
│       │               ▼               ▼              ▼       │
│       │          ┌──────────────────────────────────────┐    │
│       └──────────►         Edit Phase                    │    │
│                  │  base.png + prompts[] → variants[]    │    │
│                  └──────────────────────────────────────┘    │
│                                      │                       │
│                                      ▼                       │
│                           ┌──────────────────┐               │
│                           │  Output Manager   │               │
│                           │  base.png         │               │
│                           │  happy.png        │               │
│                           │  sad.png          │               │
│                           │  angry.png        │               │
│                           └──────────────────┘               │
└─────────────────────────────────────────────────────────────┘

CLI Interface

Generate (SDXL text-to-image)

vnasset generate \
  --checkpoint ./models/novaAnimeXL_ilV190.safetensors \
  --prompt "1girl, solo, red hair, glasses, blue eyes..." \
  --negative-prompt "deformed, ugly, bad quality..." \
  --width 1024 --height 1024 \
  --steps 20 --cfg 4.5 \
  --sampler euler --scheduler simple \
  --seed 573523050 \
  --output ./output/character_base.png

Edit (Qwen image-to-image)

vnasset edit \
  --model ./models/qwen_image_edit_2509_fp8_e4m3fn.safetensors \
  --clip ./models/qwen_2.5_vl_7b_fp8_scaled.safetensors \
  --vae ./models/qwen_image_vae.safetensors \
  --lora ./models/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16.safetensors \
  --input ./output/character_base.png \
  --prompt "make her smile happily" \
  --turbo \
  --output ./output/character_happy.png

Batch Pipeline (generate + multiple edits)

vnasset pipeline --config pipeline.yaml

Where pipeline.yaml:

models:
  sdxl_checkpoint: ./models/novaAnimeXL_ilV190.safetensors
  edit_unet: ./models/qwen_image_edit_2509_fp8_e4m3fn.safetensors
  edit_clip: ./models/qwen_2.5_vl_7b_fp8_scaled.safetensors
  edit_vae: ./models/qwen_image_vae.safetensors
  edit_lora: ./models/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16.safetensors

generate:
  prompt: "1girl, solo, red hair, glasses, blue eyes, standing..."
  negative_prompt: "deformed, ugly, bad quality, lowres..."
  width: 1024
  height: 1024
  steps: 20
  cfg: 4.5
  sampler: euler
  scheduler: simple
  seed: 573523050
  output: "character_base"

edit:
  turbo: true            # use Lightning 4-step LoRA, CFG=1
  variants:
    - name: happy
      prompt: "make her smile happily"
    - name: sad
      prompt: "make her look sad, tears in her eyes"
    - name: angry
      prompt: "make her look angry, furrowed brows"
    - name: surprised
      prompt: "make her look surprised, eyes wide open"
    - name: blush
      prompt: "make her blush, embarrassed expression"

Output: character_base.png, character_happy.png, character_sad.png, etc.

Subcommands Summary

Command	Purpose
vnasset generate	SDXL text-to-image, one shot
vnasset edit	Qwen image edit, one shot
vnasset pipeline	Generate + batch edit from config file
vnasset serve	(future) lightweight HTTP API for integration

Data Flow

Generate Phase

prompt text ──► SDXL CLIP ──► conditioning ──┐
                                              ├──► SDXL UNet (euler, N steps) ──► latent ──► SDXL VAE ──► image
noise + empty_latent ─────────────────────────┘

Edit Phase

input image ──► kontext_scale ──► Qwen VAE encode ──► latent ──┐
                                                                │
prompt text ──► Qwen VL 7B TE ──► conditioning (pos/neg) ──────┤
                                                                ├──► Qwen Edit UNet (N steps, CFG) ──► latent ──► Qwen VAE decode ──► image
input image ──► kontext_scale ──► Qwen VL 7B TE (visual tokens)┘

Turbo vs Normal Mode

Parameter	Normal	Turbo (Lightning LoRA)
Steps	20	4
CFG	4.0	1.0
LoRA	none	Lightning-4steps
Sampler	euler	euler
Scheduler	simple	simple

Technology Stack

Layer	Choice	Rationale
Language	Python 3.12+	Model ecosystem (transformers, diffusers, safetensors) is Python-only
ML Framework	PyTorch (ROCm)	Required for AMD GPU; HIP backend
Diffusion Backend	diffusers + custom pipelines	SDXL is standard; Qwen Image Edit needs a custom diffusers pipeline
Model Format	safetensors	Direct load from ComfyUI model files; no conversion
CLI Framework	click or argparse	Lightweight, no async needed
Config Format	YAML (PyYAML)	Readable; pipeline definitions are human-authored
Image I/O	Pillow + torchvision	Standard; PNG output at minimum
LoRA Loading	peft or manual merge	Weight-merging Lightning LoRA into UNet at load time

Key Optimizations (AMD-specific)

1. Persistent model registry. All models loaded at session init into a dict; references kept alive. On 128 GB unified memory this is free.

2. VAE overlap. SDXL VAE decode can run on a separate CUDA/HIP stream while the Qwen pipeline begins encoding (if generating then editing). Marginal on iGPU but worth structuring for.

3. torch.compile on UNet. The Qwen Edit UNet forward is called identically per variant (same spatial dims). Compile once, reuse. ROCm torch.compile support is improving; falls back gracefully if unavailable.

4. FP8 native. Both the Qwen UNet and CLIP are already FP8 (fp8_e4m3fn). Load them as FP8, compute in FP8 where possible on RDNA 3.5 (WMMA instructions support FP8). No upcast overhead.

5. Lightning LoRA merge. At model load time, merge the 4-step LoRA weights into the UNet base weights (simple linear addition). Avoids the extra LoraLoaderModelOnly forward overhead per step. The turbo/normal switch then becomes a steps+CFG change only.

6. Batch edit loop. All edit variants share the same input latent. The VAE encode and VL text encode (visual branch) run once, not per variant. Only the text prompt encoding and UNet denoising repeat.

Shared State Between Variants (Edit Phase)

For a batch of N variants from one input image:

• Run once: VAE encode (image → latent), VL visual token encoding
• Run N times: Text prompt encoding (short text → text tokens, cheap), UNet denoising (4 or 20 steps), VAE decode (latent → image)

This means the 7B model is invoked once for visual encoding + N times for text encoding (but text encoding is a small fraction of the 7B model, essentially a CLIP-like forward).

Output Conventions

• {output_dir}/{base_name}.png for the generated base sprite
• {output_dir}/{base_name}_{variant}.png for each variant
• Metadata saved alongside: {output_dir}/{base_name}.json with prompt, seed, model hashes, timings
• Flat green background left intact; post-processing (trimming, transparency) is out of scope for v1

Phased Roadmap

Phase 1 — Core CLI (MVP)

• vnasset generate with SDXL checkpoint
• vnasset edit with Qwen Image Edit (normal mode, no LoRA)
• Model loading from ComfyUI safetensors paths
• Single-image output
• ROCm PyTorch verified working

Phase 2 — Batch & Turbo

• vnasset pipeline YAML config
• Lightning 4-step LoRA merge-at-load
• Turbo mode (steps=4, CFG=1)
• Shared encode optimization (one VAE encode, N UNet runs)
• --seed and seed randomization
• Output metadata JSON

Phase 3 — Polish

• torch.compile UNet forward
• Progress bars (tqdm)
• Timing reports (wall time per phase)
• Checkpoint compatibility: any SDXL safetensors, any Qwen Image Edit safetensors
• Resolution from CLI/config (preserving aspect ratio through the pipeline)

Phase 4 — Future

• vnasset serve lightweight HTTP API
• Background removal integration (green-screen trim)
• Multi-character generation (multiple SDXL prompts in one session)
• Upscaling pass (optional)

Open Questions

1. SDXL CLIP encoding: The current workflow uses BNK_CLIPTextEncodeAdvanced with token_normalization=none, weight_interpretation=comfy. Is this identical to standard SDXL CLIP encoding, or does the BNK node do something special? Needs verification.

2. Qwen VAE vs SDXL VAE: The SDXL checkpoint bundles its own VAE. The Qwen pipeline uses qwen_image_vae. These are distinct VAEs with different latent spaces. The pipeline must use the correct VAE per phase — no sharing.

3. Qwen 2.5 VL 7B text encoding: The TextEncodeQwenImageEditPlus node is ComfyUI-specific. Porting this to raw HuggingFace transformers + qwen-vl-utils requires understanding exactly what tokenization/masking it does for edit-mode conditioning (positive + negative with image context). This is the riskiest port.

4. ROCm torch.compile status: Needs testing on the user's specific ROCm version. May need TORCH_COMPILE_DISABLE=1 initially.

5. FP8 compute path: RDNA 3.5 supports FP8 via WMMA. PyTorch ROCm FP8 support is maturing. If FP8 compute isn't available, the FP8 weights should be upcast to BF16 at load time — still fits in 128 GB.