# VNAsset

Fast CLI pipeline for visual novel image asset generation.

Drop-in replacement for the ComfyUI workflow loop: generate base character sprites
with SDXL, then batch-edit variants (expressions, outfits) with Qwen Image Edit —
all in one warm session, no node-graph overhead.

## Hardware

Built for **AMD Strix Halo** (Ryzen AI Max 395 Pro, Radeon 8060S, 128 GB unified
memory). Also works on discrete AMD GPUs with ROCm. NVIDIA support is untested
but should work if you swap the torch backend.

The 128 GB unified memory means VRAM is effectively unlimited (up to ~96 GB
allocatable to GPU). The bottleneck is **GPU compute throughput**, not memory
capacity — model offloading is pointless, everything stays resident.

## Install

```bash
git clone <repo> vnassets
cd vnassets

# Create venv (Python 3.12 required for ROCm torch compatibility)
python3.12 -m venv .venv
source .venv/bin/activate

# Install ROCm PyTorch from local wheels (no remote index dependency)
pip install wheels/torch-2.11.0+rocm7.2-cp312-cp312-manylinux_2_28_x86_64.whl \
            wheels/triton_rocm-3.6.0-cp312-cp312-linux_x86_64.whl \
            wheels/torchvision-0.26.0+rocm7.2-cp312-cp312-manylinux_2_28_x86_64.whl

# Install the rest
pip install -e .
```

The torch, triton-rocm, and torchvision wheels are bundled in `wheels/`. If
you don't have them yet (e.g. after a fresh clone), download them first:

```bash
mkdir -p wheels
python3.12 -m pip download torch==2.11.0 triton-rocm==3.6.0 torchvision==0.26.0 \
    --index-url https://download.pytorch.org/whl/rocm7.2 \
    --dest wheels --no-deps
```

### Models

Symlink your ComfyUI models into `models/`:

```bash
cd models
ln -s /path/to/ComfyUI/models/checkpoints/novaAnimeXL_ilV190.safetensors .
ln -s /path/to/ComfyUI/models/diffusion_models/qwen_image_edit_2509_fp8_e4m3fn.safetensors .
ln -s /path/to/ComfyUI/models/loras/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16.safetensors .
```

The Qwen VAE and text encoder are downloaded automatically from HuggingFace Hub
on first use (`Qwen/Qwen-Image` and `Qwen/Qwen2.5-VL-7B-Instruct`). No symlinks
needed for those.

Or place the actual files there — the tool just reads whatever safetensors you
point it at.

## Usage

### Pipeline (batch YAML config)

Declare multi-stage pipelines in YAML — the primary workflow for bulk asset
generation. All intermediate outputs are saved: every stage gets its own
subdirectory with images and metadata JSON files.

```bash
# Portrait pipeline: generate → emotion variants → remove bg → upscale
vnasset pipeline --config examples/portrait.yaml

# Background batch: generate many images from independent prompts
vnasset pipeline --config examples/backgrounds.yaml

# Force re-run all stages (default: skip items whose output files exist)
vnasset pipeline --config examples/portrait.yaml --force
```

Config structure:

```yaml
session:
  sdxl_checkpoint: models/novaAnimeXL_ilV190.safetensors
  edit_model: models/qwen_image_edit_2509_fp8_e4m3fn.safetensors
  edit_lora: models/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16.safetensors  # optional

output_dir: output/my_pipeline

defaults:
  generate:
    steps: 20
    cfg: 4.5
    negative_prompt: "deformed, ugly, bad quality, lowres"
  edit:
    steps: 4
    cfg: 1.0

stages:
  # Independent batch: N prompts → N images
  - id: characters
    generate:
      - id: heroine
        prompt: "1girl, red hair, school uniform, portrait"
        seed: 42

  # Fan-out cross-product: each input × each prompt
  - id: expressions
    edit:
      input: characters
      prompts:
        - id: smile
          text: "make her smile happily"
        - id: angry
          text: "make her look angry"

  # 1:1 passthrough: each input → one output
  - id: nobg
    remove_bg:
      input: expressions

  - id: final
    upscale:
      input: nobg
      scale: 2
```

Output structure (every stage saved):

```
output/my_pipeline/
  pipeline.json              ← summary (stages, timings, skip/done counts)
  characters/                ← stage 1
    heroine.png
    heroine.json
  expressions/               ← stage 2 (cross-product: {input}_{prompt})
    heroine_smile.png
    heroine_angry.png
    ...
  nobg/                      ← stage 3
    heroine_smile_nobg.png
    ...
  final/                     ← stage 4
    heroine_smile_nobg_x2.png
    ...
```

| Stage type | Input | Routing |
|-----------|-------|---------|
| `generate` | none | list of prompts → list of images (1:1 per item) |
| `edit` | previous stage | cross-product: each input × each prompt |
| `remove_bg` | previous stage | 1:1 passthrough |
| `upscale` | previous stage | 1:1 passthrough |

Resume: if an output file already exists, that item is skipped. Use `--force`
to re-run everything. This lets you add items to a stage and re-run without
regenerating existing work.

See `examples/portrait.yaml` and `examples/backgrounds.yaml` for ready-to-use
configs.

### Generate (SDXL text-to-image)

```bash
vnasset generate \
  --checkpoint models/novaAnimeXL_ilV190.safetensors \
  --prompt "1girl, solo, red hair, glasses, blue eyes, white crop top, standing, portrait" \
  --negative-prompt "deformed, ugly, bad quality, lowres" \
  --steps 20 \
  --seed 42 \
  --output output/character_base.png
```

| Option | Default | Description |
|--------|---------|-------------|
| `--checkpoint` | (required) | Path to SDXL `.safetensors` |
| `--prompt` | (required) | Positive prompt |
| `--negative-prompt` | `""` | Negative prompt |
| `--width` | `1024` | Image width |
| `--height` | `1024` | Image height |
| `--steps` | `20` | Inference steps |
| `--cfg` | `4.5` | CFG scale |
| `--seed` | `random` | RNG seed (integer or `random`) |
| `--output` | `output.png` | Output path |
| `--raw` | `false` | Disable Compel prompt weighting (fall back to plain diffusers encoding) |

### Edit (Qwen Image Edit)

```bash
vnasset edit \
  --model models/qwen_image_edit_2509_fp8_e4m3fn.safetensors \
  --input character_base.png \
  --prompt "make her smile happily" \
  --steps 4 --cfg 1.0 \
  --lora models/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16.safetensors \
  --output character_happy.png
```

| Option | Default | Description |
|--------|---------|-------------|
| `--model` | (required) | Path to Qwen Image Edit `.safetensors` (FP8) |
| `--input` | (required) | Input image to edit |
| `--prompt` | (required) | Edit instruction |
| `--steps` | `20` | Inference steps (`4` with Lightning LoRA) |
| `--cfg` | `4.0` | CFG scale (`1.0` with Lightning LoRA) |
| `--seed` | `random` | RNG seed |
| `--lora` | (none) | Path to LoRA `.safetensors` |
| `--output` | `output.png` | Output path |

**Turbo mode:** Use the Lightning 4-step LoRA with `--steps 4 --cfg 1.0` to cut
inference time proportionally. The LoRA is fused into the transformer at load
time, so there is no per-step LoRA overhead.

### Output

Each generation produces:
- `{output}.png` — the image
- `{output}.json` — metadata (prompt, seed, model path, timing, resolution)

Directories in `--output` are created automatically.

### Session (persistent models)

For multi-call workflows, use `VnAssetsSession` to keep models loaded in GPU
memory between operations. Models are loaded eagerly at construction:

```python
from vnassets import VnAssetsSession

with VnAssetsSession(
    sdxl_checkpoint="models/novaAnimeXL_ilV190.safetensors",
    edit_model="models/qwen_image_edit_2509_fp8_e4m3fn.safetensors",
    edit_lora="models/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16.safetensors",
) as vna:
    vna.generate("1girl, red hair", output="base.png")
    vna.edit("base.png", "make her smile", output="happy.png")
    vna.edit("base.png", "make her sad", output="sad.png")
```

Either model can be omitted (`None`) for single-model sessions. Properties:
- `vna.has_sdxl` / `vna.has_qwen` — check which models are loaded
- `vna.load_time_s` — total session construction time
- `vna.close()` — manual cleanup (automatic with `with`)

The standalone `vnasset generate` and `vnasset edit` CLI commands are thin
wrappers around a one-shot session.

For bulk workflows, use `vnasset pipeline` instead — it handles the
session, outputs, and resume logic declaratively.

### Background Removal

Remove backgrounds from character sprites (output is RGBA PNG):

```bash
# Single file
vnasset remove-bg --input character_base.png --output character_transparent.png

# Batch (reuses model across files)
vnasset remove-bg --input base.png --input happy.png --input sad.png --output-dir transparent/
```

Or via the session API:

```python
with VnAssetsSession() as vna:
    vna.remove_background("base.png", output="base_transparent.png")

    # Batch
    vna.remove_backgrounds(
        ["base.png", "happy.png", "sad.png"],
        output_dir="transparent/",
    )
```

| Option | Default | Description |
|--------|---------|-------------|
| `--input` | (required, repeatable) | Input image path(s) |
| `--output` | (auto) | Output path (single mode) |
| `--output-dir` | (none) | Output directory (batch mode) |
| `--model` | `isnet-anime` | Model: `isnet-anime`, `u2net`, `u2netp`, `u2net_human_seg`, `isnet-general-use`, `sam` |

The default `isnet-anime` model is trained on anime images — ideal for the
novaAnimeXL art style. Inference runs on CPU via onnxruntime (~0.25s per
1024×1024 image on Strix Halo). The model is ~176 MB, downloaded on first use
to `~/.u2net/`.

### Upscaling (Real-ESRGAN)

Upscale images 2×, 3×, or 4× using Real-ESRGAN with the anime-optimized
`RealESRGAN_x4plus_anime_6B` model:

```bash
# Single file
vnasset upscale --input character_base.png --output character_2x.png --scale 2

# Batch (reuses model across files)
vnasset upscale --input base.png --input happy.png --input sad.png --output-dir upscaled/ --scale 2
```

Or via the session API:

```python
with VnAssetsSession() as vna:
    vna.upscale("base.png", output="base_2x.png", scale=2)

    # Batch
    vna.upscales(
        ["base.png", "happy.png", "sad.png"],
        output_dir="upscaled/",
        scale=2,
    )
```

| Option | Default | Description |
|--------|---------|-------------|
| `--input` | (required, repeatable) | Input image path(s) |
| `--output` | (auto) | Output path (single mode) |
| `--output-dir` | (none) | Output directory (batch mode; `{stem}_x{scale}.png`) |
| `--scale` | `2` | Upscale factor: `2`, `3`, or `4` |

The model is ~17 MB (auto-downloaded from GitHub on first use). The upsampler
runs in FP32 on GPU and is lazy-loaded on first call within a session — it
coexists with SDXL and Qwen without memory pressure. A 256×256 warmup tile is
run at load time to compile CUDA kernels, so first-user upscale doesn't incur
a compilation penalty.

## Architecture

```
┌──────────────────────────────────────────────────────────────────┐
│                        VnAssetsSession                            │
│                                                                   │
│  ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐      │
│  │ SDXL     │   │ Qwen     │   │ Qwen VL  │   │ Qwen     │      │
│  │ UNet     │   │ Transf.  │   │ 7B TE    │   │ VAE      │      │
│  │ (~3.5GB) │   │ (~20GB)  │   │ (~14GB)  │   │ (~1GB)   │      │
│  └────┬─────┘   └────┬─────┘   └────┬─────┘   └────┬─────┘      │
│       │              │              │              │             │
│       ▼              ▼              ▼              ▼             │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │                   Pipeline Runner                           │  │
│  │                                                             │  │
│  │  stage 1: generate ──► stage 2: edit ──► stage 3: remove_bg │
│  │       │                    │                    │            │
│  │       ▼                    ▼                    ▼            │
│  │  characters/          expressions/           nobg/          │
│  │    heroine.png          heroine_smile.png     ..._nobg.png  │
│  │    heroine.json         ...                  ...            │
│  │                                       │                     │
│  │                                       ▼                     │
│  │                               stage 4: upscale              │
│  │                                       │                     │
│  │                                       ▼                     │
│  │                                  final/                    │
│  │                                    ..._nobg_x2.png          │
│  └────────────────────────────────────────────────────────────┘  │
│                                                                   │
│  ┌─────────────┐  ┌──────────────┐                                │
│  │ Upscale     │  │ Remove BG    │   (lazy-loaded on first use)   │
│  │ Real-ESRGAN │  │ isnet-anime  │                                │
│  │ (~17 MB)    │  │ (~176 MB)    │                                │
│  └─────────────┘  └──────────────┘                                │
└──────────────────────────────────────────────────────────────────┘
```

The Qwen transformer is loaded FP8 → BF16 at construction using
`init_empty_weights` + incremental conversion to keep peak memory manageable
(20B parameters: 20 GB FP8 on disk, ~40 GB BF16 at runtime). All models fit
comfortably in 128 GB unified memory — no offloading, no swapping.

SDXL and Qwen use **separate VAEs** with different latent spaces. The SDXL
checkpoint bundles its own VAE; Qwen uses `Qwen/Qwen-Image` VAE from
HuggingFace. Both coexist in the same session without conflict.

The upsampler (Real-ESRGAN RRDBNet, ~17 MB, FP32) runs on GPU via PyTorch
and is lazy-loaded on first use. It uses tiled processing (512×512 tiles)
to keep memory modest even at 4× output (4096×4096).

## Data Flow

### Generate Phase

```
prompt text ──► Compel (SDXL CLIPs) ──► conditioning ──┐
                                                        ├──► SDXL UNet (N steps) ──► latent ──► SDXL VAE decode ──► image
noise + latent ─────────────────────────────────────────┘
```

### Edit Phase

```
input image ──► Qwen VAE encode ──► latent ──┐
                                              │
prompt text ──► Qwen VL 7B TE ──► conditioning┤
                                              ├──► Qwen Transformer (N steps) ──► latent ──► Qwen VAE decode ──► image
input image ──► Qwen VL 7B TE ──► visual tok.┘
```

The text encoder handles both text conditioning and visual token encoding from
the input image.

### Upscale Phase

```
input image (RGB) ──► tile split (512×512) ──┐
                                              ├──► RRDBNet (FP32, 4×) ──► tiles ──► blend ──► upscaled image
tile overlap padding ─────────────────────────┘
```

If the requested output scale is less than the model's native scale (e.g. 2×
from a 4× model), the 4× result is Lanczos-downsampled to the target size.
Tiled processing keeps GPU memory constant regardless of output resolution.

### Turbo vs Normal Mode

| Parameter | Normal | Turbo (Lightning LoRA) |
|-----------|--------|------------------------|
| Steps | 20 | 4 |
| CFG | 4.0 | 1.0 |
| LoRA | none | Lightning-4steps (fused at load) |
| Sampler | Flow Match Euler | Flow Match Euler |

## Performance

Hardware: **AMD Strix Halo** (Ryzen AI Max 395 Pro, Radeon 8060S, 128 GB),
bfloat16, `novaAnimeXL_ilV190.safetensors`.

### Generate (SDXL)

| Resolution | Steps | Load (s) | Inference (s) | Total (s) |
|-----------|-------|----------|---------------|-----------|
| 1024×1024 | 20 | ~2.5 | ~29 | ~31 |

Per-step breakdown (1024×1024):
- Step 1 (warmup): ~1.3–1.6s
- Steps 2–20 (steady): ~1.0–1.3s each
- VAE decode included in final step

Compel prompt weighting adds ~0.4s encoding overhead — negligible.

### Edit (Qwen Image Edit, 4-step Lightning LoRA)

| Attention | Steps | Load (s) | Inference (s) | Total (s) |
|-----------|-------|----------|---------------|-----------|
| Matmul fallback | 4 | ~28 | ~87 | ~115 |
| Flash attention | 4 | ~31 | ~53 | ~84 |

Per-step breakdown (1024×1024, flash attention):
- Step 1: ~0.3s (prefill/encoding)
- Steps 2–4: ~1.6 → ~5.1 → ~7.0s (transformer + VAE)

Flash attention (`TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1`) cuts edit
inference by **1.63×**. SDXL generate is unaffected (uses its own attention
processor, not SDPA).

### Session (persistent models)

| Phase | Wall (s) | Notes |
|-------|----------|-------|
| Session load | ~28 | SDXL + Qwen transformer + VAE + TE + LoRA fuse |
| Generate | ~31 | SDXL 20-step, 1024×1024 |
| Edit (turbo) | ~87 / ~54 | Matmul / flash attention |
| Upscale (2×) | ~1.8 | Real-ESRGAN, 1024→2048 |
| **Total (3 ops)** | **~147 / ~119** | One session load amortized |

With a session, each additional edit saves one model-load round trip (~28s).
5 edits save 112s. Flash attention adds a further 1.6× multiplier on inference.

See [`docs/stats.md`](docs/stats.md) for detailed per-run breakdowns.

### Upscale (Real-ESRGAN, anime model)

| Scale | Input | Tiles | Load (s) | Inference (s) |
|-------|-------|-------|----------|---------------|
| 2× | 1024×1024 | 4 | ~0.2¹ | ~1.8 |
| 4× | 1024×1024 | 4 | ~0.2¹ | ~1.8 |

¹ Warmup tile only; model download is ~17 MB (first use, cached thereafter).

Upscaling is tiled (512×512 input tiles) to keep GPU memory modest — the
RRDBNet forward pass processes each tile independently, and the results are
blended at tile boundaries. The 2× and 4× paths have near-identical latency
because the model is natively 4×; 2× output is achieved by upscaling to 4×
then downsampling.

A 256×256 warmup tile is run at model load time to compile CUDA kernels.
Without it, the first real upscale incurs ~80s of JIT compilation.

## Technical Notes

### bfloat16 required on RDNA 3.5

`float16` causes GPU kernel crashes (segfault) on the Radeon 8060S. The tool
uses `bfloat16` internally. This is transparent to the user.

### FP8 → BF16 conversion

The Qwen Image Edit model ships as FP8 (`fp8_e4m3fn`). It is converted to BF16
at load time using `init_empty_weights` + incremental tensor conversion to keep
peak memory manageable. RDNA 3.5 WMMA supports FP8 compute, but PyTorch ROCm FP8
support is not yet mature enough to compute in FP8 — upcast to BF16 is the safe
path and still fits in 128 GB.

### Attention backends

Two attention paths, selected automatically:

| Path | When | Performance |
|------|------|-------------|
| **Flash attention** | `TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1` set | 1.63× faster edits |
| **Matmul fallback** | Default (env var not set) | Stable, slower |

SDXL always uses a custom `AttnProcessor` (direct QKV matmul) regardless of the
env var — its attention path is separate from the Qwen SDPA dispatch. The flash
attention toggle only affects the Qwen transformer and text encoder.

```bash
# Enable flash attention for the session
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 vnasset edit ...
```

### ROCm torch version

Tested with `torch 2.11.0+rocm7.2`. Newer ROCm nightlies (2.13+, 2.14+) may
cause GPU crashes. If you encounter segfaults, try matching this version.

### LoRA loading

Lightning LoRA is loaded via `diffusers` native `load_lora_weights()` and fused
into the transformer with `fuse_lora()` at session construction. The fusion
takes ~1.5s and happens once — the turbo/normal switch is then just a
steps+CFG change with no per-step LoRA overhead.

## Prompt Syntax

VNAsset supports **ComfyUI-style prompt weighting** via the `compel` library.

### Weighting

| Syntax | Effect |
|--------|--------|
| `(word)` | Boost ×1.1 |
| `(word:1.5)` | Boost ×1.5 |
| `(word:0.6)` | De-emphasize ×0.6 |
| `[word]` | De-emphasize ×0.9 (shorthand) |
| `\(word\)` | Literal parentheses (escaped) |

```bash
vnasset generate \
  --checkpoint models/novaAnimeXL_ilV190.safetensors \
  --prompt "(masterpiece:1.2), 1girl, (red hair:1.3), blue eyes, [glasses]" \
  --negative-prompt "(bad quality, worst quality:1.4)" \
  --steps 20 --seed 42
```

### BREAK (condition chunking)

Split the prompt into independent conditioning chunks with `BREAK`:

```bash
vnasset generate \
  --prompt "1girl, red hair, standing BREAK blue sky, cherry blossoms" \
  --steps 20 --seed 42
```

Use `--raw` to bypass weighting and fall back to plain diffusers encoding.

## Current State

| Feature | Status |
|---------|--------|
| `vnasset pipeline` (batch YAML config) | ✅ Working |
| `vnasset generate` | ✅ Working |
| `vnasset edit` | ✅ Working |
| `VnAssetsSession` (persistent models) | ✅ Working |
| Compel prompt weighting + BREAK | ✅ Working |
| Lightning LoRA fuse-at-load | ✅ Working |
| Flash attention (experimental) | ✅ Working |
| `vnasset remove-bg` | ✅ Working |
| `vnasset upscale` | ✅ Working |
| Self-contained torch wheel | ✅ Working |
| `vnasset serve` (daemon/HTTP API) | 🚧 Planned |
| `torch.compile` on UNet | 🔬 Needs spike (see [decisions](docs/optimization-decisions.md)) |
| Shared encode optimization | 📋 Analyzed, deferred (see [decisions](docs/optimization-decisions.md)) |

## Future Improvements

- **`vnasset serve`** — lightweight daemon with Unix socket or HTTP API for
  integrating VNAsset into external tools.
- **`torch.compile` on UNet** — the UNet forward is identical each step, but
  ROCm `torch.compile` maturity is uncertain. Needs a 30-minute spike to measure
  actual speedup and check for graph breaks / crashes on RDNA 3.5. If the spike
  shows 15%+ and stable, implement with a warmup step at session load.
- **Shared encode optimization** — for N edit variants of the same input image,
  saves ~1.2–2.3s per variant (VAE encode + vision encoder). The dominant cost
  (transformer denoising, ~45–50s) cannot be shared, so the overall gain is
  2–4%. Deferred until workload shape demands it (100+ variants). When built,
  the right place is the pipeline runner, not the session API.

See [`docs/optimization-decisions.md`](docs/optimization-decisions.md) for
full tradeoff analysis with data and cost estimates.