9c4474b93c
Detail the shared-encode and torch.compile tradeoffs with data-driven estimates from the pipeline code. Shared encode saves ~1.2-2.3s per variant (2-4% total) and is deferred; torch.compile needs a 30-minute ROCm spike before deciding. Update Current State and Future Improvements to reflect the analysis.
608 lines
23 KiB
Markdown
608 lines
23 KiB
Markdown
# VNAsset
|
||
|
||
Fast CLI pipeline for visual novel image asset generation.
|
||
|
||
Drop-in replacement for the ComfyUI workflow loop: generate base character sprites
|
||
with SDXL, then batch-edit variants (expressions, outfits) with Qwen Image Edit —
|
||
all in one warm session, no node-graph overhead.
|
||
|
||
## Hardware
|
||
|
||
Built for **AMD Strix Halo** (Ryzen AI Max 395 Pro, Radeon 8060S, 128 GB unified
|
||
memory). Also works on discrete AMD GPUs with ROCm. NVIDIA support is untested
|
||
but should work if you swap the torch backend.
|
||
|
||
The 128 GB unified memory means VRAM is effectively unlimited (up to ~96 GB
|
||
allocatable to GPU). The bottleneck is **GPU compute throughput**, not memory
|
||
capacity — model offloading is pointless, everything stays resident.
|
||
|
||
## Install
|
||
|
||
```bash
|
||
git clone <repo> vnassets
|
||
cd vnassets
|
||
|
||
# Create venv (Python 3.12 required for ROCm torch compatibility)
|
||
python3.12 -m venv .venv
|
||
source .venv/bin/activate
|
||
|
||
# Install ROCm PyTorch from local wheels (no remote index dependency)
|
||
pip install wheels/torch-2.11.0+rocm7.2-cp312-cp312-manylinux_2_28_x86_64.whl \
|
||
wheels/triton_rocm-3.6.0-cp312-cp312-linux_x86_64.whl \
|
||
wheels/torchvision-0.26.0+rocm7.2-cp312-cp312-manylinux_2_28_x86_64.whl
|
||
|
||
# Install the rest
|
||
pip install -e .
|
||
```
|
||
|
||
The torch, triton-rocm, and torchvision wheels are bundled in `wheels/`. If
|
||
you don't have them yet (e.g. after a fresh clone), download them first:
|
||
|
||
```bash
|
||
mkdir -p wheels
|
||
python3.12 -m pip download torch==2.11.0 triton-rocm==3.6.0 torchvision==0.26.0 \
|
||
--index-url https://download.pytorch.org/whl/rocm7.2 \
|
||
--dest wheels --no-deps
|
||
```
|
||
|
||
### Models
|
||
|
||
Symlink your ComfyUI models into `models/`:
|
||
|
||
```bash
|
||
cd models
|
||
ln -s /path/to/ComfyUI/models/checkpoints/novaAnimeXL_ilV190.safetensors .
|
||
ln -s /path/to/ComfyUI/models/diffusion_models/qwen_image_edit_2509_fp8_e4m3fn.safetensors .
|
||
ln -s /path/to/ComfyUI/models/loras/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16.safetensors .
|
||
```
|
||
|
||
The Qwen VAE and text encoder are downloaded automatically from HuggingFace Hub
|
||
on first use (`Qwen/Qwen-Image` and `Qwen/Qwen2.5-VL-7B-Instruct`). No symlinks
|
||
needed for those.
|
||
|
||
Or place the actual files there — the tool just reads whatever safetensors you
|
||
point it at.
|
||
|
||
## Usage
|
||
|
||
### Pipeline (batch YAML config)
|
||
|
||
Declare multi-stage pipelines in YAML — the primary workflow for bulk asset
|
||
generation. All intermediate outputs are saved: every stage gets its own
|
||
subdirectory with images and metadata JSON files.
|
||
|
||
```bash
|
||
# Portrait pipeline: generate → emotion variants → remove bg → upscale
|
||
vnasset pipeline --config examples/portrait.yaml
|
||
|
||
# Background batch: generate many images from independent prompts
|
||
vnasset pipeline --config examples/backgrounds.yaml
|
||
|
||
# Force re-run all stages (default: skip items whose output files exist)
|
||
vnasset pipeline --config examples/portrait.yaml --force
|
||
```
|
||
|
||
Config structure:
|
||
|
||
```yaml
|
||
session:
|
||
sdxl_checkpoint: models/novaAnimeXL_ilV190.safetensors
|
||
edit_model: models/qwen_image_edit_2509_fp8_e4m3fn.safetensors
|
||
edit_lora: models/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16.safetensors # optional
|
||
|
||
output_dir: output/my_pipeline
|
||
|
||
defaults:
|
||
generate:
|
||
steps: 20
|
||
cfg: 4.5
|
||
negative_prompt: "deformed, ugly, bad quality, lowres"
|
||
edit:
|
||
steps: 4
|
||
cfg: 1.0
|
||
|
||
stages:
|
||
# Independent batch: N prompts → N images
|
||
- id: characters
|
||
generate:
|
||
- id: heroine
|
||
prompt: "1girl, red hair, school uniform, portrait"
|
||
seed: 42
|
||
|
||
# Fan-out cross-product: each input × each prompt
|
||
- id: expressions
|
||
edit:
|
||
input: characters
|
||
prompts:
|
||
- id: smile
|
||
text: "make her smile happily"
|
||
- id: angry
|
||
text: "make her look angry"
|
||
|
||
# 1:1 passthrough: each input → one output
|
||
- id: nobg
|
||
remove_bg:
|
||
input: expressions
|
||
|
||
- id: final
|
||
upscale:
|
||
input: nobg
|
||
scale: 2
|
||
```
|
||
|
||
Output structure (every stage saved):
|
||
|
||
```
|
||
output/my_pipeline/
|
||
pipeline.json ← summary (stages, timings, skip/done counts)
|
||
characters/ ← stage 1
|
||
heroine.png
|
||
heroine.json
|
||
expressions/ ← stage 2 (cross-product: {input}_{prompt})
|
||
heroine_smile.png
|
||
heroine_angry.png
|
||
...
|
||
nobg/ ← stage 3
|
||
heroine_smile_nobg.png
|
||
...
|
||
final/ ← stage 4
|
||
heroine_smile_nobg_x2.png
|
||
...
|
||
```
|
||
|
||
| Stage type | Input | Routing |
|
||
|-----------|-------|---------|
|
||
| `generate` | none | list of prompts → list of images (1:1 per item) |
|
||
| `edit` | previous stage | cross-product: each input × each prompt |
|
||
| `remove_bg` | previous stage | 1:1 passthrough |
|
||
| `upscale` | previous stage | 1:1 passthrough |
|
||
|
||
Resume: if an output file already exists, that item is skipped. Use `--force`
|
||
to re-run everything. This lets you add items to a stage and re-run without
|
||
regenerating existing work.
|
||
|
||
See `examples/portrait.yaml` and `examples/backgrounds.yaml` for ready-to-use
|
||
configs.
|
||
|
||
### Generate (SDXL text-to-image)
|
||
|
||
```bash
|
||
vnasset generate \
|
||
--checkpoint models/novaAnimeXL_ilV190.safetensors \
|
||
--prompt "1girl, solo, red hair, glasses, blue eyes, white crop top, standing, portrait" \
|
||
--negative-prompt "deformed, ugly, bad quality, lowres" \
|
||
--steps 20 \
|
||
--seed 42 \
|
||
--output output/character_base.png
|
||
```
|
||
|
||
| Option | Default | Description |
|
||
|--------|---------|-------------|
|
||
| `--checkpoint` | (required) | Path to SDXL `.safetensors` |
|
||
| `--prompt` | (required) | Positive prompt |
|
||
| `--negative-prompt` | `""` | Negative prompt |
|
||
| `--width` | `1024` | Image width |
|
||
| `--height` | `1024` | Image height |
|
||
| `--steps` | `20` | Inference steps |
|
||
| `--cfg` | `4.5` | CFG scale |
|
||
| `--seed` | `random` | RNG seed (integer or `random`) |
|
||
| `--output` | `output.png` | Output path |
|
||
| `--raw` | `false` | Disable Compel prompt weighting (fall back to plain diffusers encoding) |
|
||
|
||
### Edit (Qwen Image Edit)
|
||
|
||
```bash
|
||
vnasset edit \
|
||
--model models/qwen_image_edit_2509_fp8_e4m3fn.safetensors \
|
||
--input character_base.png \
|
||
--prompt "make her smile happily" \
|
||
--steps 4 --cfg 1.0 \
|
||
--lora models/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16.safetensors \
|
||
--output character_happy.png
|
||
```
|
||
|
||
| Option | Default | Description |
|
||
|--------|---------|-------------|
|
||
| `--model` | (required) | Path to Qwen Image Edit `.safetensors` (FP8) |
|
||
| `--input` | (required) | Input image to edit |
|
||
| `--prompt` | (required) | Edit instruction |
|
||
| `--steps` | `20` | Inference steps (`4` with Lightning LoRA) |
|
||
| `--cfg` | `4.0` | CFG scale (`1.0` with Lightning LoRA) |
|
||
| `--seed` | `random` | RNG seed |
|
||
| `--lora` | (none) | Path to LoRA `.safetensors` |
|
||
| `--output` | `output.png` | Output path |
|
||
|
||
**Turbo mode:** Use the Lightning 4-step LoRA with `--steps 4 --cfg 1.0` to cut
|
||
inference time proportionally. The LoRA is fused into the transformer at load
|
||
time, so there is no per-step LoRA overhead.
|
||
|
||
### Output
|
||
|
||
Each generation produces:
|
||
- `{output}.png` — the image
|
||
- `{output}.json` — metadata (prompt, seed, model path, timing, resolution)
|
||
|
||
Directories in `--output` are created automatically.
|
||
|
||
### Session (persistent models)
|
||
|
||
For multi-call workflows, use `VnAssetsSession` to keep models loaded in GPU
|
||
memory between operations. Models are loaded eagerly at construction:
|
||
|
||
```python
|
||
from vnassets import VnAssetsSession
|
||
|
||
with VnAssetsSession(
|
||
sdxl_checkpoint="models/novaAnimeXL_ilV190.safetensors",
|
||
edit_model="models/qwen_image_edit_2509_fp8_e4m3fn.safetensors",
|
||
edit_lora="models/Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16.safetensors",
|
||
) as vna:
|
||
vna.generate("1girl, red hair", output="base.png")
|
||
vna.edit("base.png", "make her smile", output="happy.png")
|
||
vna.edit("base.png", "make her sad", output="sad.png")
|
||
```
|
||
|
||
Either model can be omitted (`None`) for single-model sessions. Properties:
|
||
- `vna.has_sdxl` / `vna.has_qwen` — check which models are loaded
|
||
- `vna.load_time_s` — total session construction time
|
||
- `vna.close()` — manual cleanup (automatic with `with`)
|
||
|
||
The standalone `vnasset generate` and `vnasset edit` CLI commands are thin
|
||
wrappers around a one-shot session.
|
||
|
||
For bulk workflows, use `vnasset pipeline` instead — it handles the
|
||
session, outputs, and resume logic declaratively.
|
||
|
||
### Background Removal
|
||
|
||
Remove backgrounds from character sprites (output is RGBA PNG):
|
||
|
||
```bash
|
||
# Single file
|
||
vnasset remove-bg --input character_base.png --output character_transparent.png
|
||
|
||
# Batch (reuses model across files)
|
||
vnasset remove-bg --input base.png --input happy.png --input sad.png --output-dir transparent/
|
||
```
|
||
|
||
Or via the session API:
|
||
|
||
```python
|
||
with VnAssetsSession() as vna:
|
||
vna.remove_background("base.png", output="base_transparent.png")
|
||
|
||
# Batch
|
||
vna.remove_backgrounds(
|
||
["base.png", "happy.png", "sad.png"],
|
||
output_dir="transparent/",
|
||
)
|
||
```
|
||
|
||
| Option | Default | Description |
|
||
|--------|---------|-------------|
|
||
| `--input` | (required, repeatable) | Input image path(s) |
|
||
| `--output` | (auto) | Output path (single mode) |
|
||
| `--output-dir` | (none) | Output directory (batch mode) |
|
||
| `--model` | `isnet-anime` | Model: `isnet-anime`, `u2net`, `u2netp`, `u2net_human_seg`, `isnet-general-use`, `sam` |
|
||
|
||
The default `isnet-anime` model is trained on anime images — ideal for the
|
||
novaAnimeXL art style. Inference runs on CPU via onnxruntime (~0.25s per
|
||
1024×1024 image on Strix Halo). The model is ~176 MB, downloaded on first use
|
||
to `~/.u2net/`.
|
||
|
||
### Upscaling (Real-ESRGAN)
|
||
|
||
Upscale images 2×, 3×, or 4× using Real-ESRGAN with the anime-optimized
|
||
`RealESRGAN_x4plus_anime_6B` model:
|
||
|
||
```bash
|
||
# Single file
|
||
vnasset upscale --input character_base.png --output character_2x.png --scale 2
|
||
|
||
# Batch (reuses model across files)
|
||
vnasset upscale --input base.png --input happy.png --input sad.png --output-dir upscaled/ --scale 2
|
||
```
|
||
|
||
Or via the session API:
|
||
|
||
```python
|
||
with VnAssetsSession() as vna:
|
||
vna.upscale("base.png", output="base_2x.png", scale=2)
|
||
|
||
# Batch
|
||
vna.upscales(
|
||
["base.png", "happy.png", "sad.png"],
|
||
output_dir="upscaled/",
|
||
scale=2,
|
||
)
|
||
```
|
||
|
||
| Option | Default | Description |
|
||
|--------|---------|-------------|
|
||
| `--input` | (required, repeatable) | Input image path(s) |
|
||
| `--output` | (auto) | Output path (single mode) |
|
||
| `--output-dir` | (none) | Output directory (batch mode; `{stem}_x{scale}.png`) |
|
||
| `--scale` | `2` | Upscale factor: `2`, `3`, or `4` |
|
||
|
||
The model is ~17 MB (auto-downloaded from GitHub on first use). The upsampler
|
||
runs in FP32 on GPU and is lazy-loaded on first call within a session — it
|
||
coexists with SDXL and Qwen without memory pressure. A 256×256 warmup tile is
|
||
run at load time to compile CUDA kernels, so first-user upscale doesn't incur
|
||
a compilation penalty.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ VnAssetsSession │
|
||
│ │
|
||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||
│ │ SDXL │ │ Qwen │ │ Qwen VL │ │ Qwen │ │
|
||
│ │ UNet │ │ Transf. │ │ 7B TE │ │ VAE │ │
|
||
│ │ (~3.5GB) │ │ (~20GB) │ │ (~14GB) │ │ (~1GB) │ │
|
||
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
|
||
│ │ │ │ │ │
|
||
│ ▼ ▼ ▼ ▼ │
|
||
│ ┌────────────────────────────────────────────────────────────┐ │
|
||
│ │ Pipeline Runner │ │
|
||
│ │ │ │
|
||
│ │ stage 1: generate ──► stage 2: edit ──► stage 3: remove_bg │
|
||
│ │ │ │ │ │
|
||
│ │ ▼ ▼ ▼ │
|
||
│ │ characters/ expressions/ nobg/ │
|
||
│ │ heroine.png heroine_smile.png ..._nobg.png │
|
||
│ │ heroine.json ... ... │
|
||
│ │ │ │
|
||
│ │ ▼ │
|
||
│ │ stage 4: upscale │
|
||
│ │ │ │
|
||
│ │ ▼ │
|
||
│ │ final/ │
|
||
│ │ ..._nobg_x2.png │
|
||
│ └────────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ ┌─────────────┐ ┌──────────────┐ │
|
||
│ │ Upscale │ │ Remove BG │ (lazy-loaded on first use) │
|
||
│ │ Real-ESRGAN │ │ isnet-anime │ │
|
||
│ │ (~17 MB) │ │ (~176 MB) │ │
|
||
│ └─────────────┘ └──────────────┘ │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
The Qwen transformer is loaded FP8 → BF16 at construction using
|
||
`init_empty_weights` + incremental conversion to keep peak memory manageable
|
||
(20B parameters: 20 GB FP8 on disk, ~40 GB BF16 at runtime). All models fit
|
||
comfortably in 128 GB unified memory — no offloading, no swapping.
|
||
|
||
SDXL and Qwen use **separate VAEs** with different latent spaces. The SDXL
|
||
checkpoint bundles its own VAE; Qwen uses `Qwen/Qwen-Image` VAE from
|
||
HuggingFace. Both coexist in the same session without conflict.
|
||
|
||
The upsampler (Real-ESRGAN RRDBNet, ~17 MB, FP32) runs on GPU via PyTorch
|
||
and is lazy-loaded on first use. It uses tiled processing (512×512 tiles)
|
||
to keep memory modest even at 4× output (4096×4096).
|
||
|
||
## Data Flow
|
||
|
||
### Generate Phase
|
||
|
||
```
|
||
prompt text ──► Compel (SDXL CLIPs) ──► conditioning ──┐
|
||
├──► SDXL UNet (N steps) ──► latent ──► SDXL VAE decode ──► image
|
||
noise + latent ─────────────────────────────────────────┘
|
||
```
|
||
|
||
### Edit Phase
|
||
|
||
```
|
||
input image ──► Qwen VAE encode ──► latent ──┐
|
||
│
|
||
prompt text ──► Qwen VL 7B TE ──► conditioning┤
|
||
├──► Qwen Transformer (N steps) ──► latent ──► Qwen VAE decode ──► image
|
||
input image ──► Qwen VL 7B TE ──► visual tok.┘
|
||
```
|
||
|
||
The text encoder handles both text conditioning and visual token encoding from
|
||
the input image.
|
||
|
||
### Upscale Phase
|
||
|
||
```
|
||
input image (RGB) ──► tile split (512×512) ──┐
|
||
├──► RRDBNet (FP32, 4×) ──► tiles ──► blend ──► upscaled image
|
||
tile overlap padding ─────────────────────────┘
|
||
```
|
||
|
||
If the requested output scale is less than the model's native scale (e.g. 2×
|
||
from a 4× model), the 4× result is Lanczos-downsampled to the target size.
|
||
Tiled processing keeps GPU memory constant regardless of output resolution.
|
||
|
||
### Turbo vs Normal Mode
|
||
|
||
| Parameter | Normal | Turbo (Lightning LoRA) |
|
||
|-----------|--------|------------------------|
|
||
| Steps | 20 | 4 |
|
||
| CFG | 4.0 | 1.0 |
|
||
| LoRA | none | Lightning-4steps (fused at load) |
|
||
| Sampler | Flow Match Euler | Flow Match Euler |
|
||
|
||
## Performance
|
||
|
||
Hardware: **AMD Strix Halo** (Ryzen AI Max 395 Pro, Radeon 8060S, 128 GB),
|
||
bfloat16, `novaAnimeXL_ilV190.safetensors`.
|
||
|
||
### Generate (SDXL)
|
||
|
||
| Resolution | Steps | Load (s) | Inference (s) | Total (s) |
|
||
|-----------|-------|----------|---------------|-----------|
|
||
| 1024×1024 | 20 | ~2.5 | ~29 | ~31 |
|
||
|
||
Per-step breakdown (1024×1024):
|
||
- Step 1 (warmup): ~1.3–1.6s
|
||
- Steps 2–20 (steady): ~1.0–1.3s each
|
||
- VAE decode included in final step
|
||
|
||
Compel prompt weighting adds ~0.4s encoding overhead — negligible.
|
||
|
||
### Edit (Qwen Image Edit, 4-step Lightning LoRA)
|
||
|
||
| Attention | Steps | Load (s) | Inference (s) | Total (s) |
|
||
|-----------|-------|----------|---------------|-----------|
|
||
| Matmul fallback | 4 | ~28 | ~87 | ~115 |
|
||
| Flash attention | 4 | ~31 | ~53 | ~84 |
|
||
|
||
Per-step breakdown (1024×1024, flash attention):
|
||
- Step 1: ~0.3s (prefill/encoding)
|
||
- Steps 2–4: ~1.6 → ~5.1 → ~7.0s (transformer + VAE)
|
||
|
||
Flash attention (`TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1`) cuts edit
|
||
inference by **1.63×**. SDXL generate is unaffected (uses its own attention
|
||
processor, not SDPA).
|
||
|
||
### Session (persistent models)
|
||
|
||
| Phase | Wall (s) | Notes |
|
||
|-------|----------|-------|
|
||
| Session load | ~28 | SDXL + Qwen transformer + VAE + TE + LoRA fuse |
|
||
| Generate | ~31 | SDXL 20-step, 1024×1024 |
|
||
| Edit (turbo) | ~87 / ~54 | Matmul / flash attention |
|
||
| Upscale (2×) | ~1.8 | Real-ESRGAN, 1024→2048 |
|
||
| **Total (3 ops)** | **~147 / ~119** | One session load amortized |
|
||
|
||
With a session, each additional edit saves one model-load round trip (~28s).
|
||
5 edits save 112s. Flash attention adds a further 1.6× multiplier on inference.
|
||
|
||
See [`docs/stats.md`](docs/stats.md) for detailed per-run breakdowns.
|
||
|
||
### Upscale (Real-ESRGAN, anime model)
|
||
|
||
| Scale | Input | Tiles | Load (s) | Inference (s) |
|
||
|-------|-------|-------|----------|---------------|
|
||
| 2× | 1024×1024 | 4 | ~0.2¹ | ~1.8 |
|
||
| 4× | 1024×1024 | 4 | ~0.2¹ | ~1.8 |
|
||
|
||
¹ Warmup tile only; model download is ~17 MB (first use, cached thereafter).
|
||
|
||
Upscaling is tiled (512×512 input tiles) to keep GPU memory modest — the
|
||
RRDBNet forward pass processes each tile independently, and the results are
|
||
blended at tile boundaries. The 2× and 4× paths have near-identical latency
|
||
because the model is natively 4×; 2× output is achieved by upscaling to 4×
|
||
then downsampling.
|
||
|
||
A 256×256 warmup tile is run at model load time to compile CUDA kernels.
|
||
Without it, the first real upscale incurs ~80s of JIT compilation.
|
||
|
||
## Technical Notes
|
||
|
||
### bfloat16 required on RDNA 3.5
|
||
|
||
`float16` causes GPU kernel crashes (segfault) on the Radeon 8060S. The tool
|
||
uses `bfloat16` internally. This is transparent to the user.
|
||
|
||
### FP8 → BF16 conversion
|
||
|
||
The Qwen Image Edit model ships as FP8 (`fp8_e4m3fn`). It is converted to BF16
|
||
at load time using `init_empty_weights` + incremental tensor conversion to keep
|
||
peak memory manageable. RDNA 3.5 WMMA supports FP8 compute, but PyTorch ROCm FP8
|
||
support is not yet mature enough to compute in FP8 — upcast to BF16 is the safe
|
||
path and still fits in 128 GB.
|
||
|
||
### Attention backends
|
||
|
||
Two attention paths, selected automatically:
|
||
|
||
| Path | When | Performance |
|
||
|------|------|-------------|
|
||
| **Flash attention** | `TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1` set | 1.63× faster edits |
|
||
| **Matmul fallback** | Default (env var not set) | Stable, slower |
|
||
|
||
SDXL always uses a custom `AttnProcessor` (direct QKV matmul) regardless of the
|
||
env var — its attention path is separate from the Qwen SDPA dispatch. The flash
|
||
attention toggle only affects the Qwen transformer and text encoder.
|
||
|
||
```bash
|
||
# Enable flash attention for the session
|
||
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 vnasset edit ...
|
||
```
|
||
|
||
### ROCm torch version
|
||
|
||
Tested with `torch 2.11.0+rocm7.2`. Newer ROCm nightlies (2.13+, 2.14+) may
|
||
cause GPU crashes. If you encounter segfaults, try matching this version.
|
||
|
||
### LoRA loading
|
||
|
||
Lightning LoRA is loaded via `diffusers` native `load_lora_weights()` and fused
|
||
into the transformer with `fuse_lora()` at session construction. The fusion
|
||
takes ~1.5s and happens once — the turbo/normal switch is then just a
|
||
steps+CFG change with no per-step LoRA overhead.
|
||
|
||
## Prompt Syntax
|
||
|
||
VNAsset supports **ComfyUI-style prompt weighting** via the `compel` library.
|
||
|
||
### Weighting
|
||
|
||
| Syntax | Effect |
|
||
|--------|--------|
|
||
| `(word)` | Boost ×1.1 |
|
||
| `(word:1.5)` | Boost ×1.5 |
|
||
| `(word:0.6)` | De-emphasize ×0.6 |
|
||
| `[word]` | De-emphasize ×0.9 (shorthand) |
|
||
| `\(word\)` | Literal parentheses (escaped) |
|
||
|
||
```bash
|
||
vnasset generate \
|
||
--checkpoint models/novaAnimeXL_ilV190.safetensors \
|
||
--prompt "(masterpiece:1.2), 1girl, (red hair:1.3), blue eyes, [glasses]" \
|
||
--negative-prompt "(bad quality, worst quality:1.4)" \
|
||
--steps 20 --seed 42
|
||
```
|
||
|
||
### BREAK (condition chunking)
|
||
|
||
Split the prompt into independent conditioning chunks with `BREAK`:
|
||
|
||
```bash
|
||
vnasset generate \
|
||
--prompt "1girl, red hair, standing BREAK blue sky, cherry blossoms" \
|
||
--steps 20 --seed 42
|
||
```
|
||
|
||
Use `--raw` to bypass weighting and fall back to plain diffusers encoding.
|
||
|
||
## Current State
|
||
|
||
| Feature | Status |
|
||
|---------|--------|
|
||
| `vnasset pipeline` (batch YAML config) | ✅ Working |
|
||
| `vnasset generate` | ✅ Working |
|
||
| `vnasset edit` | ✅ Working |
|
||
| `VnAssetsSession` (persistent models) | ✅ Working |
|
||
| Compel prompt weighting + BREAK | ✅ Working |
|
||
| Lightning LoRA fuse-at-load | ✅ Working |
|
||
| Flash attention (experimental) | ✅ Working |
|
||
| `vnasset remove-bg` | ✅ Working |
|
||
| `vnasset upscale` | ✅ Working |
|
||
| Self-contained torch wheel | ✅ Working |
|
||
| `vnasset serve` (daemon/HTTP API) | 🚧 Planned |
|
||
| `torch.compile` on UNet | 🔬 Needs spike (see [decisions](docs/optimization-decisions.md)) |
|
||
| Shared encode optimization | 📋 Analyzed, deferred (see [decisions](docs/optimization-decisions.md)) |
|
||
|
||
## Future Improvements
|
||
|
||
- **`vnasset serve`** — lightweight daemon with Unix socket or HTTP API for
|
||
integrating VNAsset into external tools.
|
||
- **`torch.compile` on UNet** — the UNet forward is identical each step, but
|
||
ROCm `torch.compile` maturity is uncertain. Needs a 30-minute spike to measure
|
||
actual speedup and check for graph breaks / crashes on RDNA 3.5. If the spike
|
||
shows 15%+ and stable, implement with a warmup step at session load.
|
||
- **Shared encode optimization** — for N edit variants of the same input image,
|
||
saves ~1.2–2.3s per variant (VAE encode + vision encoder). The dominant cost
|
||
(transformer denoising, ~45–50s) cannot be shared, so the overall gain is
|
||
2–4%. Deferred until workload shape demands it (100+ variants). When built,
|
||
the right place is the pipeline runner, not the session API.
|
||
|
||
See [`docs/optimization-decisions.md`](docs/optimization-decisions.md) for
|
||
full tradeoff analysis with data and cost estimates.
|