97ac841518
- Extract model loading from generate()/edit() into VnAssetsSession class - Session eagerly loads SDXL + Qwen Image Edit at construction (28s, once) - Both models held in GPU memory across calls; generate()/edit() reuse them - generate.py and edit.py become thin wrappers (backwards compatible CLI) - Context manager (with VnAssetsSession(...) as vna:) for library use - Metadata backwards-compatible: all fields preserved including lora_load_s - load_time_s now reflects total session construction, amortized across calls - Add performance stats for edit path (Qwen Image Edit + Lightning LoRA) - Benchmark matmul fallback (86.8s) vs flash attention (53.3s, 1.63x speedup) - Session vs cold start comparison: 2 ops save one 28s load, 5 edits save 112s - Flash attention via TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 documented
4.3 KiB
Performance Stats
Hardware: AMD Strix Halo (Ryzen AI Max 395 Pro, Radeon 8060S, 128 GB unified memory).
All runs use novaAnimeXL_ilV190.safetensors (SDXL), bfloat16, cfg=4.5.
Generate (SDXL)
| Run | Resolution | Steps | Prompt Syntax | Load (s) | Inference (s) | Size | Notes |
|---|---|---|---|---|---|---|---|
character_base |
1024×1024 | 20 | BREAK |
2.15 | 28.47 | 1.3 MB | Baseline, no weighting |
character_weighted |
1024×1024 | 20 | (word:weight) + BREAK |
2.57 | 29.94 | 1.4 MB | Full Compel syntax |
background_classroom |
1280×720 | 20 | (word:weight) |
2.08 | 181.85 | 1.2 MB | VAE decode dominated (flash attn enabled) |
Per-step breakdown (1024×1024)
| Step | Time |
|---|---|
| 1 (warmup) | ~1.3–1.6s |
| Steady state | ~1.0–1.3s |
| Total (20 steps) | ~20–22s UNet + VAE decode |
Larger resolutions
1280×720 and 1920×1080 trigger flash attention kernel compilation on first run (up to 250s for the first step). Subsequent runs reuse cached kernels. VAE decode at these resolutions is the dominant cost — 1920×1080 decode can exceed 2 minutes.
Compel overhead
Prompt weighting via compel adds negligible overhead (~0.4s for encoding
long prompts with BREAK). The embedding path is identical to raw encoding
once tensors reach the UNet.
Edit (Qwen Image Edit + Lightning LoRA)
All edits use qwen_image_edit_2509_fp8_e4m3fn.safetensors + Lightning 4-step
LoRA with turbo settings (steps=4, cfg=1.0) on 1024×1024 input images.
| Run | Steps | Load (s) | LoRA (s) | Inference (s) | Notes |
|---|---|---|---|---|---|
base_smile |
4 | 28.16 | 1.49 | 86.81 | Happy smile variant (matmul fallback) |
base_smile_flash |
4 | 31.23 | 1.36 | 53.33 | Happy smile variant (flash attention) |
Per-step breakdown (1024×1024, turbo)
Matmul fallback (no flash attention):
| Step | Time |
|---|---|
| 1 | ~0.3s |
| 2 | ~4.9s |
| 3 | ~11.5s |
| 4 | ~13.7s |
Flash attention (TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1):
| Step | Time |
|---|---|
| 1 | ~0.3s |
| 2 | ~1.6s |
| 3 | ~5.1s |
| 4 | ~7.0s |
Flash attention cuts edit inference from 86.8s → 53.3s (1.63× speedup). SDXL generate is unaffected (uses its own attention processor, not SDPA). Step 1 is fast (prefill/encoding). Steps 2–4 engage the full transformer and VAE; flash attention reduces the attention bottleneck.
Session (persistent models)
Both SDXL and Qwen loaded eagerly into a single VnAssetsSession.
Models held in GPU memory across calls.
| Phase | Wall (s) | Details |
|---|---|---|
| Session load | 28.2 | SDXL UNet + Qwen transformer + VAE + TE + LoRA fuse |
| Generate | 30.7 | SDXL 20-step, 1024×1024, Compel encoding |
| Edit (turbo) | 87.2 | Qwen 4-step, 1024×1024, Lightning LoRA |
| Total wall | 146.2 | One session, 2 operations |
Session with flash attention
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 throughout.
| Phase | Wall (s) | vs matmul fallback |
|---|---|---|
| Session load | 31.2 | ~same |
| Generate | 33.1 | ~same (SDXL uses own attn processor) |
| Edit (turbo) | 53.7 | 1.62× faster |
| Total wall | 118.2 | 1.24× faster overall |
Session vs cold start
| Approach | Generate | Edit | Total |
|---|---|---|---|
| Standalone (cold) | ~33s | ~116s | ~149s |
| Session (matmul) | ~31s | ~87s | ~146s |
| Session (flash) | ~33s | ~54s | ~118s |
| Saved vs cold | — | ~62s | ~31s |
With 2 operations the session saves one model-load round trip (~28s). The saving grows linearly with more edits: 5 edits save 4×28s = 112s. Flash attention adds a further 1.6× multiplier on edit inference time.
Notes
inference_time_sincludes VAE decode, which is disproportionately expensive at non-square resolutions on this hardware.TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1enables ROCm flash attention, cutting per-step UNet time roughly in half after kernel compilation.- First run at a new resolution incurs kernel compilation cost; subsequent runs at the same resolution are fast.
- Session
load_time_sin metadata reflects total session construction (all models loaded); individual operation inference times exclude loading. - LoRA fuse time (~1.5s) is included in session load, once.