97ac841518
- Extract model loading from generate()/edit() into VnAssetsSession class - Session eagerly loads SDXL + Qwen Image Edit at construction (28s, once) - Both models held in GPU memory across calls; generate()/edit() reuse them - generate.py and edit.py become thin wrappers (backwards compatible CLI) - Context manager (with VnAssetsSession(...) as vna:) for library use - Metadata backwards-compatible: all fields preserved including lora_load_s - load_time_s now reflects total session construction, amortized across calls - Add performance stats for edit path (Qwen Image Edit + Lightning LoRA) - Benchmark matmul fallback (86.8s) vs flash attention (53.3s, 1.63x speedup) - Session vs cold start comparison: 2 ops save one 28s load, 5 edits save 112s - Flash attention via TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 documented
117 lines
4.3 KiB
Markdown
117 lines
4.3 KiB
Markdown
# Performance Stats
|
||
|
||
Hardware: **AMD Strix Halo** (Ryzen AI Max 395 Pro, Radeon 8060S, 128 GB unified memory).
|
||
All runs use `novaAnimeXL_ilV190.safetensors` (SDXL), bfloat16, cfg=4.5.
|
||
|
||
## Generate (SDXL)
|
||
|
||
| Run | Resolution | Steps | Prompt Syntax | Load (s) | Inference (s) | Size | Notes |
|
||
|-----|-----------|-------|---------------|----------|---------------|------|-------|
|
||
| `character_base` | 1024×1024 | 20 | `BREAK` | 2.15 | 28.47 | 1.3 MB | Baseline, no weighting |
|
||
| `character_weighted` | 1024×1024 | 20 | `(word:weight)` + `BREAK` | 2.57 | 29.94 | 1.4 MB | Full Compel syntax |
|
||
| `background_classroom` | 1280×720 | 20 | `(word:weight)` | 2.08 | 181.85 | 1.2 MB | VAE decode dominated (flash attn enabled) |
|
||
|
||
### Per-step breakdown (1024×1024)
|
||
|
||
| Step | Time |
|
||
|------|------|
|
||
| 1 (warmup) | ~1.3–1.6s |
|
||
| Steady state | ~1.0–1.3s |
|
||
| Total (20 steps) | ~20–22s UNet + VAE decode |
|
||
|
||
### Larger resolutions
|
||
|
||
1280×720 and 1920×1080 trigger flash attention kernel compilation on first run
|
||
(up to 250s for the first step). Subsequent runs reuse cached kernels. VAE
|
||
decode at these resolutions is the dominant cost — 1920×1080 decode can exceed
|
||
2 minutes.
|
||
|
||
### Compel overhead
|
||
|
||
Prompt weighting via `compel` adds negligible overhead (~0.4s for encoding
|
||
long prompts with `BREAK`). The embedding path is identical to raw encoding
|
||
once tensors reach the UNet.
|
||
|
||
## Edit (Qwen Image Edit + Lightning LoRA)
|
||
|
||
All edits use `qwen_image_edit_2509_fp8_e4m3fn.safetensors` + Lightning 4-step
|
||
LoRA with turbo settings (steps=4, cfg=1.0) on 1024×1024 input images.
|
||
|
||
| Run | Steps | Load (s) | LoRA (s) | Inference (s) | Notes |
|
||
|-----|-------|----------|----------|---------------|-------|
|
||
| `base_smile` | 4 | 28.16 | 1.49 | 86.81 | Happy smile variant (matmul fallback) |
|
||
| `base_smile_flash` | 4 | 31.23 | 1.36 | 53.33 | Happy smile variant (flash attention) |
|
||
|
||
### Per-step breakdown (1024×1024, turbo)
|
||
|
||
**Matmul fallback (no flash attention):**
|
||
|
||
| Step | Time |
|
||
|------|------|
|
||
| 1 | ~0.3s |
|
||
| 2 | ~4.9s |
|
||
| 3 | ~11.5s |
|
||
| 4 | ~13.7s |
|
||
|
||
**Flash attention (`TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1`):**
|
||
|
||
| Step | Time |
|
||
|------|------|
|
||
| 1 | ~0.3s |
|
||
| 2 | ~1.6s |
|
||
| 3 | ~5.1s |
|
||
| 4 | ~7.0s |
|
||
|
||
Flash attention cuts edit inference from 86.8s → 53.3s (**1.63× speedup**).
|
||
SDXL generate is unaffected (uses its own attention processor, not SDPA).
|
||
Step 1 is fast (prefill/encoding). Steps 2–4 engage the full transformer
|
||
and VAE; flash attention reduces the attention bottleneck.
|
||
|
||
## Session (persistent models)
|
||
|
||
Both SDXL and Qwen loaded eagerly into a single `VnAssetsSession`.
|
||
Models held in GPU memory across calls.
|
||
|
||
| Phase | Wall (s) | Details |
|
||
|-------|----------|---------|
|
||
| Session load | 28.2 | SDXL UNet + Qwen transformer + VAE + TE + LoRA fuse |
|
||
| Generate | 30.7 | SDXL 20-step, 1024×1024, Compel encoding |
|
||
| Edit (turbo) | 87.2 | Qwen 4-step, 1024×1024, Lightning LoRA |
|
||
| **Total wall** | **146.2** | One session, 2 operations |
|
||
|
||
### Session with flash attention
|
||
|
||
`TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1` throughout.
|
||
|
||
| Phase | Wall (s) | vs matmul fallback |
|
||
|-------|----------|--------------------|
|
||
| Session load | 31.2 | ~same |
|
||
| Generate | 33.1 | ~same (SDXL uses own attn processor) |
|
||
| Edit (turbo) | 53.7 | **1.62× faster** |
|
||
| **Total wall** | **118.2** | 1.24× faster overall |
|
||
|
||
### Session vs cold start
|
||
|
||
| Approach | Generate | Edit | Total |
|
||
|----------|----------|------|-------|
|
||
| Standalone (cold) | ~33s | ~116s | ~149s |
|
||
| Session (matmul) | ~31s | ~87s | ~146s |
|
||
| Session (flash) | ~33s | ~54s | ~118s |
|
||
| **Saved vs cold** | — | **~62s** | **~31s** |
|
||
|
||
With 2 operations the session saves one model-load round trip (~28s).
|
||
The saving grows linearly with more edits: 5 edits save 4×28s = 112s.
|
||
Flash attention adds a further 1.6× multiplier on edit inference time.
|
||
|
||
## Notes
|
||
|
||
- `inference_time_s` includes VAE decode, which is disproportionately expensive
|
||
at non-square resolutions on this hardware.
|
||
- `TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1` enables ROCm flash attention,
|
||
cutting per-step UNet time roughly in half after kernel compilation.
|
||
- First run at a new resolution incurs kernel compilation cost; subsequent runs
|
||
at the same resolution are fast.
|
||
- Session `load_time_s` in metadata reflects total session construction
|
||
(all models loaded); individual operation inference times exclude loading.
|
||
- LoRA fuse time (~1.5s) is included in session load, once.
|