Add VnAssetsSession for persistent model lifecycle

- Extract model loading from generate()/edit() into VnAssetsSession class
- Session eagerly loads SDXL + Qwen Image Edit at construction (28s, once)
- Both models held in GPU memory across calls; generate()/edit() reuse them
- generate.py and edit.py become thin wrappers (backwards compatible CLI)
- Context manager (with VnAssetsSession(...) as vna:) for library use
- Metadata backwards-compatible: all fields preserved including lora_load_s
- load_time_s now reflects total session construction, amortized across calls

- Add performance stats for edit path (Qwen Image Edit + Lightning LoRA)
- Benchmark matmul fallback (86.8s) vs flash attention (53.3s, 1.63x speedup)
- Session vs cold start comparison: 2 ops save one 28s load, 5 edits save 112s
- Flash attention via TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 documented
This commit is contained in:
Michele Rossi
2026-07-08 10:18:42 +02:00
parent e7cde842b3
commit 97ac841518
8 changed files with 1072 additions and 212 deletions

116
docs/stats.md Normal file
View File

@@ -0,0 +1,116 @@
# Performance Stats
Hardware: **AMD Strix Halo** (Ryzen AI Max 395 Pro, Radeon 8060S, 128 GB unified memory).
All runs use `novaAnimeXL_ilV190.safetensors` (SDXL), bfloat16, cfg=4.5.
## Generate (SDXL)
| Run | Resolution | Steps | Prompt Syntax | Load (s) | Inference (s) | Size | Notes |
|-----|-----------|-------|---------------|----------|---------------|------|-------|
| `character_base` | 1024×1024 | 20 | `BREAK` | 2.15 | 28.47 | 1.3 MB | Baseline, no weighting |
| `character_weighted` | 1024×1024 | 20 | `(word:weight)` + `BREAK` | 2.57 | 29.94 | 1.4 MB | Full Compel syntax |
| `background_classroom` | 1280×720 | 20 | `(word:weight)` | 2.08 | 181.85 | 1.2 MB | VAE decode dominated (flash attn enabled) |
### Per-step breakdown (1024×1024)
| Step | Time |
|------|------|
| 1 (warmup) | ~1.31.6s |
| Steady state | ~1.01.3s |
| Total (20 steps) | ~2022s UNet + VAE decode |
### Larger resolutions
1280×720 and 1920×1080 trigger flash attention kernel compilation on first run
(up to 250s for the first step). Subsequent runs reuse cached kernels. VAE
decode at these resolutions is the dominant cost — 1920×1080 decode can exceed
2 minutes.
### Compel overhead
Prompt weighting via `compel` adds negligible overhead (~0.4s for encoding
long prompts with `BREAK`). The embedding path is identical to raw encoding
once tensors reach the UNet.
## Edit (Qwen Image Edit + Lightning LoRA)
All edits use `qwen_image_edit_2509_fp8_e4m3fn.safetensors` + Lightning 4-step
LoRA with turbo settings (steps=4, cfg=1.0) on 1024×1024 input images.
| Run | Steps | Load (s) | LoRA (s) | Inference (s) | Notes |
|-----|-------|----------|----------|---------------|-------|
| `base_smile` | 4 | 28.16 | 1.49 | 86.81 | Happy smile variant (matmul fallback) |
| `base_smile_flash` | 4 | 31.23 | 1.36 | 53.33 | Happy smile variant (flash attention) |
### Per-step breakdown (1024×1024, turbo)
**Matmul fallback (no flash attention):**
| Step | Time |
|------|------|
| 1 | ~0.3s |
| 2 | ~4.9s |
| 3 | ~11.5s |
| 4 | ~13.7s |
**Flash attention (`TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1`):**
| Step | Time |
|------|------|
| 1 | ~0.3s |
| 2 | ~1.6s |
| 3 | ~5.1s |
| 4 | ~7.0s |
Flash attention cuts edit inference from 86.8s → 53.3s (**1.63× speedup**).
SDXL generate is unaffected (uses its own attention processor, not SDPA).
Step 1 is fast (prefill/encoding). Steps 24 engage the full transformer
and VAE; flash attention reduces the attention bottleneck.
## Session (persistent models)
Both SDXL and Qwen loaded eagerly into a single `VnAssetsSession`.
Models held in GPU memory across calls.
| Phase | Wall (s) | Details |
|-------|----------|---------|
| Session load | 28.2 | SDXL UNet + Qwen transformer + VAE + TE + LoRA fuse |
| Generate | 30.7 | SDXL 20-step, 1024×1024, Compel encoding |
| Edit (turbo) | 87.2 | Qwen 4-step, 1024×1024, Lightning LoRA |
| **Total wall** | **146.2** | One session, 2 operations |
### Session with flash attention
`TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1` throughout.
| Phase | Wall (s) | vs matmul fallback |
|-------|----------|--------------------|
| Session load | 31.2 | ~same |
| Generate | 33.1 | ~same (SDXL uses own attn processor) |
| Edit (turbo) | 53.7 | **1.62× faster** |
| **Total wall** | **118.2** | 1.24× faster overall |
### Session vs cold start
| Approach | Generate | Edit | Total |
|----------|----------|------|-------|
| Standalone (cold) | ~33s | ~116s | ~149s |
| Session (matmul) | ~31s | ~87s | ~146s |
| Session (flash) | ~33s | ~54s | ~118s |
| **Saved vs cold** | — | **~62s** | **~31s** |
With 2 operations the session saves one model-load round trip (~28s).
The saving grows linearly with more edits: 5 edits save 4×28s = 112s.
Flash attention adds a further 1.6× multiplier on edit inference time.
## Notes
- `inference_time_s` includes VAE decode, which is disproportionately expensive
at non-square resolutions on this hardware.
- `TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1` enables ROCm flash attention,
cutting per-step UNet time roughly in half after kernel compilation.
- First run at a new resolution incurs kernel compilation cost; subsequent runs
at the same resolution are fast.
- Session `load_time_s` in metadata reflects total session construction
(all models loaded); individual operation inference times exclude loading.
- LoRA fuse time (~1.5s) is included in session load, once.