vnassets/docs/comfyui-prompt-style.md

# ComfyUI Prompt Style Support

## What is ComfyUI Prompt Style?

ComfyUI's prompt encoding extends standard CLIP text encoding with two
capabilities that plain diffusers does not provide:

1. **Per-token prompt weighting** (`(word:weight)`)
2. **Condition chunking** (`BREAK`)

VNAsset currently uses standard diffusers SDXL encoding (equivalent to ComfyUI's
`BNK_CLIPTextEncodeAdvanced` with `token_normalization=none` and
`weight_interpretation=comfy`), but only for plain comma-separated prompts.
ComfyUI-specific syntax `(word:1.2)` and `BREAK` are **not yet supported**.

---

## How Prompt Weighting Works

### Syntax

Users write weight annotations directly in the prompt string:

```
1girl, (red hair:1.3), [glasses], (smile)
```

| Syntax | Effect |
|--------|--------|
| `(word)` | Boost ×1.1 (default) |
| `((word))` | Nested boost ×1.21 (1.1²) |
| `(word:1.5)` | Explicit weight 1.5 (boosted) |
| `(word:0.6)` | Explicit weight 0.6 (de-emphasized) |
| `[word]` | Shorthand de-emphasis ×0.9 (same as `(word:0.9)`) |
| `\(word\)` | Literal parentheses (escaped, not weighted) |

Nested and multi-word weighting also works:

```
((masterpiece, best quality:1.3))
(red hair, blue eyes:1.2)
```

### What happens under the hood

This is **not** prompt rewriting. The weighting is applied at the **embedding
tensor level** after CLIP encoding:

```
Prompt string → Tokenize → CLIP encode → Scale embeddings by weights → UNet
```

For each token tagged with a weight, its embedding vector is multiplied by that
weight. The rest of the pipeline (UNet, VAE) sees the same tensor shapes and
operates normally.

For example, `(red hair:1.5)` means:

1. Tokenize `red` and `hair` as usual
2. Get their embedding vectors from CLIP (each is a vector of floats)
3. Multiply each vector by 1.5
4. Pass the scaled embeddings to the UNet

The UNet then pays 1.5× more "attention" to those tokens.

### How SDXL makes this trickier

SDXL has **two** text encoders:

| Encoder | Tokenizer | Pooled output? |
|---------|-----------|----------------|
| CLIP-L (ViT-L) | `tokenizer` | No |
| OpenCLIP-G (ViT-bigG) | `tokenizer_2` | **Yes** — used for global conditioning |

Prompt weighting must be applied to the outputs of **both** encoders. The
pooled embedding from OpenCLIP-G also needs to be weighted consistently.

---

## How BREAK Works

`BREAK` splits a single prompt into multiple independent conditioning vectors,
which are then **concatenated** along the sequence dimension:

```
1girl, red hair, standing BREAK blue sky, cherry blossoms, daytime
```

Instead of one CLIP encoding that mixes the character and background concepts
into a single tensor, this creates **two separate conditioning tensors**:

```
Chunk 1: "1girl, red hair, standing"     → conditioning tensor A (shape [1, N₁, 2048])
Chunk 2: "blue sky, cherry blossoms, daytime" → conditioning tensor B (shape [1, N₂, 2048])
                                                                                ↓
                                                          Concatenate: [1, N₁+N₂, 2048]
```

Each chunk gets its own CLIP forward pass, so the character description doesn't
bleed into the background encoding and vice versa. The UNet receives a longer
conditioning sequence with the two concepts cleanly separated.

### BREAK vs `.and()`

The `compel` library uses `.and()` as its native concatenation operator:

```
"a cat .and() a dog"   ← compel native
"a cat BREAK a dog"    ← ComfyUI syntax
```

Both produce the same result (two concatenated conditionings). We'll support
both forms.

---

## Implementation Plan

### Library: `compel`

We'll use the [`compel`](https://github.com/damian0815/compel) library from the
InvokeAI/diffusers ecosystem. It is:

- The standard implementation for A1111/ComfyUI prompt weighting
- Well-tested across millions of generations
- Maintained alongside diffusers
- Already mentioned in the README as the planned approach

### Step 1: Add dependency

**`pyproject.toml`** — add `compel` to `dependencies`.

### Step 2: New module `vnassets/prompt.py`

A thin Compel wrapper for SDXL. Responsibilities:

- Accept a loaded `StableDiffusionXLPipeline`
- Extract `tokenizer`, `tokenizer_2`, `text_encoder`, `text_encoder_2`
- Build a `Compel` instance configured for SDXL's dual-encoder setup with
  `ReturnedEmbeddingsType.PENULTIMATE_OR_LAST_HIDDEN_STATES_NON_NORMALIZED`
  (matches ComfyUI's `token_normalization=none`)
- Parse the positive prompt into `(prompt_embeds, pooled_prompt_embeds)`
- Parse the negative prompt into `(negative_prompt_embeds, negative_pooled_prompt_embeds)`
- Handle `BREAK` by translating to `.and()` or splitting + concatenating manually

API:

```python
from .prompt import build_compel, encode_prompts

compel = build_compel(pipe)
pos_embeds, pos_pooled, neg_embeds, neg_pooled = encode_prompts(
    compel, prompt, negative_prompt
)
```

### Step 3: Modify `vnassets/generate.py`

Replace the raw-string path with pre-computed weighted embeddings:

```python
# Before
image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    ...
)

# After
compel = build_compel(pipe)
pos_embeds, pos_pooled, neg_embeds, neg_pooled = encode_prompts(
    compel, prompt, negative_prompt
)
image = pipe(
    prompt_embeds=pos_embeds,
    pooled_prompt_embeds=pos_pooled,
    negative_prompt_embeds=neg_embeds,
    negative_pooled_prompt_embeds=neg_pooled,
    ...
)
```

Compel will be initialized **after** the pipeline is loaded to device (since
text encoders must be on the correct device).

### Step 4: Add `--raw` flag (optional opt-out)

Add a `--raw` flag to `vnasset generate` that bypasses Compel and uses the
plain string path. Useful when:

- The prompt contains literal parentheses that shouldn't be parsed
- Debugging — comparing weighted vs unweighted output

### Step 5: Update `README.md`

- Remove the "not currently supported" note under Prompt Compatibility
- Add a **Prompt Syntax** section documenting `(word:weight)` and `BREAK`
- Add examples showing weighted prompts

---

## What's NOT affected

The `vnasset edit` command is **unchanged**. Qwen Image Edit uses natural
language instructions (`"make her smile"`) rather than keyword-prompt
weighting. Compel does not support Qwen's text encoder (Qwen2.5-VL), and
weighting makes no sense for editing instructions anyway.

---

## Files Changed

| File | Change |
|------|--------|
| `pyproject.toml` | Add `compel` dependency |
| `vnassets/prompt.py` | **New** — Compel wrapper for SDXL |
| `vnassets/generate.py` | Use Compel embeddings instead of raw strings |
| `vnassets/cli.py` | Add `--raw` flag to `generate` command |
| `README.md` | Document new syntax support |

---

## References

- [Compel on GitHub](https://github.com/damian0815/compel)
- [ComfyUI BNK_CLIPTextEncodeAdvanced](https://github.com/BlakeOne/ComfyUI-BNK-CLIPTextEncode-Advanced)
- [SDXL dual text encoder architecture](https://huggingface.co/docs/diffusers/using-diffusers/sdxl#the-dual-text-encoder-architecture)