vnassets/docs/comfyui-prompt-style.md

ComfyUI Prompt Style Support

What is ComfyUI Prompt Style?

ComfyUI's prompt encoding extends standard CLIP text encoding with two capabilities that plain diffusers does not provide:

1. Per-token prompt weighting ((word:weight))
2. Condition chunking (BREAK)

VNAsset currently uses standard diffusers SDXL encoding (equivalent to ComfyUI's BNK_CLIPTextEncodeAdvanced with token_normalization=none and weight_interpretation=comfy), but only for plain comma-separated prompts. ComfyUI-specific syntax (word:1.2) and BREAK are not yet supported.

---

How Prompt Weighting Works

Syntax

Users write weight annotations directly in the prompt string:

1girl, (red hair:1.3), [glasses], (smile)

Syntax	Effect
(word)	Boost ×1.1 (default)
((word))	Nested boost ×1.21 (1.1²)
(word:1.5)	Explicit weight 1.5 (boosted)
(word:0.6)	Explicit weight 0.6 (de-emphasized)
[word]	Shorthand de-emphasis ×0.9 (same as (word:0.9))
\(word\)	Literal parentheses (escaped, not weighted)

Nested and multi-word weighting also works:

((masterpiece, best quality:1.3))
(red hair, blue eyes:1.2)

What happens under the hood

This is not prompt rewriting. The weighting is applied at the embedding tensor level after CLIP encoding:

Prompt string → Tokenize → CLIP encode → Scale embeddings by weights → UNet

For each token tagged with a weight, its embedding vector is multiplied by that weight. The rest of the pipeline (UNet, VAE) sees the same tensor shapes and operates normally.

For example, (red hair:1.5) means:

1. Tokenize red and hair as usual
2. Get their embedding vectors from CLIP (each is a vector of floats)
3. Multiply each vector by 1.5
4. Pass the scaled embeddings to the UNet

The UNet then pays 1.5× more "attention" to those tokens.

How SDXL makes this trickier

SDXL has two text encoders:

Encoder	Tokenizer	Pooled output?
CLIP-L (ViT-L)	tokenizer	No
OpenCLIP-G (ViT-bigG)	tokenizer_2	Yes — used for global conditioning

Prompt weighting must be applied to the outputs of both encoders. The pooled embedding from OpenCLIP-G also needs to be weighted consistently.

---

How BREAK Works

BREAK splits a single prompt into multiple independent conditioning vectors, which are then concatenated along the sequence dimension:

1girl, red hair, standing BREAK blue sky, cherry blossoms, daytime

Instead of one CLIP encoding that mixes the character and background concepts into a single tensor, this creates two separate conditioning tensors:

Chunk 1: "1girl, red hair, standing"     → conditioning tensor A (shape [1, N₁, 2048])
Chunk 2: "blue sky, cherry blossoms, daytime" → conditioning tensor B (shape [1, N₂, 2048])
                                                                                ↓
                                                          Concatenate: [1, N₁+N₂, 2048]

Each chunk gets its own CLIP forward pass, so the character description doesn't bleed into the background encoding and vice versa. The UNet receives a longer conditioning sequence with the two concepts cleanly separated.

BREAK vs .and()

The compel library uses .and() as its native concatenation operator:

"a cat .and() a dog"   ← compel native
"a cat BREAK a dog"    ← ComfyUI syntax

Both produce the same result (two concatenated conditionings). We'll support both forms.

---

Implementation Plan

Library: compel

We'll use the compel library from the InvokeAI/diffusers ecosystem. It is:

• The standard implementation for A1111/ComfyUI prompt weighting
• Well-tested across millions of generations
• Maintained alongside diffusers
• Already mentioned in the README as the planned approach

Step 1: Add dependency

pyproject.toml — add compel to dependencies.

Step 2: New module vnassets/prompt.py

A thin Compel wrapper for SDXL. Responsibilities:

• Accept a loaded StableDiffusionXLPipeline
• Extract tokenizer, tokenizer_2, text_encoder, text_encoder_2
• Build a Compel instance configured for SDXL's dual-encoder setup with ReturnedEmbeddingsType.PENULTIMATE_OR_LAST_HIDDEN_STATES_NON_NORMALIZED (matches ComfyUI's token_normalization=none)
• Parse the positive prompt into (prompt_embeds, pooled_prompt_embeds)
• Parse the negative prompt into (negative_prompt_embeds, negative_pooled_prompt_embeds)
• Handle BREAK by translating to .and() or splitting + concatenating manually

API:

from .prompt import build_compel, encode_prompts

compel = build_compel(pipe)
pos_embeds, pos_pooled, neg_embeds, neg_pooled = encode_prompts(
    compel, prompt, negative_prompt
)

Step 3: Modify vnassets/generate.py

Replace the raw-string path with pre-computed weighted embeddings:

# Before
image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    ...
)

# After
compel = build_compel(pipe)
pos_embeds, pos_pooled, neg_embeds, neg_pooled = encode_prompts(
    compel, prompt, negative_prompt
)
image = pipe(
    prompt_embeds=pos_embeds,
    pooled_prompt_embeds=pos_pooled,
    negative_prompt_embeds=neg_embeds,
    negative_pooled_prompt_embeds=neg_pooled,
    ...
)

Compel will be initialized after the pipeline is loaded to device (since text encoders must be on the correct device).

Step 4: Add --raw flag (optional opt-out)

Add a --raw flag to vnasset generate that bypasses Compel and uses the plain string path. Useful when:

• The prompt contains literal parentheses that shouldn't be parsed
• Debugging — comparing weighted vs unweighted output

Step 5: Update README.md

• Remove the "not currently supported" note under Prompt Compatibility
• Add a Prompt Syntax section documenting (word:weight) and BREAK
• Add examples showing weighted prompts

---

What's NOT affected

The vnasset edit command is unchanged. Qwen Image Edit uses natural language instructions ("make her smile") rather than keyword-prompt weighting. Compel does not support Qwen's text encoder (Qwen2.5-VL), and weighting makes no sense for editing instructions anyway.

---

Files Changed

File	Change
pyproject.toml	Add compel dependency
vnassets/prompt.py	New — Compel wrapper for SDXL
vnassets/generate.py	Use Compel embeddings instead of raw strings
vnassets/cli.py	Add --raw flag to generate command
README.md	Document new syntax support

---

References

• Compel on GitHub
• ComfyUI BNK_CLIPTextEncodeAdvanced
• SDXL dual text encoder architecture