01 · The core idea
A third option between “allow” and “block”
Most prompt filters make a binary decision. PromptSan asks a different question: can the system remove the unsafe direction while retaining the subject, composition, style, lighting, and other benign instructions?
Conventional gate
unsafe → reject
Simple to reason about, but it discards the entire request—even when much of the creative direction is harmless.
PromptSan
unsafe → sanitize → generate
Changes selected text embeddings or appends a learned safety suffix, then lets the same frozen image model continue.
“Preserve intent” is an optimization goal, not a guarantee. The paper measures safety reduction and benign-content preservation, but a production system still needs policy checks before and after generation.
02 · How it works
One classifier, two intervention paths
Both variants start with a CLIP-derived text representation and a small category-specific NSFW classifier. Safe inputs pass through unchanged. Unsafe inputs are routed to either inference-time token repair or a pre-trained suffix.
03 · PromptSan-Modify
Repair only the tokens driving the risk
Modify computes how sensitive the classifier loss is to every token embedding. The highest-impact tokens receive gradient updates toward the classifier’s safe region; the rest of the prompt stays fixed.
risk = classifier(prompt_embeddings)
token_scores = norm(∂ safety_loss / ∂ token_embedding)
selected = top_p(token_scores, p=0.1)
for step in range(10):
prompt_embeddings[selected] -= 0.03 * gradient[selected]
if classifier(prompt_embeddings) < threshold:
break0.03
AdamW learning rate
10
Inference updates
0.1
Top-p threshold
04 · PromptSan-Suffix
Train once, append at runtime
Suffix learns a short sequence of continuous embeddings for a harmful category. Training uses both image and text classifiers; inference first classifies the prompt, then attaches the corresponding suffix without running ten prompt-optimization steps.
Human-readable concept
+
<SAFE_01> … <SAFE_20>
The token names are illustrative. The useful object is the learned embedding sequence, not readable safety wording.
Why it is product-friendly
- No per-request gradient loop after the suffix has been trained.
- Category-specific suffixes can be selected by the same lightweight classifier.
- The base diffusion model remains frozen.
05 · The classifier
Small enough to sit in front of generation
The appendix describes a roughly 1.1-million-parameter binary network. It consumes a 768-dimensional CLIP feature and outputs a risk probability. Rather than one universal classifier, the authors train separate detectors for categories such as nudity, violence, and disturbing content using the VISU dataset.
CLIP feature
768 dimensions
Safety network
≈ 1.1M parameters
Category score
Probability 0–1
06 · Results and limits
The reported reduction is large. The scope is narrow.
On the I2P benchmark with Stable Diffusion v1.4, the paper reports the following NudeNet detection totals. These are detected exposed-region counts—not the number of prompts, images, or policy violations.
SD v1.4 baseline
659
PromptSan-Modify
43
93.5% fewer detections
PromptSan-Suffix
38
94.2% fewer detections
What the paper does not establish
07 · Runnable alternatives
Use PromptSan as a design pattern, not a package name
If the goal is a working prototype rather than a paper reproduction, these official repositories cover adjacent parts of the same safety stack.
| Project | Best fit | Safety layer | Published support |
|---|---|---|---|
| PromptGuard | Closest to a reusable soft-suffix approach | Prompt embedding | SD v1.4 example |
| SAFREE | Training-free removal of unsafe concept directions | Embedding + generation | SD 1.4, SDXL, video code |
| Safe Latent Diffusion | Mature denoising-time safety guidance | Diffusion process | Safe SD pipeline |
| LatentGuard | Fast prompt detection, not prompt repair | Input classifier | SD/SDXL text encoders |
08 · Production blueprint
A gateway should fail in layers
Prompt sanitization is best treated as one recoverable step inside a larger control plane. Detection decides what to repair; output moderation catches what the prompt layer misses.
Text policy
Hard rules and obvious abuse
Risk model
Category + calibrated score
Sanitizer
Embedding repair or rewrite
Generator
Local model or image API
Image check
Return, retry, or reject
When you control the model
Diffusers, Stable Diffusion, SDXL, or a local workflow can expose prompt embeddings. That makes PromptSan-style gradient repair, soft suffixes, and SAFREE-like projection technically possible.
Build a local ComfyUI workflowWhen the API accepts only text
You cannot pass a modified CLIP embedding into most hosted APIs. Use classifier → constrained rewrite → second classifier → generation → image classifier, and measure semantic similarity as a guardrail.
Explore prompt structure examples{
"safe": false,
"risk": 0.91,
"categories": { "sexual": 0.91, "violence": 0.04 },
"action": "sanitized",
"sanitizedPrompt": "…",
"semanticSimilarity": 0.93,
"outputCheck": "pending"
}Recommended MVP
Classify → rewrite → verify → generate → inspect
Start with measurable layers that work across model providers. Add learned suffixes or embedding-space repair only after you have category data, failure examples, thresholds, and a review loop.
Quick answers
PromptSan FAQ
Is PromptSan a new image generation model?
No. It is a prompt-level safety intervention tested on Stable Diffusion v1.4. The base image model stays frozen while the prompt embedding or a learned suffix is changed.
Can PromptSan be added to a closed image API?
Not in its original embedding-based form if the API accepts only a prompt string. A closed API usually needs a classifier, a text rewrite step, a second classifier pass, and an output image checker instead.
Does PromptSan replace output moderation?
No. The paper reports weaker behavior for some concepts and possible bypasses for composite harmful concepts. A production system should still inspect generated images and keep policy enforcement outside the model.
Is there an official PromptSan package?
The paper's arXiv record does not link an author-maintained code repository as of August 27, 2026. PromptGuard, SAFREE, Safe Latent Diffusion, and LatentGuard are useful open-source references for adjacent layers.
References
Primary paper and official implementations
- Xie, Zeng, Zhang & Fu — NSFW-Classifier Guided Prompt Sanitization for Safe Text-to-Image Generation · arXiv:2506.18325v1, June 23, 2025.
- PromptGuard official repository · Closest to a reusable soft-suffix approach.
- SAFREE official repository · Training-free removal of unsafe concept directions.
- Safe Latent Diffusion official repository · Mature denoising-time safety guidance.
- LatentGuard official repository · Fast prompt detection, not prompt repair.
Last fact-checked August 27, 2026. This guide is an independent technical interpretation and is not affiliated with the paper authors or the referenced projects.