A policy turns the categories the engine scored into one action. The engine never decides on its own: the model, when it is asked, returns numbers, and the policy code you can read on this page turns them into allow, review or block.
The categories
| Category | What it covers |
|---|---|
mild_profanity |
damn, hell, crap and their kin |
profanity |
The strong words |
sexual |
Sexual language and anatomy used sexually |
insult |
Attacks on a person |
slur |
Attacks on a group |
threat |
Threats of harm |
rude |
Rude, disrespectful or harsh text short of an insult: “shut up”, “what nonsense”, scornful anger at a party or a public figure. Only the model reads it, so an Edge-only project or mode: "fast" never scores it |
sexual_minors |
Sexual content involving a child or a minor. Every preset blocks it. Only the model reads it |
self_harm |
The writer’s own self-harm or suicidal thoughts, plans or acts; urging someone else to hurt themselves is threat. Every preset holds it for review, so a person who needs help is seen by someone rather than silenced. Only the model reads it |
reserved |
Names your project reserves (admin, staff, your brand), username contexts only |
The presets
Each preset is an action per category, with thresholds fitted per version on the benchmark’s development split against a false-positive ceiling on innocent text.
| Preset | For | Behaviour |
|---|---|---|
kids |
Under-13 products | Everything blocked, mild profanity and rudeness included; it reviews early and blocks early. |
teen |
13 to 17 | Mild profanity allowed; everything else blocked, rudeness included. |
standard |
The default for adults | Mild profanity allowed; profanity, sexual talk and rudeness reviewed; insults, slurs, threats and sexual harassment aimed at a person (“suck my dick”, “send nudes”) blocked. |
strict |
Brand-safe surfaces: engraving, printing, marketplaces | As kids, for adults. |
In every preset, sexual content involving a minor and a name your project reserves are blocked, and the writer’s own self-harm is held for review.
Your own policy
Send an object instead of a preset. Every category you leave out takes the standard action; leave out sexual and sexual harassment aimed at a person is blocked, as standard blocks it, while a sexual you set applies to it too.
{
"text": "…",
"policy": { "mild_profanity": "allow", "profanity": "allow", "insult": "review", "slur": "block", "threat": "block" }
}A slur is a slur on the page whoever wrote it: quoted or reported (“he called me a …”), it counts as the slur, as quoted profanity does. A slur someone claims for their own group, and a hateful sentence with no slur that the writer quotes to condemn it, go to the model to read.
Save it on the project in the dashboard and it gets an id, pol_…, that a request can send instead of the object. Up to twenty saved policies per project.
What review means
Review is for text a person should see before it is shown, or for a product that shows it and flags it. If your product has no queue, treat review as block for usernames and as allow for comments, and send corrections through feedback when it gets one wrong.