An open benchmark for profanity filters where blocking Scunthorpe counts against you. A public development set anyone can run, a private set we keep back, per-technique and per-language scores, and the false-positive rate printed beside every F1.
Public split: scunthorpe-test.jsonl, 15,530 items. Run of 2026-10-12. Result files in bench/results/. For the full engine beside the Perspective API and OpenAI moderation on judged real text, see head to head.
F1, rules alone
0.918
precision 0.996, recall 0.851, 2026-10-12-11-25-45-dev.json
False positives on innocent text
0.0%
Scunthorpe, Dickinson, cockatoo, breast cancer and the rest, bare and in sentences
Published npm result
0.364
obscenity, scunthorpe-test.jsonl, 21.2% false positives on innocent text
Newest public run for each system. Different corpora are not directly comparable. Systems whose terms forbid benchmarking do not appear.
| System | Kind | Corpus | Items | F1 | Precision | Recall | False positives, innocent | p50 ms | $ per 1,000 |
|---|---|---|---|---|---|---|---|---|---|
| Unprofane, rules alone (fast)Unprofane · scunthorpe-test.jsonl | Unprofane | scunthorpe-test.jsonl | 15,530 | 0.918 | 0.996 | 0.851 | 0.0% | <1 | 0 |
| Llama 3.3 70B alonemodel alone · pilot.jsonl | model alone | pilot.jsonl | 600 | 0.909 | 0.957 | 0.866 | 6.0% | 673 | 0 |
| openai-moderationAPI · scunthorpe-test.jsonl | API | scunthorpe-test.jsonl | 15,501 | 0.553 | 0.948 | 0.390 | 3.2% | 821 | 0 |
| obscenitynpm library · scunthorpe-test.jsonl | npm library | scunthorpe-test.jsonl | 15,501 | 0.364 | 0.930 | 0.227 | 21.2% | <1 | 0 |
| perspectiveAPI · scunthorpe-test.jsonl | API | scunthorpe-test.jsonl | 15,501 | 0.314 | 0.990 | 0.187 | 3.0% | 237 | 0 |
| Llama Guard 3 8B alonemodel alone · pilot.jsonl | model alone | pilot.jsonl | 600 | 0.235 | 0.932 | 0.134 | 1.8% | 219 | 0 |
| bad-wordsnpm library · scunthorpe-test.jsonl | npm library | scunthorpe-test.jsonl | 15,501 | 0.142 | 0.855 | 0.077 | 12.5% | <1 | 0 |
| 2toad-profanitynpm library · scunthorpe-test.jsonl | npm library | scunthorpe-test.jsonl | 15,501 | 0.128 | 0.878 | 0.069 | 11.5% | <1 | 0 |
| leo-profanitynpm library · scunthorpe-test.jsonl | npm library | scunthorpe-test.jsonl | 15,501 | 0.057 | 0.793 | 0.030 | 8.7% | <1 | 0 |
| Gemma 4 26B alonemodel alone · pilot.jsonl | model alone | pilot.jsonl | 600 | 0.000 | 0.000 | 0.000 | 0.0% | <1 | 0 |
Latency is wall clock on the machine the run was made on: the rules in-process, hosted models over the network, the self-hosted models on the laptop the bench ran on. Cost is the measured token price for hosted models and zero for anything local. A system whose F1 reads zero errored on every item in its run and is kept for the record.
Recall on disguised profanity, per evasion technique. Each item was produced by the named technique applied to a graded seed.
| Technique | Unprofane, rules alone (fast) | obscenity |
|---|---|---|
| homoglyph | 100% | 35% |
| zero_width | 100% | 22% |
| separator | 90% | 0% |
| leetspeak | 96% | 23% |
| repeated_chars | 89% | 31% |
| diacritics | 95% | 43% |
| mixed_script | 100% | 41% |
| fullwidth | 100% | 50% |
| combining_marks | 100% | 6% |
| phonetic | 96% | 21% |
| reversed | 100% | 0% |
| concatenation | 95% | 33% |
| case_mixing | 94% | 40% |
F1 per language of the item. Non-English items are the curated cores and the LDNOOBW lists in plain and disguised spellings; English carries the bulk of the set.
| Language | Unprofane, rules alone (fast) |
|---|---|
| Arabic ar | 0.947 |
| German de | 0.938 |
| English en | 0.915 |
| Spanish es | 0.978 |
| French fr | 1.000 |
| Hebrew he | 0.867 |
| Hindi hi | 0.846 |
| Indonesian id | 0.968 |
| Italian it | 0.987 |
| Japanese ja | 0.955 |
| Dutch nl | 0.923 |
| Polish pl | 0.977 |
| Portuguese pt | 0.968 |
| Russian ru | 0.969 |
| Turkish tr | 0.982 |
On the deployed path the model tier answered cleanly at 300 requests a minute and began failing at 450 with 2018: Invalid User Credentials (load-2026-09-24-11-11-35.json). That is why the rules carry the product and the model is consulted only where they are unsure.
Items are generated from graded seeds and templates, so a label is a function of the data rather than an annotator's mood. Disguised items are produced by the same technique functions the engine names, one technique per item, and tagged with it. The innocent set is the engine's own innocent list (a word or phrase that contains a profane string: a place, a surname, an animal, a tool, anatomy in a medical sentence) bare and inside sentences, and a system that flags one is counted as wrong. Every system answers the same items on the same day under the same runner; a hosted system is called with its documented client and a recorded cache so a rerun is free.
The public split is what you can download below and what the numbers above are measured on. A private split of the same size is held back and never published; its numbers appear here once a version is promoted to stable, beside the public ones, so a filter that fits the public set is caught. Running text with no word-list hit comes from a Civil Comments sample (Jigsaw, CC0): comments the annotators marked toxic at half or more are flagged, comments under a fifth are clean, and the middle is left out, so that label too is a function of the data.
Competitor numbers appear only where a system's terms allow benchmarking. What is not here is not hidden; it is not allowed.
The benchmark on GitHub Download the public split (15,530 items, JSON lines)
Each line is an item: id, text, context, lang, kind, expected (flagged, action by policy, categories, severity, evasion tags), source and license. The runner, the adapters, the metrics and the method are public in unprofane/scunthorpe-test (the code under MIT, the data under Creative Commons Attribution): npm install and npm run bench -- run score the open-source word-list libraries, an Unprofane adapter runs the same split through the API with your key, and adding your own filter is one adapter file.