Where the words come from
Every term in the lexicon carries its source and licence into the build. This page is read from the same manifest.
Sources
- Unprofane curated lexicon, CC BY 4.0.
packages/lexicon/data/lang/ - cuss 2.2.0, MIT. github.com/words/cuss
- LDNOOBW, CC BY 4.0. github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words
- Unicode confusables (UTS #39), Unicode License v3. www.unicode.org/Public/security/latest/confusables.txt
- FrequencyWords (OpenSubtitles 2018), 18 languages, MIT. github.com/hermitdave/FrequencyWords
- dsojevic/profanity-list (consulted while grading, not imported), MIT. github.com/dsojevic/profanity-list
- chucknorris-io/swear-words (consulted while grading, not imported), CC BY 4.0. github.com/chucknorris-io/swear-words
What the build holds
- 55,840 surface forms under 3,096 head terms, across 17 languages: ar 5022, de 3484, en 3874, es 2854, fr 2634, he 2426, hi 2970, id 2716, it 2358, ja 3572, ko 5594, nl 3247, pl 3449, pt 2089, ru 3253, tr 3429, zh 2869.
- 1,566 frame patterns that recognise abuse built from ordinary words ("you are worthless", "go back to your country"), written per language over word classes: ar 69, de 122, en 119, es 108, fr 89, he 103, hi 61, id 73, it 87, ja 66, ko 62, nl 121, pl 95, pt 107, ru 87, tr 84, zh 113.
- 1,447 innocent words and phrases that contain a profane string, each one a test: Scunthorpe, Dickinson, cockatoo, breast cancer, shiitake, and the rest.
- 342 common words across the covered languages that contain a profane stem and protect it, and 14,402 common words one letter from a profane term that are never read as a disguised spelling, from film-subtitle frequency lists.
- 1,633 look-alike characters from the Unicode confusables table, restricted to those that fold to a Latin letter.
- 20,000 English words by frequency, from film subtitles, for splitting usernames into words.
How grades are decided
A curated entry wins over an imported one. Imported entries arrive with the grade their source gives them and are lowered, never raised, by the curated list. Every entry names a category (profanity, sexual, insult, slur, threat, reserved) and a severity (mild to extreme), and the policy presets act on those, never on the word itself.
Corrections
The feedback endpoint and the dashboard's "Wrong result" button feed the curated list, the innocent list and the calibration. A change to any of them is a new dated engine version with a changelog line, never a silent change inside a version you have pinned. A word you think is graded wrongly can also be sent to hello@unprofane.com.