Profanity, Hate, Spam and Junk: Why One Score Isn't Enough

Wire a single-score moderation API into a form and you have bought yourself exactly one lever. Above the threshold, reject; below it, accept. Every kind of unwanted submission gets the same treatment, and the number you pick is a compromise between letting abuse through and rejecting customers.

That is a bad trade, and it is avoidable, because the things arriving through your forms are not one problem wearing four hats. They are four problems with different consequences, different audiences and different correct responses.

#Four Categories, Four Different Jobs

Read down the last column and the case makes itself. A furious but genuine customer swearing in a support form is a submission you want, urgently. A slur aimed at your team is one you want blocked and recorded. A crypto pitch is one you want gone without ceremony. Collapse those into one number and you have to choose which of the three you get wrong.

Most teams choose badly in a predictable direction. They set the threshold high enough to avoid rejecting real customers, which means the abuse lands in the queue anyway, and the tool ends up catching only the most obvious junk.

#What The Research Says About Collapsing Them

This is not a vendor argument. It is one of the oldest findings in the content moderation literature.

Davidson and colleagues, presenting at ICWSM in 2017, framed the central problem as separating hate speech from other kinds of offensive language. They found that lexical methods, meaning anything that flags text because it contains particular terms, have low precision precisely because they cannot tell the two apart. The distinction is not cosmetic; it is the hardest and most consequential judgement in the whole task.

The cost of getting it wrong is not evenly distributed either. Sap and colleagues, at ACL in 2019, found that tweets written in African American English were up to twice as likely to be labelled offensive by widely used models and by the annotators who trained them. A system with one score and one threshold inherits that bias wholesale and applies it as a single yes-or-no verdict on every submission.

Separating hate speech from merely offensive language is the core difficulty in automated detection, and term-matching approaches have low precision because they cannot make the distinction. Source: Davidson et al., ICWSM 2017

#And Spam Is Not Toxicity At All

There is a second, more practical problem with single-score tools: most of them score the wrong thing entirely.

The moderation services developers reach for were built for comment sections and community feeds, where the risk is abuse. They return a toxicity judgement. But the overwhelming majority of what arrives through a business contact form is commercial rather than abusive. Promotional pitches, backlink offers, agency outreach, phishing dressed as a project brief. A toxicity classifier reads all of that as perfectly civil text, because it is.

Teams migrating off Google's Perspective API, which sunsets at the end of December 2026, hit this immediately. Swapping one toxicity API for another replaces the smaller half of the problem and leaves the larger half untouched.

#What Per-Category Scoring Buys You

Four numbers instead of one changes what you can build.

Different thresholds for different categories. Strict on hate speech, moderate on spam, deliberately loose on profanity. One integration, four dials.

Different profiles for different forms. The same API call, configured differently per surface:

Routing instead of a wall. With separate scores you can act differently on each: drop the spam, quarantine the profanity for a human glance, block and log the hate speech, hold the junk for review. A single verdict cannot express any of that.

Tuning without collateral damage. When something gets through, you can see which category missed it and move that dial. With one score, every adjustment moves everything, and fixing a spam problem quietly tightens your rejection rate on real customers.

An audit trail that means something. "Blocked: hate 0.91, spam 0.04" is a defensible record. "Blocked: toxicity 0.78" tells an auditor, a regulator or an annoyed customer almost nothing about why.

#One Call, Four Decisions

In code the difference is small and immediately obvious. A single-score integration has one branch. A multi-category one has as many as the situation needs:

const { spam, profanity, hate, junk } = await score(message);
if (hate > 0.5) {
await audit.record({ decision: "blocked", reason: "hate", scores });
return silentlyDrop();          // never reaches a human reviewer
}
if (spam > 0.8 || junk > 0.85) {
return silentlyDrop();          // no notification, no CRM record
}
if (profanity > 0.7) {
return quarantine({ message, scores });  // a person decides
}
return deliver({ message, priority: spam < 0.2 ? "high" : "normal" });

(Field names and thresholds are illustrative. Check the current Babel Shield docs and tune to your own forms.)

Note the last line. Once you have separate scores you are not only deciding what to reject; you are deciding what to prioritise. The submissions that score near zero on every category are your cleanest, highest-intent leads, and they can go to the top of the queue automatically.

#The Economics Of Being Wrong

It is worth pricing the two failure directions, because they are not symmetric.

A spam submission that gets through costs you a few minutes and a polluted record. A real enquiry that gets blocked costs you the deal, and you will never know it happened, because people who hit a rejection message do not usually try again. They go to whoever is next in the search results.

That asymmetry is the argument for per-category control rather than one cautious global threshold. You can afford to be aggressive about categories where a false positive is nearly impossible, such as gibberish and overt promotional spam, precisely because you are being conservative in the categories where human language is ambiguous.

#What This Looks Like In Practice

One call on submit, four scores back, thresholds you set per form. Babel Shield was built to this shape deliberately: spam, profanity, hate speech and junk scored separately, so the decision your code makes can be as specific as the problem in front of it.

The categories exist because the responses differ. A tool that returns one number has already made the judgement for you, and it has made it without knowing whether the message was a sales pitch, a slur, or a customer having a bad day.

#References

Davidson, Warmsley, Macy & Weber. "Automated Hate Speech Detection and the Problem of Offensive Language." ICWSM, 2017.

Sap, Card, Gabriel, Choi & Smith. "The Risk of Racial Bias in Hate Speech Detection." ACL, 2019.

Google Jigsaw. Perspective API attribute reference and sunset notice, perspectiveapi.com.

Imperva/Thales. "2026 Bad Bot Report: Bots in the Agentic Age."