Building Profanity Filters: clbuttic sh!t

BUILDING PROFANITY FILTERSclbuttic sh!t

Framework Days. IT Saturday. 5.09.2015

INTERNET CENSORSHIP

childrenreligionsexual

https://en.wikipedia.org/wiki/Censorship

https://en.wikipedia.org/wiki/Censorship


WHY FILTERING GOOD

ETHNICITY

PROTECTS

CHILDREN

RELIGIONSEXUAL ORIENTATION


WHY FILTERING BADlack of trust to your users

their willing to break rules

→


COCK

https://www.youtube.com/watch?v=y9LpgWSg-As








































































































































































































































































LET’S BUILD FILTER!


filter =list of dirty words +

list of replacements +filter rule

– George Carlin, 1972

Shit, piss, fuck, cunt, cocksucker, motherfucker, and tits.

“Seven Words You Can Never Say on Television”


FILTER RULES


1. search by entry

psss…

RANGE OF WORD


NSRange range = [text rangeOfString:badWord options:NSCaseInsensitiveSearch];

BOOL hasDirtyWord = [text localizedCaseInsensitiveContainsString:badWord];

RANGE OF WORD


- (NSArray * )rangesOfBadWordsWithSpaceInString:(NSString * )text { __block NSMutableArray * result = [NSMutableArray array];

[self.listOfBadWordsWithSpace enumerateObjectsUsingBlock:^(NSString * badWord, NSUInteger idx, BOOL * stop) {

NSRange range = [text rangeOfString:badWord options:NSCaseInsensitiveSearch]; while (range.location != NSNotFound) { [result addObject:[NSValue valueWithRange:range]]; NSRange nextRange = NSMakeRange(range.location + 1, [text length] - range.location - 1); range = [text rangeOfString:badWord options:NSCaseInsensitiveSearch range:nextRange]; } }];

return result; }

SEARCH BY ENTRY


Get your ass down here! The grass around the creek was new, giving it a velvety look. Dusty, his heartless assassin, had found his mate.

FALSE POSITIVES



assart

assault

association

assurance

‘ASS’ WORDSharassment

hassel

hourglass

impassable

pass

passion

piassaba

preassign1250 words found

http://www.morewords.com/contains/ass/

http://www.morewords.com/contains/ass/


ass → butt

REPLACEMENT RULES…


classic → clbuttic

…FAILS


Constitution → Consbreastution

AND FAILS AGAIN…

medieval → medireview

Tyson Gay → Tyson Homosexual

FILTER RULES


1. search by entry

2. search whole word don’t u know me?

SEARCH WHOLE WORD


NSString * scanned; if ([scanner scanCharactersFromSet:wordCharacters intoString:&scanned]) {

if ([wordSet containsObject:[scanned lowercaseString]]) {

NSRange range = NSMakeRange(scanner.scanLocation - scanned.length, scanned.length); [result addObject:[NSValue valueWithRange:range]]; } }

NSSet * badWordsSet = [NSMutableSet setWithArray:self.listOfBadWords];

NSScanner * scanner = [NSScanner scannerWithString:text]; NSCharacterSet * wordCharacters = [NSCharacterSet alphanumericCharacterSet];

SEARCH WHOLE WORD



SPACE!


AND OTHER PUNCTUATION


Get your a s s down here! You'd probably fire my a.s.s the first day on the job. You've covered my a_s_s every time I screwed up.

PUNCTUATION

FILTER RULES


1. search by entry

2. search whole word

3. handle punctuation

don’t tell anyone…

1337 59341<


L33T SPEAK


HOW MANY DIFFERENTSPELLINGS HAS ONE WORD?

BITCH


B1TCH


I → 1

BITCH

B!TCH


I → !

BITCH B1TCH

BI+CH


T → +

BITCH B1TCH B!TCH

I3ITCH


B → I3

BITCH B1TCH B!TCH BI+CH

BITCH B!TCH B1TCH 8ITCH ßITCH

13ITCH L3ITCH BI7CH BI+CH BI†CH

BIT[H BIT¢H BIT<H BITC# BITC:

B1T¢H 8!†C# 8ITC/-/ 817[# (3][+(:


FILTER RULES


1. search by entry



4. handle l33t speak

my name is…

SCUNTHORPE PROBLEM


https://en.wikipedia.org/wiki/Scunthorpe_problem

https://en.wikipedia.org/wiki/Scunthorpe_problem

NICE TITSIn 2007, the Royal Society for the Protection of Birds blocked ornithological terms such as cock (male bird) and tit, shag and booby from its discussion forums


FILTER RULES


1. search by entry



4. handle l33t speak

5. remember about exceptions

blue-footed booby!

http://www.sebastienmillon.com/I-Touched-a-Booby-Art-Print





























































































































































TEXT FILTERING ON IOS


words dictionary (boobs, b00bs,

b00b5)

whole word scan NSScanner, NSSet



phrases dictionary (b o o b s, b.o.o.b.s,

b!o!o!bs)

substring scan rangeOfString



words dictionary (boobs, b00bs,

b00b5)

whole word scan NSScanner, NSSet

phrases dictionary (b o o b s, b.o.o.b.s,

bo!obs)

substring scan rangeOfString+

HOW FAST IS IT?


time,

sec

onds

0

0,1

0,2

0,3

0,4

user text, characters count

1000 5000 10000 20000

range scanner both dirty words dictionary

contains 455 words

LIVE FILTERING


use RAC and run filter every time user inputs character

https://dl.dropboxusercontent.com/u/11487658/itsaturday/profanity.gif

IMPROVE FILTERING• Keep dictionary up to date

• Whitelist

• Levenshtein distance

• Soundex functions (where a word sounds like another)

• Naive bayesian inference filtering of phrases/terms


https://en.wikipedia.org/wiki/Levenshtein_distance

https://en.wikipedia.org/wiki/Soundex

https://en.wikipedia.org/wiki/Bayesian_inference

POST-MODERATION


alive moderatorssolid community

flag abuse

DIRTY WORDS• list of dirty words in different languages

https://github.com/shutterstock/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words

• list of dirty words i’ve used

https://gist.github.com/vixentael/5ce4168e3e94d9686405


https://github.com/shutterstock/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words

https://gist.github.com/vixentael/5ce4168e3e94d9686405

LAST SLIDE

@vixentael


iOS developer at Stanfy

https://twitter.com/vixentael


























mailto:[email protected]?subject=

THANK YOU FOR WATCHING!


Building Profanity Filters: clbuttic sh!t

Mobile

Transcript of Building Profanity Filters: clbuttic sh!t