SYMBOLS AND RULESRSPAMD

WHAT ARE SYMBOLS AND RULES

DEFINITIONS

RULE SYMBOL

SCOREGROUP

DESCRIPTION

WEIGHT*

true/false

OPTIONSDynamic part

Static part

∑ Results

FLAGS

WHAT ARE SYMBOLS AND RULES

WHY DO WE NEED SYMBOLS

RULE

SYMBOL_ALLOW

SYMBOL_DENY

SYMBOL_UNKNOWN

Either of symbols

WHAT ARE SYMBOLS AND RULES

WHY DO WE NEED SYMBOLS

RULE

MAP1

MAP2

MAP3

Multiple symbols

WHAT ARE SYMBOLS AND RULES

WHY DO WE NEED SYMBOLS

RULE1 SYMBOL1

RULE2

Dependency

WHAT ARE SYMBOLS AND RULES

RULES

▸ Rules define what is executed:

▸ Regexps expression

▸ Lua code

▸ Plugin logic

▸ Each rule can be associated with one or many symbols

▸ Rule can depend on other rules identified by associated symbols

▸ Each rule can define the current dynamic weight (usually from 0 to 1)

WHAT ARE SYMBOLS AND RULES

SYMBOLS

▸ Symbols define meta-information of a rule:

▸ Name

▸ Static score

▸ Other data (description, group, flags, etc)

▸ Symbols can be:

▸ Normal: associated with exactly one rule

▸ Virtual: are not associated with rules but grouped with normal symbol)

▸ Callback: do not have name or score, just define common rule

▸ Special: have special purpose (e.g. composite symbols)

SYMBOLS

SYMBOLS GROUPS

SYMBOL1

SYMBOL2

SYMBOL3

SYMBOL4

Group1

SYMBOL3

SYMBOL4

SYMBOL5

SYMBOL6

Group2GROUP LIMIT

GROUP DESCRIPTION

SYMBOLS

SYMBOLS GROUPS

▸ Groups join common symbols logically

▸ Groups can set joint limit for symbols scores enclosed

▸ Groups can be used in composite rules:

▸ SYMBOL5 && G:GROUP1

▸ SYMBOL5 && (G:GROUP1 || !G:GROUP2)

RULES

EXPRESSIONS IN RULES

▸ Expressions are used in:

▸ Regexp rules

▸ Composite symbols

▸ Expressions have common syntax:

▸ Logic operations: AND (&&), OR (||), NOT (!)

▸ Braces

▸ Limit operation: A + B + C > 2

▸ Elements are called atoms

RULES

REGEXP EXPRESSIONS

▸ Atoms are regular expressions (/re/flags):

▸ Header: Header=/re/H

▸ Mime (/P): scan text parts

▸ Body (/B): scan full undecoded body

▸ URL (/U): scan URLs found

▸ There is no order of regexps execution within an expression

▸ Same expressions are cached and executed once

COMPOSITE EXPRESSIONS

COMPOSITES STRUCTURE

SYMBOL3

SYMBOL4

SYMBOL5

SYMBOL1

Group2

NOT SYMBOL6AND

SYMBOL1 AND GR:GROUP2 AND !SYMBOL6

AND

COMPOSITE EXPRESSIONS

COMPOSITES OPERATIONS

SYMBOL3

SYMBOL4

SYMBOL5

SYMBOL1

Group2

NOT SYMBOL6AND AND

SYMBOL1

SYMBOL3

Symbols to remove

Stage 1: Check

Stage 2: Remove symbols

COMPOSITE EXPRESSIONS

COMPOSITES STRUCTURE

▸ Composite atoms can include:

▸ Other symbols

▸ Groups (gr:)

▸ Other composites (with recursive references check)

▸ Composite operations can be the following:

▸ Remove symbol and weight (SYMBOL)

▸ Remove weight only (~SYMBOL)

▸ Remove symbol but preserve weight (-SYMBOL)

▸ Always remove symbol and weight (^SYMBOL)

COMPOSITE EXPRESSIONS

COMPOSITES OPERATION

▸ If any composite proposes that a symbol should NOT be removed, then it is NOT removed:

▸ A & ~B and C & B: B will NOT be removed because of the first rule, but its weight will be removed

▸ A & -B and C & ~B: neither weight, nor symbol B will be removed

▸ Removal could be forced by “^” symbol:

▸ A & ^B and C & -B: weight and symbol B are both removed

PRACTICAL EXAMPLES

A SIMPLE REGEXP EXPRESSION

local reconf = config['regexp'] -- Define alias for regexp module

-- Define a single regexp rulereconf['PRECEDENCE_BULK'] = { -- Header regexp that detects bulk email re = 'Precedence=/bulk/Hi', -- Default score score = 0.1, description = "Message marked as bulk", group = 'upstream_spam_filters'}

rspamd.local.lua:

PRACTICAL EXAMPLES

A MORE COMPLEX EXAMPLErspamd.local.lua:

local reconf = config['regexp'] -- Define alias for regexp module

-- Define encodings types-- /X is undecoded headerlocal subject_encoded_b64 = 'Subject=/=\\?\\S+\\?B\\?/iX' local subject_encoded_qp = 'Subject=/=\\?\\S+\\?Q\\?/iX'-- Define whether subject must be encoded (contains non-7bit characters)local subject_needs_mime = 'Subject=/[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1f\\x7f-\\xff]/X'

-- Final rulereconf['SUBJECT_NEEDS_ENCODING'] = { -- Combine regexps re = string.format('!(%s) & !(%s) & (%s)', subject_encoded_b64, subject_encoded_qp, subject_needs_mime), score = 3.5, description = "Subject contains non-ASCII chars but it is not encoded", group = 'headers'}

PRACTICAL EXAMPLES

A MORE COMPLEX EXAMPLErspamd.local.lua:

local reconf = config['regexp'] -- Define alias for regexp module

-- Define encodings types-- /X is undecoded headerlocal subject_encoded_b64 = 'Subject=/=\\?\\S+\\?B\\?/iX' local subject_encoded_qp = 'Subject=/=\\?\\S+\\?Q\\?/iX'-- Define whether subject must be encoded (contains non-7bit characters)local subject_needs_mime = 'Subject=/[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1f\\x7f-\\xff]/X'

-- Final rulereconf['SUBJECT_NEEDS_ENCODING'] = { -- Combine regexps re = string.format('!(%s) & !(%s) & (%s)', subject_encoded_b64, subject_encoded_qp, subject_needs_mime), score = 3.5, description = "Subject contains non-ASCII chars but it is not encoded", group = 'headers'}

PRACTICAL EXAMPLES

A MORE COMPLEX EXAMPLErspamd.local.lua:

local reconf = config['regexp'] -- Define alias for regexp module

-- Define encodings types-- /X is undecoded headerlocal subject_encoded_b64 = 'Subject=/=\\?\\S+\\?B\\?/iX' local subject_encoded_qp = 'Subject=/=\\?\\S+\\?Q\\?/iX'-- Define whether subject must be encoded (contains non-7bit characters)local subject_needs_mime = 'Subject=/[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1f\\x7f-\\xff]/X'

-- Final rulereconf['SUBJECT_NEEDS_ENCODING'] = { -- Combine regexps re = string.format('!(%s) & !(%s) & (%s)', subject_encoded_b64, subject_encoded_qp, subject_needs_mime), score = 3.5, description = "Subject contains non-ASCII chars but it is not encoded", group = 'headers'}

PRACTICAL EXAMPLES

COMPOSITES EXAMPLElocal.d/composites.conf:

# Ignore forged recipients in case of mailing list composite "FORGED_RECIPIENTS_MAILLIST" { # MALLIST symbol is preserved expression = "FORGED_RECIPIENTS & -MAILLIST"; }

# Ignore forged sender if a message has been forwarded composite "FORGED_SENDER_FORWARDING" { # Symbols from `forwarding` group are removed expression = "FORGED_SENDER & g:forwarding"; }

# Ignore forged sender if a message has been from the mailing list composite "FORGED_SENDER_MAILLIST" { # Symbol 'FORGED_SENDER' is forced to be removed expression = "^FORGED_SENDER & -MAILLIST"; }

PRACTICAL EXAMPLES

COMPOSITES EXAMPLElocal.d/composites.conf:

# Ignore forged recipients in case of mailing list composite "FORGED_RECIPIENTS_MAILLIST" { # MALLIST symbol is preserved expression = "FORGED_RECIPIENTS & -MAILLIST"; }

# Ignore forged sender if a message has been forwarded composite "FORGED_SENDER_FORWARDING" { # Symbols from `forwarding` group are removed expression = "FORGED_SENDER & g:forwarding"; }

# Ignore forged sender if a message has been from the mailing list composite "FORGED_SENDER_MAILLIST" { # Symbol 'FORGED_SENDER' is forced to be removed expression = "^FORGED_SENDER & -MAILLIST"; }

PRACTICAL EXAMPLES

COMPOSITES EXAMPLElocal.d/composites.conf:

# Ignore forged recipients in case of mailing list composite "FORGED_RECIPIENTS_MAILLIST" { # MALLIST symbol is preserved expression = "FORGED_RECIPIENTS & -MAILLIST"; }

# Ignore forged sender if a message has been forwarded composite "FORGED_SENDER_FORWARDING" { # Symbols from `forwarding` group are removed expression = "FORGED_SENDER & g:forwarding"; }

# Ignore forged sender if a message has been from the mailing list composite "FORGED_SENDER_MAILLIST" { # Symbol 'FORGED_SENDER' is forced to be removed expression = "^FORGED_SENDER & -MAILLIST"; }

PRACTICAL EXAMPLES

COMPOSITES EXAMPLElocal.d/composites.conf:

# Ignore forged recipients in case of mailing list composite "FORGED_RECIPIENTS_MAILLIST" { # MALLIST symbol is preserved expression = "FORGED_RECIPIENTS & -MAILLIST"; }

# Ignore forged sender if a message has been forwarded composite "FORGED_SENDER_FORWARDING" { # Symbols from `forwarding` group are removed expression = "FORGED_SENDER & g:forwarding"; }

# Ignore forged sender if a message has been from the mailing list composite "FORGED_SENDER_MAILLIST" { # Symbol 'FORGED_SENDER' is forced to be removed expression = "^FORGED_SENDER & -MAILLIST"; }

PRACTICAL EXAMPLES

COMPOSITES EXAMPLElocal.d/composites.conf:

# Ignore forged recipients in case of mailing list composite "FORGED_RECIPIENTS_MAILLIST" { # MALLIST symbol is preserved expression = "FORGED_RECIPIENTS & -MAILLIST"; }

# Ignore forged sender if a message has been forwarded composite "FORGED_SENDER_FORWARDING" { # Symbols from `forwarding` group are removed expression = "FORGED_SENDER & g:forwarding"; }

# Ignore forged sender if a message has been from the mailing list composite "FORGED_SENDER_MAILLIST" { # Symbol 'FORGED_SENDER' is forced to be removed expression = "^FORGED_SENDER & -MAILLIST"; }