Post on 17-Oct-2020
Sympa Mailinglist Manager
Stefan Hornburg
GP/RW Erlangen 4th March 2020
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 1 / 56
Introduction
About Sympa
Mailing list managerAlmost 23 years oldLegacy Perl
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 2 / 56
Introduction
Applications
Mailing list managerNewsletterMarketing
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 3 / 56
Introduction Users
Users
UniversitiesRiseup (riseup.net)
20 thousand lists3.3 million users4.8 million subscriptions
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 4 / 56
Introduction Customers
Customers
HTWK LeipzigOpen Hardware Repository (CERN)Shop for riderseShop (US government)Social Security Law (USA)
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 5 / 56
Introduction Competition
Email Marketing
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 6 / 56
Introduction Motivation
Motivation
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 7 / 56
Introduction Motivation
Motivation
PrivacyInfrastructureOpen SourceFlexibilityCosts
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 8 / 56
Introduction Flexibility
Flexibility
ConfigurationTemplatesScenariosPlugins
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 9 / 56
Sympa basics
Sympa basics
List typesList creation templatesScenarios
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 10 / 56
Sympa basics List types
List types
Public discussion mailing listNewsletterConfidential list...
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 11 / 56
Sympa basics List creation templates
Template files
config.tt2 (Configuration)comment.tt2 (Description)
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 12 / 56
Sympa basics List creation templates
Extract of newsletter configuration
visibility noconceal
send newsletter
process_archive on
archiveweb_access publicmail_access owner
subscribe open
unsubscribe open,notify
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 13 / 56
Sympa basics Scenarios
Incoming email for newletters
# send.newsletter scenario
title.gettext Newsletter, restricted to moderators
is_editor([listname],[sender])smtp,dkim,smime,md5 -> do_it
true()smtp,dkim,smime,md5 -> reject(reason='send_editor')
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 14 / 56
Sympa basics Scenarios
Authentication methods
smtp From: address in emailmd Web interface authenticationdkim Valid DKIM signaturesmime Valid S/MIME signature
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 15 / 56
Sympa in production
Sympa in production
HTWK LeipzigSympa university wide
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 16 / 56
Sympa in production Installation and requirements
Installation
Sympa 6.2.16PostgreSQL 9.4PostfixNginx
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 17 / 56
Sympa in production Installation and requirements
Requirements
Migration from MajordomoAuthentication AD/LDAPAutomated mailing listsListmastersAdaption of CD
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 18 / 56
Sympa in production Authentication AD/LDAP
Authentication AD/LDAP
LDAP only
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 19 / 56
Sympa in production Authentication AD/LDAP
Authentication AD/LDAP
ldapregexp .*host ldap.htwk-leipzig.de:636bind_dn itsz-sympa@htwk-leipzig.debind_password nevairbesuffix dc=htwk-leipzig,dc=deget_dn_by_uid_filter (uid=[sender])get_dn_by_email_filter(&(objectcategory=person)(mail=[sender])(!(userAccountControl:1.2.840.113556.1.4.803:=2)))
email_attribute mailscope subuse_tls ldapsca_file /etc/ssl/certs/htwk-chain.pemca_verify required
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 20 / 56
Sympa in production Automated mailing lists
Automated mailing lists
LDAP queriesList templates
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 21 / 56
Sympa in production Automated mailing lists
LDAP query for specific seminar groups
(&(objectcategory=person)(extensionAttribute3=08FOO15)(mail=*)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 22 / 56
Sympa in production Automated mailing lists
Get all seminar groups
(&(objectcategory=person)(extensionAttribute3=*)(mail=*)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 23 / 56
Sympa in production Automated mailing lists
LDAP with Perl
Net::LDAPNet::LDAP::Control
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 24 / 56
Sympa in production Automated mailing lists
LDAP with Perl
# apply page and sort controls$page = Net::LDAP::Control::Paged->new( size => 500 );$sort = Net::LDAP::Control::Sort->new( order => "cn" );
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 25 / 56
Sympa in production Automated mailing lists
Adjust list names
# replace / and ä ö ü ßuse Encode 'decode';use Lingua::DE::ASCII 'to_ascii';
$group =~ s/\//_/g;$group = lc( to_ascii( decode( 'UTF-8', $attribute ) ) );
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 26 / 56
Sympa in production Automated mailing lists
List creation
List directoryList from templateSync subscribersUpdate email aliases
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 27 / 56
Sympa in production Adaption of CD
Adaption of CD
Title and Logologo_html_definition<a title="HTWK home" href="http://www.htwk-leipzig.de/"><img alt="HTWK logo" src="/sympa/icons/htwk-logo.svg"></a>CSS templateweb_tt2/css.tt2Header templateweb_tt2/header.tt2
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 28 / 56
Sympa in production Problems
Synchronisation problem
lots of lists at the same timeLDAP/AD server error
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 29 / 56
Sympa in production Problems
Synchronisation problem
ttl 3 hoursrandom 0-59 minutes
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 30 / 56
Sympa in production Problems
Caching problem
stale dataowners, moderatorspermissions
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 31 / 56
Integration
Integration
AuthenticationData Sources
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 32 / 56
Integration Authentication
Authentication
LDAPCASSSOTLSShibbolet
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 33 / 56
Integration Data Sources
Data Sources
LDAPSQLSympa ListFile
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 34 / 56
Integration Data Sources
Data Sources
SubscribersModeratorsOwners
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 35 / 56
Integration Data Sources
Data Sources
Update in regular intervals (ttl, distribution_ttl)Marker for included recordsUser can unsubscribe
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 36 / 56
Integration Data Sources
Data Sources
include_file /home/sympa/subscribers.txt
test@linuxia.de Test Testerbar@linuxia.de Foo Barnobody@linuxia.de
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 37 / 56
Features
Features
PersonalizationProtectionHierarchyScenariosPlugins
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 38 / 56
Features Personalization
Enable personalization
Web interface: Sending/receiving setupList config: merge_feature on
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 39 / 56
Features Personalization
Example
<p>Welcome to our great workshop, [% user.gecos %]!</p>
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 40 / 56
Features Personalization
Example
<p>Welcome to our great workshop, Stefan Hornburg!</p>
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 41 / 56
Features Personalization
Add custom attributes
custom_attributeid levelname Your game leveltype integer
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 42 / 56
Features Personalization
Add custom attributes
<p>Welcome to our great workshop, [% user.gecos %]
</p><p>Your game level is [% user.custom_attribute.level.value %].
</p>
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 43 / 56
Features Personalization
Add custom attributes
<p>Welcome to our great workshop, Stefan Hornburg!
</p><p>Your game level is 25.
</p>
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 44 / 56
Features Protection
Protection
DKIMDMARC / ARCS/MIME
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 45 / 56
Features Hierarchy
Hierarchy
ServerVirtual hosts (Robots)List
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 46 / 56
Features Plugins
Plugins
Template pluginsModlist/Whitelist pluginDistribution notification pluginmessage_hookpost_archive DistributionNotification
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 47 / 56
Challenges
Challenges
Reverse DNSSPF/DKIMBouncesAddress verificationLog parsing (Sisimai)Blocking
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 48 / 56
Challenges Vetting email addresses
Vetting email addresses
my $validator = Email::Valid->new(-mxcheck => 1,
);
unless ($validator->address($email)) {warn "Email $email invalid: " .
$validator->details . "\n";}
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 49 / 56
Future Projects
Projects
REST / GraphQL APIContainerAnsibleArchive search
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 50 / 56
Future API
API
SOAP APIREST APIGraphQL APIDBIx::Class schema
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 51 / 56
Future Container
Container
SympaDatabase (PostgreSQL, MySQL, ...)MTA / IMAP / getmailWeb frontend
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 52 / 56
Future Ansible
Ansible
ProvisioningTesting
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 53 / 56
Future Ansible
Archive Search
setup Elasticsearch and index structurewrite indexer that parses the emails in the archivesetup very simple website for searchauthentication for website is list membership
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 54 / 56
Finish Questions
Questions
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 55 / 56
Finish The end
The end
Stefan Hornburg Sympa Mailinglist Manager GP/RW Erlangen 4th March 2020 56 / 56