MAIL MANAGEMENT - AdminCamp...Tell DAOSMgr Status mail\gdavis.nsf how many NLOs etc in use in my...
Transcript of MAIL MANAGEMENT - AdminCamp...Tell DAOSMgr Status mail\gdavis.nsf how many NLOs etc in use in my...
Why Mail Management?
Use of mail whether it’s Notes, Web, Traveler or even IMAP/POP is probably the most significant interaction with Domino that your users have
Many things affect Mail performance and the user experience of mail and you can do a lot to optimise it and make user’s happier
Much of the work is done on the server and to the databases and server configuration and is nothing to do with their client
Mail Databases
Database Size
Database size isn’t just about disk space, it’s also about performance
It takes exponentially more work for a Domino server to perform a view rebuild or update a full text index on a 5GB database than it does on a 500MB one
This is also reflected in client performance, slowness to open views and move between folders as well as create new folders and file messages
Maintaining a managed database size can be done using quotas but you should never deploy a quota without a warning level and the warning level should be no more than 80% of the quote size
Previously we used quotas and archiving to maintain databases sizes
both of these things punished the users
DAOS is a better alternative
Inbox MaintenanceKeeping down the number of messages in an inbox also helps performance
Once someone has 6000 inbox messages they really aren’t going to file them in folders ever
Inbox Maintenance is an agent in the Mail file
Run by Adminp on Mail Server
Defined by Server Document or Mail Policy
Inbox Maintenance - Requirements
Mail8 template
Directory on server based on v8 template
Adminp process to be running
Server document schedule defined
Mail Policy ConfigurationDifferent inbox maintenance settings can be applied to users via the polices Mail settings - Mail - Basics*
Tell Adminp Process mb - forces an immediate application of the policy for each mail user
The schedule is still determined by the server document
DDM.NSF
LOG.NSF
ADMINP_VERBOSE_POLL_TASK=2
*Overridden by Server document settings
Design
Maintaining up to date database designs impacts not just the user experience but also performance
Updated mail templates have more efficient code and behaviour
OOO for example in v8 mail designs or later
The design version should never be newer than either the server or client versions but can always be older
Database Properties
Keeping databases on the newest ODS (currently 51) enables you to take advantage of the newest server features and performance improvements
Some database properties won’t be set in older files that have never been changed. You’ll need to set those as soon as you can
LZ1 , replicating unread marks
Always compact and rebuild the indexes when you upgrade the ODS of a database
compact -c (copy style)
updall -RX (rebuild view indexes and full text indexes)
Managed Mail ReplicasLocal replicas managed by desktop profiles
Users don’t need to understand the concepts of replication or outboxes
Behaves as though you were on a server replica
Better performance as it avoids network latency issues
The local replica is automatically created by the policy
If it needs fixing or compacting it does this automatically too and fails the user over to a server replica
Existing users with local replicas can be changed to managed replicas via a policy settings
For remote users it’s a better alternative to trying to access server based mail over a WAN connection
DAOS
Domino Attachment Object Store (DAOS)
DAOS was new with Domino 8.5
but really started working well in 8.5.1 and later
In a DAOS enabled mail file, all the attachments are removed and stored as flat encrypted files on the file system called NLOs
the Domino mail file contains a file reference to that file in each message
DAOS is controlled by DAOSManager which is a Domino task that acts at a “sub Domino” level.
to the user and to the Domino administrator nothing really has changed
Some DAOS Behaviours
If I send a message to 10 people with a file attachment, that message will be removed and stored as a NLO only once
Each person’s copy of the message will have a pointer to the same file
Only when everyone who had a pointer to that file, deletes the message, will the file itself be marked for purging
The default is to prune deleted / unused NLOs after 30 days
If I send a message to Chris with a file attachment, that file attachment will be removed from the message and stored as a NLO if the file size is above a minimum size limit set in the server document
A local replica of a DAOS enabled database will have all the attachments still in it. As the replica is created the replication task creates each document with its attachment taken from the DAOS store
File Sizes
DAOS enabled files have both a physical and logical file size
The logical file size is how big the database would be if it contained all attachments. Such as if you replicated it locally or to a non DAOS server
The physical file size is how big the NSF actually is
The difference between the two is your DAOS disk saving
I have seen anywhere from 40 - 90% reduction in NSF size
Do you need to enable archiving if you can simply reduce the NSF size using DAOS?
Backups
Backups are the one thing that changes when you enable DAOS
If you backup a DAOS enabled file you are only backing up the NSF, unless you backup software is “DAOS aware”.
The NLOs are read only files so they would usually only be backed up incrementally and they sit outside the Domino data directory so your regular backup routine will miss them
By default deleted NLOs are only pruned after 30 days so any backups restored within 30 days will still have NLOs in place on the server
If you don’t have DAOS aware backup software then you have to backup both the NSFs and the NLOs and then match them up on restore
DAOS aware backup software will backup both the NSF and all its matching NLOs and can restore the two together
How To Enable DAOSMake sure your servers are 8.5.1 or later (you can do it on 8.5 but I wouldn’t recommend it)
DAOS needs a disk separate from the location of NSFs
You will kill the server with disk I/O and performance issues otherwise
DAOS needs transaction logs enabled. I often use the same disk for DAOS and transaction logs
The database needs to be ODS51 or later
Add the value Create_R85_Databases=1 to the server notes.ini and then compact the databases with a -c switch to upgrade the ODS
Update the server document with the location of the transaction logs and where the NLOs should be stored
Compact the required database with the -DAOS ON switch
Useful DAOS Commands
Tell DAOSMgr Status
Tell DAOSMgr Status Catalog
Tell DAOSMgr Resync
if Catalog needs rebuilding
Tell DAOSMgr Status DBSummary
all databases being managed by DAOS
Tell DAOSMgr Status mail\gdavis.nsf
how many NLOs etc in use in my mail file
Tell DAOSMgr LISTNLO ALL mail\gdavis.nsf (-o filename.txt)
List of all NLOs referenced in my mail file (for restore purposes)
Some Things To Remember
Replicating a DAOS enabled database to another server that doesn’t have DAOS will result in all the file attachments being stored inside the replica on that destination server
The disk space required will be the logical file size
The server document has a setting for the minimum attachment size before the attachment is removed. The default is 64k
on 8.5 it was 4k so verify your settings if you deployed on 8.5 originally
Quotas apply to the logical, not physical file size
In a cluster NLOs won’t be resent with messages if they already exist on the destination server
Standardising on LZ1 compression for attachments as a database property means that duplicate NLO files won’t be created for both Huffman and LZ1
More Things To Remember
Servers don’t all have to be DAOS enabled
even servers in a cluster
Not all databases on a DAOS enabled server need to be enabled for DAOS
The design of the database doesn’t matter to DAOS which is sub domino so you can DAOS enable a v6 database as long as its ODS is 51
Performance for Domino server tasks is greatly improved when the databases physical size is so much smaller
view indexes, user interaction, folder moves and searches all quicker
Directories
What Have Directories To Do With Mail?
Addressing and duplicate addressing
Vulnerability for directory harvesting or spam attacks
Mostly though - it’s about client performance
Type ahead addressing uses the directories, if the directories are slow so will the “Notes client” be
Finding matches for addresses or duplicate addresses is reliant upon the Directories
Make sure you complete the Directory server field on location document
Avoid local replicas of names.nsf
If you must use a mobile directory catalog but only for “offline” users otherwise it will become out of sync and conflict with the server replica of names.nsf
Directory Assistance
If you use Directory Assistance, make sure your directories are all working using the settings you expect. Not using a replica on a server in Singapore
Sh XDir
Authentication-only / authorisation-only secondary directories prevents them being used for mail addressing or routing
You have both a Domino™ and an LDAP directory that contain some identical names.
You do not want to use the LDAP directory's names for mailing.
Your mail clients are experiencing "Ambiguous name" dialog boxes when sending mail.
DA Modified Form for LDAP
Dirlint
Server Task (Load Dirlint)
Scans a directory for inconsistencies
inconsistencies in naming hierarchy
invalid syntax in directory names
valid group members
Load Dirlint -<directory> <directory>
Load Dirlint -NoDAorCascaded
Traveler and Syncing
Traveler Under the Hood
What’s All This Syncing About?
It is how the Traveler server scans/syncs data to devices
Consists of three different tasks
Server thread
Prime sync thread
Worker threads
Note
From 8.5.2 onwards, threads are mostly dynamic and now rarely need modifying
We have had to do this, however, when doing large Traveler server moves
From one server to another
Server ThreadThere is a SINGLE thread on the Traveler server that scans target servers
Target servers are servers that house users utilizing the Traveler service
Traveler issues a call to each server in turn
NSFGetChangedDB
Lists all changed databases since last scan
Very fast/efficient request
Traveler is served the list of all changed databases
Parses list and keeps changed databases that it is interested in
I.e., mail files
Passes the list of changed mail files to the Prime Sync Thread
By default, Traveler will scan the same server at a minimum of three seconds
Prime Sync Thread
Scans the target mail files
One prime sync can work with one mail file at a time
Identifies what has changed in the mail file
I.e., what is out of sync
Passes to device sync thread/worker thread
By default, there are 20 prime sync threads on a Traveler server
Can be increased using NTSConfig.xml
TSS_PrimeSync_Threads=n
Once complete, passed to device sync thread/worker thread
Device Sync Thread/Worker Thread
The thread that does the work
Sends changed data to device
Retrieves changed data from device
Touchpoint thread between mail file and device
Limit of 5,000 device threads
Worker thread is for internal Traveler communication
Limit of 5,000 worker threads
Traveler Threads
Possible Traveler Problems
Traveler uses a non-replicating Derby database that is critical, specific to a server, and ensures only incremental updates are sent to devicesThe Traveler task is a sub-task that is reliant on HTTP. It should be started after HTTP and stopped before HTTP.Lookups are performed against each user's mail server and honor Directory Assistance, but can instead be done entirely on the Traveler server to improve performanceSince the server threads will query for every changed database on every home mail server the Traveler server knows of for its users, keeping minimal hops between the Traveler server and the mail servers is important
As is having mail servers without thousands of non mail databases on themThe worker thread is reliant upon disk performance for writing and reading data from the Derby database
beware of disk fragmentationpoor performing disk affects traveler performance
Delete ntsclcache.nsf if you have cluster failover, routing or corruption issues. It will recreate itself
Common Mail Problems
Mail Delivery - What Do We Want
Mail from who you want to where you want
SMTP servers just managing inbound mail intended for you
A significant reduction of receiving spam from something like 90% of mail traffic to less than 10%
Ability for POP or IMAP users to relay mail without compromising security and risking blacklisting
From people you want delivered to the servers you want
Why is this a problem
Your MX records determine the inbound route of externally addressed mail but they can also be a bottleneck on your network
Any server with a SMTP listener which is running under default configuration can vulnerable
If you’re inside a firewall and one of your servers starts relaying mail you entire ip range can be blacklisted
Dictionary attacks and partial matches for SMTP addresses overload your servers and users with mail you don’t want and haven’t anticipated
Allowing authenticated relaying exposes your server to authentication by any identity with a person document and http password
Spammers know this and will try password cracks against common names
Your users hold the mail system responsible for delivering them unwanted mail and making it more difficult for them to find good mail
You network, disk space and bandwidth are not there to process mail you don’t even want
What Can We Do To Fix?Enable ‘Internet Sites’ on each server document so SMTP can’t be started by accident
Have multiple MX entries and at least one ‘failover’ entry with a very low priority
Server Configuration Document - Router - SMTP Inbound Restrictions
Restrict connecting hosts if you can
Configure fullname lookups only not partial matches and verify that name exists in a Directory before receiving mail
Reject ambiguous names and if you can group names
Perform anti relay checks for authenticated users
SMTPVerifyAuthenticatedSender=1
Even with this, Domino has to handle rejecting invalid inbound mail. Consider moving that initial SMTP listener to another source such as the Postini service , Lotus Protector or similar
What Do We Want?
Controlling where the mail comes into allows you to load balance traffic around your network.
Multiple MX records are part of your DR solution, once mail is received anywhere on your network you can route it or deliver it to the user
Being responsible in how you handle outbound mail protects you from being blacklisted
Pre-emptively blocking or denying all but complete valid addresses makes you unappealing for spammers
What Could Be Going Wrong?Spammers rely on connecting to your server and sending mail to users without knowing their addresses in advance (a DHA)
Domino servers that are enabled for SMTP listening have port 25 listening for any connecting hosts wanting to deliver mail
Server configuration documents accept ‘fuzzy matches’ on usernames unless you configure otherwise
Group names (which are often very common) are accepted for inbound mail unless you specify otherwise
If your servers are found to be sending out spam mail especially that didn’t originate from your domain you will find yourself quickly blacklisted
Enabling authenticated users to relay validates any user with a HTTP Password in any of your directories
If one of your servers behind your firewall does this then the NAT address (which could be the address of all your servers) will be blocked
What Can We Do To Fix?
Enable ‘Internet Sites’ on each server document so SMTP can’t be started by accident
In the server configuration document
Restrict connecting hosts if you are using a spam filter service in the server configuration document
Configure fullname lookups only not partial matches and verify that name exists in a Directory before receiving mail
Reject ambiguous names and, if you can, group names
Perform anti relay checks for authenticated users
SMTPVerifyAuthenticatedSender=1
Why This Works
Pre-emptively blocking or denying all but complete valid addresses makes you unappealing for spammers
Being responsible in how you handle outbound mail protects you from being blacklisted
Busy Mail Servers?
You have a constantly busy mail server and you don’t know if it’s running as efficiently as it could.
What settings can you change to improve mail routing speeds and could they make things worse?
What Do We Want?
The right number of mailboxes for the router to perform optimally
The right number of delivery threads to ensure local mail is being delivered and transfer threads to ensure mail is pushed to destination servers
Making it easy for you to track mail routing problems
Notifications of any mail bottlenecks
What Could Be Going Wrong?
The default configuration for a new Domino server is a single mailbox created and used by the router task
The Domino server will automatically set the number of Delivery and Transfer threads on your server according to memory availability
but that assumes your server is performing other Domino tasks too and isn’t dedicated for mail
the default setting for transfer threads is for one thread per destination at a time
If you chose to hold undeliverable mail as a way of combatting Spam you are overloading the mailbox with dead mail which effects how well the router task can process the live mail
You don’t have delayed mail notifications configured or mail probes configured to tell you if there are bottlenecks or problems
How Do We Fix It?In the server configuration document
Set number of mailboxes to be 2, 3 or 4
Mail.Mailbox.AccessConflicts / Mail.Mailbox.Accesses > 2% means time for a new mailbox
Delivery and Transfer threads can be set here but if not are calculated by the router based upon server memory and resources
Tell Router Sh Queues displays current settings
If you set the server to hold undeliverable mail you will need to write a mailbox agent to clear that mail out periodically so it doesn’t effect router performance
Setting notifications for delayed mail can let you know early if there are routing problems to a particular destination
Using Notes Named Networks means there are no connection documents and no easy to follow mail topology
Configure the DDM probes for messaging such as the
“Mail flow statistic check”
“Transfer Queue check”
Why This WorksAccepting default options won’t stop your server from delivering mail, but it will stop it from being as efficient as it can be
The router task uses threads to access the mailboxes, the more efficiently it can do this the faster your mail will be delivered
If your server does nothing other than route mail you can manually increase the number of transfer threads and concurrent transfer threads to improve throughput
Being pre-warned of bottlenecks allow you to deal with a problem or re-route the traffic before it becomes a backlog of too much mail
Having a defined mail topology, even in a small organisation makes it easier to trace mail and customise routing
If you have a cluster you can route the mail via a cluster mate without it being specifically defined in a connection document. Doing this for all hops can give you an entire alternate mail routing topology.
Don’t Go Too Far
Beyond what’s needed, more mailboxes is not always better. Giving the router additional mailboxes to process can be detrimental to performance
More transfer threads don’t always do what you think if you haven’t configured multiple concurrent threads to the same destination
Forcing the server to have more delivery and transfer threads than it thinks it needs, takes resources from other server activity
Ugly Emails
Your users complain of writing beautifully formatted mail and yet to people outside the company it looks completely wrong.
Internal messages forwarded between users sometimes lose a lot of formatting
What Do We Want?
Mail created in Notes or iNotes by your users, with custom formatting and layout shouldn’t be noticeably altered in transit
Internal mail should retain formatting as it is forwarded and replied to
You need to explain to your users the behaviour and limitations of mail routing
What Could Be Going Wrong?The default configuration for routing mail externally is to convert to MIME as plain text
User Mail Files contain mixed message formats
The default setting for internal mail is to be generated and delivered as Notes Rich Text
but if read via iNotes/Browser it converts to MIME on the fly
The default setting for inbound mail is that it remains and is stored in MIME format in the user’s mail file.
but if read via Notes it converts to Rich Text on the fly
The default setting for a Notes user is for an internet addressed message to be converted to MIME by the client as it sends
Different versions of Notes and Domino will convert differently so a mixed environment will return mixed results
How Do We Fix It?In the server configuration document
Set outbound MIME conversion to HTML or HTML and Plain Text (if you send to very old mail systems who can’t read HTML messages)
If your users are on different client versions then set their location documents to send internet mail as rich text
The server will then handle the mail conversion to MIME for everyone
Make users aware of limitations / behaviour of mail rendering
If they read a Notes mail message via iNotes and reply to it, they are replying with a converted MIME version which will look similar but not identical. Notes features such as sections and tabbed tables are most noticeable
Similarly if a Notes user receives a MIME message and opens it, the quality of the rendering is dependent upon their Notes client version. Two users with differing versions may see different results.
Some features are not compatible with sending internet mail, these include ‘letterhead’ and ‘mood stamp’
If a user complains that a message they sent ‘looks terrible’ get the message headers before doing anything else
Never accept a forwarded copy of the message as proof
Why This Works
Understanding how mail is stored, formatted and rendered for reading explains why you will see different effects from different messages and clients
You can only control your own mail, so making sure you send out as MIME and HTML is the best you can do to preserve fidelity
User dissatisfaction stems often from unrealistic or misconceived expectations.
Don’t Go Too Far
Standardise mail if you have radically different client and server versions and some of them are pre v7
Configure clients to send as Rich Text so the server does the conversion
Don’t let all servers do MIME conversion, just the newer ones
most sites have outbound routing hubs they can use
Make sure you allow MIME routing of messages within your domain so mail isn’t converted to Rich Text as it moves between servers