Open Mic Q&A: Implementing Domino Attachment and Object Service (DAOS) - 16 June 2010

News

Abstract

IBM recently hosted an Open Mic conference call with Lotus Development and Support Engineers to discuss "Implementing Domino Attachment and Object Service (DAOS)". This document contains Q&A from that call.

Content

IBM recently hosted an Open Mic conference call with Lotus Development and Support Engineers to discuss "Implementing Domino Attachment and Object Service (DAOS)".

General DAOS resources

- DAOS FAQ- DAOS Quick Start guide- DAOS best practices- DAOS backup and restore- IBM Lotus Domino going green: The new Lotus Domino attachment and object service- Achieving ultimate storage and server cost savings with DAOS in IBM Lotus Notes and Lotus Domino 8.5- Download the Domino Attachment and Object Service Estimator Tool version 1.5

Q: How does DAOS determine duplication?A: It is based on the content of the file

Q: Can DAOS find duplication across the message, and if so, does it checksum the content, compare name and size or something else? A: Duplicates are across NSFs and are based on a SHA key generated from file content. The attachment name and date are not included in the checksum process. If you were to detach a.jpg and rename it and send it out as b.jpg, it would be recognized as having duplicate content.

Q: Is Replication now called "synchronization"?A: No, synchronization refers to the DAOS synchronization process which examines all of the references to the shared attachment data from all of the NSF files to ensure it is counted properly.

Q: When will Notes/Domino 8.5.2 be available? A: It is expected to ship in Q3, see the Notes/Domino Fix List website.

Q: Will 8.5.2 require any service packs to implement DAOS?A: You will be able to upgrade your existing Domino environment to 8.5.2 without change to your host operating system

Q: Is Notes/Domino 8.5.1 FP3 available?A: Yes, see Download options for Notes/Domino 8.5.1 Fix Packs (Technote #4025721)

Q: What is DAOS?A: Domino Attachment Object Service. See IBM Lotus Domino going green: The new Lotus Domino attachment and object service

Q: If local replicas are created, I understand that the local replicas get a copy of the attachment. If we later restore a database from a local replica, will the file attachments be reposited and what is this process? A: Yes, if you restored from a local replica you would need to run compact again to get attachments back into DAOS

Q: Are attachments stored in their native form, in the HTML encoded form, compressed or something else? A: Attachments are stored in their native format, but are encrypted with the server's key by default so they are protected

Q: Does DAOS needs its own independent server?A: No. DAOS can be enabled on each Domino server to take advantage of its benefits.

Q: How and when do the files get removed from the NLO file? When the last person deletes their email with the attachment? A: NLOs are deleted after no one is referencing them and the "deferred delete interval" has expired

Q: Is there any detection and notification if a compact needs to be run to reprocess messages for attachments?A: Once DAOS is enabled on an NSF, all subsequently written attachments will be considered for DAOS participation automatically.

Q: If I enable DAOS on a Domino cluster with 3000 users, what is the size of transactional logging?A: If you are running circular logging, 4GB would be the recommendation

Q: Can I enable DAOS on a Domino server on RHEL? A: Yes, DAOS is supported on all platforms

Q: If a user with attachments in DAOS is deleted, how does DAOS know it can delete the attachment of that user and that person was the last user of the attachment? A: DAOS will decrement the reference count for the NLOs, and if deceremented to 0, each NLO will be subject to the deferred deletion interval

Q: Can we enable DAOS on databases ODS 43 (Domino 7 ODS)?A: No. ODS 51 (new to 8.5) is required. You can upgrade via create_r85_databases=1 INI parameter and running copy-style compaction

Q: Does DAOS need its own separate storage space much like transaction logging or is it stored within Domino? A: Check out the wiki article, DAOS best practices for storage location.

Q: How does DAOS impact antivirus (Trend Micro, Symantec, etc ) software? How does the encryption impact the scanning?A: There are multiple techniques used by antivirus products. Typically, for Domino, the antivirus product uses an add-in or "hook" to monitor the attachments as they are received into mail.box, or possibly, when attachments are sent by a user. These methods continue to work with DAOS when an attachment is deposited in a Notes database. If an attachment does not need to be sent

because it is already in DAOS (so Notes does not bother sending it again) then it is not scanned or, put another way, attachments will only be scanned the first time they are received. Most major antivirus software provide a method for scanning mail files where they actually open and scan each attachment. These scanners continue to work as long as they open the database on the Domino server (as opposed to doing a local database open ). The final method that antivirus vendors use is to do a direct file scan. This method is ineffective when DAOS is used because DAOS encrypts the NLO disk files. We do not recommend using file scans with NSF or NLO files, use the vendor's mail file scanner instead.

Q: Is copy-style compaction recommended for all databases, not only the mail file?A: Yes, a copy-style compaction is necessary to DAOS-enable existing attachments for all database types.

Q: We activated DAOS on a cluster and we experienced that the DAOS directory on one server is 10 GB larger than on the other. How could we resolve this?A: Check if the DAOS catalog is in-sync on both cluster mates. When the DAOS catalog goes out of sync, DAOS suspends object deletions. So it is possible that the server using more disk space needs to have the DAOS catalog resynchronized: On the Domino console, type SHOW STAT DAOS.ENGINE.*

Q: Can a Notes 7 client access a database in ODS 51 with DAOS enabled? A: Yes, all clients can access DAOS-enabled databases. The technology is server only and invisible to the client.

Q: If there is only one unique attachment, will it be extracted from the NSF or will it only be extracted if there are two or more?A: All attachments are considered for DAOS based on size. A single unique attachment will be eligible for DAOS storage if it meets the minimum size setting.

Q: What is the best backup software to use with DAOS and transactional logging? A: Assuming you are running archive style txn logging, then any of the backup vendors that support the Domino APIs is acceptable. Working for IBM, we recommend Tivoli Data Protection.

Q: If DAOS puts files in by size, is that what the DAOS min size report is talking about? What determines the size of the attachments going in? A: It is the size of the attachment after compression. This is configured in the Server Document for the server and the default minimum size is 64000 bytes.

Q: If I created a replica on another server which is not DAOS-enabled, will the attachment come back with the NSF?

A: Yes, per database option. You can also pack the attachments back into the NSF using compact -daos of dbname

Q: In the Lotus Foundations environment, are there any differences in default settings or caveats? A: There are no considerations specific to that environment. The general DAOS guidelines apply.

Q: Can we only enable DAOS for only some users and not for all users of a server?A: DAOS is enabled on an NSF basis (such as an individual mailfile), not per user. All attachments in a DAOS-enabled NSF created by all users will be considered for DAOS storage.

Q: When restoring a DAOS-enabled database, how can we find what is the NLO for a missing email being restored? A: The tool, LISTNLO, will tell you what NLOs are missing in the restored database. See the DAOS backup and restore wiki article.

Q: Is each NLO file encrypted as an attachment?A: By default, all NLO files are encrypted with the server key. There is a notes.ini variable that you can set to disable the NLO encryption. This encryption is in addition to any other compression/encryption options that you had set at the NSF level.

Q: If you send an email to another Lotus server in the same domain with DAOS enabled on both, is the attachment saved in both DAOS stores on each server? A: Currently, DAOS does not share NLOs across servers, so if both servers are DAOS-enabled, then the attachment will be stored on both servers.

Q: How or when are the attachments cleaned up? What are the options for removing the attachments permanently?A: Every morning at 2am, a DAOS prune process runs to remove NLOs with zero references after the deferred delete interval has passed.

Q: If we enable DAOS on the messaging gateway server that handles all messages, it accumulates all messages entering to the domain because mail.box is DAOS-enabled. Even if these messages are being routed to other servers, they create a massive storage of attachments. In the best practices, it recommends to DAOS-enable the mail.box, but is this necessary? What are the advantages?A: No. DAOS-enabling the mail.box is optional. It has no storage advantage, but it optimizes peformance by not needing to copy bytes. Also, if you set the deferred deletion interval on the gateway server to 0, DAOS objects will be deleted when no longert needed

Q: Is there any relation between DAOS and streaming replication? I think I've heard that if the main and cluster are DAOS-enabled the attachment will not be replicated. Is this true? A: If your servers are 8.5.2 servers, DAOS will not send the attachments if they already exist on the destination server. They will only be replicated if they are needed.

Q: Do Notes clients need to be at FP3, or just the server? A: The 8.5.1 FPx DAOS fixes to date are all in the server code. There is no minimum version requirement/recommendation for the client.

Q: Does the synchronization add the manually restored NLO files to the DAOS index so that they get removed when the restored NSF is removed? A: Yes. Synchronization takes care of restored NLO files

Q: If I only have a hub server or routing server and no mail databases, can I enable DAOS on the mail.boxA: Yes, you can enable DAOS on hub servers. If the server does *only* mail routing, just reduce the deferred deletion interval to 0 or 1 . You would not want a longer deferred deletion interval in this scenario because you do not presumably back up or restore to hub server so there is no reason to keep DAOS objects around after they are not needed.

Q: Would any new copies of DAOS-enabled databases have full copies of attachments? Also, when you locally archive mail, if a message had an attachment, then the archive has the full document, right? A: Any replica/copy that was not DAOS-enabled would have a full copy of the attachment stored inline in the NSF.

Q: Is there documentation about the disk I/O with DAOS and without DAOS? A: There is some information on that in the following developerWorks article Achieving ultimate storage and server cost savings with DAOS in IBM Lotus Notes and Lotus Domino 8.5

Q: Does a Domino 8.5.1 or 8.5.2 server need a service pack to implement DAOS ? A: No

Q: What happens if a user is migrated from DAOS-enabled environment to a non-DAOS server? Do attachments get moved into the NSF before the move?A: If the user's database is moved via replication (preferred method), the attachments will be stored inline on the destination server.

Q: Any issues with a Notes 7 client accessing application databases that are DAOS-enabled on a

Domino 8.5.2 server?A: No, attachment access is transparent to the client.

Q: Are there any issues with other people getting access to attachments if you grant others access to your mail file through the delegation profile or the ACL?A: No

Q: We have Domino 8.5.1 servers and the DAOS catalog seems to get out of sync frequently. Why is this?A: Enabling DAOS logging will help determine which database caused the catalog to go out of sync and when it happened, for example, when the mail.box was opened. Domino 8.5.2 will detect and automatically correct many of these issues without requiring a resync. Enable the following Notes.ini file parameters for debugging:

debug_threadid=1console_log_enabled=1daos_logging=<path/daos.txt> STATE_CHANGE

Q: Will messages that are sent to a BlackBerry device recognize the NLO file and know where to reference to get the attachment? A: DAOS is transparent to the client, and it will receive the attachment data in the same way regardless of if it is stored in DAOS or not.

Q: On a Domino server, is there a benefit to activating DAOS on mail.box and on user's mail files? A: Yes, it reduces the amount of data that is needed to copy from the mail.box to the user's mail file. If you do not enable DAOS on the user's mail files, the attachments will be copied from the mail.box (DAOS) to the user's mail file (inline)

Q: Are quotas enforced against the logical or physical size of the mail database with DAOS enabled? A: The logical size. DAOS attachments are charged against database quotas just as inline attachments would be.

Q: Does Lotus Traveler work with DAOS?A: Traveler is a client and DAOS is transparent to all clients.

Q: How is DAOS related to mail journalling? A: It is actually not recommended to enable DAOS for your mtstore. Mail journaling should not use DAOS because:1) journal files are typically encrypted with a specified ID file and therefore do not have any shared content. Using DAOS will create a unique, non-shared attachment in DAOS. No disk savings are possible when using encrypted journals.

2) using DAOS for journals makes them non self-contained. This is typically *undesirable* for journals. Using DAOS means that journals cannot just be "copied off", which is what many customers do with journals. Journals are often moved physically (OS level) to a separate store for archiving/auditing.3) Journals use Huffman compression for intermediate objects used during the journaling process. These intermediate objects will be stored in DAOS temporarily, but assuming you have a 30 day deferred deletion interval, they will be kept on disk for 30 days after the last reference is deleted. In general, DAOS is not advantageous in situations where temporary recompressions are happening in different formats (this is also the reason we recommend running compact -Zu before enabling DAOS)

Q: Would it be beneficial for Domino to have two different prune settings: one for the mail.boxes and one for mail files? A: The prune setting can only be configured for the server, not individual databases.

Q: Do I use the convert task to DAOS-enable a database?A: It is the compact task which DAOS-enables a database, not convert

Q: For iNotes/Webmail users, is there a significant wait time when retrieving a DAOS attachment? A: No

Q: When restoring a DAOS-enabled database, do we need the transaction logs at all? Does it matter how we back them up?A: If you are restoring to the backup (not applying the txn logs) you can run circular logging and not worry about them being backed-up or archived

Q: Do all databases on the server need to be at ODS 51 or only the ones that are DAOS-enabled?

A: Only databases participating in DAOS storage need to be at ODS 51.

Q: Is there a max data retention period? A: The attachment data will be retained as long as there is at least one valid reference existing to it. After that goes to 0, the retention period is controlled by the deferred deletion interval. You can't disable the interval entirely, but you can set it to a very large number like 32000.

Q: How can you tell if a database NSF file is at ODS 51?A: Several ways: Database properties, Admin client -> Files tab, and SHOW DIR task

Q: If you have a cluster and DAOS is enabled on both servers and you restore a database from Server A to Server B, is it able to access the attachments? A: Yes

Q: Is there documentation regarding the encryption issue within a clustered environment? A: The encryption settings are mentioned in the wiki article, DAOS Best Practices

Q: Is there anything to watch out for implementing DAOS on the iSeries? A: iSeries seems to be more sensitive to a (potentially) large number of NLO files. For that reason, the default minimum participation size is set higher (1MB) for iSeries than the other platforms to reduce the number of NLO files created. Also, due to the architecture of the iSeries disk subsystem, the physical location of the DAOS repository and transaction logs is less critical. We recommend for all platforms that the DAOS repository be not logically located underneath the data directory, as some utilities may traverse the repository unnecessarily while scanning the data directory.

Q: Is there documentation or recommendations on how to move a Domino server from one platform to another if DAOS has been enabled?A: There is no known doc that discusses this, but with some configuration and the moving of all NLO files along with the server's databases, this should be doable.

Q: Are there any size limitations for attachments in DAOS (other than operating system limitiations)? A: There are no size limits on attachments

Q: What is the nature of the DAOS catalog file? Is there any essential information in that file that can be lost or that is not stored in the NLO files?A: The catalog file tracks all NLO objects and databases that participate in DAOS storage

Q: How do I restore a database that has thousands of NLO files? Our archive server is full with over 3 TB of data with some huge archive files. Problem is, if I had to restore one of these files from backup tape, I will have to manually restore over a thousand NLO files which will be nearly impossible. Any suggestions? A: You only need to restore NLO files that are missing from the current DAOS repository. If there are other references to the same attachments by other NSF files, the NLO files will still exist in the repository.

Q: Can I expect any issues with an upgrade/move from Domino on Windows 32-bit to Windows 64-bit? A: DAOS should not be affected by this upgrade

Q: Can you set DAOS to never delete attachments or is the limit 90 days? A: There is no setting to never delete attachments (0 will cause it to delete them immediately) but you can set the deletion interval to a very high value such as 32000.

Q: If we have mail databases, Document library, and Discussion databases on the same server, will DAOS remove the attachments on all databases? A: DAOS is enabled per database; however, you can run a copy-style compaction against a directory to enable all databases in the specified directory.

Q: What is the file daos.cfg on the data folder? Should it be moved if migrating servers? A: The daos.cfg file contains information about the daos directory and subdirectories. If all paths are staying the same you can move it, otherwise, it will be regenerated on server startup.

Q: We have run daos -off on a particular database, but when I tell daosmgr dbsummary, the database still shows synchronized but with 0 files pulled.A: Yes, but as long as there are no references, and DAOS has been turned off for the NSF, there will be no DAOS activity for that NSF.

Q: If I migrate a DAOS-enabled Server A to Server B (also DAOS-enabled), and create replicas of all databases from Server A to Server B, run cmpact -c on Server B, to make sure DAOS is used on Server B, then I shut down both servers, move the server ID from Server A to Server B and copy the DAOS store from Server A to Server B. would this work? A: It is not the recommended or supported process, but yes it should work. The problem with that method is that as NLO files are created, the location of those NLO files are stored as 'hint' information in the DAOS ticket stored in the NSF in place of the attachment. Because the order of creation of the NLO files would be different between the two servers, the location of the NLO files would be different, so many of the the hints would be wrong. DAOS handles that situation by consulting daoscat.nsf to find the correct location, so the attachment would be accessible, but you would get increased traffic on daoscat dealing with all the lookups due to the bad hints. Running fixup on an NSF will update the hint information using information from daoscat. If you did a DAOS resync to ensure the NLO locations were cataloged correctly, and then did a fixup on all of the NSF files, that should update the hints properly. Again, although it would probably work, it is not a supported approach.

Q: Can a database physical size be used for quota enforcement instead of database logical size? A: There is currently no way to do that, but we have heard that request before, and are considering that for a future release.

Q: We have a cluster on three servers, all of the files are on the third server. Can we only enable DAOS on the first and second servers? A: DAOS is enabled independently on each server (and each NSF on that server) regardless of cluster status. You can have DAOS enabled on some, none, or all of the servers. Each server that has DAOS enabled must have its own DAOS repository.

Q: With transaction logging required for DAOS, what is the recommended logging style for DAOS? A: It depends on what your requirements are, but either circular or archive are supported and work

Q: Is there a difference in the backup restore process for circular versus archive style logging? A: As for backup, if you are running circular, you can take the server down and just backup the NSF files, but archive requires the use of a Domino backup 3rd party vendor (uses the domino APIs). Circular only supports restoring to the backup of the database, whereas with archive style you can recover a database to any point in time between the backup and currency. See also the DAOS backup and restore wiki article.

Q: I only have 79 GB available in the mail directory. Can I still enable DAOS and do the compacts to turn it on without running out of space?A: One benefit of DAOS is that it will save you space, so that should work fine. Three suggestions: 1) Run the DAOS Estimator, that will give you an estimate of the before/after space usage so you can plan accordingly. 2) The DAOS repository does not need to reside in the same directory (or even on the same filesystem) as the data directory. In a typical mail environment, the traffic to the DAOS repository is a small fraction of the traffic to the data directory, so you can use cheaper storage (including network storage) for the DAOS directory if that helps. 3) You do not have to enable DAOS on everything at once. Enable DAOS on the server, then enable DAOS on mailfiles a few at a time. That will give you an idea of the space usage trend.

Q: Are there any known issues with DAOS with Tivoli Storage Manager and Tivoli Data Protection? A: TSM and TDP work with and are supported with DAOS. There are no known problems.

Q: I have heard mentioned several times that NLO files are encrypted by server certificate. Please verify if this refers to the server.id file and corresponding values found in the server document in the NAB or if you are referring to an actual x.509 or CA certificate installed on the server. A: Server.ID

Q: I have several Domino servers on an AIX platform. Are there any plans to allow "sharing" of a DAOS directory (or server) across them?A: We have considered this, but there are no firm plans to implement it.

Q: If a DAOS-enabled file is deleted from the server and then restored from backup, rather than restoring the NLOs , can you just re-enable the restored file for DAOS? A: If you restore that database before the deletion interval then all NLOs will still be present

Q: Why is transaction logging required for DAOS? A: It is required because of the need to keep database actions and object actions within the same transaction.

Q: Is the suggested FixPack level for optimum DAOS functionality 8.5.1 FP3? A: Yes. FP3 or FP2 IF1 are the minimum versions we recommend for production deployments. There are no DAOS-related fixes in FP4.

Q: Do you need a certain level of ODS to run the DAOS Estimator? A: DAOS Estimator will run on any databases and any versions of the server Release 6 or higher

Q: If an item gets deleted after no activity for 30-90 days, what happens for an e-discovery?A: Unreferenced objects only are deleted after the prune interval. If you need objects that are no longer referenced, your backup plan for NLOs should take that into account. It's a good practice to ensure that your NLOs, along with your NSFs, are backed-up regularly

Q: When I run DAOSEST -O FILENAME from the console, where is the output file?A: The output to the console will tell you the path to the file

Q: We recently had corruption in the daoscat.nsf file. The recommendation was to bring the server down, delete the file, then bring server up so it could build a new one. Was this the right thing to do?A: Yes. If the database is corrupt it should be deleted, but it may take a significant amount of time to recreate depending on how many NLOs and databases you have. We strongly advise that customers do contact IBM Support before deleting the daoscat to make sure there is on alternative solution.