engage 2015 - IBM Notes Traveler Daily Business

1

IBM Notes Traveler Daily Business

engage 2015

René Winkelmeyer midpoints GmbH

2

René Winkelmeyer Head of Development

midpoints GmbH http://www.midpoints.de IBM Advanced Business Partner IBM Design Partner (Notes Domino, Mobile, Verse) Apple Enterprise Developer und MDM Group Member Samsung Enterprise Alliance Partner Services - Enterprise Mobility Service - Mobile Device und Application Management - IBM Notes Traveler und IBM Mobile Connect

About me

3

•  Skype

muenzpraeger •  Twitter

muenzpraeger •  LinkedIn

muenzpraeger •  Slideshare

muenzpraeger

•  Web http://blog.winkelmeyer.com http://www.midpoints.de

•  Mail [email protected] [email protected]

OpenNTF •  File Navigator •  Generic NSF View Widget for IBM Connections

About me

René Winkelmeyer Head of Development

4

Agenda

§  Administration basics

§  Monitoring

§  Tuning

§  Troubleshooting

§  NEXT

5

Agenda


§  Monitoring

§  Tuning


§  NEXT

6

Administration Basics – High Availability

IBM Notes Traveler allows two operating modes:

§  Standalone Traveler Server

§  LotusTraveler.nsf

§  Local Java Derby Database

§  High Availability (HA) Traveler Server Pool

§  Traveler-“Cluster“

§  Remote RDBMS (IBM DB2 or M$ SQL)

7

Administration Basics – Big Picture Traveler

Traveler Server

HTTP TASK

Traveler

OSGi SERVLET

Traveler TASK SSL

Domino Directory

Server Config

LotusTraveler

Default Settings

Derby Rel.-DB /ntsdb

Notes

notes.ini

Domino & HTTP Security

SQL

8


§  Benefits of a HA environment

§  Better scaling (Standalone should operate to max. 2.500 devices)

§  Failover capabilites, no downtime of Traveler services

§  Servers can be updated any time (there‘s never a good time for downtime...)

9


Domino Mail

Domino Mail

Domino Mail

Traveler

Server HTTPS

Notes Traveler

Server

DB2 / SQL Server

DB2 / SQL Server

DB2/SQL

HTTP(S)

IBM Notes Traveler

Service Pool

Load Balancer IP-Sprayer

Reverse Proxy (i.e. IBM Mobile Connect)

DB2 / SQL Backend

Server in HA

10

IBM Notes Traveler – High Availability

§  Two or more Domino servers with installed Traveler addon work in a so called „Traveler Service Pool“

§  All pool members use the same state database

§  The state database is centrally organized (IBM DB2 or M$ SQL)

11

IBM Notes Traveler – High Availability

What‘s a pool?

12

IBM Notes Traveler – Traveler Service Pool

§  Every user can be served by every Pool member server

§  All Pool member servers are equal (in terms of service)

§  Travelers own Availabiltiy Index (AI) is used for internal Load Balancing using TCP ports 50125/50126

§  Each user is bound to a single server within the pool for his Master Monitoring Sesssion (MM or User Session)

§  This Master Monitor Server is responsible for mail database synchronisation

13

IBM Notes Traveler – HA Load Balancing

tell traveler HADR show Domino ID Host IP:SrvrPort,SrvltPort Alive Server Servlet Last HB AI Users Devices L1/NETWORK 330 s1.network.com 10.3.1.1:50125,50126 true true true 2014-08-26 96 2315 1179 L2/NETWORK 337 s2.network.com 10.3.1.2:50125,50126 true true true 2014-08-26 100 556 1102 L3/NETWORK 585 s3.network.com 10.3.1.3:50125,50126 true true true 2014-08-26 99 1630 1140 L4/NETWORK 580 s4.network.com 10.3.1.4:50125,50126 true true false 2014-08-26 100 0 346 L5/NETWORK 505 s5.network.com 10.3.1.5:50125,50126 true true true 2014-08-26 100 311 1106

AI = Traveler internal Availability Index (AI) Users = Master Monitor Session / one per user Devices = HTTP Sessions (Devices last seen) / one session per device Note: L4/NETWORK has been restarted

14

IBM Notes Traveler – HA Load Balancing

•  Traveler Availability Index (AI) •  Master Monitor Server (MM) per user

•  User Load Balancing Bias

+ 10 Bias for local server + 20 Bias for current MM

•  Load Balancing algorithm •  AI calculated per server •  Chooses the highest AI (incl.

Bias) and defines that server as MM

•  All devices (of this user) are routed to the current MM

•  The MM isn‘t allowed to re-balance within 10 minutes

HTTP –Task Servlet

HTTP –Task Servlet

Traveler –Task AI = 75

Traveler –Task AI = 80

Server 1 Server 2

AI 75 + BIAS 10 + BIAS 20 > AI 80 AI 75 + BIAS 20 > AI 80 + BIAS 10

TCP 50125

15

Administration Basics – Webfrontend

Starting with 8.5.3. UP1 Traveler administration is done by using a XPages application

16

Administration Basics

The webfrontend is unfortunately not sufficient – you‘ll often need the Domino server console for troubleshooting

17


Current user status – tell traveler user <user name>

tell traveler user Detlev Poettgen CN=Detlev Poettgen/O=midpoints does not have sufficient access rights to the database mail/dpoettge.nsf. …

tell traveler user [email protected] IBM Notes Traveler has validated that it can access the database mail/rwinkelm.nsf. Encrypting, decrypting and signing messages are not enabled because the Notes ID is not in the mail file or the ID vault. ……

18


IBM Notes Traveler has validated that it can access the database mail/rwinkelm.nsf. Encrypting, decrypting and signing messages are not enabled because the Notes ID is not in the mail file or the ID vault.

Canonical Name: CN=Rene Winkelmeyer/O=midpoints Internet Address: [email protected] Master Server: Traveler01/srv/midpoints-trav, version 9 Master Server Locked: Jan 21, 2015 11:05 AM, type=Soft Home Mail Server: CN=mail01/OU=srv/O=midpoints Home Mail File: mail/rwinkelm.nsf Current Mail Server: CN=mail01/OU=srv/O=midpoints Release 9.0 Current Mail File: mail/rwinkelm.nsf Mail File Replicas: [CN=mail02/OU=srv/O=midpoints, mail/rwinkelm.nsf], [CN=mail01/OU=srv/O=midpoints, mail/rwinkelm.nsf] ACL for Rene Winkelmeyer/midpoints: Access=Editor Capabilities=create,update,read,delete,copy Missing Capabilities=none ACL for Traveler01/srv/midpoints-trav: Access=Manager Capabilities=create,update,read,delete,copy Missing Capabilities=none Notes ID: Mail File does not contain the Notes ID. Auto Sync User State: Monitoring disabled Last Prime Sync: Monday, Jan, 2015 2:31:11 PM CEST Banned Documents: 0…

19


Devices: Device ID: ApplC38JCFABDTWG Device Description: ApplC38JCFABDTWG Security Policy Status: No policy Security State: Clear Approval State: Not required Last Sync: Never Auto Sync Device State: Inactive Device offline time: Monday, Jan 20, 2015 2:39:42 PM CEST Auto Sync Connection State: Disconnected Auto Sync Applications to Synchronize: folder, mail, calendar, contact, serviceability, security Auto Sync Change Flags: folder:add, mail:add(4:add), serviceability:configGet/configSet

20


Pipe command output to file – tell traveler –f <filepath> user <user name> Watch out: Command leaves zero byte validation file .sem in the directory which doesn‘t get cleaned up automatically.

tell traveler –f /tmp/user.txt user rwinkelm Output for command ‘-f /tmp/user.txt show rwinkelm’ can be found at /tmp/user.txt.

[root@incinerate tmp]# ls -lrt insgesamt 3240 -rw-rw-r--. 1 domino domino 424 20. Jan 13:19 user.txt -rw-rw-r--. 1 domino domino 0 20. Jan 13:19 user.txt.sem

21


Get SQL content via Domino console – tell traveler sql „<QUERY>“

tell traveler sql "SELECT HOSTNAME FROM TS_GLOBAL“ [04683:00036-3357230848] Command 'SELECT HOSTNAME FROM TS_GLOBAL' was completed successfully. [04683:00036-3357230848] HOSTNAME | [04683:00036-3357230848] incinerate.midpoints.net | [04683:00036-3357230848] hellfire.midpoints.net | [04683:00036-3357230848] aryastark.midpoints.net |

22

Agenda


§  Monitoring

§  Tuning


§  NEXT

23

Monitoring – Server-Task

Notes Traveler is part of the „Server Tasks“ within Domino Administrator (added in 2013, check if your domadmin.nsf is updated)

24

Monitoring

You‘ll find the default statistic values for yellow/red statuses in an IBM wiki

§  http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Status_command_considerations_and_examples_LNT853

Traveler server status – tell traveler status

tell traveler status The IBM Notes Traveler task has been running since Tue May 14 12:31:09 BST 2014. The IBM Notes Traveler availability index is currently 100 while servicing 431 users. The last successful device sync was on Sat Jan 22 15:05:15 BST 2015. The overall status of IBM Notes Traveler is Green.

25

Monitoring – Statistics

§  Issuing tell traveler stat show on the Domino console brings you all Traveler statistics

§  http://www-10.lotus.com/ldd/dominowiki.nsf/xpDocViewer.xsp?lookupName=Administering+IBM+Notes+Traveler+9.0.1#action=openDocument&res_title=System_stat_results_A901&content=pdcontent

26


Statistics are also available using Domino Administrator. Some are very useful – some have only an informational level.

27


Analyzing connection issues and latencies

tell traveler stat show [0A8C:0068-0924] Availability.Index.060-070 = 1 [0A8C:0068-0924] Availability.Index.090-100 = 48 [0A8C:0068-0924] Availability.Index.Current = 100 …. [0A8C:0068-0924] DCA.DB_CLOSE = 54 [0A8C:0068-0924] DCA.DB_OPEN = 41 [0A8C:0068-0924] DCA.DB_OPEN.Time.Histogram..000-001 = 37 [0A8C:0068-0924] DCA.DB_OPEN.Time.Histogram.CN=Traveler01/OU=srv/O=midpoints-trav.000-001 = 4 ….

28


§  CPU.Pct.070-080 CPU.Pct.080-090 CPU.Pct.090-100

§  Will be set when the CPU usage is within the defined range (i. e. 70-80% of the first parameter)

§  Important parameter as Traveler performance can be affected heavily if CPU usage is 78% or more.

29


§  DCA.C.DB_OPEN.Time.Histogram.<server>.<bucket>

§  A „bucket“ defines the time in seconds which was needed to open a mail file.

§  This statistic helps to analyze latencies and connection issues to remote mail servers.

tell traveler stat show …. [0A8C:0068-0924] DCA.DB_OPEN.Time.Histogram..000-001 = 37 [0A8C:0068-0924] DCA.DB_OPEN.Time.Histogram.CN=Traveler01/OU=srv/O=midpoints-trav.000-001 = 4 ….

30

Monitoring – End-To-End

§  All shown mechanisms for analyzing Traveler health and status are missing some key elements:

§  Is Traveler available from external and can a device synchronize?

§  Are all involved components available (i. e. Internet connectivity, Firewall, Load Balancer, Reverse Proxy, Network to mail servers, mail server itself etc.)

§  An administrator needs to know issues before the CxO calls during BBQ.

31


§  Check if Traveler infrastructure works from external

§  HTTP(S) request to /traveler?action=getStatus §  HTTP response code 200 mean: Traveler is available

§  Validations

§  Traveler access (Load Balancer, Proxy, authentication) §  HTTP task active §  Traveler task active §  Mail server reachable §  Traveler database (SQL backend) available

32


33


§  Traveler Mail Delivery Confirmation Message

§  Send a mail to a specific mail account (which is set on a device)

§  Device fetches mail via Traveler

§  The sender will receive a confirmation mail if the mail has been delivered to the device

§  If the sender doesn‘t get the confirmation mail – Houston, we have a problem!

34



§  Available since Traveler 9.0.0.1 IF2

§  Must be explicitly activated via notes.ini NTS_MAIL_DELIVERY_CONFIRMATION=true

§  Allowed senders can be explicitly set NTS_MAIL_DELIVERY_CONFIRMATION_SENDERS=MonitorMail1/Company

35



§  Mail-Subject must begin with <$Confirm>

§  Additional keywords/combinations: <$Confirm,RemoveOnDelivery> <$Confirm,SuppressSaveInSentItems> <$Confirm,RemoveOnDelivery,SuppressSaveInSentItems>

§  http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Using_Mail_delivery_confirmation_messages_with_IBM_Notes_Traveler

36

Monitoring – Port checks

§  80 / 443 - HTTP/HTTPS §  50125 - Communication

Servlet => Traveler task Traveler task => Traveler task

§  50126 - Communication Traveler task => Servlet

§  Port checks should always be combined with further/other checks. §  Port 50125 may i. e. answer but the Traveler task may not reach

the SQL backend

37

Monitoring – Know your devices

Knowing your devices (os, os version type etc) is critical.

38

Agenda


§  Monitoring

§  Tuning


§  NEXT

39

Tuning

§  Always watch the amount of devices in your infrastructure and update your configuration accordingly.

§  Important parameters are i. e. §  CPU and RAM §  HTTP threads §  Memory cache §  Maximum memory size §  Address cache §  Request size

40

Tuning

Assign CPU and RAM more then needed. 64bit is always recommended.

41

Tuning

§  HTTP threads => 1,2 * number of devices per server (default: 100 32bit / 400 64bit)

§  Watch out: RAM for all threads will be allocated a HTTP task start

42

Tuning

§  Maximum cached users: Number of users per server

§  Cached user expiration level: Recommendation: 28.800 sec (8h)

43

Tuning

§  Maximum Memory Size: varies depending of the user/device numbers

§  Should be minimum 1.024 MB (rule of thumb: always ¼ of available memory)

44

Tuning

Memory – tell traveler mem

CPU and Memory (MB) Usage History Date CPU Pct Java Mem C Mem Avl Indx # Users # Errors # DB Conn 2015-01-02 15:01:53 BST 0.01 56 1307 100 5 1 0 2015-01-02 15:16:53 BST 0.01 44 1307 100 5 1 0 2015-01-02 15:31:53 BST 0.01 73 1307 100 5 1 0 2015-01-02 15:46:53 BST 0.01 39 1306 100 5 1 0 2015-01-02 16:01:53 BST 0.01 53 1306 100 5 1 0 2015-01-02 16:16:53 BST 0.01 66 1307 100 5 1 0 2015-01-02 16:31:53 BST 0.01 87 1307 100 5 1 0 … Current Memory Usage Java Memory Usage

Max Total 1024 MB Current Total 96 MB Free 940 MB (92 percent of Max Total) Allocated 84 MB (8 percent of Max Total)

C Memory Usage Allocated 1293 MB (33 percent of Total Physical)

Current Usage Java 84 MB C 1293 MB

45

Tuning

§  Traveler performance varies depending on multiple factors like mail database size and sync filter settings.

§  You‘ll see that in the size of the Derby / SQL database

§  850 User, 500 MB quota, no filter => 1 GB §  850 User, no quota (Ø 2 GB), no filter => 5 GB §  2.000 User, no quota (Ø 1,5 GB), filter ”1 year” => 4 GB

§  Largest seen Derby database: 17 GB (1.000 users, no filter) U * (D * (700 * V + 4000)) = Recommended DB space available in bytes

46

Tuning

Usage – tell traveler dbusage tell traveler dbusage IBM Notes Traveler Database Statistics Accounts: 11 Devices: 23 Total device documents: 35307 Device documents synced: 12124 Device documents filtered: 23183 Domino documents: 11056 Highest Total Usage Documents Percentage ------------------------------------------------------------------------------------------------ Rene Winkelmeyer/midpoints 7229 65.39 Petar Petrov/midpoints 1471 13.30 Detlev Poettgen/midpoints 1302 11.78 Michael Ingendoh/midpoints 819 7.41 Benjamin Gaisser/midpoints 235 2.13

47

Tuning

Usage – tell traveler dbusage

Mail documents: 14321 Highest Mail usage Documents Percentage EMail filter ------------------------------------------------------------------------------------------------------------ Rene Winkelmeyer/midpoints 6773 72.66 unlimited Petar Petrov/midpoints 1254 13.45 unlimited Detlev Poettgen/midpoints 743 7.97 30 days Michael Ingendoh/midpoints 399 4.28 14 days Benjamin Gaisser/midpoints 152 1.63 30 days Calendar documents: 10085 …

48

Tuning

49

Tuning

§  Standalone (Derby) §  Frequent defragementation

§  HA (IBM DB2 / M$ SQL)

§  Frequent Runstats for table and index status §  Index ReOrg => TALK TO YOUR RDBMS-ADMIN §  Translog check (size and storage)

50

Tuning – DON‘T DO THIS AT HOME

§  Get all Traveler configuration parameters – tell traveler config

ADMINP_POLL_INTERVAL = 0 DEBUG_OUTFILE = "” LOGFILE_DIR = "” NTS_64_BIT = false (default = true) NTS_ACCESS_ALLOW = "*:TravelerUsers" (default = "") NTS_ACCESS_DENY = "” NTS_ACCESS_ENABLED = true …. NTS_FONT_CONVERSION = 0 NTS_FORCE_GC_MEMORY_LEVEL = 5 NTS_FORCE_OUTBOX_MAIL_TO_DRAFTS = false NTS_FORCE_START = false NTS_HOST_IP_ADDR = "11.11.11.100" (default = "") NTS_HTTP_HEADERS_RESPONSE_X_IBM_TRAVELER_HOST = "" NTS_IGNORE_TIMEZONE_ERROR = false NTS_INSTALLATION_TYPE = ON_PREMISE NTS_INSTALL_INSTANCE = "" NTS_INTERNAL_BATCH_UPDATES = false …

51


§  Nearly 800 (mostly undocumented) parameters!!!

52


§  You should change them only if you they are documented or IBM support tells you to use them

§  NTS_AUTOSTART_HTTP §  NTS_PUSH_APNS_SERVER §  NTS_ROUTE_LOCAL_BIAS §  NTS_STATUS_DATA_DIR_FREE_GIGABYTES_RED

53

Agenda


§  Monitoring

§  Tuning


§  NEXT

54

Troubleshooting – Device setup

§  Can the device connect to the Traveler server?

§  Can the device open the Traveler website (/traveler)?

§  Can the user authenticate himself (wrong password)?

§  Is Internet Lockout active (you should use it if you don‘t have a secure reverse proxy in front of Traveler)?

§  Is the user allowed to use the Traveler server?

55


§  Can the Traveler server connect to the user‘s mail server?

§  Is the Traveler server allowed to connect to the user‘s mail server?

§  Is a cross-certificate for the Traveler server missing (if Traveler is hosted in another domain)?

§  Has Traveler Manager access rights (incl. Delete) for the mail database?

§  Has the user Editor access rights (incl. Delete) for the mail database?

56


§  Is „Replication of Unread Marks“ set in the mail database properties?

§  Has the mail database quota been reached (Traveler creates/uses two profile documents in the mail database)?

57

Troubleshooting – Logs

§  Central Log Directory IBM_TECHNICAL_SUPPORT/traveler/logs

§  Enable Logging per User tell traveler log adduser finest <username> tell traveler log removeuser <username>

§  Dump user information tell traveler dump <username>

58

Troubleshooting – Logs

§  Collect Information for a PMR and upload tell traveler pmr <pmr_number>

§  If you cannot do that use the following

tell traveler systemdump tell traveler log collect Check IBM_TECHNICAL_SUPPORT\traveler\logs\<timestamp>

59

Agenda


§  Monitoring

§  Tuning


§  NEXT

60

NEXT – IBM Verse iOS App

61

NEXT – IBM Verse iOS App

62

NEXT – FreeTime-Search on iOS

63

NEXT – More stuff

§  Trash folder synch on iOS

§  GCM Push Notifications for Android

§  ...

64

Q & A!

engage 2015 - IBM Notes Traveler Daily Business

Software

Transcript of engage 2015 - IBM Notes Traveler Daily Business