BEST AND WORST PRACTICES DEPLOYING IBM CONNECTIONS

Toronto, June 6-7 2016

Best and Worst Practices Deploying

IBM Connections

Christoph StoettnerNico Meisenzahl

PLATINUM&SPOTLIGHTSPONSORS

GOLDSPONSORS

SILVERSPONSORS

BRONZESPONSORS

Christoph Stoettner

• Senior Consultant – panagenda• IBMNotes/Dominosince1999• IBMConnectionssinceversion2.5/2009

• Many years of experience in:• Migrations• Administrationundinstalls• Performanceanalysis

• Joined panagenda in 2015 focusing in:• IBMConnectionsdeploymentund

optimization• IBMConnectionsmonitoring

• Husband of one & father of two, Bavarian

3

@stoeps linkedin.com/in/christophstoettnerwww.stoeps.de christophstoettner [email protected]

Nico Meisenzahl

• Consultant at panagenda• IBM Notes / Domino since 2008• IBM Connections since version 3.0 / 2010• Many years of experience in:

• Consulting• Migrations&Administration

• Joined panagenda in 2016 focusing in:• IBMConnectionsConsulting• ICSdeployment&optimization

4

@nmeisenzahl linkedin.com/in/nicomeisenzahlmeisenzahl.org nico.meisenzahl [email protected]

IBM Connections Request Flow

5

Databases

WebSphereApplication Server

IBMHTTPServerWebSpherePlugins

Application DBsPEOPLEDB

LDAPServer

TDI

Forward toApplicationServerandPort

(LoadbalancingandFailover)

Redirectunknown

URL

UploadandDownloadofFiles,Attachments

Common:AccessCustomization,Webressources

ReadandWrite

Authentication

Users/Groups

ReadsUserdataforProfiles

Create,Update,DeleteandInactivateProfiles Shared

Directory

Linkto

Attachments

ProfilechangessynchronizetoMembertablesthroughJMSQueue

Optional:

DirectAccesstoAttachments


Installation & Requirements

System Requirements

• Regularly check requirement documents

• All versions• http://short.stoeps.de/vwzrv

• IBM Connections 5• http://short.stoeps.de/mspdi

• IBM Connections 5.5• http://short.stoeps.de/cnx55sysreq

• Check all notes, Download PDF

Connections5.0CR3

Connections5.5

Connections5.5CR1

7

System Requirements

8

• Be careful with installation documents• Sometimeswrongdependenciesmentioned• Supportedstatementdoesnotmeanit’slicensed

• http://www-01.ibm.com/support/docview.wss?uid=swg21683118• Supported: DB210.5FP4->butConnections5.0LicenseAgreement:

• YouwillnotfindaDB210.5licenseinyourPAAccount

Sizing

• Be prepared for future growth• Do not overact

• Afewhundredusersdoesn’tmeanyouneedalargedeployment

• Not fans of multi-instance database machines• IfIrunindatabaseperformanceissuesIsplitthedatabasesto

differentmachines• Performancetuningguide

• Multi-instanceisbestpractice,ifyouhaveenough resources

PerformanceTuningGuide5CR1– page10

9

Sizing

• A word on requirements• 4|8GBmemoryminimumisoftentooless,

bettertostartwith10or12GB• Memoryswappingkillsalltuningefforts

• CPU cores• 2coresminimumonlyonsmalldeployments• Thumbrule:calculateonecoreforeachjvm

(expensivewithPVUlicense)

• Disk• Usingnetworkstorageorvirtualizedservers• Easiertoextend

Connections5.0Connections5.5

10

Prepare for your Installation

• Download all software packages• CheckSystemRequirements!

• Paths shouldn't contain spaces• Nospacesinsourceanddestinationfolders

• Use a dedicated administration user • EspeciallyonWindowsavoiduserswithappliedgrouppolicies• IfpossibledisableUserAccountControl(UAC)• RunallInstallerandScriptswithoption“RunAsAdministrator”

11

Security Settings

• During installation you should disable all "Security" Software• SELinux• AppArmor• Antivirus• Firewalls• Selfdevelopedscriptsandextensions

• It's not fun, when a script deletes databases, because you forgot to add the directory to the script exclusions

12

Operating System specific settings -Linux

13

• Different operating systems need special settings• Always use the operating system where you have the best

skills• Linux

• /etc/security/limits.conf• Increasenofileandnproc(seetuningguides)

• Example from tuning guide

• Defaultnproc(maxnumberofprocesses)foruserroot2047• Youcanextendthenprocwithulimit–pupto16384(e.g.within.bashrc

or.profile)• Orsetsoftandhardlimittoequalsizes,avoidsadditionalchangeswith

profile

root soft nproc 2047 root hard nproc 16384

Operating System specific settings -Windows

14

• Always use UNC path as Shared Directory• EasiertoaddadditionalWebSphereNodes

forfailoverorloadbalancing

• WebSphere services • Technicaluseraccount

• Enable“Passwordneverexpires”• Disable“Mustchangepasswordonnextlogin”

• Default:LocalSystemhasnonetworkaccess

• Check access rights on Shared Directory

Network

• Name lookup / DNS• Allserversmustberesolvable(hostsisnotasuitableworkaround)• Knowingtheprotocol

• AvoidRoundRobin• NoAuthentication failoverinWebSpherewithRoundRobin!

• Network storage (file locking is important)• NFSv4/SMB|CIFS• NoDFS

• Reverse Proxies / Proxies• Alwaystestyourdeploymentwithoutproxies• Activateaftersuccessfultesting

15

Register WAS as a service

• Register WAS as a service • Services for Deployment Manager and

NodeAgent(s)• wasservice.bat|sh

• Map service to a technical user• anyActiveDirectoryUserispossible• allowedtoread/writenetworksharewithShared

Content

• Service can parse commands to nodeagent• -stopArgs "<NAcommands>"

• Configure monitoring policy (if required)

16

WasService.bat|sh – Register service

17

cd D:\IBMCNX\WebSphere\AppServer\bin

WASService.exe -add CnxNode01 -serverName nodeagent -profilePath d:\ibmcnx\websphere\appserver\profiles\CNXNode01 -stopArgs "-username wasadmin -password password -stopservers" -userid cnxtec -password password -encodeParams -restart true -startType automatic

parsed to nodeAgentstops AppServer

Monitoring Policy

• Each Application Server• ChangeNoderestartstateto

"RUNNING"• Large deployment on Windows

• Defaulttimeoutforserviceshutdown=20seconds

• IncreaseValueat:HKEY_Local_Machine:SYSTEM\CurrentControlSet\Control\WaitToKillServiceTimeout

• Must set this to STOPPED before performing updates• OrusesyncNode.bat|shafterapplyingtheupdate

18

Directories & Synching

• Prepare your LDAP• BetterdatawithinLDAP→betterProfiles

• Switching Authentication directories is possible• Needsomeplanning

• Dependencies• QualityofLDAPdata• PlanstoactivateSPNEGO• DominoMailIntegration

19

Federated Repositories Best Practice

• Leave the file based wasadmin with WebSphere Application Server• FallbackifLDAPBindCredentialschanged• Solvingproblemswith

FederatedRepositories

• Default does not allow this (you have to disable security to change configuration)

Checkthisbox

20

Logs – adjust language WebSphere

• Change log language to English (IBM will love you for this)• WebSphere

Add "-Duser.language=en –Duser.region=US" to Generic JVM arguments of• Eachapplicationserver(Processdefinition – JavaVirtualMachine)• dmgr (SystemAdministration – DeploymentManager– ProcessDefinition ...)• nodeagents (SystemAdministration– Nodeagents– nodeagent– ProcessDef

...)

21

Logs – adjust language TDI

22

• TDI• editibmdisrv.bat|sh• add-Duser.language=en–Duser.region=UStoLOG_4Jvariable• Linux:

• Windows:

Rotate Logs

• WebSphere Logs too small for Troubleshooting• Default:5Logs1MBeach(SystemOutandSystemErr)• Better5-10Logs20MBeach

• SettingforeachApplicationServer• rememberNodeagentsandDmgr

• Change this as soon as your servers have been created

23

Rotate Logs

• IBM Connections 5.5 – SET BY DEFAULT!!• Install.log

• Result:

• Soyourlogsarestored30days,independentofsize

24

Rotate IBM HTTP Server Logs

• Default: no max size for access_log and error_log• Often some GB of Log files

• OpenwithanEditor?• Disksize

• Search for this lines in httpd.conf:

• Comment out:

CustomLog log/access_log common

ErrorLog logs/error_log

# CustomLog log/access_log common

# ErrorLog logs/error_log

25

Rotate IBM HTTP Server Logs

• Add:

• Delete Log Files older than x days• Linux

• Windows(BatchthroughTaskSchedulerorPowershell)

Linux:CustomLog "|/opt /IBM/HTTPServer/bin/rotatelogs /opt/IBM/HTTPServer/logs/access_log.%Y%m%d 86400" commonErrorLog "|/opt/IBM/HTTPServer/bin/rotatelogs /opt/IBM/HTTPServer/logs/error_log.%Y%m%d 86400“

Windows:CustomLog "|D:/IBM/HTTPServer/bin/rotatelogs.exe D:/IBM/HTTPServer/logs/access_log.%Y%m%d 86400" commonErrorLog "|D:/IBM/HTTPServer/bin/rotatelogs.exe D:/IBM/HTTPServer/logs/error_log.%Y%m%d 86400"

crontab -e# Delete logfiles older than 3 days in logs10 0 * * * find /opt/IBM/HTTPServer/logs/*_log.* -mtime +3 -exec rm -rf {} \;

forfiles -p "D:\IBM\HTTPServer\logs" -s -m *_log.* -d -3 -c "cmd /c echo @file"

26

Rotate Logs DB2

• db2diag.log• Default: no maximum size

• Default:%PROGRAMDATA%\IBM\DB2\instancename\DB2• FullC-PartitioninWindowsstillhardtosolve

[db2inst1@cnx-db2 ~]$ db2 get dbm cfg |grep -i diagsizeSize of rotating db2diag & notify logs (MB) (DIAGSIZE) = 0

[db2inst1@cnx-db2 ~]$ db2 update dbm cfg using DIAGSIZE 1024DB20000I The UPDATE DATABASE MANAGER CONFIGURATION command completed successfully.

[db2inst1@cnx-db2 ~]$ db2 get dbm cfg |grep -i diagsizeSize of rotating db2diag & notify logs (MB) (DIAGSIZE) = 1024

27

HTTP Server Keystore

28

• Several Reviews showed• Keystore ofWebSpherePluginusedforIHSSSLKey• Whyisthisworse?• WhatwouldyoudowhenyougetSSLErrorswithinConnections?

• Thisoverwritesplugin-key.kdb onyourWebserver• SSLKeydeleted• Backup?

HTTP Server Key store

29

• When you want to reuse Plugin Key store• ImportSSLKeyintoCMSKeyStore• Butneverseenthisinthewild

HTTP Server Keystore

• Best Practice - Create a separate key store for IHS• Ikeymanwillhelpyou• Possibletouseawildcard

• Thenyoucanjustcopyittouseondev/testmachines

• Backupthekeystore beforechanges• Don’tactivate“Expirationtime”• “Stashpasswordtoafile”

• Createsacnx-key.sth file• UsedbyIHStoopenkeystore

30


Security

J2EE Roles

• Some Applications are public readable after installation• Profiles• Communities• Blogs• Wikis

• Check after Updates• Google:“Site:myconnections-host”• Shouldonlyshowaloginpage

• Use the Community Scripts to do this or change in the ISC

32

Harden HTTP

• DisableSSLv2/v3• Automaticallydisabledwith8.5.5.4• SSLProtocolDisable SSlv2SSLv3

• Checkwithhydra,nmap orssllabs.com/ssltest/

• Defaulthttpd.conf uses:TLS_RSA_WITH_3DES_EDE_CBC_SHA

# Ciphers TLS1.0, 1.1SSLCipherSpec TLS_RSA_WITH_AES_256_CBC_SHASSLCipherSpec TLS_RSA_WITH_AES_128_CBC_SHA# Additional Ciphers TLS1.2SSLCipherSpec TLS_RSA_WITH_AES_128_GCM_SHA256 SSLCipherSpec TLS_RSA_WITH_AES_256_GCM_SHA384 SSLCipherSpec TLS_RSA_WITH_AES_128_CBC_SHA256 SSLCipherSpec TLS_RSA_WITH_AES_256_CBC_SHA256

33

Harden HTTP

• IfyouuseSSLKeyslongerthan2048bit,youmustreplaceJavapolicy• DownloadandreplaceJava(unrestricted)policyfiles• https://www-01.ibm.com/marketing/iwm/iwm/web/reg/pick.do?source=jcesdk

• AlsoneededifDomino(MailIntegration)orSametimeProxyuselongerkeys

• RemoveServerInformation(HTTPHeader,Errorpages)• ServerSignatureOff• ServerTokens Prod(DEFAULT)• AddServerHeaderOff

Default

34

Remove Index

• RemoveallFilesexceptindex.htmlfrom<IHS_ROOT>/htdocs• Renameindex.html(e.g.0815.html)

• echo1>0815.html• Fortestingyoucanaccess thefile

• Addrobots.txt

35


Tuning

Performance Tuning Guides

• 4.0• http://www-

10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.0_Performance_Tuning_Guide

• 4.5 Addendum• http://www-

10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.5_Performance_Tuning_Guide_Addendum

• 5.0 CR1• http://www-

10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connection_V5_CR1_Tuning_guide• Read everything carefully• check and understand dependencies

37

Worst Practise Example - Tuning

• Customer showed me a system with following infrastructure

• WebSphere• Largedeployment• 16GBRAM• 4Cores

• DB2• 12instances• 8GBRAM• 4Cores

• Connections restart 22 minutes

Web Serverihs.example.local

WebSpherewas1.example.local Db2 / TDI

db2.example.localFreigabe

LDAPdomino1.example.local

User SynchronisationAuthentication

38

Solving the problem

• Large deployment means about 15 JVM on the machine• Restartshows15min100%CPUusage• Adding4coresandrestarttimegetdownto7minutes• Otheroptionwouldbemidsizedeployment,butthenyouhavetoreinstall

Connections• Java Heap Sizes set to default (256 MB and 768 MB) ->

increase to 1.5 – 2.5 GB• Perf Guide mentions that multiple instances on DB2 only

increase performance with enough resources• Butthatwasnottherealproblem

• DataSource connectionPool Sizes are set to Default 1/10 • Increasethisvaluestotheproposalsintheguideand...• Restarttimecomesdownunder3minutes

• Key point: read the complete guide

39

Java Heap

• Default Java Heap Sizes on Midsize Deployment: 2506 MB / application server

• Large Deployment depends on application: 0.5 to 2.5 GB• Main point in memory tuning

• Neverexceedthesystemmemory• Swappingkillsallyour tuningefforts

• Counting the JVM Heap sizes is not enough• Maximumheapisnotthemaximumamountofmemorythejvm uses!• Libraries,jarsandsooncountadditional tomemoryusage• JVMmemoryusagemaybe3*JVMmaximumHeap

• Initial and maximum Heap Size should be equalized• mentioned inIBMConnections5.0tuningguide

40

IBM HTTP Server

• Enable compression• Important!!!!!• SeeSlidesfromBP307- IBMConnect2014• Saveupto70%networktraffic• MinimalincreaseofCPUload

• Enable file download through IHS• Dependonyourdeployment• OftensecurityforbidsstorageaccessfromDMZ

• If you have no access to file share from IHS -> Files should be installed in a separate Cluster

41

Midsize Deployment Files

• Often IHS positioned in the red zone (DMZ)• Mostly No Access to SHARED DIRECTORY

• CreateaClusterforFiles• NoProblemwithLargeDeployments• WithMidsizeyoucanaddanadditionalClusterduringSetup

(LooksdifferentonConnections5.5!)

http://www-01.ibm.com/support/docview.wss?uid=swg21317658

42

Activate Synchronous File transfer

• Servers -> Application Servers -> serverName -> Web Container Settings -> Web Container -> Custom Properties• com.ibm.ws.webcontainer.channelwritetype=sync

43


Enhance User experienceHappy admins with happy users

Single Sign On - LtpaToken

45

• Single Sign On within IBM portfolio• Domino only supports one domain per Web SSO Document

• Youcancopy&pasteWebSSODocumentsandchangeDomainnames(seee.g.PaulMooney- AdminBlast2012– Tip#4

• DNSDomainismultivalue(worksuntilDomino8.5.x,butnotwithDomino9.x)

• ServerswithmixedInternetSiteandNon-InternetSiteusage:copy&pastetoo!

• Often internal servers use local domains, when Connections is external accessible SSO needs workaround• addingadditionalhostnamestodomino• YoucanuseIHS(IBMHTTPServer)asareverseproxytoaccessiNotes

Single Sign On - LtpaToken

46

• It’s possible but not recommended to change the cookie name

• Default one are• LtpaToken• LtpaToken2

• Name dependencies• Web-SSOdocumentnameforDomino• WebSSOwithinWebSphere• sametime.ini

• ST_TOKEN_TYPE=CustomLtpaName• Connectionsscheduler

Single Sign On - SPNEGO

47

• Requirements• Windows2003/2008/2012ActiveDirectory

• Configure use documentation and http://de.slideshare.net/david_hay/dave-hay-desktop-single-signon-in-an-active-directory-world?related=1

• Real additional value for users• Easy to deploy, when you have the rights and clue what to do• Do not test Browser Single Sign On with Chrome, because process

does not end when you close the last window• Tip: SPNEGO is not working with a SPN which is a CNAME DNS

alias

Mail integration

48

• Use IBM HTTP Server as reverse proxy to access iNotes

LoadModule rewrite_module modules/mod_rewrite.so <IfModule mod_ibm_ssl.c>

Listen 0.0.0.0:1443<VirtualHost *:1443>

ServerName connections.example.com SSLEnable RewriteEngine onProxyRequests OffProxyPass / http://inotes.example.local/ProxyPassReverse / http://inotes.example.local/

</VirtualHost> </IfModule>

iNotes Web Mail Redirect

49

https://connections.example.com:1443

Socialmail-config.xml

50

• When you use reverse proxy to access iNotes• Mailintegrationworksonlywhenyouusehttporhttps• AddUseConfiguredProtocol toyourconfiguration

• Problem when you need to access multiple iNotesServers

<ServerConfig name="domino-redirect"><ConfigType>REDIRECT</ConfigType><RedirectURL>https://connections.example.com:1443/iwaredir.nsf</RedirectURL><MailPattern type="example.com" />

</ServerConfig><GadgetConfig>

<GadgetPreference id="UseConfiguredProtocol">true</GadgetPreference></GadgetConfig>

Mail integration and SPNEGO

51

• LtpaToken contains AD $DN• Lookup in Domino Directory with this DN -> user is not

allowed to open mail• Solution

• AddAD$DNtoACL• OraddAD$DNtoDominoFullname (ADDNcontains,asdelimiter

betweenou)

• Or: • http://tdiblog.anderls.com/2015/02/adding-user-active-directory.html• ThanksAndreasArtner


Backup

What to Backup

• UsinganexampleConnections installationguiderarelyexplainsbackups• Theseguidesnormallydonotmentionbackup,orwhattobackup• Diskcrashmeansdataloss

• Databasebackupsthrough filebackuparenotsupported andmostlynotrestorable

• Important!!!• DatabaseBackupthroughOnlineBackupscanbetakenwhenConnectionsisup• Offlinebackupsarealsopossible

• Ensurethefilesystem&DBbackuparerunatthesametimeofday• DBandFilesystemdatawill stayinsync– ifyoutakeyourDBbackupatmidnightandthefile

systematmiddaytheywillbeoutofsync

53

Backup

• Most important (minimum daily)• Databases(offlineoronline)• Sharedcontent

• Important• Configuration

• WebSphereApplicationServer• Connections• IBMHTTPServer• TDISolution

• Test if restore is possible!!!!• SeveralissueswithWebSphererestores,wherebinariesweren'ton

thetape

54


Checklists

Checklist

Do• Documentyourinstallationsteps

• Theofficialdocumentationissometimesconfusing,becauseallOSwithinonedocument

• UseaLDAPuserforconnectionsAdmin• Bepreparedforscaling

• ShareddirectoryonUNCpath• Nosmalldeploymentinstallations

• Tuneyourenvironment• READTHEDOCUMENTATION!!!!

Don’t• Usemultiple instancesDB2withsmall

resources• Installonasinglemachine (unlessthe

environmentisverysmallorfortest)• Copycustomizationstonewerversions

• jsp,ftl copywillbreaksomething• Useunstablefile shares• TestdeploymentwithserverIE• Testwithonlyonelanguage

56

Install Checklist• WebSphereApplicationServer

• ConfigureFederatedRepository• LtpaToken,enablesecurity

• WebSphereApplicationServerSupplements(IHS,Plugins)• DB2(orotherDBM)• TDI• AddWebserver toDmgr (useconfigurewebserver.bat)• EnableSSLonIHS• ImportIHSRootKeywithinWebSphere cell trustkeystore (retrieve fromport)• ConfigureCCM• InstalloptionalAddons likeFormsExperience Builder,IBMDocs,Cognos

57

Documentation

DocumentEVERYTHING!!!becauseyoucanremembereverythingyoudid….

58

Documentation

• Everyone hates writing documentation• BUT – make notes as you go, it doesn’t need to be a full

step by step guide with screenshots• Document all customizataions• Any additional changes made• Anything of note that deviates from the guides• Lessons learnt or how you solved issues• Use the scripts to output some of it

59


Resources

Useful Tools

• Editor with syntax highlighting• vim,geany• notepad++,pspad,UltraEdit

• Tail• baretail• multitail• mtail

• Proxy• Fiddler(oftenaskedforbyIBM

Support)• Burpsuite (interceptproxy)

• Browser• Firefox(portable)/Firefox

ESR• Chrome• IE(downloadvmwith

differentversions)• https://www.modern.ie

• Network analyzer• Wireshark• tcpdump

• Unzip / Unarchiver• 7-zip• WinRar

61

Links and References

• IBMConnectionsSystemRequirements• http://www-01.ibm.com/support/docview.wss?uid=swg27012786

• IBMConnectionsFamilyDocumentation• http://www.ibm.com/support/knowledgecenter/SSYGQH/welcome

• IBMConnections4PerformanceTuningGuide• https://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.0_Performance_Tuning_Guide

• IBMConnections4.5PerformanceTuningGuideAddendum• https://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.5_Performance_Tuning_Guide_Addendum

• IBMConnections5CR1PerformanceTuningGuide• https://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connection_V5_CR1_Tuning_guide

62

Useful Blogs• http://ibmconnections.com• http://turtleblog.info• http://portal2portal.blogspot.de• https://www.urspringer.de• http://socialconnections.info• http://blog.robertfarstad.com• http://www.curiousmitch.com• http://www.ramsit.com/category/blog• http://techblog.gis-ag.info• https://milanmatejic.wordpress.com• http://ibmdocs.com• http://domino.elfworld.org• https://dontforgetthe0.com

• http://dilf.me.uk/socialshazza• http://www.stoeps.de• http://scripting101.org• http://meisenzahl.org• http://martin.leyrer.priv.at• http://kbild.ch• http://www.notesgoddess.net• http://www.dominodiva.com• http://notesbusters.com• https://rob59blog.wordpress.com• http://connections101.info• http://brandlrainer.blogspot.de• https://collaborationben.com

63

Thank you very much for your attention!

panagenda GmbH – MakeYourDataWorkforYou

Lahnstr. 17● 64646 Heppenheim (Germany)

Skype:christophstoettner ● Cell:+49173 8588719

E-Mail:[email protected]

ChristophStoettnerSeniorConsultant

panagenda GmbH – MakeYourDataWorkforYou

Lahnstr. 17● 64646 Heppenheim (Germany)

Skype:nico.meisenzahl ● Cell:+49170 7355081

E-Mail:nico.meisenzahl@panagenda .com

NicoMeisenzahlConsultant

64

PLATINUM&SPOTLIGHTSPONSORS

GOLDSPONSORS

SILVERSPONSORS

BRONZESPONSORS

BEST AND WORST PRACTICES DEPLOYING IBM CONNECTIONS

Technology

Transcript of BEST AND WORST PRACTICES DEPLOYING IBM CONNECTIONS