Client Hardware and Infrastructure Suggested Best Practices

This document was last updated September 21, 2010 and may be further updated from time to time.

A current copy of the Hardware Pretty Good Practices may be found at www.rehabdocumentation.com/hardwarepgp

Client Hardware and Infrastructure Suggested Best Practices While it is the responsibility of our Clients to support their hardware and infrastructure, the Pretty Good Practices below are provided for informational purposes (see the ReDoc Service Level Agreement for details). The intent of this document is to help our Clients understand some of the key areas for infrastructure planning, installation, configuration, monitoring, maintenance, upgrading, and budgeting for equipment lifecycles. Note that if ReDoc provides support for an issue caused by a lack of infrastructure maintenance or poor design, it will be on a time and materials basis. Well thought out and thoroughly monitored IT practices lead to good system performance and minimal down-time and, once established, do not require significant time or expense to maintain. Inattention to the subjects below, on the other hand, can result in a variety of serious problems ranging from sub-optimal system performance (slowness) to system „crashes‟ with loss of data. Given the total overall investment you have made in your ReDoc system and the total benefits realized, the time and costs associated with effective preventive maintenance are very modest relative to the value they deliver. For example, one of the single leading causes of downtime for ReDoc clients is running out of hard drive space and crashing. Monitoring hard drive space as part of a weekly or monthly systems check takes minutes; truncating or archiving transaction logs to save disk space can be an automated maintenance routine; and eventually adding an additional hard drive can cost ~$100. One of the leading causes of poor system performance is highly fragmented hard drives. Automating a periodic disk defragmentation and confirming the results during a weekly or monthly systems check takes minutes. These two actions alone could save the disruption and significant loss of productivity caused by system down time. Most of the concepts addressed below are subjective, and our primary recommendation is that our Client‟s use their own IT staff, processes, and procedures to align their own hardware and infrastructure best practices for all internal systems. For example, in addition to ReDoc, most of our Clients use and maintain a billing system, email, accounting/payroll systems, basic desktop applications (like Microsoft Office), and other applications. Decisions on topics like networking, security, server management, and backups should be consistent throughout the organization, and must take into account all supported applications. The checkpoints below should be compared to and used to augment your weekly, monthly, quarterly, and annual monitoring and maintenance checklists. The Suggested Best Practices below consider only ReDoc, and are, in many cases, only one of several ways in which basic tasks (such as backups) can be accomplished. Thus, this should not be considered a complete and comprehensive list. If you are not familiar with the tasks below, or if you are not comfortable with developing your own internal IT processes and procedures, we strongly urge you to seek local IT support services. If they are not available, TRDC can provide these services on a time and materials basis, but for the reasons listed above TRDC should be considered a vendor of last resort for IT support services. Single User (on a single computer)

- Don‟t “suspend” your computer; When you need to exit ReDoc, completely exit and shut down. Configure your power settings to actually shut down, and not suspend. Suspending (or sleeping/hibernating) is effectively the same as withdrawing power from your system and can cause corruption of your ReDoc database.



All Servers (including application, SQL, and Thin Client)

- Monitor and maintain manual or automated OS, SQL and .NET framework service pack updates - Monitor hard drive space and project capacity utilization over time - Periodically check redundant components (ie power supplies, network cards, RAID hard drives,

etc) for fail over preparedness. Review the cost/risk/benefit analysis of redundant components where they do not exist. For example, a redundant network card and power supply are relatively inexpensive, and have relatively high failure rates. A modest investment could keep the server accessible and would leave normal work uninterrupted in the event of a failure.

- Spot check Performance Monitor for overall capacity utilization. Monitor PerfMon over time to trend CPU, memory, disk I/O, and network traffic and review for bottlenecks

- Monitor Uninterruptable Power Supplies for battery life under no-power circumstances - Monitor UPS or line regulator for clean power - Defragment drives and files often enough to maintain less than 8% fragmentation - Review physical security measures for protection of PHI - Monitor anti-virus and other security related utilities (anti-worm, firewalls, etc) for appropriate

updates - Monitor and maintain backup routines (incremental, full, and offsite). Periodically restore from

each type of backup to ensure integrity - Periodically re-boot (complete power down for 30+ seconds) - Reliability („up-time‟) can be optimized with:

1. Redundant network cards. 2. Redundant power supplies. 3. Redundant hard drives in a RAID array, with the OS on either a separate drive or a

separate array. Monitor environmental factors (temperature and humidity) for proper levels.

- Performance can be optimized by: 1. Gigabit network connectivity. 2. Fast (10,000+ rpm) hard drives. 3. Additional memory.

Microsoft SQL Server (applies whether on a shared, dedicated, or farm server)

- Recommend MS SQL 2008 Standard edition when there are less than 10 concurrent users and not using Terminal Services; recommend Enterprise version when using Terminal Services, when supporting multiple instances, when doing replication, or when clustering or farming. MS SQL 2005 may be used on existing servers with the same recommendations for Enterprise edition as stated for 2008. SQL Express can be used for a single user, or very small multi-user environment (< 5 concurrent users). Note that SQL Express does not come bundled with a maintenance utility.

- Monitor and maintain (archive or truncate) TempDB - When Recovery mode is set to „Full‟, monitor and periodically truncate and/or archive all SQL

transaction log files - Periodically re-index and defragment indices within each SQL instance - Ensure that virus scanning is set to allow exceptions for SQL - Reliability („up-time‟) can be optimized with automated, periodic maintenance plans set up in MS

SQL Maintenance plans, and monitored frequently for proper execution. - Performance can be optimized in an Enterprise environment (more than 15 concurrent users ) on

your SQL server by: 1. Having the OS on its own physical drive.



2. Install TempDB on a physical drive other than the drive the OS is running from (usually the c:\ drive).

3. Move the OS Swap file to a physical drive other than the OS drive. 4. Set the size of your OS swap file to be a fixed size of 50% of the server‟s total RAM. 5. Move the MDF to a different physical drive than the LDF 6. If a conscious decision is made to recover from a backup only (under disaster recovery

circumstances), with the loss of the transaction log file data considered acceptable, you may set the recovery mode to „simple‟ rather than „full‟, which will result in a log file of a fixed max size and no need to truncate or archive.

Local Area, Wide Area, and Wireless Networking

- Monitor overall network utilization, including saturation, packet loss and latency assessments. Monitor the same for server backbones, and review for redundant connections where appropriate.

- Periodically check main and redundant switches and/or wireless access points for fail over preparedness. Identify single points of failure (ie switches and wireless access points) and install redundant hardware based on the cost/risk/benefit analysis. A modest investment in an extra switch or wireless access point could keep the server accessible and would leave normal work uninterrupted in the event of the loss of one switch or wireless access point.

- Monitor and maintain firewall (hardware and/or software) and any intrusion detection or other security related hardware and applications.

- Reliability („up-time‟) can be optimized with: 1. Redundant network cards in all servers. 2. Redundant, corporate grade (ie Cisco or equivalent) wireless access points instead of

small office/home office (ie Linksys) class devices. - Performance can be optimized with:

1. Gigabit throughput (network cards, switches, etc) for the wired LAN. 2. Wireless 802.11G/N network devices for wireless. Corporate grade (ie Cisco or

equivalent) are recommended over small office/home office (ie Linksys) class devices. Thin Client

- Performance is impacted by the total number of apps supported via thin client. - Recommend starting with a low number of users and monitoring all system performance metrics

before finalizing on a scaling plan. - If only ReDoc is deployed via thin client, a Dual Quad Core server with 48 gig of memory is likely

to support ~40 users. A Quad Core with 16 gig of memory is likely to support ~20 users. Light use by admin or clerical staff will require lighter memory allocation (128-256 meg), while heavy use of print preview or wound care by clinical staff will benefit by higher memory allocation (256-512 meg).

Workstation PC’s – one of the most important gains that can be realized with the ReDoc system is Point of Care documentation; allowing the therapists to document at or near where they are treating their patients, so that daily notes (and in some cases re-evals and discharge summaries) can be completed between patients instead of at the end of the day or later. By doing point of care documentation, it is far less likely that treatments delivered will go undocumented (and un-billed). To facilitate point of care



documentation, the therapists will usually need mobile PC‟s (laptops or tablet PC‟s) and a wireless network.

- Laptops generally offer the most flexibility, since they include keyboards. If Tablet PC‟s are used, companion keyboards are encouraged since therapy initial evaluations and, in some cases, other documentation can include extensive free text typing.

- Extra power cords and/or batteries for laptops or tablet PC‟s can significantly enhance usability. - Desktop PC‟s are generally best suited only for desktop space and often are not ideal for point of

care documentation, since they are completely immobile and take up significant space. Printing – The ReDoc application uses the Operating System (Windows) print spooler. If printing appears to be slow, it can be diagnosed by right clicking on the target printer in Printer Setup and selecting Properties. Any delay experienced there contributes to the delay in printing a document from within ReDoc. Budgeting for hardware lifecycles. As a general rule, workstations can be expected to last approximately three years, and servers about 4-5 years. Financially, most organizations will budget for cycling out their hardware at these intervals, but only actually purchase new equipment when absolutely necessary. That decision requires a careful analysis of the risk of losing a single piece of hardware. For example, workstations can often be lost, and replaced within a week or two, without significant loss of productivity. The unexpected loss of mission critical servers may so substantially hurt productivity, that it makes financial sense to have 100% redundancy so any single failure has a backup. In multi-server environments, some IT professionals will proactively replace and „retire‟ or repurpose older servers to less mission critical roles after 3-4 years and run them there until they fail beyond repair. Minimum System Specifications. A current copy of the Minimum System Specifications for ReDoc is available at www.rehabdocumentation.com/hardware . Appendix A: Performance Assessment and Troubleshooting Guide Appendix B: Draft Hardware and Systems Preventative Maintenance Checklists (should be subordinated

to the Client‟s existing IT processes and procedures, or customized based on the needs and use volume of each Client)

http://www.rehabdocumentation.com/hardware



Appendix A: Performance Assessment and Troubleshooting Guide As a guide for establishing expected performance standards for the end user experience, the following guidelines are provided:

- Load the Patient List in <4 seconds - Preview a Progress Note in <6 seconds - Filter the Patient List in <2 seconds - Create a ReExam in <4 seconds - Create a Progress Note in <6 seconds - Load the Patient File Menu in <2 seconds

For the purposes of determining where performance improvements might be made within the IT infrastructure, it can help to consider what are normally the three main parts of a network; the Server Room (and all components between the servers and the firewall or last router in that building), the Wide Area Network connecting to User Site(s), and the User Site(s) themselves.

RouterSwitch

Wireless Access Point

Application, database, and Thin Client Servers.

May be one or more, depending on number

of End Users.

End User with ReDoc Client

(Wireless)

Router Switch

End User with ReDoc Client

(Wired)

ReDoc Client on Either a PC or

One Server

Server Room User Site(s)

Wide Area Network

Firewall Firewall

If the user tasks described above can be executed within the response times stipulated by an end user connected wirelessly at the User Site, then the infrastructure as a whole is sufficient in terms of hardware, bandwidth, and latency. If longer than anticipated response times result, the tests should be repeated on a wired workstation (isolating the wireless network). If response times are then within limits, the wireless network should be



diagnosed and fixed. If response times are still not within limits, then the tests should be repeated using a copy of the ReDoc Client installed either on one of the servers or on a PC inside the server room (isolating the WAN and all components at the User Site). If response times are within limits when tested from the server room, then the WAN should be diagnosed for latency, bandwidth, and overall Quality of Service. If response times are not within limits inside the server room, check CPU, memory, disk I/O, fragmentation, and networking metrics on the servers, and confirm that database maintenance routines are being executed properly. Performance Measurement and Troubleshooting Tools. To augment your use of normal network monitoring and troubleshooting tools, ReDoc can collect and provide metrics from inside the ReDoc Suite application to help you benchmark performance on your network. Beginning in September 2010, this assessment will be done once as part of a new Client‟s implementation process to validate performance prior to scheduling your initial go-live. Reassessments (or other network troubleshooting services) may be requested at any time by the Client, but they are not part of the Service Level Agreement and will be charged on a time and materials basis. Below are some examples …

Applying the Patient List Filter – many Clients aggregated This is an aggregate of many clients showing response times (on the x-axis) by frequency of occurrence (on the y-axis) for Applying a new filter to the Patient List. For all instances measured for all Clients,



>99% of the response times were <1 sec. Some, however, were > 4 sec. Not all results could be shown in this one screen shot; in fact, some response times went out as far as 39 seconds.

Applying the Patient List Filter – one Client This is an example of the same metric - response times for Applying the Patient List Filter – but at just one Client. It illustrates that the vast majority of all instances of this event had response times of < 1 sec, which is as expected. However, a notable number of instances longer than 2 sec, and ranging up to almost 26 seconds, indicates significant variation in response time over this network. Comparing this individual Client‟s response time to all Clients seems to indicate that this one Client is the source for most, if not all of the response times above 1-2 seconds in the all Clients view. This would merit further troubleshooting by this Client to examine latency and possibly routing or other factors that could cause such a significant variation in response times over their network. Accurately diagramming your Network. One of the most valuable things you can do to help ensure accurate execution of your network infrastructure planning, and later evolving or troubleshooting that



network, is to create an accurate and detailed network schematic. This should include details such as Server names, OS versions, domain organization, router/switch/WAP make and model numbers, etc.

DeltaComm Data only T1

1.544

100bt

100bt

1 gigbt1 gigbt

4 Cisco 9876 802.11N WAP’s with 80% overlapping coverage zones

16 Mobile users on Dell Latitude laptops running TS Client

with .11n wireless cards

Juniper M7i GigBT Router

Cisco 1234 24 port GigBT Switches (A and B)

8 Desktop Users running TS Client

with 100mBT NIC’s

West Side Office

Barracuda 300nSpam/AV/IntDect

Appliance

Portion of a Sample Network Diagram



Appendix B: Draft Hardware and Systems Preventative Maintenance Checklists

(should be subordinated to the Client‟s existing IT processes and procedures, or customized based on the

needs and use volume of each Client)

Initial Setup (either during a ReDoc implementation, or at the initial creation of a Preventative Maintenance Plan)

Managing hardware lifecycles.

Compare existing server and network hardware to minimum and recommended system specifications;

o Identify existing hardware that can be applied to ReDoc tasks o Assess and advise on the cost/benefit of hardware enhancements for performance or

reliability o Propose new hardware acquisition where necessary o Project hardware lifespan for server and network components


Assess the current version of the Operating System for service pack currency and growth potential. If Windows Server 2003 is in use, establish a plan to update to 2008 before Microsoft sunsets 2003.

Determine and configure for OS and .NET framework service pack updates for automated or manual

Establish an automated maintenance plan for: o Monitor hard drive space and alert for pending shortfalls. Project capacity utilization

over time o Defragmenting drives and files weekly o Daily incremental (ReDoc app, ReLeaf Directories, etc) and weekly full backups

Check redundant components (ie power supplies, network cards, RAID hard drives, etc) for fail over preparedness.

Spot check Performance Monitor for overall capacity utilization. Trend CPU, memory, disk I/O, and network traffic over an average work day and review for bottlenecks

Check Uninterruptable Power Supplies for battery life under no-power circumstances Check UPS or line regulator for clean power Review physical security measures for protection of PHI Check Storage and Hardware backup plans (for more detail on suggested Backup Plans, see

the Minimum Systems Specifications at www.rehabdocumentation.com/hardware ) Check anti-virus and other security related utilities (anti-worm, firewalls, etc) for appropriate

updates Periodically re-boot (complete power down for 30+ seconds).




Microsoft SQL Server

Assess the current version of SQL for service pack currency and growth potential. If SQL 2005 Standard or Enterprise Edition are in use, establish a plan to update to 2008 before Microsoft sunsets 2005.

Determine and configure for SQL service pack updates for automated or manual Determine whether Recovery mode will be set to „Full‟ or „Simple‟ Establish an automated maintenance plan for;

o Daily incremental and weekly full backups (for more detail on Backup Plans, see the Minimum Systems Specifications at www.rehabdocumentation.com/hardware )

o Archiving truncating TempDB

o If recovery mode is set to „Full‟, monitor and periodically truncate and/or archive all SQL transaction log files

o Re-index daily

o Defragment indicies weekly Ensure that virus scanning is set to allow exceptions for SQL Server hardware permitting, evaluate the following steps for potential performance

enhancement: o Having the OS on its own physical drive.

o Install TempDB on a physical drive other than the drive the OS is running from (usually the c:\ drive).

o Move the OS Swap file to a physical drive other than the OS drive.

o Set the size of your OS swap file to be a fixed size of 50% of the server‟s total RAM.

o Move the MDF to a different physical drive than the LDF

o If a conscious decision is made to recover from a backup only (under disaster recovery circumstances), with the loss of the transaction log file data considered acceptable, you may set the recovery mode to “simple” rather than “full”, which will result in a log file of a fixed max size and no need to truncate or archive.


Assess network for bottlenecks (ie low bandwidth NIC‟s, hubs, or switches) Monitor overall network utilization, including saturation, packet loss and latency

assessments. Monitor the same for server backbones, and review for redundant connections where appropriate.

Check main and redundant switches and/or wireless access points for fail over preparedness. Identify single points of failure (ie switches and wireless access points) and install redundant

hardware based on the cost/risk/benefit analysis. A modest investment in an extra switch or wireless access point could keep the server accessible and would leave normal work uninterrupted in the event of a most failures.

Monitor and maintain firewall (hardware and/or software) and any intrusion detection or other security related hardware and applications.

Thin Client

Monitor initial use (first three weeks) of different types of users to access memory allocations o Overall server performance (memory, CPU, network bandwidth, and disk I/O)

o Memory allocation for admin or clerical staff (initial settings likely to be 128-256 meg)




o Memory allocation for heavy use of print preview or wound care by clinical staff (initial settings likely to be 256-512 meg)

Project the growth capacity (future additional staff) of the server hosting the Thin Client sessions

Workstation PC’s

Discuss different form factors (laptop, tablet, and desktop) based on client-specific physical layout of the workspace

Review the availability of power, and consider the impact of additional power supply points for laptops or additional shared batteries

Establish an automated maintenance plan for periodic disk defragmenting

Printing

Test print from each workstation using ReDoc Discuss the conditions under which routine printing will take place. For example, if a Result

Reporting interface is in place, clinical users may not do any routine printing, but medical records (or other admin staff) may print Plans of Care or other documents routinely.

Review the physical locations of printers accessible to ReDoc users, and optimize based on routine usage

Managing hardware lifecycles

Compare existing server and network hardware to minimum and recommended system

specifications; o Identify existing hardware that can be applied to ReDoc tasks o Assess and advise on the cost/benefit of hardware enhancements for performance or

reliability o Propose new hardware acquisition where necessary o Project hardware lifespan for server and network components



Draft Daily Checklist


Check the integrity of the previous day’s backup (for more detail on suggested Backup Plans, see the Minimum Systems Specifications at www.rehabdocumentation.com/hardware )

Draft Weekly Checklist


Check the integrity of all automated maintenance plans Check anti-virus and other security related utilities (anti-worm, firewalls, etc) for appropriate

updates Check RAID drives (if used) for failures Conduct a hard re-boot (complete power down for 30+ seconds).

Draft Monthly Checklist


Check the integrity of Operating System, SQL, and .NET framework service pack updates (whether configured for automated or manual)

Spot check Performance Monitor for overall capacity utilization. Trend CPU, memory, disk I/O, and network traffic over an average work day and review for bottlenecks

Draft Quarterly Checklist


Restore recent backups to ensure backup integrity Check Automated Maintenance Routines for appropriate configuration, intervals, and the

possible deletion of ineffective routines or the addition of new routines. Check redundant components (ie power supplies, network cards, etc) for fail over

preparedness. Check Uninterruptable Power Supplies for battery life under no-power circumstances Check UPS or line regulator for clean power Review physical security measures for protection of PHI Check anti-virus and other security related utilities (anti-worm, firewalls, etc) for appropriate

updates.


Monitor overall network utilization, including saturation, packet loss and latency assessments. Monitor the same for server backbones, and review for redundant connections where appropriate.




Monitor and maintain firewall (hardware and/or software) and any intrusion detection or other security related hardware and applications.

Thin Client

Monitor use of different types of users to access memory allocations o Overall server performance (memory, CPU, network bandwidth, and disk I/O)

o Memory allocation for admin or clerical staff (initial settings likely to be 128-256 meg)

o Memory allocation for heavy use of print preview or wound care by clinical staff (initial settings likely to be 256-512 meg)

Draft Annual or Semi-annual Checklist (should be scheduled in sync with the Client’s annual budgeting process)

Managing hardware lifecycles

Compare existing workstation, server and network hardware to current and projected (for the next 12 months) minimum and recommended system specifications;

o Assess and advise on the cost/benefit of hardware enhancements for performance or reliability

o Propose new/replacement hardware acquisition budget where necessary o Project hardware lifespan for server and network components


Assess the current version of the Operating System for service pack currency and growth potential. Plan and budget for version updates if appropriate.

Microsoft SQL Server

Assess the current version of SQL for service pack currency and growth potential. Plan and budget for version updates if appropriate.


Assess network for bottlenecks (ie low bandwidth NIC‟s, hubs, or switches) Check main and redundant switches and/or wireless access points for fail over preparedness. Identify single points of failure (ie switches and wireless access points) and install redundant

hardware based on the cost/risk/benefit analysis. A modest investment in an extra switch or wireless access point could keep the server accessible and would leave normal work uninterrupted in the event of a most failures.

Thin Client

Project the growth capacity (future additional staff) of the server hosting the Thin Client

sessions

Client Hardware and Infrastructure Suggested Best Practices

Documents

Transcript of Client Hardware and Infrastructure Suggested Best Practices