Rapid Machine Rapid Machine DeploymentDeploymentThomas Kunka & Jon Weber

JON WEBERJON WEBERNetwork Administrator for Instructional

Computing

Electrical and Computer Engineering (ECE)

University of Illinois at Urbana-Champaign

ECE Instructional EnvironmentECE Instructional Environment

235 Dell Precision Workstations and Optiplex USFF desktops (4 year lifecycle)

17 labs and 14 officesPredominantly Windows XPUse Campus AD for auth

Deployment IssuesDeployment Issues

Large number of machinesDifferent hardware between old and

newVaried application needs across labsSome engineering applications very

largeTwo remote facilities

Current Deployment MethodsCurrent Deployment Methods

Ghost (Multicast)WDS (formerly RIS)Windows PreInstallation Environment

(PE)GPOs for application installs

GhostGhost

Primary deployment method since 2005Currently running Ghost Solutions Suite 2.0

(11.0.1)◦ Ghost Console on Server 2003 R2 file

server◦ Ghost Client software installed on each

workstationUse to create images, push images, and

install applicationsAverage image size is ~ 25 GBCurrently maintain 6 customized XP images

Ghost ConsoleGhost Console

Ghost Server BenefitsGhost Server Benefits

Multicast – can push large images out to multiple labs in a relatively short time

Central Management – one click deployment

Image creation/cloning means all machines are the same

Extra features like application deployment (AIBuilder), inventory, user migration

Ghost Server DrawbacksGhost Server Drawbacks

Requires extra setup time◦ Workstations need client software to

communicate – no bare-iron support * No native support for different hardware in

cloned images◦ But Sysprep works really, really well for that

Works well in primary building, no multicasting to other buildings though

Extra licensing cost ($10 per client)

Windows Deployment Services Windows Deployment Services (WDS)(WDS)

Formerly RIS – introduced in Server 2003 SP2 and updated in Server 2008

Mainly used in ECE to do quick OS installs when standard image not necessary

WDS ConsoleWDS Console

WDS Pros and ConsWDS Pros and Cons

Benefits◦ PXE makes bare installs easy◦ Native cross-hardware support◦ No additional licensing cost◦ Images can be modified off-line

Drawbacks◦ Still network based, requires DHCP and PXE◦ Multicast support not available until Server

2008 releases

Windows PEWindows PE

Bootable Windows CD (like Knoppix)Can be combined with Ghost

executables for installs◦ We store images on file servers or external

hard drives for portabilityOr booted from WDS/PXE server

Application InstallsApplication Installs

Combination of:◦ Ghost AIBuilder (captures changes installer

makes into an .exe)◦ GPOs for .msi files◦ Manual installs

What’s Next for ECEWhat’s Next for ECE

Vista / Office 2007◦ When?◦ How? – will re-examine Ghost and WDS

Linux / Alternative OS deployment?Application Virtualization?

THOMAS KUNKATHOMAS KUNKACoordinator, Network Systems and OperationsCITES Classroom TechnologiesUniversity of Illinois at Urbana-Champaign

The supreme art of war is to subdue the enemy without fighting. Sun Tzu

OverviewOverview

Setting the StageGeneral DirectionsFirst LifecycleSecond LifecycleDrive DuplicatorsLessons LearnedCosting ModelFuture DirectionsFuture ChallengesFinal Thoughts

The Wayback Machine – June 1The Wayback Machine – June 1stst 20012001

Resident Computers in ‘Smart Classrooms’

Approx 70 PCsApprox 40 buildings17 different hardware configurationsLocal vendor – custom builtNo software licensing records

Distinctive NeedsDistinctive Needs

Unlike a labUnlike an officeUnlike a kioskCriticalityTime constraintsLogistics

Needs AssessmentNeeds Assessment

Standardize equipmentImprove inventory processesStandardize software & configurationImplement rigorous configuration

testingDevelop new duplication/deployment

modelImprove update and management

abilities

GENERAL DIRECTIONGENERAL DIRECTION

The general who wins the battle makes many calculations in his temple before the battle is fought. The general who loses makes but few calculations beforehand. Sun Tzu

New hardware◦ Dell GX400

Software licensing◦ MCA, campus-wide, etc.

Symantec Ghost◦ 200 seat license

Problem tracking system◦ SharePoint

Active Directory◦ Classroom OU◦ Standard naming conventions

Group Policy◦ Look n’ feel◦ Security◦ Update services

Windows Update Services◦ Campus?◦ Departmental?

Registry Scripts◦ Lockdowns◦ Login

WMI Scripts◦ Status◦ Settings◦ Restart/Shutdown

FIRST LIFECYCLEFIRST LIFECYCLE2001-2004

First Lifecycle: 2001-2004First Lifecycle: 2001-2004

Dell GX400◦ Custom factory assembly◦ Storcase removable drive system◦ IDE Drives

20 extra drives

DuplicationDuplication

Plan A: Ghost Over NetworkPlan B: Removable Drives

Quickly realized that Plan A may not be feasible due the size of the image, varied building networks, and required fieldwork.

The ‘Ah-Ha’ MomentThe ‘Ah-Ha’ Moment

Deployment◦ Drive to drive duplication◦ Preconfiguration◦ Walk-out & turn on deployment

Maintenance◦ Drive failure = replace drive, configure for

room◦ Other failure = replace chassis, keep drive◦ Either option = 10-15 minute solution◦ Able to react quickly & maintain service◦ Not a lot of thinking involved

Standardized GX400 System Standardized GX400 System ImagesImages

“Standard”◦ Windows XP◦ Standard applications◦ Media players, plugins, etc.

“Engineering”◦ Standard + engineering applications◦ AutoCAD, Mathmatica, ProE, MatLAB, etc.

Several months configuring & testing

Drop all day – Clone all nightDrop all day – Clone all night

During major rollouts◦ We would duplicate 20 drives ahead of

time◦ Drop the 20 new & bring back 20 old◦ Repeat until complete

First Lifecycle ResultsFirst Lifecycle Results

Problem reports reduced 71%Attributed to…

◦ Expanding breadth of software available◦ Expanding depth of configuration settings◦ Extensive system testing

Outcomes…◦ Increased responsive to problem reports◦ Increased responsive to customization

request

The Right Tools?The Right Tools?

A good proactive support model will◦ Improve customer service◦ Reduce required support staffing

Allows time for ◦ Even more proactive measures◦ Further improvement of customer service◦ Further reduction in staff costs◦ Other IT projects

SECOND LIFECYCLESECOND LIFECYCLE2004-Present

Second Life CycleSecond Life Cycle

Dell GX280CRU removable drive systems40 extra drives

DRIVE DUPLICATORDRIVE DUPLICATORLogicube OmniClone 5u

Drive Duplication SystemDrive Duplication System

Hardware-based drive to drive duplication

OmniClone 5u◦ 1 Master & 5 Targets◦ 3 GB/min transfer rate

(new models are 3.5GB/min)

◦ Mirror Copy & Clever Copy◦ IDE/SATA◦ Adapters for various drive types◦ Drive diagnostics & wiping functions

OmniClone5Xi

Classtech ‘Rig’Classtech ‘Rig’

Wanted to use removable drive cagesNeeded IDE to SATA adaptersPlugs modified with a DremmelNeeded Molex power to SATA power

adaptersUgly but functional

Results with the Results with the OmniCloneOmniClone

First Cycle Second Cycle

Good, about 1GB/minTwice as fast as GhostVery few failed runsVery happy

Duplicator had issuesSlow duplicationMany failed runsBecame unusableProblems still

unresolved

What went wrong?What went wrong?

Nothing wrong with duplicator itselfThe problems worsened as the drives

agedDiagnostics on the drives revealed bad

sectors

TRICKLE-DOWNTRICKLE-DOWNGX400 Redeployment

Budget Conscious DecisionsBudget Conscious Decisions

3+ years oldReallocated old classroom systems

internallyAble to gain standardizationUtilized removable hard drivesConsolidated RAM to get 512MBExtra parts to cannibalize

Impact on Office ComputingImpact on Office Computing

Met needs of most staffRemovable drives paid off again

◦ Drive failure = replace drive, user configuration

◦ Other failure = replace chassis, keep drive◦ Either option = 10-15 minute solution◦ Very little staff downtime◦ Very little thinking involved

For to win one hundred victories in one hundred battles is not the acme of skill. To subdue the enemy without fighting is the acme of skill. Sun Tzu

LESSONS LEARNEDLESSONS LEARNED

#1 – Removable Drives #1 – Removable Drives

They just workThe most likely component to failAble to get systems up quicklyMore drives = faster, scalable

deploymentMake sure they are ruggedConsider cooling fans

#2 - Duplication#2 - Duplication

Limit the number of images/buildsTest twice, clone onceRun speed testsAlways have a backup plan

◦ Drive to Drive, Network, etc.Always leave some slack time

#3 - Purchasing#3 - Purchasing

Custom configurations take longer to getAim high on the technical specsGet development units in ahead of timeEnroll in ‘parts’ service plans when

possibleSpecs must be articulated very clearlyBuild in development/replacement unitsBuild in units for projected growthOrder enough extra drives

#4 – Processes#4 – Processes

Keep a good inventory database ◦ pas tag, service tag, serial number, MAC◦ if under $500, consider tracking anyway

Configure drives in the shopLabel each drive and keep good

checklistsWMI Scripts provide operational statusDevelop standardized nomenclature

#5 – More Processes#5 – More Processes

Get a handle on software licensingDisposal of packing materialsCarting things around

◦ Systems◦ Drives

Storing drivesWiring plans

COSTING MODELCOSTING MODEL

The Case for Removable The Case for Removable DrivesDrives

With Removable Drive Without Removable Drives

100 units$810/unit

◦ Custom integration & routing

◦ Removable drive hardware

$81,000 systems total$2750 extra drives

◦ $??/each

◦ Qty. 50

$XX,XXX total cost

100 units$706/unit$70,600 systems total

Time-Driven Activity-Based Time-Driven Activity-Based CostingCosting

Define support processes◦ With removable drives◦ Without removable drives

Estimate duration and frequency of events

Relate all process costs to time to determine cost of support activities

Justified?Justified?

Do removable drives save enough support time to justify added expense of hardware?

Current model is easy to costOther models are difficult to conceive

FUTURE DIRECTIONSFUTURE DIRECTIONS

Next Lifecycle?Next Lifecycle?

Removable drivesExternal SATA portsVirtualizationImproved management toolsContinue to develop lifecycle cost

models

FUTURE CHALLENGESFUTURE CHALLENGES

Rapidly changing technology◦ Virtualization, Windows Vista, etc.◦ AutoCAD, Multimedia, iTunes, Skype, etc.

Need for increased scalabilityBalancing R&D vs. OperationsDecreasing quality of hardwareTighter budgets / Recurring fundingLonger lifecycles

FINAL THOUGHTSFINAL THOUGHTS

Decidedly low-tech 6 year old solutionIncremental improvementsSatisfied instructorsVery scalableDoubled the number of machines

(117)Six years and counting on GX400sActually reduced the required staffingEased support burdenTook on more projects

Goal for next lifecycleGoal for next lifecycle

Questions?Other Deployment Experiences?