JBoss in the Trenches - Red HatJBoss In The Trenches Red Hat's EAP6 Migration 250 Applications One...
Transcript of JBoss in the Trenches - Red HatJBoss In The Trenches Red Hat's EAP6 Migration 250 Applications One...
JBoss in the Trenches
Andrew BlockSenior Consultant, Red Hat
Tim BielawaSenior Release Engineer, Red Hat
JBoss In The Trenches
Presenters
Andrew Block
•2nd year presenting JBoss in the Trenches
•JBoss master
Tim Bielawa
•Emacs Guru
•Team “point man” during the EAP6 migration
•Has an actual Tux mascot tattoo (watch for it)
JBoss In The Trenches
JBoss In The Trenches
Session Topics
•Beginning the Migration
•Configuration Management
•Best Practices
•Production Support
•Advance to The Cloud
JBoss In The Trenches
Red Hat's EAP6 Migration
250 Applications
One Year
Legacy EAP 4.3/5.1 Clusters
Dense to Sparse Clustering 50 Red Hatters
JBoss In The Trenches
Legacy Infrastructure
Client initiatesREST request
iRule determines ifclient is allowed
Request balancedto a member of the cluster.
An iRule is a TCLscript running onthe BIG-IP device.
It's like iptableson steroids.
JBoss In The Trenches
?
New Infrastructure
JBoss In The Trenches
Migration - Timeline
ApplicationAnalysis
InitialPorting
OperationsConsulting
DeploymentScripting
MonitoringTooling
Release,Cutover
January December
IN THE BEGINNINGSTARTING THE MIGRATION
JBoss In The Trenches
Application Analysis
What applications do we need to migrate?
What are the Application dependencies?
•Databases
•Messaging
•Other Applications
How are applications currently built, deployed and managed and how should they be in the future?
JBoss In The Trenches
Common Migration Tasks
•Updating libraries
•Adding/updating container specific deployment descriptors
•Updating JNDI references
•Determine enterprise wide logging framework
What if there was a tool which could aid in this process?
Good news, there is: Windup
JBoss In The Trenches
Windup
●Application migration tool●Can be run on source code or compiled archives●Generates reports
● Provides an overall level of effort in “Story Points”● Displays XML and Java hints and classifications to aid in migration effort
●Extensible
•New hints and classifications can be defined
JBoss In The Trenches
Windup – Story Points
JBoss In The Trenches
Windup - Classifications
JBoss In The Trenches
Windup - Hints
CONFIGURATION MANAGEMENT
JBoss In The Trenches
JBoss Management
How do we manage the JBoss platform?
How can we manage our configurations efficiently?
What tools can we use?
How can we manage our application deployments?
JBoss In The Trenches
Core Tools
Puppet●IT configuration management tool
libeap6 (custom RHIT Internal)●Puppet module for managing and deploying EAP6
jcliff (custom tool, open source)●Tool for applying configurations to EAP6
JBoss In The Trenches
jcliff In Brief
Begin End
Puppet installs or updates jcliff config files
jcliff compares expected vs. actual (live) configuration.
Applies diffs if necessary via jboss cli.
Notify admin if JBoss needs reloading.
JBoss In The Trenches
Run Puppet
Artifact Workflow
Builder packages JARs in RPMs
UploadTo Pulp
JBoss In The Trenches
JBoss Configuration Management
The management of platform configurations are as important as the configurations themselves
Configurations should be stored and versioned in a source control management system such as Git
Similar to applications, configurations should be deployed to each environment in an automated fashion
Externalize environment specific configuration values
JBoss In The Trenches
JBoss Configuration Management (cont.)
Red Hat IT stores puppet modules in Git. Each repository has branches for each server environment
Git hooks are in place to ensure configurations have been merged into all lower environments before being promoted
We call this the parenting hook
BEST PRACTICES
JBoss In The Trenches
Deployment
Deployments should be handled in an automated scripted process
Have an established rollback strategy
Prepare for the worst (pre-deployment):
• Snapshot server configurations and backup file systems
Can be a component of incremental builds in a Continuous Integration Environment
JBoss In The Trenches
High Availability
Critical component of any infrastructure
Common methods: Clustering and Load Balancing
Enables Rolling Updates during deployments
JBoss In The Trenches
Red Hat's HA Strategy
Pre-Migration(Dense Clustering)
Post-Migration(Sparse Clustering)
Many apps,single cluster
Many clusters,fewer apps on each
JBoss In The Trenches
JBoss Platform Management
Web Management Console
User friendly web interface for managing basic JBoss configurations and runtime settings
Only exposes a subset of management capability
Command Line Tool (CLI)
Tool for interacting with the JBoss platform providing access to the full set of management operations
JBoss In The Trenches
JBoss CLI
Uses a Detyped Management Model to execute operations
GUI component one of the best “Easter Eggs” of JBoss
$ jboss-cli.sh -c --gui
Public CLI API can be used to build robust applications● Can be written using Java, Groovy, Javascript, etc
JBoss In The Trenches
JBoss In The Trenches
JBoss Management Access
Limit access as you get closer to production
Migration Effort● Admin console/management interface disabled in production
● Configurations can only be initiated through Puppet/libeap6
JBoss
• Limit access by use of Role Based Access Control (RBAC)
JBoss In The Trenches
Role Based Access Control
New feature of JBoss EAP 6.2
Provides separation of duties among users
7 standard roles with varying access
Fine tune access to resources using sensitivity, vault, and application level constraints
PRODUCTION SUPPORT
JBoss In The Trenches
Day-to-Day Operations
Question: How do you manage the day to day operations of the JBoss platform and the applications deployed?
Answer: Monitor the right resources:● Application● System● Thresholds
JBoss In The Trenches
Production Support
Have a plan in place to support application and overall server infrastructure. We call ours the Pager Playbook
•Description and example of the issue
•Areas to diagnose
•Steps to resolve
•Escalation personnel
JBoss In The Trenches
Management Tools
jconsole
Splunk
Munin
Naigos
Taboot
Custom groovy scripts for monitoring
JBoss Operations Network
JBoss In The Trenches
Metrics Via Munin
What happened
here?
JBoss In The Trenches
Splunk – Old/New Cluster Requests ChartsEAP 6.1
EAP 5.1
JBoss In The Trenches
Stats Via Splunk
93.6% came from monitoring!
JBoss In The Trenches
Patch Management
EAP 6.2 contains new patch management feature
• Application of individual and cumulative patches from Red Hat
• Simplified patching strategy from previous releases
Executing through the CLI tool● Ability patches● View applied patches● Rollback patches
ADVANCE TO THE CLOUD
JBoss In The Trenches
Advancing Towards a Cloud Infrastructure
IaaS PaaSIaaS
JBoss In The Trenches
Advancing Towards a Cloud Infrastructure
●Offers technological and financial flexibility
●Additional considerations must be accounted for:
• Limitations
• Security●Leverage additional tools while
reusing the migration toolset●Endless possibilities!
WRAP UP
JBoss In The Trenches
Bryan Madaras
A “THANKS!” of considerable
magnitude goes out to Bryan Madaras
for his efforts preparing this session
with us and for his machine-like focus
during IT’s migration.
JBoss In The Trenches
Download this Slide Deck!
http://people.redhat.com/~tbielawa/summit2014/
Most slides have additional notes with links to any referenced material
JBoss In The Trenches
JBoss In The Trenches
Thank You!
Andrew Block
@sabre1041
http://blog.andyserver.com
Tim Bielawa
@tbielawa
https://blog.lnx.cx
JBoss In The Trenches
Questions?
JBoss in the Trenches
Andrew BlockSenior Consultant, Red Hat
Tim BielawaSenior Release Engineer, Red Hat
JBoss In The Trenches
Presenters
Andrew Block
•2nd year presenting JBoss in the Trenches
•JBoss master
Tim Bielawa
•Emacs Guru
•Team “point man” during the EAP6 migration
•Has an actual Tux mascot tattoo (watch for it)
SHARED
http://www.redhat.com/summit/speakers/session.html#tbielawa
http://www.redhat.com/summit/speakers/session.html#ablock
Tim:● Avid Blogger of technical things● Open Source contributor/maintainer for a decade● Writing a book● Co-creator of a scholarship at WVU to encourage
young adults to become involved in open source.
See also: “Thank You!” at the end of this presentation.
JBoss In The Trenches
TIM
Thanks for joining us!
Over the next hour we're going to move through the story of how Red Hat IT migrated legacy 4.3/5.1 clusters to EAP6.1
Along the way we'll be identifying important implementation decisions, as well as best practices.
And now Andy's going to explain what that covers in greater detail.
JBoss In The Trenches
Session Topics
•Beginning the Migration
•Configuration Management
•Best Practices
•Production Support
•Advance to The Cloud
ANDY
JBoss In The Trenches
Red Hat's EAP6 Migration
250 Applications
One Year
Legacy EAP 4.3/5.1 Clusters
Dense to Sparse Clustering 50 Red Hatters
TIM - CLICK THORUGH
Before we continue I want to give you an idea of the scope of the migration project, beginning with explaining why it even happened at all.
Why Migrated: We were running unsupported versions of JBoss in production.
How much migrated: Rough estimate of 250 apps.
What changed: Cluster with many apps turned into more clusters with fewer apps.
How many people: Nearly every team in IT was involved. 3-4 development towers, project management office, sysadmins, production control, networking, qa testing..
How much time: Took about a year.
JBoss In The Trenches
Legacy Infrastructure
Client initiatesREST request
iRule determines ifclient is allowed
Request balancedto a member of the cluster.
An iRule is a TCLscript running onthe BIG-IP device.
It's like iptableson steroids.
TIM
Services call out to a known REST/SOAP endpoint.
Load balanced. Originally via mod_jk + httpd, now via F5 balancer pools.
services-security is an F5 iRule that operates as a firewall of sorts.
If you were allowed through the firewall then you were balanced to a member of the pool.
All services were hosted on the same cluster.
JBoss In The Trenches
?
New Infrastructure
TIM
Similar to the original setup, however we have two routing points involved now.
Top is the ‘director’, basically an httpd ProxyPass. It routes traffic by inspecting the incoming request to determine the specific service endpoint requested and then forwards your request to the correct cluster.
Now services are partitioned into three separate clusters.
We kept the “Old SOA VIP” around during the migration as a catch-all. If a service wasn’t ported to EAP6 yet, then the request was directed back to the 5.1 cluster.
JBoss In The Trenches
Migration - Timeline
ApplicationAnalysis
InitialPorting
OperationsConsulting
DeploymentScripting
MonitoringTooling
Release,Cutover
January December
TIM
CLICK THROUGH
IN THE BEGINNINGSTARTING THE MIGRATION
JBoss In The Trenches
Application Analysis
What applications do we need to migrate?
What are the Application dependencies?
•Databases
•Messaging
•Other Applications
How are applications currently built, deployed and managed and how should they be in the future?
ANDYIntroduction slide to application analysis section
* What applications fall under the scope of the migration effort?
How are the applications currently being built?* Ant/Maven?
How are the applications currently being deployed?* Manually* Scripted process* Automated?
How are the applications managed?* Artifact repository?* Package manager?
JBoss In The Trenches
Common Migration Tasks
•Updating libraries
•Adding/updating container specific deployment descriptors
•Updating JNDI references
•Determine enterprise wide logging framework
What if there was a tool which could aid in this process?
Good news, there is: Windup
ANDY - CLICK THROUGH
* Libraries include core JDK/JBoss libraries but also 3rd party libraries which may no longer be supported by the JDK or JBoss. * Deployment descriptors have evolved through JBoss versions. These must be updated to support the most recent container * jboss-classloading.xml -> jboss-deployment-structure.xml * JNDI - java:/TransactionManager -> java:/jboss/TransactionManager* Various logging frameworks exist. Being able to determine the particular logging framework to utilize up front will allow for a single strategy moving forward
JBoss In The Trenches
Windup
●Application migration tool●Can be run on source code or compiled archives●Generates reports
● Provides an overall level of effort in “Story Points”● Displays XML and Java hints and classifications to aid in migration effort
●Extensible•New hints and classifications can be defined
ANDY -
* Story point is a unit of work. It can equate to 1 hour of work in a shop with senior resources, but can vary depending on the skill set* Hint can alert the developer that a modification may be required as part of a migration effort* Classifications are ways of identifying a type of file (eg: jboss-ejb3.xml = JBoss EJB descriptor)
Windup also features support for users to define their own set of hints and classifications. This can be useful when teams are migrating several applications and want t
JBoss In The Trenches
Windup – Story Points
ANDY - When you run the windup tool, it generates a report to help you migrate your application. When viewing the report, you are presented with an overall diagram of what the migration entails. The story points are on the left as previously discussed. On the right is a depiction of the classes and number of references in which they appear in the classes within the application.
JBoss In The Trenches
Windup - Classifications
ANDY - Classifications will appear as informative queues above certain files containing steps required for successful migration. In this example of a jboss.xml file, the classification describes the usage for this file and the requirement to update this file to a jboss-ejb3.xml specification.
JBoss In The Trenches
Windup - Hints
ANDY -
Hints will appear in Java and XML classes as potential places which require migrations
CONFIGURATION MANAGEMENT
JBoss In The Trenches
JBoss Management
How do we manage the JBoss platform?
How can we manage our configurations efficiently?
What tools can we use?
How can we manage our application deployments?
ANDY - CLICK THROUGH 4x
SEVERAL AREAS OF CONSIDERATION
For the migration effort, we need to begin to think how we would go about managing the JBoss platform * In previous versions of JBoss, you would utilize JMX/twiddle to view statistics on the container and applications
These are the critical questions which the migration team pondered and lead to the development of several core tools.
SEGUE: .... Which Tim will describe .
JBoss In The Trenches
Core Tools
Puppet●IT configuration management tool
libeap6 (custom RHIT Internal)●Puppet module for managing and deploying EAP6
jcliff (custom tool, open source)●Tool for applying configurations to EAP6
TIM -
Internet Resources:
• Puppet: http://puppetlabs.com/
• Libeap6 is currently an internal only module. There are no active plans to open it up at this time.
• Jcliff: https://github.com/bserdar/jcliff/
JBoss In The Trenches
jcliff In Brief
Begin End
Puppet installs or updates jcliff config files
jcliff compares expected vs. actual (live) configuration.
Applies diffs if necessary via jboss cli.
Notify admin if JBoss needs reloading.
TIM - 1. Puppet run initiated2. Puppet determines is any configuration has been modified since last run via puppet managed config files in a format understood by jcliff, these files are managed by types in libeap6 so they can be controlled3. jcliff queries jboss using the cli and does comparison4. decision point if updated needs to happen5. jcliff has configurations files to match differences it finds and then configurations to create cli commands to rectify those differences using input from the found difference, then executes the cli commands to update the configuration. FYI they do have ordering of commands if anyone asks6. Puppet queries the state of jboss via cli to determine if it needs to be reloaded7. decision if it needs to notify anyone8. Notify via nagios (Tim check on this)
JBoss In The Trenches
Run Puppet
Artifact Workflow
Builder packages JARs in RPMs
UploadTo Pulp
TIM -
CLICK THROUGH - 3x
Pulp– Repository manager
JBoss In The Trenches
JBoss Configuration Management
The management of platform configurations are as important as the configurations themselves
Configurations should be stored and versioned in a source control management system such as Git
Similar to applications, configurations should be deployed to each environment in an automated fashion
Externalize environment specific configuration values
ANDY - Configuration management is just important as the management of your applications.Jboss EAP 6 has consolidated the configurations from previous versions The key is to ensure each configuration is properly versioned to allow changes to be efficiently tracked and to allow for a rollback of previous configurations. The application of these configurations to the various servers, like the deployment archives themselves, should be handled in an automated fashion.
Externalize as many environment specific configuration variables into system properties or stored in properties files which can be loaded by the JBoss server at startup. These would also include sensitive resources such as passwords to provide another layer of abstraction (datasources, login modules). This provides a standard configuration file across multiple environments
JBoss In The Trenches
JBoss Configuration Management (cont.)
Red Hat IT stores puppet modules in Git. Each repository has branches for each server environment
Git hooks are in place to ensure configurations have been merged into all lower environments before being promoted
We call this the parenting hook
TIM -
Rollback strategy:
Revert back to the original jboss config (from installation time).
Revert the configurations stored in the puppet modules. Which are in git.
Reapply the previous configuration state via puppet+jcliff.
Up next: General Best practices
BEST PRACTICES
JBoss In The Trenches
Deployment
Deployments should be handled in an automated scripted process
Have an established rollback strategy
Prepare for the worst (pre-deployment):
• Snapshot server configurations and backup file systems
Can be a component of incremental builds in a Continuous Integration Environment
ANDY - Numerous tools can be used to leverage the deployment process. From as simple as bash/batch files to an entire orchestration process such as the aforementioned automation with puppet:* Artifacts should never be manually deployed to servers
* Snapshots can be taken with the CLI tool. * :take-snapshot
* A Continuous Integration tool such as Jenkins can perform a deployment (especially in a lower environment and on feature branches) to shorten the development lifecycle
JBoss In The Trenches
High Availability
Critical component of any infrastructure
Common methods: Clustering and Load Balancing
Enables Rolling Updates during deployments
ANDY - Critical piece in any infrastructure to ensure services are readily available
High availability centers around ensuring services achieve a prearranged level of operational performance and availability. Red Hat IT has several service levels it must meet from service to service.
As part of the migration effort, several JBoss components related to high availability were configured. The biggest change during the migration involved load balancing of the services (SOA) tier.
JBoss features support for both software and hardware based load balancers
JBoss traditionally leverages Apache httpd to serve as a front end load balancer to effectively route web traffic to a cluster of JBoss servers. In versions prior to EAP 6, mod_jk was the default and preferred module. In EAP 6, the mod_clustermodule became the default implementation, as it provides advantages over other connectors. * Dynamic configurations of HTTP proxies which allows EAP to adapt on the fly without additional configurations* Advanced server side load balancing calculations through the use of a separate communication channel (Mod-Cluster Management Protocol [MCMP]) not found in other connectors * Fine grained control of web applications by dynamically enabling and disabling contexts without resulting in 404* AJP protocol is optional. Support for http, https
JBoss In The Trenches
Red Hat's HA Strategy
Pre-Migration(Dense Clustering)
Post-Migration(Sparse Clustering)
Many apps,single cluster
Many clusters,fewer apps on each
TIM - As part of the migration effort, several JBoss components related to high availability were configured. The biggest change during the migration involved load balancing of the services (SOA) tier.
In the dense scenario, there is no need for location-based pool routing. All contexts were served from the same cluster.
However, in the sparse scenario, not all contexts are available on every cluster. Therefore a director is necessary.
This director component lives between the requesting client and each cluster load balancer.
See New Infrastructure slide 7 for a more detailed diagram.
JBoss In The Trenches
JBoss Platform Management
Web Management Console
User friendly web interface for managing basic JBoss configurations and runtime settings
Only exposes a subset of management capability
Command Line Tool (CLI)
Tool for interacting with the JBoss platform providing access to the full set of management operations
ANDY - Before discussing any specific implementations which might configure or manage the JBoss platform, lets discuss the most common methods of interacting with the JBoss platform
* Web Administration Console * The Web Administration Console is a GWT based user interface for interfacing with the JBoss platform. It fe
* CLI Tool* Introduced in EAP 6. Exposes the full set of
functionality of the management API
JBoss In The Trenches
JBoss CLI
Uses a Detyped Management Model to execute operations
GUI component one of the best “Easter Eggs” of JBoss
$ jboss-cli.sh -c --gui
Public CLI API can be used to build robust applications● Can be written using Java, Groovy, Javascript, etc
ANDY - The API is described as "detyped" to reflect the fact that the API only exposes a small number of types besides JDK classes.
Red Hat IT created several application and tools for configuring, managing and monitoring servers. More on that later though.
SEGUE: <andy> CLI gives you great power, however with great power</andy>
CUT TO TIM : <tim> <click to spiderman /> comes great responsibility <click to access mgmt /></tim>
JBoss In The Trenches
TIM -
Comes great responsibility.
JBoss In The Trenches
JBoss Management Access
Limit access as you get closer to production
Migration Effort● Admin console/management interface disabled in production
● Configurations can only be initiated through Puppet/libeap6
JBoss• Limit access by use of Role Based Access Control (RBAC)
TIM -
As soon as code leaves DEV and enters the managed environments (qa, stage, prod) access is restricted to system operators.
Also, the admin console/mgmt interface is disabled.All application server configuration takes place strictly through puppet and jcliff.
Next Andy will tell you some more about RBAC.
JBoss In The Trenches
Role Based Access Control
New feature of JBoss EAP 6.2
Provides separation of duties among users
7 standard roles with varying access
Fine tune access to resources using sensitivity, vault, and application level constraints
ANDY - RBAC Roles --- PLUS YOU CAN DEFINE YOUR OWN* Monitor* Operator* Maintainer* Administrator* SuperUser* Deployer* Auditor
Constraints* Sensitivity (attribute granulatiry) – Defines a set of resources which are considered sensitive. Can be a secret value (password), networking, JVM or System Property* Vault - Reading and writing to the security vault* Application - Set of resources that can be accessed by those in the deployer role. Can provide additional functionality to support the applications they serve
PRODUCTION SUPPORT
ANDY
And now Tim will tell us about day-to-day operations.
JBoss In The Trenches
Day-to-Day Operations
Question: How do you manage the day to day operations of the JBoss platform and the applications deployed?
Answer: Monitor the right resources:● Application● System● Thresholds
TIM - CLICK THROUGH x4
After applications are up and running, you need to make sure that it continues to function
Application – Any application specific resourcesSystem – JVM, CPU, SWAP etcThresholds – Determine the list of thresholds for both your applications, application servers and operation system
JBoss In The Trenches
Production Support
Have a plan in place to support application and overall server infrastructure. We call ours the Pager Playbook
•Description and example of the issue
•Areas to diagnose
•Steps to resolve
•Escalation personnel
TIM - CLICK THROUGH x4
JBoss In The Trenches
Management Tools
jconsole
Splunk
Munin
Naigos
Taboot
Custom groovy scripts for monitoring
JBoss Operations Network
TIM - CLICK THROUGH x7
* jconsole in the <EAP_HOME>/bin features a traditional jconsole but also includes the CLI GUI tool. 2 for the price of 1!* JBoss Operations Network * Monitoring * Deployment * Drift Management
Groovy scripts supply the information needed by nagios to determine health, and the metrics for munin to create charts from
JBoss In The Trenches
Metrics Via Munin
What happened
here?
TIM - CLICK THROUGH x1
Example of a munin chart showing SOAP access times per service (in miliseconds).
Something looks off though
CLICK
JBoss In The Trenches
Splunk – Old/New Cluster Requests ChartsEAP 6.1
EAP 5.1
TIM -
This slide shows two custom splunk queries.
The top chart shows the number of requests to the new EAP6 Services cluster since the cut over.
The bottom chart shows the number of requests to the legacy Services cluster since the cut over.
Unfortunately, it seems that after the cutover the legacy cluster is still receiving requests. But from where? And for which contexts?
JBoss In The Trenches
Stats Via Splunk
93.6% came from monitoring!
TIM -
Still using splunk, we can “drill down” into this data and identify exactly which contexts are receiving the most requests on the deprecated cluster.
We went even further than that still and broke each context request down by requesting client IP address.
JBoss In The Trenches
Patch Management
EAP 6.2 contains new patch management feature
• Application of individual and cumulative patches from Red Hat
• Simplified patching strategy from previous releases
Executing through the CLI tool● Ability patches● View applied patches● Rollback patches
ANDY -
Ensuring the JBoss server is kept up to date is essential and a component of system administration
No longer are you required to manually extract and apply patches to update the JBoss platform * patch apply –patch <path>* patch info * patch history* patch rollback
Because patch mgmt is CLI based it can easily be integrated into automated deployments and maintenance.
ADVANCE TO THE CLOUD
JBoss In The Trenches
Advancing Towards a Cloud Infrastructure
IaaS PaaSIaaS
ANDY -
When thinking about Jboss in a cloud environment, there are several areas and options to consider
JBoss In The Trenches
Advancing Towards a Cloud Infrastructure
●Offers technological and financial flexibility
●Additional considerations must be accounted for:
• Limitations
• Security●Leverage additional tools while
reusing the migration toolset●Endless possibilities!
ANDY -
Red Hat internally has been driving to migrate a number of its services to the cloud platform
Limitations – UDP traditionally not available
In AWS there are Security groups – Additional level of consideration and complexity that must be accounted for
Tools that you currently use to configure and manage JBoss can be reused in the cloud. Migration related tools such as libeap6 and jcliff can still be utilized
WRAP UP
JBoss In The Trenches
Bryan Madaras
A “THANKS!” of considerable
magnitude goes out to Bryan Madaras
for his efforts preparing this session
with us and for his machine-like focus
during IT’s migration.
JBoss In The Trenches
Download this Slide Deck!
http://people.redhat.com/~tbielawa/summit2014/
Most slides have additional notes with links to any referenced material
http://people.redhat.com/~tbielawa/summit2014/
JBoss In The Trenches
http://people.redhat.com/~tbielawa/summit2014/
JBoss In The Trenches
Thank You!
Andrew Block
@sabre1041
http://blog.andyserver.com
Tim Bielawa
@tbielawa
https://blog.lnx.cx
JBoss In The Trenches
Questions?