An Introduction to Palomino

Laine Campbell,CEOJay Edwards, CTOCharlie Killian, VP of Professional ServicesKevin Bowman, VP of Operations

*

Who we are

Palomino staff have been managing large production systems since before the dot com era began.

Technorati, Twitter, Mozilla Firefox, Digg, Travelocity, Friendster, Chegg, Obama for America, Activision and

StumbleUpon are only part of our pedigree. We've scaled companies through years of explosive growth.

*

Our Partners

Palomino builds partnerships that make sense for our clients and staff, allowing us to augment our expertise.

● Amazon Web Services● SkySQL● MariaDB● Basho● Couchbase● 10GenWe also build key relationships with products we believe are useful and appropriate to our clients, such as Scalearc, Continuent and TokuTek.

Palomino never resells, or takes commission on any products they recommend.

*

Professional Services

● Choosing your DBMS● High availability and disaster recovery● Search, caching, queuing and security● BI, ETL and real-time analytics● Large distributed database systems (aka Big Data)● Configuration management via Chef, Puppet and Ansible● Scripting and automation● Cloud Solutions, AWS and Joyent● Operational tools development and implementation● Backup, recovery and data management

*

Operational Support

● Tier 1 Oncall, integration with customer monitoring● Tier 2 Escalation and problem solving.● Team integration, including email, chat room and IM collaboration, as

well as regular stand-ups and project mgmt meetings.● Proactive health checks, recovery tests, capacity and security reviews.● Project level work as needed.● Ownership of release management and configuration management.

○ Query Reviews.○ DDL and DML Reviews○ Safe production change management

● Ongoing improvements of incident and problem management.

*

Remote Engineering Services

● Pre-deployment ● Data modeling● Data migration● Cluster sizing● Load testing and QA● Application profiling● Performance tuning

*

How we can help

● 24x7 operational support - Chegg.com● MySQL revamping and support - Echosign● Operations team needed - SoHalo● Verifying systems will scale under load - Contest Factory● Lousy application performance - Zeitbyte● Tools development - Zendesk and Slideshare● Proactive services - SendGrid ● Release management and tool development - Zendesk● DBMS tuning and performance - Slideshare and Houzz● Search solutions - Zenlok and Discover Books

*

24x7 operational support needed - Chegg.com

● 48 MySQL Nodes in 10 separate clusters● Tier 1 support for web, search, application, middleware and RDBMS

layers● Tier 1 and 2 support for MySQL, Cassandra and MongoDB● 24x7 operational support with 30 minutes SLA● Amazon EC2 tooling and architectural management● Manage backup and recovery, database release, incident and

configuration management processes.● Onsite support during Amazon EC2 region migrations● Advanced DBMS tuning and management● Upgrades of tooling and diagnostics for alerting and trending

*

MySQL revamping and support - Echosign

● Monolithic MySQL with MMM to start.● New version upgrades.● New hardware and capacity improvements.● Functional partitioning and data management.● Migration to Tungsten for HA and DR.● BI/ETL to reduce DB load, support business needs and reduce load

times for key processes.● Aggregation and denormalization for performance improvements.

*

Operations team needed - SoHalo

● Startup with no operations team● Provided DBA and techops resources● Architected environment ● Using Amazon MySQL RDS as datastore● Fulfilled the entire operations function● Running both their US and EU sites

*

Verifying systems will scale under load - Contest Factory

● Configuration review● Hardware and O/S review● Metrics and logs review● AWS architecture review● Implementation of recommended configuration changes

*

Tools development - SlideShare and Zendesk

● Replication aware rolling DDL implementations● Shard migration and monitoring utilities● Chef recipes for configuration management of configurations● MHA and HAProxy for availability● Cacti for MySQL Trending

*

Lousy application performance - Zeitbyte

● Customer blamed the DBMS (MongoDB)● Configuration review● Application profiling (Django)● Identify problem as code, not DBMS● Fix code● Integrate with ongoing monthly support post - tuning

○ New backups○ New HA

*

Proactive services - SendGrid

● System health checks● Backup and recovery● Upgrades to monitoring and trending.● Chef recipes and support.● Query reviews, indexing, benchmarking● Upgrades● Support of Sharding and NoSQL evaluation

○ Sharding for Short URLs.○ HBase for Counters

*

Release management and tool development - Zendesk

● Own the weekly migration process● Identify and manage risk, push changes throughout sharded

infrastructure● Publish views to production● Built shard rebalancing utilities● Manage regular shard rebalancing and capacity planning● Slow data deleter tool

*

DBMS tuning - Slideshare and Houzz

● Improved slideshare site load times by factor of 3● Benchmarked InnoDB/XtraDB conversions for both clients● Tuned InnoDB and XtraDB to manage background IO, concurrency

issues and optimal memory management● Eliminated locking and replication lag for Houzz.com

*

Search solutions - Zenlok and Discover Books

● Installed, configured and maintain Solr and Sphinx clusters● Gathered search requirements● Created and optimized index and delta queries● Created custom ranking formulas● Improved morphology and relevance● Reviewed and improve indexing speed by over 10x● Reduced query response over 4x by splitting and distributing indexes

across the cluster● Configure HA and failover

*

Engagement process

● Palomino is a boutique offering bespoke solutions for our customers. How we interact is thus hard to predict.

● We make ourselves available via email, IM, phone and chat.● Your primary DBA can spend their working hours available in your chat

rooms.● Daily standups for busy times, and weekly project mgmt meetings are

all part of the service.● How else can we help you?

*

Who we are working with

*

Zendesk & Palomino

“Palomino got up to speed very quickly. Palomino works successfully on complex [data migration] issues that

have stumped other DBAs.”

*

SoHalo & Palomino

“Palomino has been helping us build our Development and Production systems from the beginning. Early on, they built the dev

servers, set up our release management system, migrated us to MySQL and Couchbase, and architected our Amazon Web

Services cluster. Now, day to day, they help us run both our US and EU sites, and continue to support us in projects, such as a recent

German/EU Data Privacy compliance effort. We couldn't have done it all without Palomino.”

*

Slideshare & Palomino

“Palomino was instrumental in helping SlideShare develop an architecture that supported arbitrary schema

updates without downtime.”

*

Rent the Runway & Palomino

“Palomino has shown persistence, talent and a great breadth of experience as they continue to bring us to

new levels of stability and performance.”

*

Technorati & Palomino

“Palomino helped us to rethink our database architecture and to automate processes that were

formally manual, if done it all. Their deep experience in designing large database systems for scale and stability

and their ability to work side-by-side with operations staff and technology management, made them a huge

asset.”

*

Chegg & Palomino

“Palomino are hands-down the best DBAs I have worked with. Their knowledge, experience, and tenacity

make a huge difference. I would highly recommend them.”

*

WorldReader & Palomino

Palomino’s partnership with WorldReader has allowed us to focus on our core efforts. Finding talented, reliable

partners is crucial for a non-profit, and we are quite grateful for Palomino.

*

Palomino Leadership

Laine CampbellChief Executive

Officer

Charlie KillianVP, Professional

Services

Jay EdwardsChief Technical

Officer

*

Palomino Leadership

Kevin BowmanVP of Operations

Tim EllisPrincipal Big Data

Architect

Kaichi SungAccount and Project Management Lead

*

Laine CampbellOwner/CEO

Laine is the Owner/CEO and Principal Consultant at Palomino. She is a proven database expert with a history of scaling high-performance and high-availability web applications.

Laine leverages extensive operational experience and best practices with extensive technical knowledge, troubleshooting skills and a passion for results to ease you through the life-cycles of your company.

*

Charlie KillianVP of Professional Services

Charlie has created the back-bone technology for multiple million dollar companies. An expert in web applications, Charlie has built web application frameworks, setup and integrated numerous disparate systems, built long term client relationships, and contributed to open-source projects.

Key to Charlie's success has been a love of cyberspace, hardware, software, an unwavering patience and an unstoppable urge to figure out how things work.

*

Jay EdwardsChief Technology Officer

Jay was the first dedicated DBA at Twitter. He later served as the lead database engineer at Obama for America during the 2012 election. In his role as CTO for Palomino, he focuses on technology strategy, evangelism and mentoring/training staff and customers.

*

Kevin BowmanVP of Operations

Kevin began his career at Garmin offering desktop and network/server support and was soon serving as IT Manager, supervising areas including service desk, Internet application development, systems administration, network engineering, DBA, architecture and devops. He also worked on delivering internet-enabled products. He contributed significantly on application and system architecture and devops for the backend systems that provide data services to products as well as for Garmin Connect.

*

Kaichi SungAccount Manager and Scrum Master

Kaichi occupies the dual role of Account Manager and Scrum Master at Palomino. During her time in Biotech, she focused on business analysis and project mgmt, supporting process and product improvements. She rolled out Agile Scrum and is certified as both a Scrum Master and Product Owner. A multi tasker and planner, she loves coordinating projects and helping teams deliver on customer requirements.

*

Tim EllisPrincipal Big Data Architect

Tim has been working with database clusters that serve billions of database operations daily since the year 2000 and has been involved in the NoSQL community from the earliest stages. He has a broad background in relational database systems. He was a core operations component to the Digg NoSQL rollout, and an active, participating operations member of the NoSQL community at StumbleUpon, Mozilla, and Riot Games.

*

Contact Info

Laine Campbell, laine@Palomino.comCharlie Killian, charlie@Palomino.com

Jay Edwards, jedwards@palominodb.comKevin Bowman, kbowman@palominodb.com

Kaichi Sung, kaichi@Palomino.comTim Ellis, time@Palomino.com

www.Palominodb.com@Palominodb on Twitter