Solving Enterprise Integration Challenges With Biz Talk Server
Solving enterprise challenges through scale out storage & big compute final
-
Upload
avere-systems -
Category
Technology
-
view
55 -
download
1
Transcript of Solving enterprise challenges through scale out storage & big compute final
![Page 1: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/1.jpg)
WEBINAR
Solving Enterprise Business ChallengesThrough Scale-Out Storage & Big ComputeMichael Basilyan, Product Manager, Google Cloud PlatformScott Jeschonek, Director of Cloud Products, Avere SystemsRob Futrick, CTO, Cycle Computing
![Page 2: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/2.jpg)
Housekeeping
• Slides
• Questions
• Recording
• Attachments
![Page 3: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/3.jpg)
Presenters
Michael BasilyanProduct Manager
Scott JeschonekDirector of Cloud Products
Rob FutrickCTO
![Page 4: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/4.jpg)
Introduction to Google Cloud PlatformFocusing on Compute Engine & Storage
Michael Basilyan [email protected] Manager, GCE
![Page 5: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/5.jpg)
Agenda
• Google Cloud Overview
• Compute Engine VMs:
• GCE VM Instances & Managed Infrastructure
• Storage:
• Block Storage
• Cloud Storage
![Page 6: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/6.jpg)
What is Google Cloud Platform?
![Page 7: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/7.jpg)
7
Google Cloud Platform Services
VIRTUAL NETWORK
LOAD BALANCING
CDN
DNS
INTERCONNECT
Management Compute Storage Networking Data Machine Learning
STACKDRIVER
IDENTITY AND ACCESS
MANAGEMENT
CLOUD ML
SPEECH API
VISION API
TRANSLATE API
NATURAL LANGUAGE API
![Page 8: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/8.jpg)
8
Google Cloud Platform Services
VIRTUAL NETWORK
LOAD BALANCING
CDN
DNS
INTERCONNECT
Management Compute Storage Networking Data Machine Learning
STACKDRIVER
IDENTITY AND ACCESS
MANAGEMENT
CLOUD ML
SPEECH API
VISION API
TRANSLATE API
NATURAL LANGUAGE API
![Page 9: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/9.jpg)
GCE: Compute & VM Features
![Page 10: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/10.jpg)
VM Live Migration = No Downtime
![Page 11: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/11.jpg)
Custom Machine TypesAverage Savings: 19%
Create VMs shaped for your workloads instead of shaping your workloads to fit pre-defined VMs.
![Page 12: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/12.jpg)
Preemptible VMsIdeal for batch, grid, and fault-tolerant workloads
Save 80% off regular VM list prices: flat $0.01 per core hourFlat pricing with no complex bidding or competitionSame performance (CPU, I/O, Net) as regular VMs
Example uses: Hadoop, Rendering/Transcoding, Genomics, Monte Carlo Simulations, etc.
![Page 13: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/13.jpg)
Managed Infrastructure - zero devops for IaaSCreate Groups of Instances- Define Instance Template- Deploy Docker containers or
apps directly- Automatically connect new
instances to load balancer
Autoheal- Use app level healthcheck to
signal issue- Get machine recreated or
restarted
Autoscale- Add/Remove instances automatically
based on scaling policy (CPU utilization, LB load, Custom Metrics)
- Scale pool of workers with task queue
Update- Deploy new version of your software
with rolling update while serving traffic
- Do cannary, % rollout, control pace, roll-back
- Recreate in place or surge instances
![Page 14: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/14.jpg)
Ways we save you money● Preemptible VMs
● Custom Machine Types
● Per-minute billing
● Sustained Use Discount○ The more you use, the bigger
the discount. Automatically.
● Instance right-sizing ○ Instance recommendations
displayed on VM Instances Page
○ Single Button Actuation
![Page 15: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/15.jpg)
Block & Object Storage
![Page 16: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/16.jpg)
Cloud Storage
Cloud Bigtable
Cloud Datastore
Cloud SQL
Good for:Binary or object data (BLOB)
Such as:Media, analytics, archive/backup
Good for:Hierarchical,mobile, web
Such as:User profiles,Game State
Good for:Web frameworks
Such as:CMS, eCommerce
Good for:Heavy read + write, events,
Such as:AdTech, Financial, IoT
Where do I store my data?
Big Query
Good for:Data Warehouse
Such as:Analytics, Dashboards
Relational NoSQL Object Warehouse
Good for:Local VM file storage
Such as:Application data/binaries
Block
Persistent Disk (GCE)
![Page 17: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/17.jpg)
Cloud Storage
Cloud Bigtable
Cloud Datastore
Cloud SQL
Good for:Binary or object data (BLOB)
Such as:Media, analytics, archive/backup
Good for:Hierarchical,mobile, web
Such as:User profiles,Game State
Good for:Web frameworks
Such as:CMS, eCommerce
Good for:Heavy read + write, events,
Such as:AdTech, Financial, IoT
Where do I store my data?
Big Query
Good for:Data Warehouse
Such as:Analytics, Dashboards
Relational NoSQL Object Warehouse
Good for:Local VM file storage
Such as:Application data/binaries
Block
Persistent Disk (GCE)
![Page 18: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/18.jpg)
Block StorageReliable, high-performance block storage for virtual machine instances on GCE
Standard Persistent Disk SSD Persistent Disk Local SSD
Targetscenarios
Large data processing workloads and some enterprise applications
Genomics processing, video transcoding in GCE
High performance database and enterprise applications
MySQL, SQL Server, Oracle
In-memory databases
High-performance scratch space
Features
Persistent storage
Cost sensitive ($.04 GB)
Persistent storage
Performance sensitive ($0.17GB)
Ephemeral storage
Highest-performance ($0.218 GB)
Encryption, Snapshots64 TB, Disk Size sets performance
(Attach larger VMS for max SSD performance)
Encryption3TB
![Page 19: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/19.jpg)
Cloud Storage: Object/Blog store● Google Cloud Storage is a scalable
object storage service suitable for all kinds of unstructured data.
● Cloud Storage vs Perst. Disk:○ Scales to exabytes.○ Accessible from anywhere. ○ REST interface; higher latency
than locally attached block storage (PD)
○ Write semantics include insert and overwrite file only.
○ Offers versioning. ○ Cheaper!
● Lots of guidelines on picking storage on our site.
![Page 20: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/20.jpg)
Regions and Zones
![Page 21: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/21.jpg)
–––– 20182018
Current regions and number of zones
Edge points of presence
Network
Committed regions for 2017 and number of zones
#
# https://peering.google.comhttps://cloud.google.com/compute/docs/regions-zones/regions-zones
Google Cloud Platform InfrastructureGoogle Cloud Platform is built on a datacenter network infrastructure that supports Google scale, performance, and availability
2
3
Singapore2
S Carolina
N Virginia
BelgiumLondon
Tokyo
TaiwanMumbai
Sydney
OregonIowa
Frankfurt
São Paulo
Finland
3
3
33
3
3
2
43
3
3
![Page 22: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/22.jpg)
Cloud HPC: Data Access ChallengesScott Jeschonek, Director of Cloud Products
![Page 23: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/23.jpg)
HPC in the Cloud
• Bring 100s or 1000s of cores online, quickly and efficiently
• Networking within the Cloud Compute environment minimizes compute latency
• Creative use of preemptible / spot market VM instances allow large numbers of worker nodes at reasonable cost
![Page 24: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/24.jpg)
“Pure” Cloud HPC
• Entire grid in Compute Cloud
• Data is located locally
•Cloud Storage options may be used
• 3rd party Data may be incorporated (from their cloud storage)
![Page 25: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/25.jpg)
Hybrid HPC
Existing HPC clusters:
Capital investment- Possibly sunk cost already
Logical investment:- Hardware Tuned- Storage optimized- Network optimized- Daily ops dependent on status
quo
Cloud HPC Clusters:
Transient investment:- Can build on demand
infrastructure
Expand on-prem:- Use orchestration and grid
management to extend jobs into cloud
- Schedule jobs based on performance / cost requirements
![Page 26: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/26.jpg)
Hybrid HPC
![Page 27: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/27.jpg)
Grids On-Demand
![Page 28: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/28.jpg)
Latency “Kills”• Access to Data is the main challenge for HPC
• Amplified in the cloud:
- Data has to be located on or near the worker nodes
- Data may be in your datacenter
- Copy it all to the cloud?
- Costs for workers grows if data has to be copied to local disks
- Pipelines may require multiple writes (of results)
- Writes to local storage increases consistency risks
- Writes back to on-prem storage introduces significant latency
![Page 29: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/29.jpg)
Using a Data Access Layer
![Page 30: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/30.jpg)
Advantages of Data Access Layer
Keep your data on prem! – Data in cloud is only there while the compute nodes work the jobs.
- Reduce the security objections, simplify the move to cloud
Increase cloud compute performance – using file system caching, most of the data will be in RAM, close to the nodes
- Avoids ingest latencies and slashes transit latency after first read
Scale out – Using solution that facilitates 10s of 1000s of core file system connections
![Page 31: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/31.jpg)
Hybrid Cloud / Hybrid HPC Using Avere Technology
Customer Needs Avere Delivers
Low-latency file access Edge-Core Architecture
Scalable Performance and Availability
Scale-out Clustering
NFS & SMB interfaces FlashCloud File System for Object
Single pool of storage Global Namespace
High Security AES-256 Encryption, KMIP
Flexibility Physical and virtual products
![Page 32: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/32.jpg)
Lessons Learned from 10 YearsHow Cloud Changes Big ComputeRob Futrick, CTO
![Page 33: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/33.jpg)
33
The BroadInstitute
Need: 270,000 hours of computing
Why: Machine learning to map relationships among cancer datasets
![Page 34: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/34.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 34
Internal cluster queue too long
Up & running in 1 hour, scaled and completed project in 2 weeks
30 years of Computing in 6 hours!
Submit jobs, orchestrate ML application
Encrypt, route data to Cloud, return results
51,200 cores To run R ML framework
Secure Cluster
Cell Line Data, RNA,
DNA
Scaling Machine Learning @ The Broad Institute
![Page 35: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/35.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 35
Manufacturing& Electronics
Pharma &Biotech
Financial & Insurance
Media &Entertainment
Oil & Gas
65% of G2000 are limited by access to Big Compute
![Page 36: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/36.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 36
The Challenges: Cloud & HPC Big Compute
User Inputs
Existing Workflows
Data Dependencies Instance types
Applications
Scalability
Budget Controls
AuthorizationSecurity Stack
IT LOB Inputs
Job scripts & data
Cloud accounts
Storage / Data sources
OS variations
AD / LDAP Authorization
![Page 37: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/37.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 37
The Solution: CycleCloud for Cloud HPC & Big Compute
User Inputs
Existing Workflows
Data Dependencies Instance types
Applications
Scalability
Budget Controls
AuthorizationSecurity Stack
IT LOB Inputs
Job scripts & data
Cloud accounts
Storage / Data sources
OS variations
AD / LDAPAudit/Compliance data
Usage data (User, Group, App)
Job run-time by instance data
AppServer platform
Internal
![Page 38: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/38.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 38
Who is Cycle Computing?
• Leader in Cloud Big Compute/HPC • Pioneering Cloud Management Software for 10 years• 370M compute-hours managed• Compute hour growth: 7x every 2 years
• CycleCloud Value Proposition• Simple Managed Access to Big Compute• Accelerating Innovation for the Enterprise
=> Faster time to result, with cost control
• Our customers• Fortune 500, startups, and public sector• Life sciences & pharma, financial services,
manufacturing, insurance, electronics
![Page 39: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/39.jpg)
© 2016 Copyright | All rights reserved
7 Lessons Learned
![Page 40: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/40.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 40
#1 – Zero waiting in line for compute
![Page 41: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/41.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 41
#2 – Ask questions of any scale
Ask the right question,
regardless of scale
Think about the problem first
Then the system
![Page 42: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/42.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 42
#3 – Users with unique requirements are OK
Trivial to support different use cases
Different GPU, RAM, SSD, OS needs can be created easily
Move workloads that don’t fit internally to Cloud
![Page 43: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/43.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 43
#4 – Cloud gets faster/cheaper over time
![Page 44: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/44.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 44
#5 – Time & cost are the sole metrics that matter
![Page 45: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/45.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 45
Everythingyou don’t think about!
![Page 46: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/46.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 46
#6 – Accelerating answers, accelerates people
720 (hours) 720 720
Computing Analysis
2880 hours /120 Days to Decision
Computing720
AnalysisSCALABLE COMPUTING (in hours)
720
Computing Analysis Analysis
1456 hours /60.6 Days to Decision
7208
Computing
ANTICIPATED BENEFIT (in hours)
8
![Page 47: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/47.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 47
#6 – Accelerating answers, accelerates people
720 (hours) 720 720
Computing Analysis
2880 hours /120 Days to Decision
Computing720
AnalysisSCALABLE COMPUTING (in hours)
Higher Quality Output, Iterative Analysis, Less Context Switching
Computing & AnalysisPOST ADOPTION: AGILE DESIGN PROCESS
8
![Page 48: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/48.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 48
#7 – Every smart person gets their own workspace
Old: Shared internal cluster
• Competition for resources• Waiting in line for compute• Zero sum game between users
New: Cluster Per Researcher
• Remove bottlenecks• Cost controls to manage $• No waiting = 2x faster users
User
User User UserUser User UserUserUser
User
User User
![Page 49: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/49.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 49
Lessons Learned Summary
1. Zero Queue Wait for computing
2. Any scale, Any time
3. Users with unique requirements are ok
4. Performance goes up over time, same cost
5. Time and Cost are the sole metrics
6. Faster iterations
7. Every researcher gets their own workspace
49
![Page 50: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/50.jpg)
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 50
The Solution: CycleCloud for Cloud HPC & Big Compute
User Inputs
Existing Workflows
Data Dependencies Instance types
Applications
Scalability
Budget Controls
AuthorizationSecurity Stack
IT LOB Inputs
Job scripts & data
Cloud accounts
Storage / Data sources
OS variations
AD / LDAPAudit/Compliance data
Usage data (User, Group, App)
Job run-time by instance data
AppServer platform
Internal
![Page 51: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/51.jpg)
Questions& Answers
![Page 52: Solving enterprise challenges through scale out storage & big compute final](https://reader031.fdocuments.in/reader031/viewer/2022030121/58a211f51a28ab40098b6ec1/html5/thumbnails/52.jpg)
Contact Information
Michael BasilyanProduct Manager
Scott JeschonekDirector of Cloud [email protected]
AvereSystems.com
Rob FutrickCTO