#ABBYYSummit15 - Training (1/6): FlexiCapture Best Practices of System Configuration: Scaling,...
Transcript of #ABBYYSummit15 - Training (1/6): FlexiCapture Best Practices of System Configuration: Scaling,...
TechnologySummit 2015
© Copyright 2015 ABBYY Technology Summit
BEST PRACTICES OF SYSTEM CONFIGURATION. SCALING, REDUNDANCY
Andrew Zyuzin, PM of DataCapture Dev Team
#ABBYYSummit15
TechnologySummit 2015
How to setup a production system for desired performance?
● Performance
#ABBYYSummit15 2
● Scalability & Redundancy
● Bottlenecks
TechnologySummit 2015Choose a hardware
#ABBYYSummit15 3
BWpages per
day
GRAYpages per day
COLORpages per day
Verifiers
5K 1K 1K 3
1M 300К 200К 100
600K 300
Unlimited Unlimited Unlimited Unlimited
10Gb/s
ABBYY FlexiCapture 12
TechnologySummit 2015
Environment
ABBYY FlexiCaptureServers
ABBYY FlexiCaptureStations
TechnologySummit 2015Calculate amount of cores
#ABBYYSummit15 5
10Gb/s0
100
200
300
400
500
600
700
800
900
1000
8 16 24 32 40 48 56 64 72 80
Tho
usn
ad o
f p
age
s p
er
24
ho
urs
Processing CPU cores
bw
TechnologySummit 2015ABBYY QA Farm Performance
#ABBYYSummit15 6
10Gb/s0
100
200
300
400
500
600
700
800
900
1000
8 16 24 32 40 48 56 64 72 80
Tho
usn
ad o
f p
age
s p
er
24
ho
urs
Processing CPU cores
bw
gray
color
TechnologySummit 2015Approach
7#ABBYYSummit15
Test
Analyze
Design
Deliver
TechnologySummit 2015Explore the Business Process
How many documents? Seasonality? How many pages in documents?
#ABBYYSummit15 8
mount
ime
ource
ntegrity
erification
What is SLA about time to process,
time to store?
What is SLA about quality?
What are the sources of images?
What is the smallest
integral part to process?
TechnologySummit 2015Parameters of the System
#ABBYYSummit15 9
Metric UoM
Average, PeakPerformance
pages per 24 hours,pages per hour
Input flow MB/s
File Storage size GB
Amount of verifiers guys
Metric UoM
Document size pages
Batch size pages
Optimal
up to 100
10 - 1000
Limitations
1M pages / day,50K pages / hour
1 Gb/s [10 Gb/s]
Cost of Hardware
Up to 100 (300)
TechnologySummit 2015Architecture
#ABBYYSummit15 10
• Application Server• Licensing Server
• Processing Servers• Processing stations
• Database Server• FileStorage
• Clients• Network
TechnologySummit 2015Interaction of components
#ABBYYSummit15 11
TechnologySummit 2015Automatic Processing
#ABBYYSummit15 12
TechnologySummit 2015Automatic Processing
#ABBYYSummit15 13
TechnologySummit 2015Automatic Processing
#ABBYYSummit15 14
TechnologySummit 2015Manual Processing
#ABBYYSummit15 15
TechnologySummit 2015Export Results
#ABBYYSummit15 16
TechnologySummit 2015
Network utilization
Application Server (Web Servie)
Image EnhancementInput Flow * upload times, Input Flow * download timesMAX ( )
vs. Network Capacity
TechnologySummit 2015
File Storage
Application Server (Web Servie)
Input Flow * upload times
+Input Flow * download times
vs.
IOPS * bytes per operation
RedundancyRAID 10 from SATA2,
SAN, NAS
Volume
Input Flow * Time to Store
TechnologySummit 2015
Application Server
Application Server (Web Servie)
Clients
Clients
Processing Stations
Critical resources:Network (2+ NIC)Connection to FileStorageProcessorRAM
TechnologySummit 2015
Application Server
Application Server (Web Servie)
Clients
Clients
Processing Stations
Critical resources:Network (2+ NIC)Connection to FileStorageProcessorRAM
Network Load Balancing Cluster
TechnologySummit 2015
Application Server
Application Server (Web Servie)
Clients
Clients
Processing Stations
Critical resources:Network (2+ NIC)Connection to FileStorageProcessorRAM
Network Load Balancing Cluster
TechnologySummit 2015
Database ServerCritical resources:
RAMFast Physical DiskProcessorNICs
Fault Tolerance & High Availability
RecommendationsDisable MirroringUse Simple Recovery ModelStore database log at separate physical driveRebuild indexes
TechnologySummit 2015
Processing, Licensing Servers
Microsoft Failover Cluster
Active Passive
Critical resources:RAM: 4GBProcessor: 2 coresNICs
Highly Availability
TechnologySummit 2015
Processing Station
#ABBYYSummit15 24
10Gb/s
Critical resources:Processor: - frequency,- up to 12 coresFast Hard DriveRAM: 2 GB per core
TechnologySummit 2015
Operators
#ABBYYSummit15 25
10Gb/s
● Scanning
● Verifiers
Tho
usn
ad o
f p
age
s p
er
24
h
ou
rsProcessing CPU cores
100 cores
TechnologySummit 2015
Production - Starter
TechnologySummit 2015
Production - Redundant
#ABBYYSummit15 27
10Gb/s
TechnologySummit 2015
ABBYY FlexiCapture 12
#ABBYYSummit15 28
10Gb/s
Load Balancer
TechnologySummit 2015
Performance
#ABBYYSummit15 29
10Gb/s0
100
200
300
400
500
600
700
800
900
1000
8 16 24 32 40 48 56 64 72 80
Tho
usn
ad o
f p
age
s p
er
24
ho
urs
Processing CPU cores
bw
color
color*
TechnologySummit 2015
Help & Support
#ABBYYSummit15 30
10Gb/sAlexander Timoshenko,Head of Large Projects Support Dept.
Guide about High Performance Configurations
TechnologySummit 2015
LOOKING FOR BOTTLENECKSBest practices of system configuration
#ABBYYSummit15
TechnologySummit 2015Performance Monitor
#ABBYYSummit15 32
10Gb/s
● Web Performance Monitor (free)
● System Center Operations Manager
TechnologySummit 2015General System Parameters
#ABBYYSummit15 33
10Gb/s
● Memory● Memory\Available Bytes● Memory\Pages/sec
● Process(<All instances>)\Working Set
● Processor● Processor(_Total)\% Processor● Time System\Processor Queue Length
● Process(<All instances>)\% Processor Time
TechnologySummit 2015General System Parameters
#ABBYYSummit15 34
10Gb/s
● Disk● LogicalDisk(<All instances>)\Free Megabytes
● PhysicalDisk(<All instances>)\% Disk Time ● PhysicalDisk(<All instances>)\Avg. Disk Queue Length
● Network● Network Interface(<All instances>)\Output Queue Length ● Network Interface(<All instances>)\Packets Outbound Discarded● Network Interface(<All instances>)\Bytes Total/sec ● Network Interface(<All instances>)\Current Bandwidth
TechnologySummit 2015Application Server
#ABBYYSummit15 35
10Gb/s
● Latency of App Server● FlexiCapture(All instances)\ASCT
Latency --- 30 sec = Critical Mode
● Activity● W3SVC_W3WP(_Total)\Active
Requests
● W3SVC_W3WP(_Total)\Requests/
● Web Service(Default Web Site)\Current Connections
● Network● Web Service(Default Web Site)\Bytes
Received/sec
● Web Service(Default Web Site)\Bytes Sent/sec
● Threads● W3SVC_W3WP(_Total)\Active Threads
Count
● W3SVC_W3WP(_Total)\Maximum Threads Count
TechnologySummit 2015Processing Server
#ABBYYSummit15 36
10Gb/s
● Latency of Processing Server● FlexiCapture(Processing Server)\Primary Thread Latency
10 sec is critical
● Cores● FlexiCapure(Processing Server)\TotalCores
● FlexiCapture(Processing Server)\Free Cores
TechnologySummit 2015
QUESTIONS
#ABBYYSummit15 37