Post on 19-Nov-2014
description
Managing Large Data VolumesManaging Large Data Volumes
Suchin Rengan, Director, Salesforce Services
@Sacrengan
Mahanthi Gangadhar, Senior Solutions Technical Architect, Salesforce Services
Safe harborSafe harbor statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services. The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site. Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
We do a lot of things with data…
Data Creation (Loads/ Manual)
Data Searches/ Reporting/ List
Views
Data ExtractsData Archival
Data Integration (Out and In)
Am I using the platform’s features optimally?
How do we ensure we keep up with performance?
What factors do we need to consider across each topic?
How can I ensure I have a scalable process?
SFDC Cloud Computing for the Enterprise
Infrastructure Application Operations Innovation Services Services Services Development
Network Storage Security Sharing Authentication Data ModelOperation System Integration Availability Business LogicDatabase Customization Monitoring User InterfaceApp Server Web Services Patch MgmtWeb Server API UpgradesData Center Multi-Language Backup / NOC
Customer Avoids: Tuning OS, Capacity Management Tuning Web Servers, Certificates, Log file Mgmt, etc Tuning App Servers, threads, Java stack, memory/log mgmt Tuning DB Servers, memory mgmt, disk distribution Network management, bandwidth
Lets understand the underlying Platform
UserAccount
Custom Object
…
Your System
APIApex
Custom Object
Data LoadsData Loads
Suchin Rengan
Mahanthi Gangadhar
Suchin Rengan
Mahanthi Gangadhar
Considerations at every layer and level
File Storage
Sandbox
Apex Triggers
Data Storage
Logic / Application Layer
Storage Layer
Data ObjectsSharing Tables
IndexesSkinny Tables
Validation Rules
Workflow Rules
API
VF
Apex
SOAP/REST API
API
SOAP API
Bulk API
Real TimeRelatively slow for large volumesLoads for up to 250K records
Batches per dayParallel modeLarger than 250K records
Rolling 24 clock for available batchesTime to download results
Batch sideTime out / failure
Bulk API – Asynchronous Process
Data streamed to temporary storageData streamed to
temporary storage
ClientClientProcessing
ThreadProcessing
Thread
Processing Thread
Processing Thread
Processing Servers
Processing Servers
Data Batches
Data Batches
Dequeue batch
Dequeue batch
Insert/updateInsert/update
Save resultsSave
results
Send all data
Send all data
Check StatusCheck Status
Retrieve ResultsRetrieve Results
Job updated in Admin Setup
Job updated in Admin Setup
Dataset processed in parallel
Dataset processed in parallel
ResultsResults
Bulk API
The “go-to” option for tens of thousands of records and up
Up to 10,000 records in a batch file
Asynchronous loading, tracked in Setup’s “Monitor Bulk
Data Load Jobs” section
Walkthrough time!
Example: American Insurance Co. ?230 million records processed in 33 hours,14 hours ahead of schedule
Some tips for loading•Difficult to extrapolate performance in a Sandbox
• At Par or Better in Production
•Sharing Calculations
•Indexing
•File Storage
•Triggers – Act Judiciously
•Upserts – Avoid Them!
•Parent References
Sequence of Events/Logic
System Validation
Before Trigger
Custom Validation
AssignmentRules
Auto-responseRules
Workflow Rules
Escalation Rules
Parent RollupSummary
After Trigger
BeginTransaction
End Transaction
Database Layer
APEX Layer
Initial Load – Incremental or Big Bang Obj 1 Obj2 Obj3
Option 1
Option 2
Legend
Pre-Implementation Activities
Initial Load ValidationCatch up and Ongoing Sync
User Activation
All Objects
Audience Question!
Scenario: Data Load on …
Click icon to add pictureData ExtractionData Extraction
Mahanthi Gangadhar
@twittername
Mahanthi Gangadhar
@twittername
Data Extraction – Bulk Query Current LimitationsBulk query works exactly like data loads
Create a job (Job Id)
Each query is a batch (Batch Id)
Close the Job and fetch results when job is complete
Limitations Query optimizer has 100mins processing time (timeout issue)
Informatica currently does not support Bulk Query
Other tools like data loader can submit only one query
Currently requires a custom client for submitting multiple queries for a
given job
Data Extraction - ChunkingAuto Number Chunking
Query smaller chunks of the entire data
Use Auto number and formula field for internal indexing
Find chunk boundaries (25K) and issue queries for each chunk
PK Chunking Use primary key to chunk (ID)
Usually better performance when entire object is extracted
Find chunk boundaries (250K) and issue queries for each chunk
Auto Chunking (“safe harbor”) Provide single query and everything happens magically!
Data Extraction - Chunking
Job Id (1234)
Q1 -> Batch Id (123)
Q2 -> Batch Id (234)
…..
Qn -> Batch Id (789)
Close Job Id (1234)
Find Chunk Boundaries -> Create Job ->
Submit each batch/query -> Get Results -> Close Job
Q1
Q2
Q3
Q4
Q5
Qn
1
150 M
Data Cleanup - General Considerations
Deletion is one of the most resource intensive data operations in Salesforce
and can perform even slower than data load in some cases (objects with
complex sharing, with Master/Detail relationships, with Rollup Summary fields
etc.) even in bulk hard delete mode.
Custom objects can be cleaned up by truncation. Note that truncation cannot be
performed on OOB standard objects.
Data in standard objects can be deleted only using Delete API, faster
performing Bulk Hard Delete option is available
Records from User object cannot be deleted, only deactivated.
Data Cleanup - Truncation and Hard DeleteWhere Truncation is not Possible?
Are referenced by another non empty object through a lookup field
Are referenced in an analytic snapshot
Contains a geo location custom field
Have a custom index or an external ID field
Hard Delete
Hard Delete option is disabled by default and must be enabled by an
administrator
Observed about 4.5M/hr on Task object delete, versus 18M/hr for load time,
indicative of how slow of a process it is…
Data Cleanup - General Recommendations
General Recommendations When removing large amounts of data from a sandbox org consider Refresh option first. This is
the fastest way to clean up an org. Additional benefit of Refresh option is that data in User
object is removed also.
To remove data from a custom object use truncate function. Truncation deletes an object
entirely and places it into the recycle bin. Use Erase function to physically remove data from
the system it frees up storage space. Note it might take several hours to erase data from a
large custom object.
To remove data from standard object use bulk hard delete option. Note that performance of
bulk hard delete is quite low so plan sufficient time to remove large amounts of data using this
option. Recent test on Task object in Shadow production org demonstrated performance of
~2M records deleted per hour. On Account object the rate might be even lower.
When planning large scale tests in production environment consider possibility of getting 2 orgs
that can be used in “round robin” mode: when a test is performed in one org, cleanup is being
performed in another.
Tips Test early and often and as big data sets as possible
Split initial load set to smaller subsets (we used 10MM records). This allows for greater flexibility in load
monitoring and control
Queries: for aggregate data validation queries (since bulk query option is not available for them) consider use of
Work Bench with special asynchronous background processing configuration that prevents early timeout on client
side (more info: http://wiki.developerforce.com/page/Workbench). Publicly available WB app
https://workbench.developerforce.com/login.php can be utilize
ETL Tool Timeout settings
Bulk query support
Handling of Success and Error files
Monitoring of Job Status
General Guidelines – Tips / Partner Tools
Audience Question!
Scenario: Data Load on …
Click icon to add pictureOther Areas Other Areas
Suchin Rengan
@twittername
Suchin Rengan
@twittername
Searching and Reporting and List Views• Indexes
• Filtered Queries
• Skinny Tables
• Data Scope
• Roles and Visibility
• Report Folders
Data Governance• Data Model
• Data Management and Administration
• Security and Access Controls
Data Archival and Retention• Storage
General Guidelines for other areas
Speaker NameSpeaker Name
Speaker Title,@twittername
Speaker NameSpeaker Name
Speaker Title,@twittername
Speaker NameSpeaker Name Speaker NameSpeaker Name
Speaker Title,@twittername
Speaker Title,@twittername
Initial Load - Parent References (* goes into notes)General LDV recommendations XLDV considerations
Avoid reference to parent record via External Id if possible, use SFDC Id instead.
Particularly important on large objects with multiple parent references
Consider preparing source data sets with native SFDC id for parent reference. This might be achieved by querying key mapping data from parent objects and performing joins on client side to retrieve SFDC Id for parent reference vs. using External Id reference. Additional benefit – this is a good data validation step.
Note that querying data from extra large objects might be a challenge by itself and often takes hours. Consider alternative approaches, for example collecting mapping keys from initial upload log files.
Referencing parent via native SFDC GUID will also allow the use of FAST LOAD option when available (next release, “safe harbor”)
Initial Load - Data ValidationGeneral LDV recommendations XLDV considerations
The following post load data validation checks are generally can be considered:
Attribute to attribute comparison
Total counts comparison
Aggregate values/subtotals comparison (numeric and currency values)
Spot checking
Negative validations
Data validation on XLDV is a challenging task. Queries on tables with hundreds of millions of records are slow and often time out (even bulk queries). Consider options for splitting (chunking ) your data validation queries by filtering on indexed attributes.
Special chunking techniques: auto number chunking, PK chunking
Do not underestimate and plan for enough time for performing post load data validation on XLDV.
Incremental synch’sGeneral LDV recommendations XLDV considerations
For SOAP API’s use the largest batch size (200).
During incremental syncs on objects with large amount of processing the biggest batches can fail due to time out (10 min per batch). In this case batch size might need to be reduced.
It is possible to “batch” standard SOAP API calls by submitting several jobs in parallel. When loading data into the same object using several SOAP API batches in parallel consider to group child records by the parent in the same batch to minimize DB locking conflicts.
For incremental synchs of larger data sets (over 50K-100K) use bulk API’s.
Consider configuring client application to programmatically set load mode (SOAP vs. Bulk) based on the size of incremental data set in each load session.
Data Loads - Current Volumes - REMOVE
Object approx. 1A # of rows approx. 1B # of rows User Role 60,000 User 200,000 Account (+Contact) 186,000,000 Agent Role 155,000,000 Relationship 11,000,000 Household 96,000,000 Household Member 155,000,000 Opportunity 100,000,000 Task 104,500,000 120,000,000Remark 16,500,000 18,000,000Note 15,000,000 Attachment 6,000,000 Life Event 900,000External Agreement 33,540,000Agent Marketing Inf. 26,000,000Policy 149,000,000Policy Role 165,000,000Agent Agreement Role 172,000,000Billing Account 23,220,000Billing Account Role 23,220,000Total 845,413,000 730,880,000
• Initial volume close to 3.5 B• Worked with customer to reduce volumes• Split the implementation into two phases
Data Loads - Org/Environment preparation and tuning
For XLDV it is recommended to perform test loads in Production like environments To get a true representation of the performance in Production
To test loads in real production environment for environment specific settings
Allows for more accurate planning of data load in terms of timing and dependencies
Concerns:
Large scale deletion / environment cleanup after tests. Deletion is an resource intensive data
operation and the slowest (even using bulk hard delete)
Cannot predict in advance all possible issues and consequences related to multiple mass
data deletions in actual production environment
Users cannot be deleted from Production environment, only deactivated. Multiple User load
tests can produce a “garbage pile” of inactive User records that would clutter Production
environment.
General best practice recommendation: Request a dedicated production test environment to avoid testing in actual Production org.
Work with SFDC Operation on plans for re-provisioning test production environment because
erasing XLDV data from an org for subsequent test might not be feasible
Coordinate with SFDC Operations on any large scale test activities in Prod environment
(systems can be shut down by SFDC Operations as a suspected DDOS attack)
Plan accordingly and factor in the time required for setting up and configuring test production
environments when multiple production tests rounds are required (production org cannot be
refreshed as sandbox org so it needs to be re-provisioned and configured from scratch)
Data Loads - Org/Environment preparation and tuning (Contd..)
Data Loads – Org/Environment preparation and tuningGeneral LDV recommendations XLDV considerations
For large volume data loads it is possible to request additional temporary special changes on SFDC side to expedite load performance
Request increase of batch API limit
Request Increase of file storage
Notify SFDC Operations about upcoming loads with approximate data volumes
Request turning off Oracle Flashback
Use FASTLOAD option (future option, “safe harbor”)
Note that some settings/perms can technically be tweaked only in Production environments, not in Sandbox environments
Initial Load - Use Bulk APIs for massive Data LoadsGeneral LDV recommendations XLDV considerations
SFDC Bulk API are specifically designed to load large volumes of data. It is recommended to use bulk API for LDV initial data load.
Use the Salesforce Bulk API in parallel mode when loading more that few hundred thousand records. Main performance gain of bulk API’s is executing batches in parallel.
Note that generally bulk API (w/o batching, in sequential mode) might perform slower than SOAP standard API so if number of records is less than certain threshold (that depends on number of factors: object being loaded, processing on SFDC side, number of attributes loaded etc…) using bulk API’s might become counterproductive.
Use bulk API’s in parallel mode with maximum allowed batch size whenever possible to maximize number of records that can be loaded within 24 hour period (based on bulk API batch limit – standard limit 2K batches per 24 hours)
Use standard SOAP APIs for incremental ongoing data synchs (data sets less than 50-100K) and bulk APIs for larger incremental data sets (over 50K-100K rows)
Using standard SOAP API’s for incremental synchs has a benefit of reducing the risk of database contention during uploads of child objects
Allow enough time for collecting Request and Result log files (takes ~15 min for INFA to extract load logs for 10MM rows loaded)
Initial Load - Planning Bulk Jobs (Parallel vs Single Object)General LDV recommendations XLDV considerations
When planning bulk jobs consider the following:
When loading large volume of data into a single object it is recommended to submit all the batches in one job. Splitting (partitioning) data across several parallel jobs does not improve performance when each individual job is big enough. Note that batches of all jobs are placed in the same batch pool and queued for processing independently of jobs they belong to.
When loading data into several objects running several jobs for different objects in parallel is recommended when jobs are small enough to not consume all the available system resources.
For loading extra large volumes of data into single object in Salesforce it is recommended to run one object load at a time. The rationale behind this recommendation is that bulk jobs with hundreds/thousands of batches consume all available system resources and running other jobs in parallel just causes jobs to compete for the limited resources. The other consideration is DB cache. When several objects are loaded in parallel sharing DB cache is causing more frequent swapping that slows down overall performance on the DB layer.
Initial Load - Defer Sharing Calculations
General LDV recommendations XLDV considerations Reduce sharing recalculations processing during initial
data upload.
Disable sharing rules and/or use the defer sharing calculation perm for custom objects while loading data.
Note that sharing calculations will still need to be performed after data load is complete and it can take significant amount of time but it can reduce loading time by allowing sharing to be calculated in “off” hours.
Sharing tables can alternatively be uploaded directly that allows to avoid sharing recalculation on initial load altogether
Based on the XLDV performance tests general recommendation for deferring sharing calculations might not be applicable for extra large volumes. Post loading sharing calculations might take very long time to execute on bigger volumes (it is not parallel currently, on roadmap for the nearest release)
Load data in the following sequence to enforce main sharing calculations to be performed during data upload: Set OWD on objects Private where applicable
Create User Role Hierarchy and upload users with correct roles assigned
Create main user groups and group members
Create sharing rules
Upload data with correct owner specified.
Consider uploading sharing tables on some objects directly to avoid sharing calculations during upload and post load processing.
Initial Load - Triggers, WF Rules, Data Validation Rules
General LDV recommendations XLDV considerations
Disable when possible triggers, WF rules, data validations on objects being uploaded.
To avoid lengthy post load catch up processing consider performing data transformations/ validations on source data set prior to data upload or
consider batch APEX post-load processing for data transformations that can’t be performed prior to upload.
For each trigger, WF rule and validation rude individual analysis should be performed to define the best strategy.
General best practice rules apply
Initial Load - Defer IndexingGeneral LDV recommendations XLDV considerations
Reduce additional processing on SFDC during initial data upload: turn off search indexes, consider creating custom indexes after data load
Based on initial performance results index recalculations on extra large volumes of data take long time post load (not parallelized currently). It might be a better approach to configure all required indexes prior to XLDV load.
Initial Load - Use Fastest Operation PossibleGeneral LDV recommendations XLDV considerations
Use the fastest operation possible. insert is fastest then update, then upset
If possible load full set of initial data in one go using insert with only small incremental upserts when needed (failed records reprocessing for example).
To avoid loading big volumes data in upsert mode (for example when main load job fails in the middle and remaining data should be added on top of existing data in an object) consider configuring a client load job the way that joins existing records to remaining records (on external Id) outside of SFDC and then loading resulting set in insert mode.
Initial Load - Use Clean Data SetGeneral LDV recommendations XLDV considerations
Use as clean data set as possible
Note that errors in batches are causing single row processing on the remainder of the batch that slows down load performance significantly.
Perform rigorous data validation prior to upload. This will allow:
1. Loading most of the records in the fastest insert operation and avoid slow down due to the preventable errors
2. Avoid slow down when processing goes record by record within a 200 record transactional batch when number of failed record reaches a Threshold
Slide parts
Slide parts
Slide parts
Slide parts
Data Load Considerations (*- will be replaced)Topic Consideration What do I need to do?Data Model Normalized/ De-normalized
Sharing Model Sharing calculations
Target Data Does it really need to exist in SFDC? BP etc.
Test Environments Test them before you deploy!
Timelines Adequate timeline for testing
User data Cannot be deleted
API Soap versus Bulk
Batches Limit of 2K batches for Bulk API a day SF Support to increase batches
Deletes Oh No..
Extrapolation Not really possible It is at par or better, if Prod can be used even better