Nearline storage - SAP BI Netweaver
-
Upload
shyam-babu -
Category
Documents
-
view
546 -
download
1
Transcript of Nearline storage - SAP BI Netweaver
IM302 Near-line Storage With SAP NetWeaver BI
Glen Leslie, Director - SAP NetWeaver RIG Americas
Dr. Peter Zimmerer, Development Architect - SAP NetWeaver BI
© SAP 2008 / SAP TechEd 08 / <IM302> Page 2
Disclaimer
This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.
© SAP 2008 / SAP TechEd 08 / <IM302> Page 3
Session IM302 Abstract
Near-line Storage With SAP NetWeaver BI
In the context of information lifecycle management, SAP NetWeaver 7.0 offers a near-line storage (NLS) interface that allows to export operational data from the database into NLS partitions under control of a particular NLS vendor solution. Several vendors are already certified for this interface. NLS partitions of an InfoProvider are read-only and therefore have to be seen as a type of archive. Nevertheless, from the query and DTP perspective the existence of NLS partitions is transparent. The NLS concept establishes scalability for database volumes, reduces TCO for database management and administration, and complements with the SAP NetWeaver Business Intelligence Accelerator (SAP NetWeaver BI Accelerator), keeping old data - that normalynobody wants to look at - away from the valuable SAP NetWeaver BI Accelerator memory space. Finally, NLS support is offered for write-optimized DataStore Objects (7.0 EHP1) and for MultiProviders (7.2) as well. The session reflects the entire NLS concepts as well as the latest enhancements and optimizations discussing various scenarios with the help of numerous system demonstrations.
© SAP 2008 / SAP TechEd 08 / <IM302> Page 5
1. Concept of Nearline Storage1.1. Basic ideas of cross media storage managment1.2. Information Lifecycle Management
2. Nearline Solution of SAP NetWeaver BI2.1. Architecture2.2. Implementation Steps
3. Latest Enhancements3.1. NLS for write-optimized DataSore Objects and MultiProvider3.2. Look-up API 3.3. Administration Improvements
4. Summary and Outlook4.1. Roadmap and time line
Agenda
© SAP 2008 / SAP TechEd 08 / <IM302> Page 6
Information as Corporate Asset – We Do not Know What we not Know...
The Known Current BI implementations are set up
to answer known requirements
The UnknownLittle or nothing is done to be prepared for
unpredictable future information needs
© SAP 2008 / SAP TechEd 08 / <IM302> Page 7
DSS Applications Departmental Data Marts
EDW
MarketingAcctg Finance
Sales ERPERP
ERP
CRM
eComm.
Bus. Int.
ETL
GlobalODS
Oper.Mart
Exploration warehouse/ data mining
Stag
ing
Are
a
localODS
DialogueManager
CookieCognition
Preformatteddialogues
Cross mediaStorage Management
NearlineStorage
Web Logs
SessionAnalysis
Internet
ERPCorporate
Applications
ChangedData
GranularityManager
Archives
Bill Inmon’s: Enterprise Data Warehousing Concept
Source:Bill Inmon
© SAP 2008 / SAP TechEd 08 / <IM302> Page 9
Bill Inmon’s Data-Aging Strategies
Online Database Near-Line Storage Classic Archive
Frequently read / changed data
(actual)
Infrequently read data (mature)
Very rarely read data (aged)
Information Lifecycle according to Importance/Age:
© SAP 2008 / SAP TechEd 08 / <IM302> Page 10
Classic Archiving in BW vs. NLS in SAP NW 7.0 BI
Access Frequency/Possibility
Age of Data
Archiving (SAP BW 3.X)ADK-based (Archive Development Kit) archiving solution for InfoCubes and ODS objectsCost reduction using alternative storage mediaArchived data must be reloaded into the SAP NetWeaver BI database for analysis purposes
Online Archive ArchiveReloadOnline
Online Near Line Storage
NLS (SAP NW 7.0 BI)SAP NetWeaver BI has direct access to NLS data Availability of historic data for reporting and propagation with reduced costsReload of NLS-based data into the InfoCube or DataStore Object notnecessary for analysis purposes
© SAP 2008 / SAP TechEd 08 / <IM302> Page 11
1. Concept of Nearline Storage1.1. Basic ideas of cross media storage managment1.2. Information Lifecycle Management
2. Nearline Solution of SAP NetWeaver BI2.1. Architecture2.2. Implementation Steps
3. Latest Enhancements3.1. NLS for write-optimized DataSore Objects and MultiProvider3.2. Look-up API 3.3. Administration Improvements
4. Summary and Outlook4.1. Roadmap and time line
Agenda
© SAP 2008 / SAP TechEd 08 / <IM302> Page 12
Analytic Engine
BI Architecture: Platform & Data Warehouse
Dat
a Fl
ow C
ontr
ol /
Proc
ess
Cha
ins
Business Explorer Suite (BEx)
Caching
Data Acquisition
Mon
itorin
g / A
dmin
istr
atio
n
Met
a D
ata
Rep
osito
ry /
Doc
umen
ts
Calculation
Aggregation
Planning Services
Planning Functions
andSequences
Data Warehouse
Operational Data Store(volatile) Data Warehouse Layer
(historical)
Architected Data Marts
Open Hub
Service
DataSource PSA
Analysis Process Design
BI A
ccel
erat
or E
ngin
e
Master Data
Archiving Service
Arc
hive
/ N
LS E
ngin
e
© SAP 2008 / SAP TechEd 08 / <IM302> Page 13
Focusing on Archiving, NLS and Business Intelligence
Main ideas of the SAP NW 7.0 BI Nearline Storage Interface:
Data Archiving Processes (DAP) result in partitioning of InfoCube and DSO dataNearline Storage fills the gap between online storage and offline storage NLS data is directly accessable via SQL-like interfaces, but read only NLS data can be highly compressedNLS data is self-contained with respect to indexing DAP leads to Archiving (ADK) or NLS or both
Technical details depending on Partner Solutions residing on top of the interface
ERPBI
ETL/EAI
ADK+ASADK
ILM
Archive NLS NLS
DAP
© SAP 2008 / SAP TechEd 08 / <IM302> Page 14
NLS Interface – Architectural Overview
© SAP 2008 / SAP TechEd 08 / <IM302> Page 15
NLS Interface – Objects, Processes and Functions
RDBMS
InfoProvider
NLSPartner Solution
BI
DB
DAP – Design Time
DAP – Run Time
define partitioning citeriagenerate DAP
extract(DAP Menu,Process Chain)restore (DAP menu)
DAP – Scheduling /Monitoring
describe partition to be extractedgenerate request InfoProviderManage MenuProcess Chain
Data Archiving Process (DAP)
Data Archiving Process (DAP)
NLS - Interface
Add. Features Query access to NLSDTP access to NLS
© SAP 2008 / SAP TechEd 08 / <IM302> Page 16
The Near-Line Storage Solution for SAP NetWeaver BI
Separation of frequently used data and rarelyused data using Data Archiving Processes (DAP)
Data Archiving Processes guarantee data consistency over DBMS and file system boundaries using
automatic deletes write protection for removed areas in InfoCubes and DataStore ObjectsReload option
NLS support for InfoCubes and DataStore ObjectsNLS Partions of an InfoCube are not accesseble in an MultiProvider scenario(available with the next 7.2 release)
Transparent access to NLS data forQueriesData Transfer Processes, DAP Restore
Open interface for partner solutions
Present development partners Certified for SAP NW 7.0 (in the sequence of certification)
SAND Technology – Dynamic Nearline Access® yesPBS Software – CBW® yesFileTek – StorHouse® yes
© SAP 2008 / SAP TechEd 08 / <IM302> Page 17
Offline Archive
RDBMS
Design Aspects – Near-line Storage vs. BI Accelerator
InfoMarts (InfoCube)
Near-line StorageBIA
Acquisition
Acceleration Archiving
BI
Access - very frequently frequently not frequently rarely
NLS alternative storage types with direct accesscapabilities for reporting and loadingextracted partitions are deleted in RDBMS (NLS storage and Online Storage togetherconsistently reflect the BI data persistencyof an InfoProvider)NLS data is read-only NLS part of an InfoProvider are write-protectedNLS is primarily partitioned by time/age
BIA Replication of the BI Star Schema includingmaster dataDB volume not affectedRoll-Up and Change Run possible after data loads optimized for fast BI Query access (loadbalancing by hash partitioning)
© SAP 2008 / SAP TechEd 08 / <IM302> Page 18
1. Concept of Nearline Storage1.1. Basic ideas of cross media storage managment1.2. Information Lifecycle Management
2. Nearline Solution of SAP NetWeaver BI2.1. Architecture2.2. Implementation Steps
3. Latest Enhancements3.1. NLS for write-optimized DataSore Objects and MultiProvider3.2. Look-up API 3.3. Administration Improvements
4. Summary and Outlook4.1. Roadmap and time line
Agenda
© SAP 2008 / SAP TechEd 08 / <IM302> Page 19
Overview About Latest Enhancements
Archiving of write-optimized DataStore ObjectsSupport of Data Acquisition Layer ( Corporate Memory)
Nearline access through MultiProviderDecoupling of reporting layer and persistency layer
Look up of Nearline data (e.g. for staging processes)Support of Data Integration Layer
Miscellaneous enhancementsEnhanced status and job monitoring within InfoProvider management viewSuspension and selective continuation of archiving processes within Process ChainsRestore of an archiving request with all successorsSmaller Data Object size for ADK-based Nearline Solution without semantic grouping
© SAP 2008 / SAP TechEd 08 / <IM302> Page 20
Archiving of Write-Optimized DataStore – Properties and Modeling Recommendations
New DataStore type "Write-optimized DataStore Object" introduced with NW 7.0Only active data table (no ChangeLog, no ActivationQueue)Generated technical key (Request Id, Packet No, Record No)
mass insert into active data tableno activation step necessary
Additional semantic keyMust be defined, butUniqueness check can be switched off
No SID generation (but reporting is possible)Fully integrated into the staging process (can be used as source and target as well)
Recommended to be used within the Data Acquisition Layer as "Pass Thru"Finest granularity of dataHighest data volumeBest place to be prepared for the unforeseen ( Corporate Memory)
IssueNot archivable with SAP NetWeaver BI 7.0
© SAP 2008 / SAP TechEd 08 / <IM302> Page 21
Archiving of Write-Optimized DataStore – Request-Based Archiving
New obligatory Selection Profile "Request-Based Archiving"
Relative time selection will be defined in order to derive a "Less than" selection for Request IDEither based on technical request attribute creation date or loading date or …… absolute time characteristic from data
Archiving takes place with strictly increasing Request IDConsequently restore must take place with strictly with decreasing Request ID
Selection Profile depends on DSO property "Do Not Check Uniqueness of Data"Property is not set (= default)
An absolute time characteristic or a time-correlated partitioning characteristic from the semantic key must be chosen in order to support check of uniqueness by range locks
Property is set (i.e. uniqueness needs not to be checked)Get minimum request id rmin having creation/loading date greater than threshold date dnArchive all requests having request Id R < rmin
Currently being loaded
Currently being archived/restoredArchived
© SAP 2008 / SAP TechEd 08 / <IM302> Page 22
Archiving of Write-Optimized DataStore – Variants of Request-Based Selection Profile
Use creation/loading date as timeslice
char.
Archive all requests having request Id R ≥
rn-1 and R < rn (with r0 = 0)
Calculate threshold date dn from current date, time shift, and rounding unit
Convert threshold date dn to value of timeslice characteristic tn
Get minimum request Id rn having timeslice characteristic
value greater than tn
Get minimum request Id rn having loading/creation
date greater than dn
Check uniquess
of data
No
Yes No
Timeslice characteristic is part
of the semantic key
Yes
Convert threshold date dn to value of timeslice characteristic tn
pn := tn
Get minimum request Id rn having partitioning characteristic
value greater than pn
No
Get minimum value pn for partitioning characteristic
having timeslice characteristic value greater than tn
Lock range P ≥
pn and P < pn
Yes
Should be preferred
Fastest
Slowest
© SAP 2008 / SAP TechEd 08 / <IM302> Page 23
Archiving of write-optimized DataStore – Demo
Create a Data Archiving ProcessCreate and schedule archiving requestsRestore archiving requestsLoad data to subsequent Data Targets
LSA
Data Propagation Layer Corporate Memory
Data Acquisition Layer
Data Acquisition Layer
DataSource
InfoSource
InfoPackage
InfoPackage
RDA-DTP
Nearline Storage
Reporting Layer (Architected Data Marts)Reporting Layer (Architected Data Marts)
MultiProviderSAP Sales InfoCube
Propagating DataStoreStandard DSO
DAP
DTP DTP
DTP
PSA
RDA-Daemon (Pull ~ 1/min)
DTP
1
23
4
1234
Corporate Memory DSOWrite optimized DSO
© SAP 2008 / SAP TechEd 08 / <IM302> Page 24
Overview About Latest Enhancements
Archiving of write-optimized DataStore ObjectsSupport of Data Acquisition Layer ( Corporate Memory)
Nearline access through MultiProviderDecoupling of reporting layer and persistency layer
Look up of Nearline data (e.g. for staging processes)Support of Data Integration Layer
Miscellaneous enhancementsEnhanced status and job monitoring within InfoProvider management viewSuspension and selective continuation of archiving processes within Process ChainsRestore of an archiving request with all successorsSmaller Data Object size for ADK-based Nearline Solution without semantic grouping
© SAP 2008 / SAP TechEd 08 / <IM302> Page 25
Nearline Access through MultiProvider
Decoupling of Reporting Layer from Persistency Layers
Recommendation:
Usage of MultiProviders
Issue in NW 7.0:
MultiProvider does not support access of Nearline partitions of its Part Providers
Solution with NW 7.2:
Support enabled with NW 7.2Development already finishedTests are ongoing
Availability:
Ramp-Up to customer (RTC) planned for Q4/2009
Queries
MultiProvider
Nearline Storage
© SAP 2008 / SAP TechEd 08 / <IM302> Page 26
Overview About Latest Enhancements
Archiving of write-optimized DataStore ObjectsSupport of Data Acquisition Layer ( Corporate Memory)
Nearline access through MultiProviderDecoupling of reporting layer and persistency layer
Look up of Nearline data (e.g. for staging processes)Support of Data Integration Layer
Miscellaneous enhancementsEnhanced status and job monitoring within InfoProvider management viewSuspension and selective continuation of archiving processes within Process ChainsRestore of an archiving request with all successorsSmaller Data Object size for ADK-based Nearline Solution without semantic grouping
© SAP 2008 / SAP TechEd 08 / <IM302> Page 27
NLS
EMPL_ID DATE_TO DATE_FROM … SALARY …4711 31.12.2007 01.01.2007 … 50 000 …4712 31.12.2007 01.01.2007 … 62 000 …4711 31.12.2008 01.01.2008 … 51 000 …4712 31.12.2008 01.01.2008 … 64 000 ...
Look up of Nearline Data
EMPL_ID SDATE … SALARY …4711 27.12.2007 … 50 000 …4711 26.01.2008 … 51 000 …4712 27.12.2007 … 62 000 …
EMPL_ID SDATE …4711 27.12.2007 …4711 26.01.2008 …4712 27.12.2007 …
Target
Source
Lookup DSO
Target
Source
Lookup DSO
L.EMPL_ID = S.EMPL_ID AND L.DATE_TO >= S.SDATE AND L.DATE_FROM <= S.SDATE
© SAP 2008 / SAP TechEd 08 / <IM302> Page 28
Look up – without Nearline Storage Code Snippet from Transformation ... *$*$ begin of global - insert your declaration only below this line *-* ... DATA pt_lookup_data TYPE SORTED TABLE OF ... WITH UNIQUE KEY emplid date_to. ... *$*$ end of global - insert your declaration only before this line *-* ... METHOD start_routine. ... *$*$ begin of routine - insert your code only below this line *-* ... * collect keys for lookup from data package LOOP AT source_package ASSIGNING <source_fields>. MOVE-CORRESPONDING <source_fields> TO ls_lookup_key. APPEND ls_lookup_key TO lt_lookup_keys. ENDLOOP. * lookup from DSO table SELECT emplid date_to date_from salary INTO TABLE pt_lookup_data FROM (l_tabname) FOR ALL ENTRIES IN lt_lookup_keys WHERE emplid = lt_lookup_keys-emplid AND dateto >= lt_lookup_keys-sdate AND datefrom <= lt_lookup_keys-sdate. ... *$*$ end of routine - insert your code only before this line *-* ENDMETHOD. "start_routine ...
© SAP 2008 / SAP TechEd 08 / <IM302> Page 29
Look up – with Nearline Storage Adjustments to Transformation
SELECT emplid date_to date_from salary INTO TABLE pt_lookup_data FROM (l_tabname) FOR ALL ENTRIES IN lt_lookup_keys WHERE emplid = lt_lookup_keys-emplid AND dateto >= lt_lookup_keys-sdate AND datefrom <= lt_lookup_keys-sdate.
DATA: lt_lookup_conditions TYPE cl_rsda_infoprov_query=>t_entry_fields, ls_lookup_condition LIKE LINE OF lt_lookup_conditions. ls_lookup_condition-fieldname = 'EMPL_ID'. ls_lookup_condition-option = 'EQ'. ”= ls_lookup_condition-component_name = 'EMPL_ID'. APPEND ls_lookup_condition TO lt_lookup_conditions. ls_lookup_condition-fieldname = 'DATE_TO'. ls_lookup_condition-option = 'GE'. ”>= ls_lookup_condition-component_name = 'SDATE'. APPEND ls_lookup_condition TO lt_lookup_conditions. ls_lookup_condition-fieldname = 'DATE_FROM'. ls_lookup_condition-option = 'LE'. ”<= ls_lookup_condition-component_name = 'SDATE'. APPEND ls_lookup_condition TO lt_lookup_conditions. TRY. CALL METHOD cl_rsda_infoprov_query=>select EXPORTING i_infoprov = c_infoprov i_t_entries = lt_lookup_keys i_t_entry_fields = lt_lookup_conditions CHANGING c_t_data = pt_lookup_data. CATCH cx_rsda_input_invalid cx_rsda_access_error. * Do error handling ... ENDTRY.
Replace
SELECT … FOR ALL ENTRIES …
by
API call
© SAP 2008 / SAP TechEd 08 / <IM302> Page 30
Look up of Nearline Data
Available with SAP NetWeaver BI 7.0 Support Package 10
See SAP note 985609 and related notes for details
Extension to Nearline Interface as prototype for pilot customers availableto solve performance problems with temporal lookups
© SAP 2008 / SAP TechEd 08 / <IM302> Page 31
Miscellaneous Enhancements
Enhanced status and job monitoring within InfoProvider management viewIncludes monitoring of archive instances within process chains and ADK-based jobsIndicator for “active” archiving requestsAvailable with SAP NetWeaver BI 7.0 Enhancement Package 1
Suspension and selective continuation of archiving processes within Process Chains
Option: Continue only archiving requests created by predecessor instancesAvailable with SAP NetWeaver BI 7.0 Support Package 18
Restore of an archiving request with all successorsAvailable with SAP NetWeaver BI 7.0 Support Package 18
Smaller Data Object size for ADK-based Nearline Solution without semantic grouping
Data Object size reduced from 20 MByte to 1 MByteImproves performance for ADK-based Nearline Solutions like PBS CBWAvailable with SAP NetWeaver BI 7.0 Support Package 19 and Enhancement Package 1
© SAP 2008 / SAP TechEd 08 / <IM302> Page 32
1. Concept of Nearline Storage1.1. Basic ideas of cross media storage managment1.2. Information Lifecycle Management
2. Nearline Solution of SAP NetWeaver BI2.1. Architecture2.2. Implementation Steps
3. Latest Enhancements3.1. NLS for write-optimized DataSore Objects and MultiProvider3.2. Look-up API 3.3. Administration Improvements
4. Summary and Outlook4.1. Roadmap and time line
Agenda
© SAP 2008 / SAP TechEd 08 / <IM302> Page 33
Summary
General aspectsNLS fills the gap between online storage and offline storage with its data residing neither in the BI data base nor in a classical archiveNLS data can be accessed directly for analyses and data load purposes (BI Queries, DTP’s, DAP Restore Feature)NLS handling is provided for InfoCubes and DataStore Objects and processed using Data Archiving Processes (DAP)NLS data is read-only and thus promotes cost-efficient storage management
Vendor specific aspectsNLS data can be compressed (up to 95%)NLS data is intrinsically indexedNLS data can reside on a separate database or file system
© SAP 2008 / SAP TechEd 08 / <IM302> Page 34
Outlook
Medium termEnhanced lookup support especially for temporal lookups (non-equal lookup conditions)Request-based archiving for InfoCubes (avoid compression before archiving)Combined DTP extraction from online and archive partition of an InfoCube
Long termArchiving of InfoCubes with non-cumulative key figuresArchiving of master data and hierarchiesNLS support for InfoSets…
© SAP 2008 / SAP TechEd 08 / <IM302> Page 35
SDN Subscriptions offers developers and consultants like you, an annual license to the complete SAP NetWeaver platform software, related services, and educational content, to keep you at the top of your profession.
SDN Software Subscriptions: (currently available in U.S. and Germany)A one year low cost, development, test, and commercialization license to the complete SAP NetWeaver software platform Automatic notification for patches and updatesContinuous learning presentations and demos to build expertise in each of the SAP NetWeaver platform componentsA personal SAP namespace
SAP NetWeaver Content Subscription: (available globally)An online library of continuous learning content to help build skills.
Starter Kit
Building Your Business with SDN Subscriptions
To learn more or to get your own SDN Subscription, visit us at the Community Clubhouse or at www.sdn.sap.com/irj/sdn/subscriptions
© SAP 2008 / SAP TechEd 08 / <IM302> Page 36
Further Information
Related Workshops/Lectures at SAP TechEd 2008IM362 Enterprise Data Warehousing With SAP NetWeaver BI:
Designing an Effective Corporate Memory, H2
Related SAP Education and Certification Opportunitieshttp://www.sap.com/services/education/catalog/netweaver/bi.epxservice.sap.com/okp
SAP Public Web:SAP Developer Network (SDN): https://www.sdn.sap.com/irj/sdn/nw-biBusiness Process Expert (BPX) Community: www.bpx.sap.com
© SAP 2008 / SAP TechEd 08 / <IM302> Page 37
Thank you!
© SAP 2008 / SAP TechEd 08 / <IM302> Page 38
Please complete your session evaluation.Be courteous — deposit your trash,
and do not take the handouts for the following session.
Thank You !
Feedback