Download File.1 - CCSU · Title: Download File.1.1 Created Date: 20180118170653Z
Download File
Transcript of Download File
![Page 1: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/1.jpg)
BID209: Best Practices for Implementing with Sybase IQ
Mark D. Mumy
Principal Systems Consultant
April 12, 2023
![Page 2: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/2.jpg)
2
About This Presentation
Setup and Administration• Hardware Configuration
• Creating the Database
• IQ Server Settings
• Connectivity
• Maintenance Tasks
• Memory Usage
Development and Design• Data Model Recommendations
• Database Programming
• Data Manipulation
![Page 3: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/3.jpg)
3
Target Audience
Database Administrators• All Levels
Query Developers• All Levels
![Page 4: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/4.jpg)
4
For This Presentation
Let’s Keep It Interactive• I will entertain questions on a subject
• Would be happy to speak to you offline, if desired
No Question is too Basic• Chances are others may have the same question
Open Question Forum at the end• Time Permitting
![Page 5: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/5.jpg)
5
Setup and Administration
Hardware Configuration Creating the Database IQ Server Settings Connectivity Maintenance Tasks Memory Usage
![Page 6: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/6.jpg)
6
Hardware Configuration –Drive Arrays
Volume Manager• Not a necessary or recommended component for IQ
• Can use MPXIO on Sun Platform to provide controller failover, dynamic multi-pathing, and world wide naming convention
• Additional overhead and software that has no value add to an IQ installation
• Acceptable for an IQ single node operation where hardware does not have the ability to apply RAID to the disk devices
![Page 7: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/7.jpg)
7
Hardware Configuration –Drive Arrays
RAID Level• Recommend using RAID 5 as a blend between performance,
protection, and cost
Recommend using raw devices (a must for multi-node IQ)
For details on drive array specifications and configurations see the Sun Reference Architecture Whitepapers and EMC Whitepapers on array configurations with IQ
![Page 8: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/8.jpg)
8
Hardware Configuration –Memory
How Much RAM for IQ• As much as possible!
• Most systems should have at least 2 GB
• Rough rule of thumb is 2 GB per CPU
• Don’t forget additional RAM for each load being performed
• Shared memory is only needed on 32-bit systems and some 64-bit systems to wire memory into RAM
• HP-UX 64-bit should not use –iqsmem
![Page 9: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/9.jpg)
9
Hardware Configuration –Processors and Disk Controllers
Optimal CPU configuration• Queries: 1 CPU per active query – more if queries are complex and
can be run in parallel
• Loads: 1 CPU per HG index plus 1 CPU per 2-5 columns being loaded
Disk Controllers• 1 fiber controller per 5 write CPU’s
• 1 fiber controller per 10 read CPU’s
• In mixed mode operation use the 5:1 ratio
• Recommend a minimum of 2 controllers
![Page 10: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/10.jpg)
10
Creating The Database –Creation Options
Creation Options• Block Size and Page Size
• Minimum should be 128K page size (8K block)
• Use 256K page for larger databases (>200 GB or so)
• Larger the page size the more RAM that will be required
• CASE IGNORE vs. CASE RESPECT
• Use CASE RESPECT whenever possible
• Removes case comparison steps and improves performance
• Java in the Database
• Install if Java will be needed in the database
• No impact if installed and not used
![Page 11: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/11.jpg)
11
Creating The Database –File Placement
Use relative links – makes it easier to relocate files Use symbolic links for IQ MAIN and IQ TEMP
• /IQM_devs/IQ_MAIN_00
• /IQM_devs/IQ_TEMP_00
Place the transaction log and transaction log mirror on significantly large file system• Recommend 5 GB of filesystem space per IQ node for database
related storage
• Transaction logs, IQ MSG, scripts, catalog file, etc
![Page 12: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/12.jpg)
12
Creating The Database –Files Locations
Filename rules• Catalog: DATABASE_NAME.db
• Transaction log: DATABASE_NAME.tlog
• IQ MSG: DATABASE_NAME.iqmsg
• IQ Main Store: DATABASE_NAME_iqmain_000
• 000 would be an incrementing number
• IQ Temp Store: DATABASE_NAME_iqtmp_000
• 000 would be an incrementing number
• IQ Configuration File: params.cfg (default)
![Page 13: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/13.jpg)
13
Creating The Database –Top Level Directory
Use for writer and readers (multi-node configuration) to communicate
IQ 12.4.3 and earlier• Must be shared across all hosts
• Generally done via NFS, but some customers use cluster based filesystems (Veritas Cluster Manager)
IQ 12.5• No longer a shared filesystem!
• Should be local filesystem on each host that is protected accordingly
• The name and path can be unique for each node
• No need for NFS or cluster based filesystems
![Page 14: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/14.jpg)
14
IQ Server Settings –Query Performance
FORCE_NO_SCROLL_CURSORS• Should always be set to ON
• Very few applications require this to be OFF
• Can improve query performance
![Page 15: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/15.jpg)
15
IQ Server Settings –Query Plans
Query Plan Settings to Provide Optimal Query Information to DBA’s and Sybase Engineering• set temporary option query_plan='on';
• set temporary option query_plan_after_run='on';
• set temporary option query_timing='on';
• set temporary option query_detail='on';
• set temporary option dml_options10='on';
• set temporary option query_plan_as_html='on';
• set temporary option Query_Name = ‘Query Name‘
Should not be set globally as the IQ MSG file will grow rapidly
![Page 16: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/16.jpg)
16
IQ Server Settings –Storage
Append_Load• Can be used to improve load performance
• Will not reuse Row ID’s or the space occupied by those Row ID’s
• Great for systems where large, contiguous chunks of data are deleted
• May not be good if random rows are deleted as it can lead to fragmentation and allocated, but unused space
![Page 17: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/17.jpg)
17
IQ Server Settings –Storage Continued
Disk_Striping• If ON, IQ will stripe writes to all available devices
• If OFF, the first device must be full before the next is used
Disk_Striping_Packed• If ON, it forces better space usage and less fragmentation
• Fragmentation is indicated when out of space messages are returned but the main dbspace used is less than 100% full
• Allows Adaptive Server IQ to better utilize small pieces of unused space that remain after compression
![Page 18: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/18.jpg)
18
IQ Server Settings –Data Loads
LOAD_MEMORY_MB• Set to 0 (default) on systems with enough RAM
• Set to <= 500 on systems where RAM is tight or there are simultaneous loads taking place
• Rough rule of thumb for load memory requirements
• Run: select tablewidth( ‘tablename’ )
• Divide that value by 2
• New number is rough calculation of MB needed to load that table
• Remember that each load requires separate memory
![Page 19: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/19.jpg)
19
Network Connectivity –TCP/IP Ports
When configuring a TCP/IP port for IQ to listen on, use something other than the default of 2638
ASA and IQ use port 2638 as a broadcast listener Multiple servers on the same host share the default port for
broadcast listening, but not for client network traffic Broadcast listening takes precedence over client network traffic as
it is the first port started Easier and safer to assume that 2638 is in use already by ASA/IQ
and to use another port for client network traffic Many things must be considered if an IQ server is to use port
2638 for client network traffic on a host with more than one IQ server (server start order, multiplex synchronization, etc)
See http://my.sybase.com/detail?id=1012955 for details
![Page 20: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/20.jpg)
20
Client Access and Network Connectivity
Different internal environments and settings for ODBC, JDBC, and Open Client connections
JDBC and ODBC are recommended• JConnect is Sybase’s implementation of JDBC
• Use the JDBC 2 driver (JConnect 5.2/5.5)
• com.sybase.jdbc2.jdbc.SybDriver
Open Client• Use with caution
• Most applications written using Open Client expect an ASE server
![Page 21: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/21.jpg)
21
Client Access and Network Connectivity –Open Client vs. ODBC
See Chapter 32 of the ASA Users Manual as well as the ASA Reference Section “Transact-SQL and SQL/92 compatibility options” for complete list of differences
If writing stored procedures or embedded application code, make sure to explicitly make settings for compatibility as these options will get set to different values for Open Client vs. ODBC connections
ALLOW_NULLS_BY_DEFAULT QUOTED_IDENTIFIER STRING_RTRUNCATION ANSI_BLANKS
ANSINULL CHAINDED FLOAT_AS_DOUBLE
![Page 22: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/22.jpg)
22
Client Access and Network Connectivity –Open Client vs. ODBC
ODBC and JDBC connectivity support all IQ datatypes Open Client does not support all IQ datatypes
• Unsigned integer
• Depending on version of Open Client, bigint and unsigned bigint should be OK (OC 12.5 and later)
![Page 23: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/23.jpg)
23
Client Access and Network Connectivity
AutoPreCommit Within ODBC• Set registry setting AutoPreCommit to Y
• Forces applications to issue a COMMIT before each query
• Go to the registry and update the corresponding Sybase Data Source Name (DSN) created, by adding a new value 'AutoPreCommit’ with a value of ‘Y'
• HKEY_LOCAL_MACHINE/SOFTWARE/ODBC/ODBC.INI/{DSN}
Packet Sizes• Larger packet sizes will help with large data retrieval
• Use –p option in IQ configuration file to increase size
• Use CommBufferSize parameter in ODBC connection string
![Page 24: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/24.jpg)
24
Client Access and Network Connectivity
Network Speed• Data retrieval will be as fast as the network
• 100 MB of data will take 80 seconds on a 10 Mbit LAN
• 100 MB of data will take 8 seconds on a 100 Mbit LAN
• 100 MB of data will take 0.8 seconds on a Gigabit LAN
• LAN speed may be the performance bottleneck for queries that return large amounts of data
• The faster the network cards and LAN the better off concurrency will be: more available bandwidth per user
Network Interface Cards• Adding multiple network cards to the IQ node will help with network
![Page 25: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/25.jpg)
25
Client Access and Network Connectivity
ASE Component Integration Services (CIS)• Can have ASE CIS reference IQ tables (proxy tables)
• Use ASE 12.5 CIS class of ASIQ
• Prior to ASE 12.5 ASAnywhere or ASEnterprise classes had to be used
• Not a viable option if joining non-IQ tables with an IQ proxy table
• Can map multiple ASE logins to a single IQ login
• Data modifications should not be performed on proxy tables
• Can map ASE proxy tables to stored procedure calls in IQ
![Page 26: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/26.jpg)
26
Client Access and Network Connectivity
IQ Component Integration Services (CIS)• Can only be used on the Solaris and WinNT versions of IQ (12.4.3)
• All platforms support CIS with IQ 12.5
• Can map to tables in Oracle (ODBC), ASE (ODBC & JDBC), ASA (ODBC & JDBC), DB2 (ODBC), MS SQL Server (ODBC), and any ODBC source
• Not a viable option if joining IQ tables with non-IQ proxy tables
• Data modifications may be performed on proxy tables
![Page 27: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/27.jpg)
27
Maintenance Tasks –Database Consistency Checks
IQ 12.4.3 • The routine to check the database for potential corruptions is
SP_IQCHECKDB
• There are no run-time options for this command, however the DBCC options control the behavior of the SP_IQCHECKDB command
• Recommendations
• Run Level 1 (DBCC_OPTION=1) DBCC’s every 1 to 4 weeks (will process 1-2 GB per second)
• Run Level 0 (DBCC_OPTION=0) DBCC’s every 1 to 3 months (will process 50-100 MB per second)
• Use DBCC_TEAM_SIZE option to control parallel execution
![Page 28: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/28.jpg)
28
Maintenance Tasks –Database Consistency Checks
IQ 12.5• New syntax for IQ 12.5 – See the Reference Manual
• Recommendations
• Perform CHECK every 1 to 4 weeks
• Perform VERIFY every 1 to 3 months
• Perform REPAIR if errors are reported from VERIFY
• Use RESOURCE_PERCENT parameter to control CPU usage during DBCC.
![Page 29: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/29.jpg)
29
Maintenance Tasks –Parallel Create index
Create Multiple Indexes in a batch• Syntax:
• BEGIN PARALLEL IQ
• Create HG Index … ;
• Create LF Index … ;
• END PARALLEL IQ ;
Recommendation• Create no more indexes in a batch than you have CPU's
• Note: Two indexes on same column will be serial
![Page 30: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/30.jpg)
30
Maintenance Tasks – Setting the IQ Caches
Start by allocating memory to caches• 40% IQ Main Cache
• 60% IQ Temporary Cache
Adjust allocations from here based on monitoring Number of HG indexes in a table may alter this plan for
loading data Make sure not to starve other operations that need
RAM (OS, LOAD_MEMORY_MB, other applications, etc)
![Page 31: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/31.jpg)
31
Maintenance Tasks –IQ Monitoring
IQ Monitor is a diagnostic tool for DBA’s It collects and reports internal counters from the IQ
Buffer Caches• Main Cache interaction with the IQ Store
• Temporary Cache interaction with the IQ Temp Store
IQ Monitor offers a series of “views” of the counters to showing differing aspects of the server and buffer cache workload
![Page 32: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/32.jpg)
32
Maintenance Tasks –IQ Monitoring
Provides different views of buffer activity• summary report of both caches
• detailed report of one cache
• i/o activity of a cache
• a debug report of all buffer cache activity
You must specify an ‘option’ when you start the monitor to specify what view to monitor
One Monitor may be running for a cache• May have one for each (Main & Temp)
![Page 33: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/33.jpg)
33
Managing Memory –How Much Can IQ Address?
Solaris and AIX (64-bit) – limited to RAM + swap space with no kernel changes necessary
AIX (32-bit) – limited to 2.5 GB using –iqsmem to achieve this
HP-UX – limited to kernel value of maxdsiz_64bit• maxdsiz_64bit should never be set to exceed RAM + swap space
Windows NT/2000/XP – Must use a 4GT supported version• 2 GB for main and temp buffer caches• 3 GB total for the process• Load memory counts against 3 GB total, not 2 GB limit on buffer
caches
![Page 34: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/34.jpg)
34
Managing Memory –How Much Can IQ Address?
Keep in mind that more memory may mean tuning the O/S On Solaris, a system with 288 GB spent a tremendous amount of
time managing memory with standard configuration• Changed following kernel parameters to help (/etc/system):
set autoup=900
set tune_t_fsflushr=1
set lotsfree=0x10000
set bufhwm=8000
set maxphys=1048576
set maxusers=2048
set rlim_fd_max=2048• Keep in mind that above options were for a limited test and were the result of
site specific analysis performed by Sun and should be used as reference only
Refer to hardware and O/S specific tuning guides for help
![Page 35: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/35.jpg)
35
Managing Memory –Inventory Your System Memory
Assess what’s running and how much memory it uses (exclusive of IQ)
Typically these include• Operating System
• OLAP Servers
• Middleware
• Other applications
![Page 36: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/36.jpg)
36
Managing Memory
Deduct that total from RAM Allow 20% for File System Cache
• If using file systems
Deduct another 10% (just in case) The rest is for IQ and IQ Caches A picture tells this story
![Page 37: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/37.jpg)
37
Memory –The Big Picture
Start with the O/S and other applications
Determine what the IQ server needs (normal load)
IQ Overhead (Heap for loading)
Allocate IQ Caches• 40% IQ Main Cache
• 60% IQ Temp Cache
Operating System+ All Other Apps
IQ Server
IQ Overhead
IQ Main Cache
IQ Temp cache
![Page 38: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/38.jpg)
38
Development and Design
Data Model Recommendations Database Programming Data Manipulation
![Page 39: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/39.jpg)
39
Data Model Recommendations –Proper Datatype Sizing
Use the smallest datatypes possible for data Be aware of all datatypes in IQ – there may be more than you
know If hour, minute and second information is not necessary, use
DATE instead of DATETIME If the data will fit within a TINYINT or SMALLINT datatype use that
rather than INTEGER or BIGINT Allows the engine to store data in smaller units (1-byte TINYINT or
2-byte SMALLINT versus 4-byte INTEGER or 8-byte BIGINT Don’t over allocate storage when defining NUMERIC() or
DECIMAL() as it can be costly for data that doesn’t need all that space
![Page 40: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/40.jpg)
40
Data Model Recommendations –IQ UNIQUE
Use whenever possible to help minimize storage usage IQ 12.5
• Used solely for FP index type (storage)
IQ 12.4.3 and earlier• Imperative to use on all columns even if cardinality exceeds 64K
• Optimizer may make use of IQ UNIQUE value at query runtime
• Does not need to be exact, but should be close to cardinality
![Page 41: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/41.jpg)
41
Data Model Recommendations –IQ UNIQUE and the FP index
Consider using Minimize_Storage option in Sybase IQ 12.5. This will place an IQ UNIQUE(255) on every column for every table created and removes the need to use IQ UNIQUE
If the value is <= 255 then IQ will place a 1-byte FP index on the column – 1 byte of storage per row
If the value is > 255 but <= 65536 then IQ will place a 2-byte FP index on the column – 2 bytes of storage per row
May slightly hinder data loads, but improve query speeds May incur onetime slight load slowdown while 2-byte FP is
converted to flat FP, but this usually happens during the first load
![Page 42: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/42.jpg)
42
Data Model Recommendations –NULL Values
Always specify NULL or NOT NULL• Open Client and ODBC connections have different default behavior
when table is created
Allows the optimizer a better guess at join criteria NULL data does not save space on the database page
as it would in ASE Will be compressed out when stored on disk
![Page 43: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/43.jpg)
43
Data Model Recommendations –Unsigned Datatypes
Use unsigned datatypes where possible Use for surrogate keys and join columns Unsigned data comparisons are quicker The caveat to this is that Open Client may misinterpret
the value if it is too large as it does not understand large unsigned data• Can convert to signed integer, numeric, or decimal if returning data to
an Open Client application
• This caveat applies to moving data between IQ servers with INSERT FROM LOCATION
![Page 44: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/44.jpg)
44
Data Model Recommendations –Long Varchar and Long Varbinary
Can be used to store moderate amounts of text or binary data
VARCHAR() or VARBINARY() datatypes• Maximum width is 32K (64K ascii hex for VARBINARY())
LONG BINARY added in 12.5• Maximum width is 64K binary (128K ascii hex)
The WORD index is the only index allowed on VARCHAR() data wider than 255 bytes
Storage will be allocated in 256 byte chunks• A 257 byte string will require 512 bytes of storage• Much less than the 2K requirement in ASE TEXT and IMAGE
columns
![Page 45: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/45.jpg)
45
Data Model Recommendations –Large Object Storage
Can be used to store binary or text based objects Extends the long binary datatype from a maximum size
of 64K to an unlimited size!!! FP index is the only viable index Cannot be searched, only stored and retrieved Special function to return the size of an object
(byte_length64) Special function to return portions of the object, not the
entire contents (byte_substr64)
![Page 46: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/46.jpg)
46
Data Model Recommendations –Large Object Storage Continued
Create table example:• create table lob_test ( seq_no int, my_lob long binary )
No change to load table statement, so how is data loaded in the LOB?
Through a “load table” source data file:• 1|/tmp/lob_1.txt|• 2|/tmp/lob_2.txt|• 3|/tmp/lob_3.txt|
Load engine will place the contents of the file specified in the source data file (/tmp/lob_1.txt) into the long binary column for that row
Can also extract contents of a binary object to individual files with the BFILE() function
![Page 47: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/47.jpg)
47
Data Model Recommendations –Varchar vs. Char
Use CHAR() whenever possible All storage in IQ is fixed width VARCHAR() types only add overhead
• A VARCHAR(100) columns will require 101 bytes of storage
• 100 bytes for data
• 1 byte for the size of data
CHAR() data is blank padded, VARCHAR() is not
![Page 48: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/48.jpg)
48
Data Model Recommendations –When and Where to Use Indexes
Always use indexes on:• Join columns (HG index regardless of cardinality)
• Searchable columns (HG or LF index)
• Aggregation columns (HNG)
• Only if a single column is used in an aggregation
• SUM( A * B ) will not use an HNG, but rather an FP
• Date, Time, and Datetime columns (DATE, TIME, DTTM)
If uncertain, place an LF or HG index on the column Use Primary Key, Unique Constraint, or UNIQUE HG
indexes where appropriate
![Page 49: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/49.jpg)
49
Data Model Recommendations –When and Where to Use Indexes
A column with an HNG, CMP, or WD index should have a corresponding LF or HG index (very rare circumstances will negate this)
Indexes are not needed on columns whose data is ONLY returned to the client (projected)
Remove HNG indexes on date/time/datetime columns. Replace them with DATE, TIME, or DTTM indexes
Investigate the use of HNG indexes. In certain circumstances having HNG indexes on columns can hurt performance (mostly in aggregations).
![Page 50: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/50.jpg)
50
Data Model Recommendations –Simple Index Selection Criteria
Ask yourself these simple questions about each column to determine which indexes belong
• Is the cardinality greater than 1500-2000?• Yes: place an HG index on this column• No: place an LF index on this column
• Does this column contain date, time, datetime, or timestamp data?• Yes: place a DATE, TIME, or DTTM index on this column (these should
be used in lieu of the HNG index recommended in IQM 12.4.3 and earlier)• Will this column be used in range searches or aggregations?
• Yes: place an HNG index on this column (if the aggregation contains more than just the column, an HNG may not be appropriate)
• Will this column be used for word searching?• Yes: place a WD index on this column
• Will this column be compared to another column in the same table?• Yes: place a CMP index on the two columns being compared
![Page 51: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/51.jpg)
51
Data Model Recommendations –Multi-column Indexes
Currently, only HG, UNIQUE HG, UNIQUE CONSTRAINT, and PRIMARY KEY indexes support multiple columns
HG inserts are the most expensive in IQ Try to guarantee that inserts will happen at the end of
the index• Place generally incrementing data at the beginning of the index list
(sequential data)
• For instance, a transaction date or batch number
• Something that will try to guarantee a sequential key
![Page 52: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/52.jpg)
52
Data Model Recommendations –Join Column
Prefer joining integer datatypes (unsigned if possible) Integer comparisons are quicker than character
comparisons Keep the datatypes as narrow as possible to improve
join performance by reducing disk I/O and memory requirements
Prefer using HG index on join columns rather than cardinality appropriate index (LF or HG)
![Page 53: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/53.jpg)
53
Data Model Recommendations –Primary Keys
Multi-column primary keys should have an additional LF or HG index placed on each individual column• Must be done manually
UNIQUE CONSTRAINT, UNIQUE HG, and Primary Key are identical structures: HG index with no G-Array
Use when possible• Helps optimizer make more informed query path decisions even if
index is not used in joins or searches
• You get an HG Index created automatically
• This HG index has no G-Array (uses less space)
![Page 54: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/54.jpg)
54
Data Model Recommendations –Foreign Keys
Should be used with IQ 12.5 and later to aid in query join performance
Use when possible• Helps optimizer make more informed query path decisions even if
index is not used in joins or searches
• You get an HG Index created automatically on the foreign key column in the current table
• Requires that a primary key exist on referenced table
![Page 55: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/55.jpg)
55
Data Model Recommendations –Temporary Tables
3 Types of Temporary Tables• “#” tables
• create table #temp table ( col1 int )
• Local Temporary Tables
• declare local temporary table temp table ( col1 int )
• Behave just like “#” tables
• Global Temporary Tables
• create global temporary table temp table ( col1 int )
• Tables structure is static across connections and reboots
![Page 56: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/56.jpg)
56
Data Model Recommendations –Temporary Tables
On Commit Preserve Rows• Use this option so that rows in temporary tables remain after the
transaction has been committed
Temporary tables are available at the current level (parent) and all of its children
A parent cannot see a child's temporary table(s)
![Page 57: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/57.jpg)
57
Data Model Recommendations – Cursors
Avoid using cursors• Generally means row based processing
• IQ was designed for set based processing
• Sometimes they cannot be avoided
• If used, make sure to use NO SCROLL cursors
Open With Hold• Allows the cursor to remain open across transactions
• If not used, the cursor may be closed when a commit is issued (depends on connectivity type)
![Page 58: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/58.jpg)
58
Database Programming Language –Watcom SQL vs. T-SQL
IQ (ASA) is not 100% T-SQL Compatible, but very close Recommend using Watcom SQL
• All system procedures written with it• Many more code examples and more IQ people versed in it
Watcom SQL has some extensions that T-SQL does not:• Dynamic SQL• Better Loop control• Full cursor movement rather than just read next
Batches and procedures must be written in the same dialect• Cannot mix T-SQL with Watcom SQL
![Page 59: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/59.jpg)
59
Database Programming Language –Watcom SQL vs. T-SQL
Behavior differences include: Global variables
Variable Names CALL FOR ASA requires variables
to be declared immediately after a BEGIN
DECLARE CURSOR GOTO IF PRINT RAISERROR SET
WHILE (T-SQL) vs. LOOP
![Page 60: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/60.jpg)
60
Database Programming Language –Commit and Rollback
Use transaction control around logical units of work, even read only queries
Should commit before a read/write batch is started to ensure latest version of data is available
Should issue commit and rollback after batch completion to release all query resources
Rollback will free memory resources in use by previous operations
For systems with high number of connected users, freeing memory resources can aid in query performance
![Page 61: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/61.jpg)
61
Database Programming Language –Custom Functions
Custom functions can be written in either SQL or Java Great way to encapsulate business logic for
transforming data Can have a significant performance impact on queries Functions are executed in the catalog portion of the
engine• All result rows may need to be moved to ASA
• Can be time consuming for large result sets
Turn on query plans to see what impact the functions have on effective query plans
![Page 62: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/62.jpg)
62
Database Programming Language –Outer Joins
T-SQL Outer Join: *= or =* ANSI Standard/Watcom SQL Outer Join: [left|right|full] outer join Be careful of outer join syntax The T-SQL syntax can be very ambiguous or non-deterministic for
IQ to translate• All T-SQL outer joins must be converted to ANSI outer joins and then
processed
Use the ANSI standard instead as they are not ambiguous and are always clear in their meaning
Visit http://my.sybase.com/detail?id=1017447 for more details
![Page 63: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/63.jpg)
63
Data Manipulation –Load Table
Parallel Load Table• Make sure to put column delimiter after last column
• Must use ROW DELIMITED BY and DELIMITED BY options in load table command
• Column and row delimiters must be a single character
• 1 to 4 byte delimiters allowed for serial loads
• Parallel load can only have a single character delimiter
![Page 64: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/64.jpg)
64
Data Manipulation –Load Table
If possible, perform load table from binary datafiles• Can be 3 to 10 times faster than ASCII loads
Can use FILLER() clause with a delimiter or byte count Better performance achieved by casting the date or
datetime formats rather than letting IQ guess If possible, issue a single load table with multiple files
rather than 1 load table per file to be loaded into a table• Only important for tables with HG, Unique HG, or Primary Key
indexes
![Page 65: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/65.jpg)
65
Data Manipulation –Insert From Location
Great way to move data from any Open Client source Syntax:
• insert into TABLE()
• location ‘SERVERNAME.DBNAME’
• { select statement };
IQ username and login must match on remote system Interfaces entry must match the SERVERNAME Can also be used to move data quickly from an ASA
table to an IQ in the same server See section on Open Client datatype limitations
![Page 66: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/66.jpg)
66
Data Manipulation –Single Row Operations
Avoid at all costs for large data manipulation operations
Different from single statement operations that modify many rows
Individual INSERT … VALUES() will be slower than bulk load operations• Expect no more than 5,000 to 20,000 operations per hour
![Page 67: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/67.jpg)
67
Data Manipulation –Single Row Operations
If multiple single row inserts are needed there is a method to improve performance
Consider this code (1 insert per row):• insert into #my_table values ( 1, 1 )
• insert into #my_table values ( 2, 1 )
• insert into #my_table values ( 3, 1 )
Can be converted to this (always 2 inserts):• create table #iq_dummy ( a1 int )
• insert into #iq_dummy values ( 1 )
• insert into #my_table select 1, 1 from #iq_dummy
• union all select 2, 1 from #iq_dummy
• union all select 3, 1 from #iq_dummy
![Page 68: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/68.jpg)
68
Data Manipulation –Named Pipes and Flat Files
Named pipes can be faster – no disk I/O How to make a named pipe on most flavors of UNIX:
mknod PIPE_NAME p
Can use BCP, GZIP, UNCOMPRESS, or applications to push IQ formatted data into named pipe
LOAD TABLE command can read from named pipe Can also fast extract data to named pipes so that they
can be read by another application or even compressed and stored
![Page 69: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/69.jpg)
69
Data Manipulation –Partial Width Inserts
Can induce fragmentation and overly large space consumption if not watched
Row ID’s from deleted data are not reused during a partial width insert operation
Space from delete data is not reused (because Row ID’s are not reused)
Partial width inserts are analogous to APPEND_LOAD=‘on’ in terms of Row ID and space behavior
Only becomes a problem if partial width inserts are a way of life for a table
![Page 70: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/70.jpg)
70
Data Loading –The Deep Fact Table
Many databases have a “deep” table• A growing table with tens of millions of rows
• Rows typically ‘rolled out’ and replaced over time
HG indexes can be slow to load/delete• Loading time may increase as data added, but it depends
Solution (with a caveat)• Partition the table (Example – time: day, week, month)
• Build a view that is a Union of all the tables
![Page 71: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/71.jpg)
71
Data Loading –Partitioned Fact Table
Big Fact Table
t1
t2
t3
Partitioned
Create View bigtable as
Select * from t1Union AllSelect * from t2Union AllSelect * from t3
Accessed by View
![Page 72: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/72.jpg)
72
Data Loading – Why Partitioning?
Loads are faster and predictable• x million rows will load consistently
IQ may process the Union All in parallel• As long as CPU resources are available
To roll data out, truncate one table• Truncate table is much faster than Delete
• No changes to DDL required
• Load new data into the empty (truncated) table
![Page 73: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/73.jpg)
73
Data Extraction
For fastest data unloads use the TEMP_EXTRACT options
Can unload data in ASCII or BINARY format• Recommend BINARY format as it is faster to reload into IQ
Great way to archive portions of the database Can unload to one or more files, serially Avoid using ISQL or DBISQL redirection to a file
• Much slower than fast unloads
![Page 74: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/74.jpg)
74
Questions?
Best Practices for Implementing with Adaptive Server IQ
Th
![Page 75: Download File](https://reader036.fdocuments.in/reader036/viewer/2022062312/5565f85dd8b42a20158b5236/html5/thumbnails/75.jpg)
BID209: Best Practices for Implementing with Sybase IQ
Mark D. Mumy
Principal Systems Consultant
April 12, 2023