FMK2014 FileMaker Performance Under the Hood by Jon Thatcher

Performance - Under The Hood

Jon Thatcher FileMaker, Inc.

FileMaker Konferenz 2014 Winterthur !www.filemaker-konferenz.com

Jon Thatcher Performance - Under The Hood

http://www.filemaker-konferenz.com

• Lead Software Engineer on Database Server

• 25 years at FileMaker (and Claris) • Directed development of Draco engine, FileMaker Pro and

Server 7

• Helped Clay Maeckel ship the first FileMaker Server in 1994

• Worked on FileMaker Pro starting in 1993

• Previous experience at Intel, Convergent Technologies, and Esvel (database startup)

Who is Jon Thatcher?




• Under The Hood • How and when FileMaker moves and caches data

• Some Performance Dos and Don’ts

• Measuring performance in your own solutions • When and how to measure performance

• Real-world performance diagnosis examples

Performance - Under The Hood: Agenda




• Data read the first time it is needed • Read from disk into Server RAM cache

• Read from Server into the client RAM cache / temporary file

• Data uploaded from clients to Server temp file • Only Server commits data from temp file to main database

• Result: network failure during upload from client has no impact on the main database

How FileMaker moves data



Will talk more about the client temporary file in a bit…


How FileMaker moves data



Main DB

Client RAM

Host

Client

Temp FileRead / WriteRead / Write

Rea

d / W

rite

FMS RAMRead / W

rite

Read / Write

Temp File

Read / Write


• Whole record, script or layout is moved at once • All stored fields* of a record are moved at once

• All steps in a script, all objects* on layout moved at once

• *Except for container fields or layout images

• Only the container or image key in library is moved

• Unstored fields are never downloaded or uploaded

• Tip: “wide” tables, complex layouts and huge scripts are slow to load over WAN

FileMaker moves data in whole units



Unstored calculation and summary fields are just that, unstored.


“Demo” opening a wide table over WAN



Not a real demo, but the time taken to open this database over the WAN is simulated with animation here.


5 seconds

“Demo” opening a wide table over WAN



10 seconds15 seconds20 seconds

Not a real demo, but the time taken to open this database over the WAN is simulated with animation here.


What is making each record so big?



Same detail view shown by clicking TotalLen column in either version of solution, note FullText field with the scroll bar. That is where the long text is, 16KB to 85KB per record.


Wide table, notice FullText and FullLen



And just in case, here are the field definitions for the “Wide” table. Note FullText, FullLen, and that FullLen is included in TotalLen calculation.


Narrow table with 1:1 relationship instead



Use the ArticleID unique field to make a 1:1 relationship to separate table containing just the FullText and FullLen, so all that data doesn’t have to be downloaded for any layout.


Opening the narrow table over WAN




• Use Server Statistics to view Kbytes In / Out • Can use Server values if you are the only user

• Enable Statistics logging to capture Stats.log • Good idea to always have this enabled on Server

• For even better detail, use Client Statistics • See Bytes In from and Bytes Out to each active client

• Use checkbox in Statistics view to capture the ClientStats.log while panel open

How to verify results?




Tip: Use narrow tables



Narrow

Wide

Note that the number of remote calls was exactly the same, since only one remote call needed to download all of record X, it’s just that the “width” of record X was much greater, making the total bytes sent out much higher. !I will talk more about remote calls in second half of presentation.


• Whole record is also uploaded at once when client commits a change, so…

• Tip: put frequently changing fields like InventoryCount in table separate from rest of Product data (type, name, description) • Smaller, faster upload when committing change

• Smaller, faster download when displaying new count

What about frequently changing data?




Tip: Frequently changed data in own table



Whole record copied to host whenever inventoryCount is updated; very expensive if productLongHTML is large

Slow

Faster


• Multiple records may be fetched at once by client, reducing number of download requests • Form View: 25 records

• List or Table View: count of visible records + 2

• Portal: count of visible portal rows

• List/Table View with many rows and columns can be slow to load, especially over WAN

Client loads records based on view



Yet another reason to use narrow tables where possible


• Have the user find just the records they want to work with, instead of showing all records

• Speeds up display of large list or table views

• Especially important with a view that is sorted or includes a summary • Why? …

Tip: Manage the user’s found set




• Having a view automatically sort or summarize requires that ALL records in the found set be downloaded to perform the sort / aggregate

• If user can enter such a view with large found set or showing all records, the sort or summary may take a very long time and make user very unhappy

Tip: Use sorted / summary views carefully




• To limit the found set, Enter Find Mode before going to layout of a large table:

Enter Find Mode [] Go to Layout [ “Customer (customers)” ]

• This will avoid hidden download of the first 25 records (or N + 2 records for List/Table view), which the user may never want to see

Tip: Enter Find Mode first




• Multiple child records will load at same time as parent record if portal is showing on the layout

• Tip: to avoid downloading child records, • Relocate portal to a layout where it will only be shown

when it is needed, or

• Place portal on the second panel of a Slide Control so it isn’t visible by default

Tip: Load portal rows only when needed




• Import: up to 1000

• Sort / Export / Summary: 5000

• Replace: 500

• Relookup: 100

• Delete: 100

Bulk operation: many records at once




• For bulk operations, use Perform Script on Server to have Server do the work for you

• Especially useful when operating across all records in a large table or when find can easily be done on Server via script parameter(s)

• Avoids having to copy all that data to the client, a huge win for clients on the WAN

Tip: Use Perform Script on Server




• RAM cache stores all blocks read from disk • When block modified in RAM, immediately put on flush list

• Whole flush list written to disk once per second

• Block N will stay in the RAM cache until the cache gets full, then when block N is the "least recently used" some other block takes its place in the cache

• Tip: if Cache Hit % statistic stays below 100 • Increase size of Database RAM Cache on the Server

• Archive old little-used data to avoid users accessing it

Server data caching




• Client stores all downloaded data in a temporary file (one per database file), and keeps most recently used blocks in RAM cache

• Downloaded data kept cached locally until: • another user modifies the record (or script, layout, etc)

• the file is closed

• the temporary file grows too large

Client data caching




• Tip: If a user is sorting or summarizing large data sets (>100s of MB), try increasing RAM cache in FM Pro preferences for that user

• Tip: Make sure clients have plenty of free disk space, or temporary file size will be limited, and client may download same data over and over, slowing performance for everyone

Client caching tips




• Under The Hood • How and when FileMaker moves and caches data

• Some Performance Dos and Don’ts

• Measuring performance in your own solutions • When and how to measure performance

• Real-world performance diagnosis examples

Performance - Under The Hood: Agenda




• Avoid premature optimization • Without studying your solution’s usage, guesses about

what is slow are just guesses

• If solution is simple and there will be few users, just make the solution available

• When slowness is reported, measure the operations users are actually performing

Performance - When to measure




• If solution business-critical or has many users, measure before deploying • If at all possible, use actual data of appropriate size

• Otherwise, use randomized actual data or randomly generated data if you must

Performance - When to measure




• Use database of actual size, otherwise testing will miss issues • If solution will hold 10GB of active data, test with 10GB

• Use similar hardware for hosting and testing (especially CPU count/speed, RAM size and disk speed)

Performance - How to measure




• If you can’t use actual data, try randomized actual data • Randomly reassign first names to last names, randomize

all street/phone/ID numbers, etc.

• Randomly generated data WON’T have the same distribution as your real data • Real data has clusters of names, postal codes, cities,

countries, etc., which impacts indexes, finds and sorting

• All that said, generatedata.com is your friend

Performance - How to measure



generatedata.com is pretty useful, and generates up to 5000 random rows at a time…


Set Variable [ $start; Value:GetAsNumber(Get(CurrentTime)) ]Perform Script [ “NiceLongScript” ] Set Variable [ $end; Value:GetAsNumber(Get(CurrentTime)) ]Show Custom Dialog [ Title: "test time"; Message: "Seconds: " & GetAsText( $end - $start ); Default Button: “OK”, Commit: “No” ]

How NOT to measure script run time




!

!

!

!

• This timing can randomly return +/- 1 second!

• Plus, $end < $start if test run at midnight or at daylight time change

Get(CurrentTime) is inaccurate



“actual” time “actual” diff Get(CurrentTime) Get(Curr..) diff

start 50000.999 - 50000 -

point A 50001.001 0.002 50001 1point B 50001.999 0.998 50001 0

But in FileMaker 12 or 13 you can use…


• UTC = Coordinated Universal Time (aka GMT) • NO changes due to daylight savings time, so doesn’t jump

by an hour seasonally

• Doesn’t “roll over” at midnight (does in 580 million years)

• Millisecond accuracy (1/1000th of a second)

• Available in FM13, and FM12 as Get(UTCmSecs) • Same function in 13 and 12, just with a different name,

and “hidden” in 12

• Returns a Number, so no GetAsNumber needed

Use Get(CurrentTimeUTCMilliseconds)



P.S.: No, I don’t like the new name either. New name too long, old name too short


Set Variable [ $start; Value:Get(CurrentTimeUTCMilliseconds) ]Perform Script [ “NiceLongScript” ] Set Variable [ $end; Value:Get(CurrentTimeUTCMilliseconds) ]Show Custom Dialog [ Title: "test time"; Message: "Seconds: " & ( $end - $start ) / 1000; Default Button: “OK”, Commit: “No” ]

The right way to measure




• Customer says their database takes 3 minutes to open on WAN (actually an example db)

• Using WiFi + VPN, took 5 minutes to open!

Diagnose “real world” performance issue



Turns out customer got this file from a FileMaker tips site, where it was NOT intended to show good performance! http://filemakerhacks.com/2014/01/19/summary-list-fields-in-fm-13-part-1/


• Server Statistics view, Clients tab shows work Server did for client:

!

!

• 3500 remote calls to open file!?

Step #1 - Use easiest tool(s) for diagnosis!



Note that all those calls only took 1 second over LAN… So the time to open was still slightly noticeable even over Gigabit Ethernet, but nothing compared to WAN time.


• A remote call is one request from a FileMaker client to a FileMaker host, like: • Get list of hosted files

• Download layout ID 1

• Download records with IDs 5,7,9… from table Y

• Upload and perform query

• A remote call is always one or more network packets (remote call maximum size is ~1MB) • Packet size is usually 1500 bytes

An aside: what is a remote call?




• Use Manage Database, Tables tab to see how many tables + records:

!

!

!

• Under 10000 records total. How can this be so slow?

Step #2 - How large is the data?




• Use Manage Database, Relationships tab to see complexity:

!

!

!

!

• Nice and simple…

Step #3 - How complex is the solution?




• Summary or Aggregate functions are a common cause of slowness…

!

!

• Header with summary of $ across multiple invoices and customers looks “interesting”, what is in that locked grouped object with <…>?

Step #4 - Check for likely culprits




• Copied highlighted text and pasted elsewhere to find:

• In Manage Database, Fields tab for Customers this turns out to be:

!

• Unstored calculation Sum( cfs_invoices::total), and total is…?

Step #4 - Down the rabbit hole!




• Unstored calculation Sum( cfs_invoices::total) =>

• total: Unstored calculation, Sum( line_items::ext_price ) =>

• And line_items::ext_price is a stored value

• Join from 7 customers to 44 invoices to 312 line_items, retrieve >360 records, then Sum ext_price per invoice, Sum total per customer

• Looks like the cause of a lot of round trips to Server!

Step #4 continued - Keep digging!



Why cfs_invoices instead of Invoices? Does that impact performance? Quick answer is no, the performance was the same, whichever TO was used.


• Confirm the culprit by just removing it from the layout and re-test

• Or try a fix, change invoices total from unstored Calculation to stored Number field:

Step #5 - Confirm the culprit then fix it



This is probably more like how you would want your invoice total to behave in any case: the final total should get calculated only when the invoice is marked final, not 3 months later when the price of the item changes! !Using negative value to mark the non-final invoice total is just a hack. Actually, you might want to keep the old unstored calc as “totalEstimate” for display only in the Invoice layout, plus the final invoice total as calculated above or as ( isFinal * … ) so the total would be ZERO until marked final.


• Do exactly the same test as before and compare the Client stats:

• 45 remote calls versus 3541. Wow!

• Opening database over hotel WiFi + VPN went from 295 to 7 seconds

Step #6 - Verify the result




• Latency = time delay in the network between request and response

• Multiply latency by # of remote calls to roughly estimate total overhead:

Rule: always plan for the WAN



Network Latency (ms) # Calls OverheadGigabit LAN 1 3500 3½ secCorporate LAN 10 3500 35 secWAN (Internet) 100+ 3500 350 sec

This is a rule, not a tip, because even if your solution may never be used on the WAN, making your solution perform better on the WAN will save significant time for users on the LAN, too! !Overhead meaning time it will take just to send and receive all the packets, separate from any processing done by FileMaker Server or client or even the number of bytes being sent


• Customer reports Import taking too long • Data from external system requires FMP updates hourly

• But each Import process is taking 30 minutes!

• Didn’t figure out the problem at first because I skipped a step in diagnosis

A true real world diagnosis




• Recalculating price of today’s shipments, based on updated surcharge from trucking carrier

• This script took 7 minutes over slow LAN

Measure slowest part




• This part of solution was really simple!

How complex?




• Steps of script were also simple:

Likely culprits?




Use client statistics!



Why is there more data going IN to Server than OUT??? !And why are there 20000 remote calls?


• Only 20 carrier records, but 10000 trucking

• So Replace Field Contents is doing 1 join for every trucking record, or 10000 joins

• Plus 10000 replaces => 20000 remote calls

I missed: How large is the data?




• Fast:

• Slow:

Fix




• Fix: 216 seconds

• Original: 427 seconds

• Remote calls and time halved (only 20 joins)!

Measure the fix



FYI: the “RelatedLoop” is a script I did with Go To Related Record to prove that the results were the same as using the Replace Field Contents from the “carriers” context !And you can see that cutting the joins also cut in half the amount of data being sent to server (Net Bytes In).


• Why are there still >10000 remote calls? • Would initial forced download of all trucking records allow

more records to be packed in one remote call?

• Is the process doing more work than needed? • E.g., did all carrier surcharges actually change today?

• If not, find only the changed carrier records, then perform the Replace Field Contents from “carrier” context, and it will update only the related records of the found set

Take-home work: what else to try here?




1. Use easiest tools for measuring and diagnosis!

2. How large is the data?

3. How complex is the solution?

4. Check for likely culprits

5. Confirm the culprit then fix it

6. Verify the result

Steps to diagnose performance problems




• Solve the real user problem • Avoid premature optimization

• Measure to find the real problem, then fix!

• Consistently use all the diagnostic steps • Helps you find the real problem sooner

• Plan for the WAN

Performance - Summary





Vielen Dank unseren Sponsoren

Danke für das Bewerten dieses Vortrages!FileMaker Konferenz 2014 Winterthur !

www.filemaker-konferenz.com

http://http.//www.filemaker-magazin.de

http://www.guenther-bs.de/

http://www.filemaker.de

http://camelcase.com/

http://www.ingeno.ch/

http://walkingtoweb.com/

http://www.troi.com/

http://www.karo.at/index.php/fmk2013

http://www.monkeybreadsoftware.de/filemaker/

http://rem-software.ch/

http://www.cross-solution.com/


http://www.klemenso.de/

FMK2014 FileMaker Performance Under the Hood by Jon Thatcher

Technology

Transcript of FMK2014 FileMaker Performance Under the Hood by Jon Thatcher