Making the Most of the M LanguageReal World Power Query for Excel & Power BI

Greg McMurrayBusiness Solutions Developer

First - Thanks to our Sponsors!Yearly Partners

Gold Sponsors

Silver SponsorsBlogger Sponsors: @SQLvariant

Greg McMurray Currently Business Solutions Developer at WECC Manager at Aritus Computer Services for 18 years Aerospace, Branding & Marketing, Energy, Healthcare,

Software Active in many local user groups Find me online

@goyuix https://www.linkedin.com/in/goyuix https://stackoverflow.com/cv/goyuix - top 3% of usersSLC SQL Saturday #57618-Nov-

20163 |

Western Electricity Coordinating Council (WECC) Non-profit to ensure the reliability of the western

interconnection Approved Regional Entity by Federal Energy

Regulatory Commission Create, monitor & enforce reliability standards Publish various models and independent perspective Covers 14 western states, 2 Canadian provinces and

Baja Mexico We are hiring! https://www.wecc.biz/careersSLC SQL Saturday #57618-Nov-

20164 |

Agenda Power Query and M language introduction Real World Examples and Gotchas

HTML: Remove header, adjust columns, mashup CSV: Filter and Replace JSON: Expand properties JSON: List expansion gone wrong – then right OData: Expand columns, format date/time

SLC SQL Saturday #57618-Nov-20165 |

Where does this fit in? Power to the People

Power Query: Discover, import, and reshape Power Pivot: Analyze, define, and measure Power View: Visualize Power Map: Geospatial

New products (Power BI) and partnerships (ESRI) are replacing older parts of data stack

SLC SQL Saturday #57618-Nov-20166 |

Power Query – Tech Ed 2014

SLC SQL Saturday #57618-Nov-20167 |

Power Query Intro Informally known as “M” language Targets info workers and data analysts

Needed to be familiar, easy to read & remember Limited syntax – but similar flavor to Excel formulas Functional, case sensitive language similar to F#

Query Folding: Push work back to the server where appropriate

Formula Reference:https://msdn.microsoft.com/en-us/library/mt211003.aspx

SLC SQL Saturday #57618-Nov-20168 |

Some Basics of Power Query Primitive Values – your scalar values

Examples: Binary, Dates, Logical, Number, Text Structured Values – your compound objects

Examples: List, Record, Table Expressions – your formulas

Examples: let and in keywords Functions – your custom logic

Examples:let Add = (p1,p2) => (p1 + p2) in Addlet Source = Add(3,5) in SourceSLC SQL Saturday #57618-Nov-

20169 |

Lists, Records and Tables Usually Power Query figures out the right structure For Programmers:

Lists are like arrays Records are like name/value pair objects

You can define them as needed as well List: {1, 2, 3} Record: [OrderID = 123, CustomerID = 321] Table: #table( {Col1, Col2}, { {1, 2}, {2, 3} } )

SLC SQL Saturday #57618-Nov-201610 |

Real World Example: HTML Table Get HTML table results from 2012 U.S. Presidential Elections

https://www.archives.gov/federal-register/electoral-college/2012/popular-vote.html Skip header row Remove other parties columns Rename long column names ProTip: I can easily go back to modify a step (remove Total

Votes) Mashup / Combine via Table.Join – GUI shows it as Merge

QueriesCSV from https://statetable.com/SLC SQL Saturday #57618-Nov-

201611 |

Real World Example: CSV Retrieve last 90 days of phone call log Filter out bogus rows Eliminate outgoing calls Give friendly names to numbers ProTip: Notice that many library functions also take

functions as arguments. Example:Table.AddColumn(#”Step”, “NewCol”, each MyFunc([src]))

SLC SQL Saturday #57618-Nov-201612 |

Real World Example: JSON from Web Expand JSON document graph to get to data JSON Objects are treated as Records JSON Arrays are treated as Lists Records expand a single property at a time

Need to convert List of Records to a Table Finally expand columns from Table of Records ProTip: jsonlint.com is a great resource

SLC SQL Saturday #57618-Nov-201613 |

Real World Example: JSON Dataset Get energy data through Quandl from EIA

https://www.quandl.com/api/v3/datasets/EIA/INTL_116_7_USA_MK_A.json?start_date=2007-12-31

Navigate document properties Convert data property from List to Table Uh oh: Table of lists – we can’t work with this… can we?

Expanding the columns creates new rows Expanding a List object just gives me those values

Solution: Add new calculated columnsUse List.First, List.Skip to get specific values from List

Real World Example: OData from SharePoint Retrieve list of meetings for this year

https://www.wecc.biz/_vti_bin/ListData.svc Choose meetings list Expand list of associated committees Add a new column that is the Weekday representation

of dateDateTime.ToText(DateTime.FromText([Source]), "dddd"))

SLC SQL Saturday #57618-Nov-201615 |

Want to Learn More? Reminders:

Power BI does monthly drops which often include new query features Office 365 Excel isn’t quite as fast, but still very regular updates

Channel 9 has some great videos & slideshttps://channel9.msdn.com/Search?term=power%20query

Keep a copy of the language spec handyhttps://msdn.microsoft.com/en-us/library/mt211003.aspx

Power BI Communityhttp://community.powerbi.com/

Microsoft Tech Community – Excel & BI and Data Analysishttps://techcommunity.microsoft.com/

SLC SQL Saturday #57618-Nov-201616 |

Questions and Discussion Did I introduce everything you hoped to learn today? Would you like any clarifications on something we

covered? Share with us: How are you going to apply this is your

world? Feel free to reach out to me online:

Twitter: @goyuix LinkedIn: https://www.linkedin.com/in/goyuix

SLC SQL Saturday #57618-Nov-201617 |