Introduction to MS-HMSHTTP - Microsoft · 2019-11-19 · SQL Server 2019 new feature: Big Data...
Transcript of Introduction to MS-HMSHTTP - Microsoft · 2019-11-19 · SQL Server 2019 new feature: Big Data...
Introduction to MS-HMSHTTP
Bin Guo
SQL OIPI Test Team
Agenda
Background & Overview
Transport Direction & Stack
Message Structure & Examples
Apache Thrift Generation Demo
Resource
1
2
3
4
5
Protocol Background and Overview
SQL Server 2019 new feature: Big Data Cluster
Apache Hadoop cluster
Big Data Cluster
• Apache Hive
• Hive Metadata Database
Hive Metastore HTTP Server
• Hive MetastoreService
Client
MS-HMSHTTPDerby
DatabaseSQL Server
MS-HMSHTTPTheHiveMetastoreHTTPprotocol
Comparison with Apache Hive IMetaStoreClient
MS-HMSHTTP
LightweightHive
Metastore
Interface
Hive IMetaStoreClient
All kinds of management
functions
Protocol Transport Direction and Layers
Knox, the Gateway in Apache Hadoop
HTTP Post to https://{clusterIp}:{knoxPort}/path
https://ClusterIP:30443/gateway/default/hmshttpthrift/api/hms
Transport Stack (Layers)
Message Structure and Examples
All MS-HMSHTTP Message Operations
• get_all_databases• get_databases• get_database
• get_all_tables• get_tables• get_table• get_tables_by_type
• get_partition_names• get_partitions
Message Data Structures
• Database• Table• Partition
• MetaException• NoSuchObjectException
struct Database {
1: string name,
2: string description,
3: string locationUri,
4: map<string, string> parameters, //
properties associated with the database
5: optional PrincipalPrivilegeSet
privileges,
6: optional string ownerName,
7: optional PrincipalType ownerType
}
Message body in TJSON array format
Position 1
• version
Position 2
• message name
Position 3:
• message type
Position 4
• message seqid
Position 5
• payload
message type
1: CALL 2: REPLY 3: EXCEPTION 4: ONEWAY
Request body Example
Position 1
• version
Position 2
• message name
Position 3:
• message type
Position 4
• message seqid
Position 5
• payload
[1,"get_all_databases",1,1,{}]
Response Body Example
Position 1
• version
Position 2
• message name
Position 3:
• message type
Position 4
• message seqid
Position 5
• payload
[1,"get_all_databases",2,1,{"0":{"lst":["str",2,"default","hmshttptestdatabase"]}}]
Hive Metastore Thrift API signature of get_all_databases()
list<string> get_all_databases() throws(1:MetaException o1)
Thrift in Apache Hive Architecture
Protocol Client and Payload Thrift Generation
MS-HMSHTTP Thrift Appendix
ApacheThrift Tool Chain
A Hive MetastoreThrift HTTP Client
• JavaScript API , PHP API, C# API, …
Thrift Schema in Protocol Appendix
演示 Demo
Protocol client can be created by using the Apache Thrift [ApacheThrift] toolchain to query the Hive Metastore Thrift API.
Demo
On Windows 10
with VS2019
According to
https://thrift.apache.org/
C#/JS HMSHTTP Thrift Client Generation Steps
▅ 1) Download Thrift compiler EXE
▅ 2) Prepare
ThriftHiveMetastore.thrift and
fb303.thrift files
▅ 3) Use VS Developer CMD:
▅ thrift --gen <language> <Thrift
filename>
▅ thrift-0.13.0.exe --gen netstd/JS
hive_metadata.thrift
[MS-HMSHTTP]: Hive Metastore HTTP Protocolhttps://docs.microsoft.com/en-us/openspecs/sql_server_protocols/ms-hmshttp/1f53d4a7-19be-41dc-aecc-f762096f2e92
Apache Hive IMetaStoreClienthttps://hive.apache.org/javadocs/r1.2.2/api/org/apache/hadoop/hive/metastore/IMetaStoreClient.html
[ApacheThrift]https://thrift.apache.org/
What are SQL Server Big Data Clusters? https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-overview?view=sqlallproducts-allversions
SQL-On-Hadoop: Hive - Part Ihttps://www.sqlservercentral.com/articles/sql-on-hadoop-hive-part-i
Resource
Summery
• Protocol Background & Overview
• Transport Direction & Stack
• Message Structure
• Message Example
• Apache Thrift Generation Demo
This is a Sample QR Code, We will replace itwhen the new QR code created.