Transwarp Data Cloud · 2019-12-05 · 02 The data age has witnessed the transformation of...
Transcript of Transwarp Data Cloud · 2019-12-05 · 02 The data age has witnessed the transformation of...
Whitepaper
Transwarp Data Cloud
Transwarp Technology (Shanghai) Co., Ltd.
Stable, Efficient, and Intelligent Data Application Cloud Platform
01
Stable, Efficient, and Intelligent Data Application Cloud Platform
Integration of Big Data + Cloud + AI
Big data and cloud computing technologies have entered the second decade of rapid growth. As artificial intelligence technology develops and prospers, the integration of the three technologies has become a popular exploration trend. The growth of container and virtualization technology has brought big data and AI with infinite resources and computing capability. Big data and AI have extended our understanding of data scope and depth, and they also extend the application of clouds.
Recently, most Fortune 50 companies have published their plans to migrate applications to clouds. A number of start-ups have already built their IT architecture on the cloud since the very beginning and realized cloud-native applications as well. At the same time, well-known public cloud servers at home and abroad have made a deeper exploration into intelligent scenarios. They rely on big data and AI to provoke the next round of business revolution and are making every attempt to improve the way in which enterprises process big data with the support of cloud.
Against the backdrop of data cloudization, Transwarp Technology, as the leader in the development of big data technology productization, has been studying the integration of cloud computing, big data, and AI. With rich experience in big data technology, Transwarp has developed a container platform specially designed for big data. Now, a data application cloud platform product, Transwarp Data Cloud, has come into being. This platform integrates big data + cloud + AI and renders an ecosystem of data + application.
02
The data age has witnessed the transformation of enterprises, represented by Google, Facebook, and Amazon, from IT magnates to DT magnates. With technological advantages in big data, cloud computing, and AI, these enterprises are way ahead in developing data business, data assetization, and enterprise operation datafication, which speeds up the transformation of business value as well. As a result, they not only become tech leads in the industry but also achieve great business success. These big companies have evolved and transformed in several years, which can be summarized as the following.
In this phase, enterprises need to build a flexible technology platform to support large data volumes, ultra-huge data dimensions, and diverse data types. Data unification consists of building a unified computing export platform, unified metadata management and data standard, as well as data integration into the platform.
When data are unified, data integration and assetization need to be realized through data analysis. At the same time, data quality and validity should be ensured by effective data quality management. The more high-quality data accumulated on the platform, the more developers would be attracted to start the data assetization work according to the data features, which includes connecting data and business glossary as well as data management process. As a result, primitive data are converted into valuable data assets.
When data are unified and assetized, an enterprise would have a powerful computing capability and rich data assets, which makes it convenient to build data business. Currently, typical data businesses that can produce huge values are in the fields of operation datafication, intelligent applications, and online data services. These businesses integrate big data and AI technology effectively so as to quickly extract the values from massive data.
In this phase, the enterprise has created a unified platform for data, computing, and business, and a larger number of developers can conduct self-service business development on this platform. Meanwhile, businesses produce new data and assets, which attracts developers to build new business. In this sense, data, business, and developers have formed positive feedback in a complete data ecosystem.As the technologies of big data, cloud, and AI develop rapidly, although enterprises may not exactly follow the four phases above in the process of business evolution and each phase may overlap, repeat, or iterate, the technological evolution based on the four phases will surely become more mature in terms of technology and business and best suit the data-intensive strategy for enterprises.
Data Unification
Data Assetization
Data Business
Data Ecologicalization
•
•
•
Data Integration
Data Quality Management
Assetization & Measurement
Data Assetization
Operation Datafication
Intelligent Applications
•
•
• Online Data Services
Data Business
•
•
•
Closed-Loop Data Domain & Business
Operational Data
Service & Application Sharing
Data Ecologicalization
DataUnification
How does the data business of big companies evolve?
•
•
•
Unified Data Processing
Unified Metadata
Unified Computing Platform
Stable, Efficient, and Intelligent Data Application Cloud Platform
Introduction to Transwarp Data CloudIn order to help enterprises with the evolution of data business, Transwarp Technology makes use of its technological advantages in big data (big data platform TDH), container cloud platform (cloud operating system TCOS), and AI (AI platform Sophon) to develop a new-generation intelligent big data cloud platform Transwarp Data Cloud (referred to as TDC hereinafter), which provides enterprises with an efficient infrastructure technology platform, empowers department-oriented business, and helps enterprises with digital transformation.
TDC integrates three kinds of PaaS platforms, Analytics PaaS, Database PaaS, and Application PaaS; the underlying layer shares the resources on the IaaS platform. It helps enterprises to overcome the difficulties in collaboration data analysis, application development normalization, stock application governance, efficiency management, and solves the problems of data management chaos and resource conflict.
TDC Analytics PaaS meets the internal and external requirements for data analysis business, allowing data engineers and scientists to work on the platform at the same time, which facilitates team collaboration. TDC provides multiple analysis platforms across datasets, so that analysts do not need to re-define policies when using different tools or platforms. In this sense, it saves time for re-configuration and parameter adjustment when conducting analysis on different platforms among team members. It also supports systematic and efficient management for models.
03
Analytics PaaS
Finance
Analytics PaaS
Analytical PortalBig Data
Dev ToolkitTranswarp Studio
Data Catalog Application Market Security Center
Dev / Test DevOps
Service GovernanceGraphDatabaseStellarDB
Flash-BasedDatabaseArgoDB
AnalyticalDatabaseInceptor
Real-TimeComputing Engine
Slipstream
Data SearchingSearch
NoSQL DatabaseHyperbase
TransactionalDatabase
KunDB
Data SciencePlatformSophon
Container Service
ImageRepo
Network Service
StorageService
VirtualMachine
Container Orchestration System
Reporting ToolPilot
Database PaaS
Transwarp Cloud Operating System
Application PaaS CMP
CRM OA ERP Analysis Mobile APP
Data Interface Security Interface Resource Interface Microservice Interface
Operation Center
Metrics Monitor
Billing Service
Ticket Tracking
Disaster Recovery
TDC Database PaaS solves the problem of disperse data and isolation and avoids exchange barriers, which enables unified storage of all kinds of data assets and realizes data service open-up as well as data exchange. Data Catalog is provided for comprehensive data governance.
At the same time, TDC Database PaaS consists of a full set of data development tools. It provides a well-equipped data development environment, which lowers the technical requirements, simplifies data development procedures, overcomes the challenges in data development, and addresses efficiency issues, which altogether improve the data development capability.
04
TDC Application PaaS provides plentiful intermediary components and application development platforms, thus solving the efficiency problem in application development, deployment, maintenance, and governance. It helps to remove the barriers in the development and management of all kinds of applications.TDC provides essential mainstream application development tools, microservice framework, and DevOps tools, which effectively standardizes the application development procedures, improves efficiency, and controls quality. At the same time, TDC supports service deployment and service governance and features resource resilient scalability, resource isolation, and fault-tolerance.
Application PaaS
The underlying IaaS platform of TDC solves the efficiency problem in resource management and achieves unified management of system resources horizontally and vertically.It packs all kinds of resources such as computing, storage, and network to services, provides efficient configuration and scheduling, maintains the lifecycle of resources, and monitors the overall allocation conditions. It also provides accurate and reasonable billing services, which improves application maintenance efficiency and speeds up resource scheduling.
IaaS Platform
Procedures to build an enterprise data application center
Old Monolithic Business System Microservice Platform
Application Center Designing Phase
Development & Maintenance Phase
Designing Microservices
Studying Monolithic Applications
Designing Microservices Developing Microservices
Operating Microservices
Platform Publishing
Microservice Split
Planning & Research
111101101101010101011
1111011011010101010110
1111011011010101010110
11110110110101010101
1111011011010101010110
Database PaaS
Stable, Efficient, and Intelligent Data Application Cloud Platform
05
TDC covers the functions of Analytics Cloud, Database Cloud, and Application Cloud, rendering the integration of multiple platforms. TDC connects different cloud platforms and can meet the requirements to build all three kinds of cloud platforms on the same platform. It breaks the boundaries between applications and data, and resource sharing is realized on the underlying IaaS layer. As a result, unified account management is achieved, which avoids departments to build a cloud platform vertically and realizes ecological communication within the enterprise system.
Integrating Analytics Cloud + Database Cloud + Application Cloud
One platform features the capability of three cloud platforms, which avoids different departments to build business systems separately and realizes the communication between applications and data. It helps to promote innovative business and empower the construction of ecosystem.
Integrating Three Clouds
It realizes a unified platform and provides consistent network, security, and maintenance management, so that communication barriers across platforms can be avoided, and there is no need to switch between different platforms in development.
Unifying Platforms
Since the underlying layer adopts a unified container supporting platform, three cloud platforms can share resources. Resources are resilient and can be borrowed from different platforms, which improves the resource usage efficiency greatly.
Resource Resilience
TDC provides unified resource and maintenance management for the three cloud platforms. No additional third-party cloud management platform needs to be introduced, which reduces the IT costs.
Easy Maintenance
Why Transwarp Data Cloud?
Cloud Platform Portal
Infrastructure as a Service
AnalyticsCloud
DatabaseCloud
ApplicationCloud
Network Communication
DataTransmission
PermissionControl
Security Management
Integration & Communication Between Clouds
Security Perm
ission
ResourceM
aintenance
06
Meeting the Requirements for Different Roles in the Company
TDC provides one-stop cloud services covering IaaS+PaaS+SaaS and is a leading cloud platform in the industry which features comprehensive service capability. From tenant & resource management to big data, AI applications, and data analysis, to microservice development, TDC can manage all types of applications and services, which meets the requirements of different roles in a company.
TDC solves the problem of application island and data island, builds a unified application center for data entry and permission control as well as a big data and data asset center, which helps managers to control business and data from a more comprehensive perspective.
For Enterprise Managers
TDC has a built-in application center, which supports self-service installation and deployment of any applications on the cloud, including reporting tool, daily office applications, ERP, and other SaaS services.
For Enterprise Clerks
TDC provides a full set of services from application development, testing, publishing, and maintenance, which consists of a development/testing platform, DevOps tools, intelligent & agile publishing, and metrics monitoring.
For Developers
TDC provides out-of-the-box data development services, which is scalable based on needs. And the graphical development interface frees the engineers from the architecture details, which speeds up business implementation.
For Data Engineers
TDC provides diverse visualized data analysis tools, which helps online data exploration and modeling in business analysis. All models support iteration, collaboration, and publishing.
For Analysts
TDC realizes comprehensive service governance, which ensures system stability and business continuousness and at the same time lessens the workload. It also provides course tracking of maintenance service chain for trouble-shooting.
For Maintenance Staff
BusinessLayer
Data Analysis Platform
Analytical Database NoSQL Database Real-Time Computing
Transactional Database
DevOps Service
Data Storage
Service Governance Center Web Middleware
Business Business Business Business Business Development & TestingPortal
Business Analysis Portal
Data Catalog
AI PlatformSophon
Object StorageObject Storage
DistributedStorageHDFS
Node.jsApplication
Jenkins Gitlab Tomcat
ContainerService
StorageServiceBilling Service Image Market Web Service Virtual Machine Bare Metal
Security / Monitoring /Management / Maintenance
VM-Based SolutionContainer Cloud Platform
ServiceGateway
KONG
ServiceTracking
Zipkin
APP PerformanceMgmt
Pinpoint
LogMgmtMilano
ApplicationServer
ConfigCenter
Hamurapi
Metrics MonitorPrometheus
GoApplication
DataVisualization
Pilot
Flash-BasedDatabaseArgoDB
Data AnalysisAnalytical
Portal
AnalyticalDatabaseInceptor
Search EngineSearch
Real-Time Computing Engine
Slipstream
DistributedDatabase
KunDB
Message Queue Kafka
HA DatabaseTxSQL
In-Memory KVKV Storage
GraphDatabaseStellarDB
NoSQLDatabase
Hyperbase
Big DataDevelopment
Studio
Model Market APP Market Data Service Layer PaaS Interface Layer
Maintenance Portal
InterfaceLayer
DevelopmentService
DataService
ResourceService
SaaS
PaaS
IaaS
Stable, Efficient, and Intelligent Data Application Cloud Platform
07
Features of TDCBig Data Product
Artificial Intelligence Product
TDC Database PaaS provides all kinds of big data services that can be deployed by one-click. These services are supported by the basic components in Transwarp big data ecosystem, which have logics built within automatically to render all functions for common scenarios in data processing.
Build one-stop data warehouses, providing a full set of services such as data integration, processing and analysis, with the purpose of building the core of data.【Application Scenarios】 Data lake, batch processing, enterprise data warehouse, logical data warehouse.
Data Warehouse
Oriented at department-level data analysis businesses, provide OLAP Cube engine and reporting & scheduling tools; support performing interactive analysis and building automated reporting applications.【Application Scenarios】 Self-service reporting, multidimensional analysis, interactive analysis.
Data Mart
PB-level high-speed full-text search, supporting high concurrency, hot/cold data isolation, accurate/fuzzy retrieval, and quick statistics.【Application Scenarios】 Industry search engine, knowledge sharing platform, information retrieval.
Information Retriever
A stream processing platform on cloud for real-time data collection and processing, building real-time data warehouses and applications and discovering the value of data streaming.【Application Scenarios】 Real-time data analysis, online anti-fraud, sensor network analysis, intelligent device detection and failure prediction.
Real-Time Computing
Featuring scalability, high concurrency, and high availability, providing a common solution for building modern databases, and supporting business platforms of all industries.【Application Scenarios】 E-commence platform, financial distributed system, business aggregation center.
Distributed Transactional Database
A database product designed for the future all-flash servers, with delicate storage structure and efficient algorithms designed for the underlying layer.【Application Scenarios】 Public security center, financial analysis platform, targeted marketing system.
Distributed Flash-Based Database
Build an enterprise RDB to process OLTP business, supporting complex SQL query and guaranteeing high stability, scalability, and strong consistency.【Application Scenarios】 Online trading system.
Relational Database
Build a unified enterprise-level AI application platform for data cleansing, data analysis and mining, machine learning, deep learning, model management, API deployment, and workflow scheduling, thus helping enterprises with the innovation and revolution of business in the age of AI.【Application Scenarios】 Data mining and modeling, graphical modeling and feature engineering.
Artificial intelligence Platform
Application Development & Management Platform
08
TDC Application Market has all kinds of built-in applications, including intermediary components as well as tools for visualized analysis, development, and website building, which handles all tasks such as business development, data analysis, efficiency improvement, and effective manage-ment. Users can install applications according to their needs. These out-of-the-box applications do not occupy local resources and enable service sharing with others, which facilitates team collaboration and saves resources.
In addition, as an open platform, TDC Application Market supports developers to release applications at any time, so that all users with granted permissions can get informed in time. In this way, it shortens the application release cycle and improves the efficiency in the process from application design to implementation.
Application Market
TDC Development Platform provides a one-stop application publishing service for application development, release, and management. Users can complete the entire procedures from development to release on the same platform, which shortens the application release cycle. It also supports the management of application lifecycle.
DevOps Tools
Automatic deployment of DevOps tools, including code management, auto-testing, publishing, and deployment, which enables agile development, continuous integration, and continuous delivery. In this sense, a unified development and maintenance process is realized.
Application Publishing
For DevOps tools, self-built applications on other platforms, and open-source services, developers only need to do some simple configuration on the publishing platform so as to release them on the market. TDC supports scientific version management. Applications can have multiple versions and can be maintained individually. Users can feel the version change online in real time after updates.
My Applications
Display a list of all applications created, monitor the release status of all versions, control the release conditions, manage application deployment, and provide log analysis tools for trouble-shooting.
Development Platform
Stable, Efficient, and Intelligent Data Application Cloud Platform
Tenant
…Project 1
…
Project M
…
Tenant
Role1UserGroup Project 1
Data Catalog
Management
09
TDC has a built-in Data Catalog service, which provides reliable, convenient, and intelligent support for enterprise data governance. It helps enterprises to create data map, unify data standard, mark data location, analyze data relationship, manage model change, and improve data quality, so that they can extract and make use of the value more effectively, realize accurate, efficient analysis and decision, promote system change management, and reduce project risks.
TDC management platform focuses on the projects, tenants, and users, achieving reasonable division and management of permissions and resources, and multi-tenancy is supported on a unified management platform. Tenant administrator has the highest tenant management permission, who is responsible for permission management. Product instances are managed and organized in each project as a unit, so as to achieve clear and rational permission management in terms of granularity.
Multi-Tenancy Management
08
07
10
With container technology, TDC features the isolation of applications, data, resources, and execution among multi-tenancy. Tenants running on the same platform are completely transparent to each other, as if they were operating on different infrastructures. In addition, for enterprise private cloud, TDC can provide unified data management for internal tenants. Shared data are placed in the public area while high-value and sensitive data in the sensitive area, thus ensuring unified metadata management and data quality control, supporting unified data lifecycle management, and promoting data assetization.
TDC adopts an accurate billing architecture, which ensures data accuracy and security. At the same time, this architecture features high availability and scalability, and is able to conduct near real-time data computing.TDC platform brings up a reasonable and clear billing standard for users. Pricing items are categorized into four types, which are hardware resource, big data software, data service, and third-party application. At the same time, a variety of billing units are supported so as to form a fair and convenient billing model.
In addition, TDC provides powerful fee management functions, including detailed tenancy bill, unified operation analysis report, account write-off and reconciliation, tenant quota setting, charge items, price setting, and discount rule customization, to release full control of finance management to platform managers.
Accurate Billing
Business Billing Description
Computing resources + storage resources Package Billing: set several billing stalls, charge by the month, and the exceeding parts are billed on demand Accurate Billing: charge by operational events on physical resources
Resource Billing
Software Service Billing
Data Billing
Application Billing
Charge by data services of Transwarp big data and AI platform Pricing: Basic price × Advanced/Standard/Lite configuration
Each data type can be priced individually or collectivelyPackage Billing: data volume VS update frequencyAccurate Billing: provided by applications or services
Sell third-party applications at a fair price Charge by the billing agreement
稳定 、高效、智能的大数据云平台
TDC has built-in maintenance & management services, which realizes unified maintenance of the entire platform and internal management for tenants.
TDC's maintenance & management system has distinguished performance in throughput, security, and availability, which is capable of providing high-quality maintenance services for the cloud platform.Full-Course High Throughput
Maximum TB-level throughput. The number of logs collected per second on a single node reaches tens of thousands; a three-node cluster can collect up to 2 billion logs in one day.Full-Course Security
Log data is encrypted by Kerberos. Logs of different tenants are isolated from each other. Security authentication is needed when users try to access and analyze logs.Full-Course High Availability
Status monitoring is enabled within the system to ensure the high availability of data, prevent data loss and duplication.
Unified Graphical Maintenance Monitoring
Alert Message
Visualized StatusAnalysis
Resource Mgmt& Monitoring
Log Analysis &Retrieval
Stable, Efficient, and Intelligent Data Application Cloud Platform
11
Health Check
Log TimePortrait
Business SolutionsData Sharing Center
稳定 、高效、智能的大数据云平台
TDC can be used to build a data sharing center for multi-tenancy scenarios, which provides fine-grained tenant management. The head office can build a unified platform with all its departments as individual tenants. With the resources and technological capability of the head office, it can realize self-service big data and application development. TDC will conduct unified scheduling and governance for resources and services. TDC supports horizontal and vertical data flow. The head office has a global overview of all the data from lower divisions, and data sharing is enabled across departments. They can obtain the desired data in a self-service manner, which enables more comprehensive and valuable production in data platform construction.
TDC can build a unified Internet big data platform, integrating massive network and user data resources, and realizing unified development & management for applications. TDC Application Aggregation Platform provides development/testing support for developers and the underlying resources are under unified management. According to application requirements, custom services are provided with data in the storage service, intermediary and result data on the mining & analysis platform, and open data on the distribution platform. This platform can deploy third-party applications flexibly, support calling other services via API, collect and manage applications on a large scale, which altogether achieves the maximum data value.
Application Aggregation Platform
08
07
12
Data Collection
Real-TimeData Analysis Platform
Big Data Mining Platform
Typical Applications
Big Data Storage & Management Platform
Big Data Distribution Platform
All Internal/ExternalApplications
Big Data Application Aggregation Platform
Basic Information Resource
Platform PortalOpen Big Data Platform Collaborative Governance
Platform
Oriented at the general public: Oriented at enterprises & governments:
PopulationInformation Library
Legal EntityInformation Library
Basic Information Resource Service platform
Data Sharing Cloud Platform
Department 1 Department 2 Department 3 Department 4 Department 5
Spatial GeographicInformation Library
CreditInformation Library
E-LicenseInformation Library
About UsTranswarp is a world leading big data and AI platform provider, focusing on enterprise level cloud computing on
container, big data and AI core platform research and services. Transwarp is based in Shanghai, with regional
headquarters in Beijing,Guangzhou, Singapore, and support centres in Nanjing, Zhengzhou and Chengdu. Transwarp
also has numerous domestic offices in different provinces and overseas branches in the United States and Canada.
After years of R&D, Transwarp establishes major product lines with many patents: Transwarp Data Cloud (TDC), a
container-based intelligent big data cloud platform; Transwarp Data Hub (TDH), a one-stop big data platform; AI
platform Transwarp Sophon; and Hyper-Converged big data server TxData Appliance. Meanwhile, Transwarp attaches
great importance to knowledge exporting and training on big data and AI, founded the Transwarp University, which
focus on training product R&D, professional certification and training services providing.
In 2016, Transwarp was appraised by Gartner as the world's most visionary vendor in data warehouse and data
management. In 2017, Transwarp was recognized by IDC as the leader of the big data market in China. In 2018,
Transwarp became the first database vendor worldwide to complete TPC-DS testing and pass official audits over the
past 12 years. Transwarp Technology has secured its series D1 funding which is led by TCL Capital and CICC Capital.
Core Technology
Enterprise-grade data warehouse and data mart based on one-stop big data platform;
High-performance and scalable distributed database;
Streaming engine with low-latency event-driven approach and complex batch processing programming model;
AI platform with complete algorithms and practice models for statistics, machine learning and deep learning;
Container based operating system to support big data at production level;
Multi-tenant PaaS platform based on container technology.
Industry Focus
Products of Transwarp have been widely used in more than ten industries, such as finance, government, trans-
portation, manufacturing, telecommunications, logistics, education and so on. Transwarp has more than 1000
clients, including Ministry of Finance of the People's Republic of China, Bank of China, China Post, China
National Petroleum Corporation (CNPC), Airbus, China Unicom, etc.
Selected Customers
Version 2.0
11F&12F&15F, Block B, 9F, Block A, 88 Hongcao Road, Xuhui,Shanghai
86-4007-676-098
www.transwarp.io
@transwarp_data