Post on 20-May-2020
© 2019 KNIME AG. All rights reserved.
Sharing and Deploying Data Science with KNIME Server
Roland Burger
Marten Pfannenschmidt
KNIME
© 2019 KNIME AG. All rights reserved. 2
KNIME Server
• Collaboration– Share Expertise and Templates/Blueprints
• Automation– Schedule, Monitor, Update
• Deployment– To Humans: Analytical Applications (“Guided Analytics”)
– To Machines: via REST and custom APIs
• Management– Manage distributed setups
– Integrate with existing enterprise setup (authentication)
© 2019 KNIME AG. All rights reserved. 3
Delivering Data Science
• Statistics• Mining• AI/ML/DL Models • R/Python Scripts• Legacy Languages
Marketing and Sales Decisions
Financial Decisions
Manufacturing Decisions
IT Decisions
Product R&D Decisions
• Data Access• Data Blending• Transformation• In-database• “In” Big Data
Apps
Data Blending
Data Analytics
Predictive Analytics
Machine Learning
Artificial Intelligence
• Analytics as a service
• Guided Analytics• Analytical Apps &
Dashboards
Data Science
© 2019 KNIME AG. All rights reserved. 4
Creating Data Science: KNIME Analytics Platform
Partner
Extensions
KNIME Analytics Platform
Community
Extensions
KNIME
Extensions
KNIME
Integrations
Data Blending
Data Analytics
Predictive Analytics
Machine Learning
Artificial Intelligence
Data EngineersData Analysts
Data ScientistsML/AI Engineers
App DevelopersData Science
© 2019 KNIME AG. All rights reserved. 5
Productionizing Data Science: KNIME Server
Marketing and Sales
Financial
Manufacturing
IT
Product R&D
DeployManageAutomateCollaborate
KNIME Server
Data Science Teams IT Teams Business Teams Apps
REST API
Web Applications
Mobile Applications
Cloud-based Services
Guided Analytic Apps
© 2019 KNIME AG. All rights reserved. 6
Better Decision-making, Faster!
KNIMEIntegrations
KNIMEAnalytics Platform
KNIME Server
Load > Integrate > Transform > Analyze > Visualize
KNIMEExtensions
CommunityExtensions
PartnerExtensions
Collaborate Automate Manage Deploy
© 2019 KNIME AG. All rights reserved. 7
KNIME Server
Management
Deployment
Configuration &Versioning
RESTful Services
Client & User PreferencesSecurity
Guided Analytic AppsWeb Front-ends
Collaboration
Workflow Hub Workflows
Shared Repository, Versioning
Metanodes, Nodes Data
AutomationDistributed Executors
Remote, Client ControlProvisioning
© 2019 KNIME AG. All rights reserved. 8
• Connect to KNIME Server from local KNIME Analytics Platform– Authentication possible via LDAP
/ Active Directory
– Centralized administration and security
• Interact with KNIME Server:– Upload workflows for sharing and
collaboration with colleagues
– Execute workflows, create schedules, set permissions, …
– Organize your work in workflow groups (directories)
Collaboration
Workflows
Workflow Groups
Data
Metanode Templates
Jobs
Click to add new mountpoint
© 2019 KNIME AG. All rights reserved. 9
• Right-click -> Deploy to Server– Share workflows, data,
etc. for collaboration
– Continuous delivery in all steps of workflow life cycle
Deploy workflows to KNIME Server
© 2019 KNIME AG. All rights reserved. 10
KNIME Workflow Hub
• A place to share knowledge about Workflows.
– Share
– Discover
– Search
– Rate
– Comment
– Version
© 2019 KNIME AG. All rights reserved. 11
Metanode Templates
• Create template from metanode and store on server
• Templates are linked -> check for new version and apply updates automatically
• Increase efficiency through reusable workflow components
• Ensure compliance with company policies
© 2019 KNIME AG. All rights reserved. 12
Remote Execution
• Execute workflows directly on KNIME Server
• Scale beyond limitations of desktop machines
© 2019 KNIME AG. All rights reserved. 13
Scheduled Execution
• Automate workflow execution in regular intervals
• Orchestrate job pipeline by using Call Workflow nodes
© 2019 KNIME AG. All rights reserved. 14
Remote Workflow Editor
• Monitor progress of execution on server (almost) live• Very helpful for longer-running jobs
What‘s my workflow doing right now?
© 2019 KNIME AG. All rights reserved. 15
Remote Workflow Editor
• Change node configurations directly in the remote job– Resume execution if
there was an error
– Try different configuration options without re-executing the full job
– Add nodes, change node connections, all live and in the secure server environment
© 2019 KNIME AG. All rights reserved. 16
KNIME WebPortal
• All workflows on KNIME Server available as web apps
• Step-by-step execution of workflows from any browser
• Simple, clean interface for end users
– Guided Analytics
• Customize layout to match corporate design
© 2019 KNIME AG. All rights reserved. 17
Guided Analytics: Interactive Data Science
Interaction Points
© 2019 KNIME AG. All rights reserved. 18
Inside an Interaction Point
© 2019 KNIME AG. All rights reserved. 19
KNIME Server REST API
• Enables external integration
• Build applications around KNIME Server
• e.g. for Microservices and real time scoring
Output in server responseInput data
Deploy KNIME workflows as web services
© 2019 KNIME AG. All rights reserved. 20
KNIME Server REST API: Job Pool
KNIME Workflow for e.g. sentiment prediction, churn score, chemical property
Pre-loaded jobs pooled and waiting
for work
Workflow immediately
executed
External REST call
No “load latency” for client
Results
Take from pool
Pooled jobsKNIME Executor
KNIME SERVER
© 2019 KNIME AG. All rights reserved. 21
KNIME Server REST API
• Swagger UI: Explore API definition of workflow in browser
• Provides all information needed to interact with API
© 2019 KNIME AG. All rights reserved. 22
KNIME Server REST API
• Programmatically control KNIME Server– Upload/download/delete
resources
• Staging from Dev to Prod
– Upload licenses
– Empty trash/restore items
– Execute workflows
– Schedule jobs
– Set permissions
– Create users & groups, etc.All server functionality available via REST API
© 2019 KNIME AG. All rights reserved. 23
Version Control
• Create snapshots of workflow revisions
• Allow roll-back to previous versions
• Full workflow history available in client and on Workflow Hub
© 2019 KNIME AG. All rights reserved. 24
• Inspect differences between versions of the same workflow
Workflow Difference
Highlight differences:- Nodes included/excluded- Node configurations
© 2019 KNIME AG. All rights reserved. 25
Permission Management
• Manage permissions for workflows and data on KNIME Server
• Control which groups can access, modify, or execute a workflow
• Use LDAP / Active Directory groups for centralized administration
© 2019 KNIME AG. All rights reserved. 26
Managing Preferences – local
KNIME Analytics Platform
KNIME Analytics Platform
KNIME Analytics Platform
How do I install
database drivers?
How do I configure Python?
KNIME WebsiteInternal knowledgebase
© 2019 KNIME AG. All rights reserved. 27
Managing Preferences – KNIME Server
Marketing R&DFinance
Linux and macOSPythonR
Windows 7OracleMS Access
Windows 10HiveSpark
• Different departments/teams have different requirements
• Multiple OS deployments• Windows 7• Windows 10• Linux• macOS
© 2019 KNIME AG. All rights reserved. 28
Managing Preferences – KNIME Server
KNIME Server
Profiles can include: Preferences, drivers, and more
• Client-profiles• Python-Linux• Python-macOS• R-Linux• R-macOS• Databases-Win7• Big Data-Win10
client-profiles
Big Data-Win10
Databases-Win7
Python-macOS (etc)
Marketing Finance R&D
© 2019 KNIME AG. All rights reserved. 29
Client Customization Profiles
© 2019 KNIME AG. All rights reserved. 30
KNIME Server – Admin made easy
© 2019 KNIME AG. All rights reserved. 31
Distributed Executors
• Need more workflow horsepower?
• KNIME Server currently supports ‘Scale Up’
• KNIME Server Distributed Executors allows ‘Scale Out’
https://upload.wikimedia.org/wikipedia/commons/thumb/8/8b/Server.svg/2000px-Server.svg.pnghttps://upload.wikimedia.org/wikipedia/commons/5/53/Server-multiple.svg
© 2019 KNIME AG. All rights reserved. 32
Distributed Executors
KNIME Server
Executor 1
Executor 2
Executor 3
RabbitMQ
KNIME Server + Distributed Executors
Requests to execute
workflows
© 2019 KNIME AG. All rights reserved. 33
Distributed Executors
KNIME Server
Executor 1
RabbitMQ
KNIME Server + Distributed Executors
Request to execute
workflow
Scripted launch on AWS CloudFormation
© 2019 KNIME AG. All rights reserved. 34
Distributed Executors
KNIME Server
Executor 1
Executor 2
RabbitMQ
KNIME Server + Distributed Executors
More requests to
execute workflows
Launching
CPU load increases
AWS Auto-scaling Group
© 2019 KNIME AG. All rights reserved. 35
Distributed Executors
KNIME Server
Executor 1
Executor 2
RabbitMQ
KNIME Server + Distributed Executors
Requests to execute
workflows
AWS Auto-scaling Group
© 2019 KNIME AG. All rights reserved. 36
KNIME Software – On premise and in the cloud
KNIMEAnalytics Platform
KNIME Server
KNIMEAnalytics Platform
KNIME Server
KNIMEAnalytics Platform
KNIME Server
© 2019 KNIME AG. All rights reserved. 37
KNIME Server
Management
Deployment
Configuration &Versioning
RESTful Services
Client & User PreferencesSecurity
Guided Analytic AppsWeb Front-ends
Collaboration
Workflow Hub Workflows
Shared Repository, Versioning
Metanodes, Nodes Data
AutomationDistributed Executors
Remote, Client ControlProvisioning
Ready to get started with KNIME Server? Contact us for a free trial:
contact@knime.com
38© 2019 KNIME AG. All rights reserved.
The KNIME® trademark and logo and OPEN FOR INNOVATION® trademark are used by KNIME AG under license from KNIME GmbH, and are registered in the United States.
KNIME® is also registered in Germany.