Migrate and modernize SFTP file

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Migrate and modernize SFTP file transfer workflows with AWS

S T G 3 3 3

Asa Kalavade

General Manager

Amazon Web Services

Smitha Sriram

Senior Product Manager

Amazon Web Services

Atul Bahl

AVP, Cloud Infrastructure

Verisk Analytics

Douglas Scott

IT Architect

Bose Corporation

Do you build any of the following?

ERP and EDI systems for supply chain logistics

Data Lakes and analytics platforms

IoT services used in remote locations for monitoring

Digital media content aggregation and distribution

CRM applications for various business segments

Subscription based data products

Financial services

$

Retail

Healthcare ..and more

Receive third-party uploads

Distributed data exports

Transfer data internally

Widely used protocol across various industries globally

… and you’ve come to the right session!

Then you probably use SFTP!


Feature deep dive and use cases

Customer case study—Verisk Analytics

Customer case study—Bose Corporation

AWS Transfer overview

Q&A

Closing

AWS transfer for SFTP

Cost-effective

Seamless migration of

existing SFTP workflows

Native integration

with AWS services

SimpleFully managed

in AWS

Secure

and compliant

Launched re:Invent 2018, available in 16 commercial regions

Fully-managed SFTP service enabling transfer of data into Amazon S3

How it works

SFTP users

AWS Cloud

Amazon Simple

Storage Service

(Amazon

S3)

No changes to end-user

credentials, firewall

configurations, or scripts

Custom identity provider

Amazon

API Gateway

AWS

Lambda

Modernize

your workflow

using cloud

native services

Amazon

Athena

Amazon

Redshift

Amazon

EMR

Amazon

SageMaker

Your Amazon Virtual

Private Cloud (Amazon

VPC) or data center

DIY SFTP server

and data

No need to manage

SFTP infrastructure

AWS Transfer for

SFTP

In as simple as 3 steps

Your users can now use your AWS Transfer server to transfer data

Associate your hostname with the server endpoint

Map your hostname1

Create an IAM role to access the S3 bucket(s) used for storing data transferred over SFTP

Select your S3 bucket(s)2

Create and map users toIAM roles to enable them

for file operations

Set up your users3

Thank you!

Support for seamless migration

Same clients, credentials, and hostname

At launch (re:Invent 2018)

Integrate existing identity providers

(Microsoft AD, LDAP, or in-house built)

for end-user credentials

Route existing SFTP domain to service

endpoint using Amazon Route 53

Support standard SFTP clients

Logical directories to map Amazon S3

bucket paths to end-user visible paths

2019

Import your existing server’s host key

Same server identity and scripts

Elastic IP support using Network Load

Balancer

Support for security and compliance

Support for VPC endpoints (AWS PrivateLink) and AWS CloudFormation

2019

SOC 1,2,3 compliant

Amazon CloudWatch logging enhancements

Source IP logging new!


Encryption at rest options such as

SSE-S3 or SSE-KMS

Server activity tracking in Amazon

CloudWatch and AWS CloudTrail

Use AWS SFTP for your regulated workloads

Support for native AWS integrations

AWS CloudFormation templates

2019

VPC endpoints (AWS PrivateLink) support

Tag-based access control new!


Data stored in Amazon S3 bucket

Amazon S3 events for automated post-upload processing

AWS Identity and Access Management (IAM) for access control

Amazon API Gateway and AWS Lambda for identity provider integration

Easily use AWS services for a rich set of functionality

Features for seamless migration

Identity provider integration

Logical directories for Amazon S3

Host key import

Elastic IP support

Identity provider integration


AWS Cloud

No changes to credentials

SFTP users AWS Transfer for SFTP Amazon S3


AWS LambdaAmazon API

Gateway

How the service supports seamless migration

Service managed authentication

Configure your users credentials and keys using the AWS Console

Users serviced using their existing clients and credentials

Amazon S3 accessed using AWS IAM during file transfers

AWS Transfer

for SFTPAmazon S3

1

2

3

2

3

1

Store and manage user identities and keys inside the service

VPC

2

AWS Transfer for SFTP

“Bring Your Own” (Custom) authentication

Response from API

Gateway used to

authorize S3 access

API Gateway URL

supplied during

SFTP server creation

Set up an API

Gateway and

Lambda for Identity

Provider access

4

API Gateway and

Lambda are invoked

to authenticate

Amazon S3

5

Identity

provider

1AWS LambdaAmazon API

Gateway

End users login

using SFTP client

and credentials

3

Integrate an existing identity provider

Custom identity provider examples

Need password

authentication?

Use AWS Secrets

Manager solution!

Use Microsoft

Active Directory or AWS

Managed AD?

Code samples available

Need support for

third party providers?

CloudFormation templates

available

Want to integrate an in-house

built Identity Datastore?

Use the generic template

to get started!

Logical directories pair(s) 2019 feature!

✅ Credentials match

Response block :

IAM Role ARN (required)

SSH keys (for key based authentication)

Home Directory (optional)

Scope down policy text (optional)

3

How to integrate your custom identity provider?

Request :

Username

Password

(passed in the header)

1 2

How to integrate your custom identity provider?

Provide the API Gateway URL and Role to invoke the method when creating your server

Tip: Test your integration using TestIdentityProvideroperation before trying it from an SFTP client!

Logical directories for Amazon S3


AWS Cloud

✅ No changes to credentials

No changes to transfer scripts




Gateway


precipitation

temperature

history

2019

history

2019

climate_data

End user scripts with file paths

You want to store

the datasets in

your Amazon S3

data lake

What if your end users cannot

change their scripts?

Amazon S3

climate-dataset-1

Amazon S3

climate-dataset-2

Amazon S3

climate-dataset-3

SFTP users want to

be presented with

POSIX style filesystem

using symbolic links

to their files

Logical directories for Amazon S3 solves this!

Privacy and compliance✓ Hide the bucket name

✓ Customize user visible

paths to files

Share same datasets with

multiple users without

making copies

Use logical directory mappings to:

✓ Map Amazon S3 folders

and objects to preexisting

path names

No changes required to

existing transfer scripts

“/pics” “/my_bucket_name/path_to_pics”

“/doc” “/my_bucket_name/path_to_doc”

“/reporting” “/my_bucket_name/path_to_reporting”

“/anotherpath” “/my_bucket_name/path_to_anotherpath”

Logical directories on S3

How it works

Map of entry/target pairs

Entry: what path users see

Target: actual bucket location(can be file or folder)

Use IAM policy for S3 bucket access

“/” “/my_bucket_name/path_to_my_user’s home”

“chroot” the user to a designated folder

How it works

Single entry/target pair

Entry: “/”

Target: actual bucket location of home folder

Use IAM policy forS3 bucket access

Host key import


AWS Cloud




Gateway


✅ No changes to credentials

✅ No changes to transfer scripts

No changes to server identity files

Host key import

Does not match

Client’s record of

server’s host key

The above message will fire off security

alarms, and scripts will break!

How do clients identify your server to trust it?

Host key import

Bring your own host key!

(optional) Upload an RSA host key (from your existing SFTP server)

Server host key persists across server restarts

Elastic IP support


Server endpoint options

Options to host endpoint

Private (in your VPC)

Internet facing (Service hosted)

Corporate

data center

Private (VPC)

SFTP client

in VPC

Server only accessible within VPC and on-premises

environments over DX/VPN

AWS

PrivateLinkSFTP client

on-premises Amazon

Direct Connect

Amazon VPN

AWS cloud

AWS Transfer

for SFTP

Amazon S3

VPC

AWS Transfer

for SFTP

SFTP

endpoint

Associated Elastic IPs with an NLB whose target is your SFTP server endpoint

Use Network Access Control Lists (NACLs) to whitelist clients from 72.21.196.66

End users will whitelist 83.23.196.X to access your server from their firewalls

1

2

83.23.196.66

83.23.196.66

83.23.196.66

1

2

3

Elastic IP attachment

Corporate

data center

SFTP client

on-premises

3

VPC AWS cloud

Amazon S3

72.21.196.66

AWS Cloud




Gateway


✅No changes to credentials

✅No changes to transfer scripts

✅ No changes to server identity files

✅ No changes to firewall configurations

Corporate data center

Scalable media distribution platform

Use case

Need to receive video mezzanines from producers or send files to content aggregators

Offload CDN logs from CDN storage to S3

Solution

No infrastructure to manage, built-in DR

Autoscaling in real-time based on activity

CDN logs directly ingested to S3 and can be analyzed using Amazon EMR

—Huseyin YurtsevenCTO

BluTV

“Before AWS’s SFTP service was available, we were

self managing a cloud based SFTP solution for our needs and ran into lots of problems. There are

open source projects for mounting S3 to EC2, but managing those open source projects and getting good performance from them was not easy. Also there were scaling issues when we needed more

resources. After AWS announced the fully managed SFTP service, we directly started to use that. We

moved our DIY setup to AWS Transfer for SFTP and don’t need to monitor for scaling or manage any

open source projects anymore. Also in terms of cost,

we don't have to pay for unused resources.””

SFTP is a common method of data exchange, and

managing a solution for SFTP was hard

Problem

BluTV is Turkey's #1 subscription

based video-on-demand service.

Cloud based core banking engine

Use case

Securely exchange data between clients (systems) and Ohpen’s core-banking engine

Solution

No infrastructure to manage, automatic scaling, built-in DR

Minimal operational burden – no more DIY SFTP instances, OS and software to manage

Easy storage management in S3

Seamless cutover; using existing authentication systems and transfer scripts

Ohpen is a ruthlessly effective

cloud-based core-banking engine.

It is built for one purpose only:

the efficient administration of

retail investment and savings

accounts.

Problem

Ohpen AWS Transfer for SFTP solutionAWS Cloud

Region

VPC


Amazon API Gateway

Transfer Server

VPC Endpoint

(AWS PrivateLink)

AWS Secrets Manager

Amazon Simple Storage

Service

AWS Lambda

AWS KMS key

Client Bucket

SFTP client

connections

Ch

eck

cre

den

tials

& r

etr

ieve

acc

ou

nt

info

rmati

on

File operations on S3

Custom Identity Provider

Ohpen Endpoint ServicesSFTP client

connections

using PrivateLink

NLB

SFT

P L

isten

er

with

IP ta

rget g

rou

p

Ohpen VPC Endpoint

(AWS PrivateLink)

Who am I

I have been a professional in Bose IT for the past 20 years

Over that time I have worked with many technologies and business units

I have been using AWS since 2009

I am a developer, engineer, and innovator

You can’t see my three kids in this photo

S F T PA T S C A L E

I T C A N B E A N E A S Y

C L I M B W I T H

A W S T R A N S F E R F O R S F T P

© 2 0 1 9 B O S E C O R P O R A T I O N . A L L R I G H T S R E S E R V E D .

OUR BRAND PROMISE

Helping people reach their fullest human potential so they can: feel more, do more, and be more

As with many enterprises,

Bose uses SFTP for securely

transferring files between

companies. These files

include orders & data extracts.

For Bose and our partners,

the SFTP service must be

highly available.

A step along our Cloud Journey

Our IT was migrating all of our On-premise data centers to AWS

So… We needed to move our SFTP service to the cloud since some of our partners require us to host the service.

We saw opportunity…

Running SFTP on servers includes operational overhead and monitoring.

Using AWS Transfer for SFTP service allows us to focus on our business by providing us security and vigilance as a service.

We migrated hundreds of SFTP accounts with a very small migration team.

Having the AWS SDK made integration easy

Challenges during migration

No chroot jail option available at go live

• Chroot/jail is now available as of Sept 2019. Looking forward to using it!

Required more than 10 SSH keys per user

• Some legacy accounts had more than 10 SSH keys associated with them

• The SFTP Transfer Service was very accommodating and was able to increase this limit for our use

This associates the encrypted S3 bucket(s) with the AWS Transfer for SFTP, create users and associate RSA keys for authentication.

When we started testing with partners, we needed to support static IPs for our SFTP Service as some of our partners needed to whitelist our service IP addresses in their firewalls.

CNAME

Home directoriesAmazon

S3 bucket

AWS Transfer

for SFTP

AWS SDK

Amazon Route 53

Phase 1: Basic Implementation

While this is a DMZ, it does not need to be connected to our corporate network.

DNS

CNAME


S3 bucket

AWS Transfer

for SFTP

AWS SDK

VPC

Network Load

Balancer

Private endpoint

IGW

Phase 2: Static IP support

Additional requirements

Additional requirements related to security:

• Disallowing the use of any RSA key’s use 90 days after associating it with any SFTP account (key expiration)

• This requirement requires us to maintain a key history

• Maintaining consistent metadata for SFTP accounts and keys

• A desire to allow internal account owners to manage their keys to minimize administrative work by IT

• Enforce RSA expiration - use DynamoDB and Lambda for associating and disassociating RSA keys to SFTP accounts

• Self service, authenticated UI – built a web-based UI to rotate RSA keys, powered by AWS AppSync, Cognito and React

Phase 3: Adding a Serverless API

DNS


S3 bucket

AWS Transfer

for SFTP

AWS SDK

VPC

Network Load

Balancer

Private endpoint

IGW

AWS Lambda

AWS Amplify

Amazon

Cognito

AWS

AppSync

Amazon

DynamoDB

React

front end

Our serverless API

Our serverless APIDNS


S3 bucket

AWS Transfer

for SFTP

AWS SDK

VPC

Network Load

Balancer

Private endpoint

IGW

AWS Lambda

AWS Amplify

Amazon

Cognito

AWS

AppSync

React

front end

The DynamoDB table

maintains key and key

assignments. Users

indicate location of

private key to make

rotation easy

Amazon

DynamoDB

Amazon

DynamoDB



S3 bucket

AWS Transfer

for SFTP

AWS SDK

VPC

Network Load

Balancer

Private endpoint

IGW

AWS Lambda

AWS Amplify

Amazon

Cognito

React

front end

AppSync is easy to incorporate into a modern single page application. API transactions map to lambda or DynamoDB via resolvers

AWS

AppSync

AWS

AppSync

Amazon

DynamoDB



S3 bucket

AWS Transfer

for SFTP

AWS SDK

VPC

Network Load

Balancer

Private endpoint

IGW

AWS Lambda

AWS AmplifyReact

front end

Cognito is required for AppSync to ensure that all API calls from our web app are authenticated

Amazon

Cognito

Amazon

Cognito

AWS

AppSync

Amazon

DynamoDB



S3 bucket

AWS Transfer

for SFTP

AWS SDK

VPC

Network Load

Balancer

Private endpoint

IGW

AWS Lambda

React

front end

AWS Amplify brings this all together. Amplify creates the login screen and provides the bridges to call the API and expose this to React

AWS Amplify



S3 bucket

AWS Transfer

for SFTP

AWS SDK

VPC

Network Load

Balancer

Private endpoint

IGW

AWS Lambda

AWS Amplify

Amazon

Cognito

AWS

AppSync

Amazon

DynamoDB

React

front end

Realized benefits

• Conversion was fast - able to implement these patterns in weeks

• Enforced key rotation - email warnings sent before keys expire

• Logging and audition - We have great logging in CloudWatch which makes everyone happy

Next steps

• Use S3 events to notify partners that files are ready –Amazon SNS/EventBridge

• We deployed using CloudFormation… We would like to port this wrapped SFTP pattern to the AWS Cloud Development Kit (CDK)

• We plan to use this pattern for additional SFTP use cases across our company

Verisk Analytics is a leading data analytics provider serving customers in insurance, energy and specialized markets, and financial services

We’ve been delivering data, analytics, and decision support services to our customers for nearly 50 years

We're a socially responsible global company operating in 30 countries around the world

Member of the S&P 500 and Nasdaq-100

Cloud at Verisk Analytics

Verisk is a family of operating companies—distinct data owners, application owners, DevOps teams—with a central Cloud CoE

On a mission to migrate all on-premises workloads to the Cloud, with Cloud-first mindset for all new initiatives

Partnered with AWS—industry leader with a track record of innovation and responsiveness to customer feedback

Multi-account strategy at AWS

Our journey—SFTP

Traditional on-premises solution

SFTP is a commonly used, secure method to get data into and out of our infrastructure—providers, customers, employees

Accounts for external users as well as internal users and systems, with isolated network segments

Highly centralized platform with a lot of complexity and management overhead

Firewall Public segment Private segment Application

& user segments

Our journey—SFTP

Overall shift to AWS has led to more experimentation, SFTP included

Different solutions tried by various teams

Customer identity management more fragmented as a result

Hardening and managing these pockets of SFTP services is still burdensome: Network Security, HA, Storage Management, Logging

Overall lack of consistency

SFTP on AWS EC2

S3EC2

Starts out simple

Single server solution with

maybe a mounted S3 bucket

SFTP on AWS EC2

EC2 EC2

Secondary server introduced

to segregate public and

private subnets

SFTP on AWS EC2

S3

EC2

EC2 EC2

EC2

LB

HA considerations leads

to the introduction of

load balancers

SFTP on AWS EC2

S3

EC2

EC2 EC2

EC2

LB

Introduction of a

directory service to handle

identity management

AD

SFTP on AWS EC2

S3EFSAD

EC2

EC2 EC2

EC2

LB

Sometimes EFS is in the mix

SFTP on AWS EC2

S3EFSAD

EC2

EC2 EC2

EC2

LB

End result

A variety of solutions each

with many moving parts

to manage, built differently

across several accounts

and VPC’s

Doesn’t look very different than my first diagram!

Verisk’s use of AWS transfer for SFTP

Allows for a business-aligned infrastructure with centralized identity management

Federation with Okta—a strategic partner for customer identity management

Removes the infrastructure management overhead and leverages S3 for target storage—flexible and cost effective

SFTP becomes a platform that we consume rather than infrastructure to manage; focus our efforts on downstream technology


S3AWS Transfer

It’s this simple!


S3

Lambda

API Gateway

AWS Transfer

Integrated our provider—Okta


S3

Lambda

API Gateway

AWS Transfer

CloudWatch

Integration with Splunk

for monitoring

Integrated our provider—Okta


S3

Simplifies data sourcing,

authentication, logging

Quick to deploy

Allows focus on downstream

data engineering and analytics

Lambda

API Gateway

AWS Transfer

CloudWatch

Example – Analytics lab

Data consortium model for the

energy industry

Combination of private data sets and

public data

AWS transfer used for contributor

data sourcing

Dataiku on EC2

Transfer S3 RDS Analytics Visualize

Dataiku on EC2

Next steps using SFTP at Verisk

Several integration projects in the pipeline

As initial adopters succeed, we expect accelerated adoption

Continual feedback loop with AWS product team to improve the platform

Resources—blog posts and templates

Visit the product websitefor resources

aws.amazon.com/sftp

https://aws.amazon.com/architecture/icons/

Available in 16 AWS regions US East (N. Virginia)

US East (Ohio)

US West (N. California)

US West (Oregon)

Canada (Central)

EU (London)

EU (Frankfurt)

EU (Ireland)

EU (Paris)

EU (Stockholm)

Asia Pacific (Seoul)

Asia Pacific (Singapore)

Asia Pacific (Sydney)

Asia Pacific (Tokyo)

Asia Pacific (Mumbai)

South America (Sao Paulo

Consolidate all your file transfers in AWS!


Pricing:

• Endpoint @ $0.30/hour

• Data uploads and downloads @$0.04/GB

Visit console.aws.amazon.com/transfer to get started today!

https://console.aws.amazon.com/transfer

Related sessions

STG204

Get your data to AWS: How to choose and use data migration services

STG206

Stop managing SFTP servers today

STG221

SFTP workflows for data lakes and enterprise applications

STG316

Get hands-on & learn best practices for AWS data migrations


Visit aws.amazon.com/training/path-storage/

Classroom offerings, like Architecting on AWS, feature AWS expert instructors and hands-on activities

45+ free digital courses cover topics related to cloud storage, including:

Learn storage with AWS Training and Certification

• Amazon S3

• AWS Storage Gateway

• Amazon S3 Glacier

• Amazon Elastic File System

(Amazon EFS)

• Amazon Elastic Block Store

(Amazon EBS)

Resources created by the experts at AWS to help you build cloud storage skills

Thank you!


Migrate and modernize SFTP file

Documents

Transcript of Migrate and modernize SFTP file