devcon iv

Architecting with EthereumAlexander Miller Denver, Colorado CryptoKube.io

dc iv.

devcon iv!1

devcon iv

Introduction

!2 devcon iv

devcon iv!3

Based out of Denver, Colorado

2012: discovered Bitcoin, started mining w/GPU -> BFL ASIC

2014-2016: platform support engineer @ Hortonworks (Hadoop)

2016-2017: tooling engineer @ Endless OS (Linux distro)

2017: discovered Ethereum, devops @ Shapeshift

2018: open source sabbatical, distributed systems consulting

Passions: open source, P2P crypto, radical self-reliance

devcon iv

Intro: Who am I?

devcon iv!4

GOAL: build an Ethereum computing stack from re-usable modules

AGENDA:

Introduce admin tooling & major stack components

Conduct a series of exercises to demonstrate the concepts in realistic use cases. Each exercise:

builds on previous exercises

introduces one admin concept & one Ethereum node concept

devcon iv

Intro: Workshop

devcon iv!5

FOSS platform for hosting P2P crypto applications

Easily provision & deploy components to multiple environments

Currently a collection of Ansible roles & Terraform modules

Integrated stack is in development

Latest details at https://www.cryptokube.io

devcon iv

Intro: CryptoKube

devcon iv!6

Experience: remote Linux administration, and high-level understanding of Ethereum and cloud computing concepts

Software: SSH client & web browser

DigitalOcean:

Create account (use link for $100/60-day trial)

Add SSH public key for authentication on new droplets

Create a personal access token for API access

devcon iv

Intro: Prerequisites

devcon iv!7

Setup guide:

https://www.cryptokube.io

devcon iv

Intro: DigitalOcean setup

devcon iv

Administrative Tooling

!8 devcon iv

devcon iv!9

Terraform

Ansible

Git

Docker

DigitalOcean

devcon iv

Administrative Tooling

devcon iv

Administrative Tooling Terraform

!10 devcon iv

devcon iv!11

Codifies APIs into declarative configuration files that can be shared, edited, reviewed, and versioned (just like code)

Safely and predictably create, change, and improve infrastructure

Version control your resources, allowing rollback to previous state in case of error

Uses declarative syntax (HCL), fully JSON compatible but extended for easier human consumption

Maintained by Hashicorp. https://www.terraform.io

devcon iv

Administrative Tooling: Terraform

devcon iv!12 devcon iv

Describes infrastructure and sets variables. Stored as text files with .tf extension.

Configuration

Responsible for understanding API interactions and exposing resources. Generally an IaaS (AWS, DigitalOcean), PaaS (Heroku), or SaaS (CloudFlare).

Providers

Maps real world resources to your configuration, and keeps track of metadata. Stored locally as terraform.tfstate (or remotely)

State

Self-contained packages of Terraform configs that are managed as a group. Used to create reusable components and for basic code organization.

Modules

Administrative Tooling: Terraform concepts

devcon iv!13

terraform init - Initialize a new or existing Terraform configuration (install plugins, perform minimal validation)

terraform get - Import modules

terraform plan - Generate and show an execution plan

terraform show - Inspect Terraform state or plan

terraform apply - Builds or changes infrastructure

terraform destroy - Destroy Terraform-managed infrastructure

devcon iv

Administrative Tooling: Terraform commands

devcon iv

Administrative Tooling Ansible

!14 devcon iv

devcon iv!15

Simple IT automation engine that automates configuration management, application deployment, and more

Native OpenSSH remote communication when possible (no agents)

Executes ad-hoc commands or scripted “playbooks”

Extensible via modules, plugins, and Python APIs

Maintained by Red Hat. https://www.ansible.com

devcon iv

Administrative Tooling: Ansible

devcon iv!16 devcon iv

Consists of hosts and groups. May also define variables. We will generate ours from Terraform state using terraform-inventory

Inventory

Collections of related variables, tasks, templates, and files. Primary mechanism for breaking playbooks into reusable components.

Roles

Consists of plays, which map tasks and roles to a group of hosts from the inventory. YAML format.

Playbooks

Reusable, standalone scripts that can be used by Ansible. Return information as JSON to stdout. Take arguments in several ways. See ‘Module Index’

Modules

Administrative Tooling: Ansible concepts

devcon iv!17

[ethereum_nodes]geth_nodesparity_nodes[geth_nodes]geth-light.example.com geth_sync_mode=lightgeth-full[0:2].example.com[parity_nodes]parity-full.example.com

devcon iv

Administrative Tooling: Ansible Inventory

devcon iv!18

---- hosts: ethereum_nodes vars: data_dir: ‘/opt/.ethereum’ rpc_port: 8545 tasks: - include_role: name: rhel_common when: "ansible_facts['os_family'] == 'RedHat'" - include_role: name: debian_common when: "ansible_facts['os_family'] == 'Debian'"

devcon iv

Administrative Tooling: Ansible Playbooks

devcon iv!19

A few quick examples:

- lineinfile: path=/etc/selinux/config regexp=‘^SELINUX=' line=‘SELINUX=enforcing'

- template: src: etc/ssh/sshd_config.j2 dest: /etc/ssh/sshd_config owner: root group: “{{sshd_group}}” mode: ‘0600'

devcon iv

Administrative Tooling: Ansible Modules

devcon iv!20

Roles contain some (or all) of these subdirectories:

tasks - list of tasks to be executed

handlers - special tasks that can be triggered to run at end

defaults - default variables for the role

vars - other variables for the role

files - static files that can be copied

templates - dynamic files that can be altered by variables

meta - role dependencies, author info, etc devcon iv

Administrative Tooling: Ansible Roles

devcon iv!21

Ad-hoc command to display Linux distribution info:ansible all -a ‘cat /etc/lsb_release’

Run playbook ‘site.yml’ against Terraform hosts:ansible-playbook -i terraform-inventory site.yml

Install Galaxy-style roles specified in requirements.yml:ansible-galaxy install -r requirements.yml -f

Generate scaffolding for new Galaxy-style role:ansible-galaxy init <role-name>

devcon iv

Administrative Tooling: Ansible commands

devcon iv

Administrative Tooling DigitalOcean

!22 devcon iv

devcon iv!23

Infrastructure as a Service (IaaS) platform

Powerful & simple API, friendly web UI

Extensive set of documentation & tutorials

Cloud resources include:

Servers, Block storage, Object storage, Machine images

Firewalls, Load balancers, Floating IPs, DNS

devcon iv

Administrative Tooling: DigitalOcean

devcon iv!24

Git - distributed version-control system

Docker - popularized containerization, isolate components

Packer - build images for cloud, VMs, containers

Consul - service discovery & KV store

devcon iv

Administrative Tooling: additional resources

devcon iv

Infrastructure Components

!25 devcon iv

devcon iv!26

Ethereum Node

Application Server

Reverse Proxy

devcon iv

Infrastructure Components

devcon iv

Infrastructure Components: Ethereum Node

!27 devcon iv

devcon iv!28

Services & ports

Networks & chains

Node types

Light client

Full node

Implementations

Geth

Parity devcon iv

Ethereum Node

devcon iv!29 devcon iv

Ethereum Node: services & ports

Service Protocol Port Interface

JSON-RPC TCP 8545 private

Websockets TCP 8546 private

Peer-to-peer TCP 30303 public

Node discovery UDP 30301 public

devcon iv!30 devcon iv

Ethereum Node: networks & chains

Network ID Chain ID Description Consensus Clients

1 1 Ethereum mainnet PoW All

3 1 Ropsten testnet PoW All

4 1 Rinkeby testnet PoA Geth

42 1 Kovan testnet PoA Parity

1 61 Ethereum Classic mainnet PoW All

2 1 Ethereum Classic testnet PoW All

devcon iv!31

Chain data: list of blocks containing transactions

State data: results of each transaction’s state transition

Account balances, nonces, smart contract code and data

Importing state entries now takes much longer than downloading the blocks

A full node stores all blocks and a pruned account state

A light client only stores the headers of all blocks, and downloads state as needed

devcon iv

Ethereum Node: synchronization

devcon iv!32

all "full clients" except for archive nodes (intended to be run by businesses, block explorers, etc) will eventually be set up as "partially light clients" with respect to all history older than a few thousand blocks

- https://github.com/ethereum/wiki/wiki/Light-client-protocol

Full node - download all blocks, keep pruned state

Light client - download block headers, state on-demand

Archive node - download all blocks, keep unpruned state

Bootstrap node - no blocks, provides directory of other nodes

devcon iv

Ethereum Node: node types

devcon iv!33

An Ethereum client implements the

protocol specified in the yellow paper.

Many implementations are available, but

the vast majority of mainnet is running

Geth or Parity.

devcon iv

Ethereum Node: implementations

EtherNodes.org

devcon iv!34

Written in Go

Developed by Ethereum Foundation

Licensed under GPLv3

Separate implementation for each chain (ETH, ETC, Ubiq, etc)

Best configured using only command-line arguments

devcon iv

Ethereum Node: Geth (go-ethereum)

devcon iv!35

Written in Rust

Developed by Parity Tech (UK)

Licensed under GPLv3

Single implementation supports many networks & chains

Configurable by file and/or command-line arguments

devcon iv

Ethereum Node: Parity Ethereum

devcon iv!36

Feather - wallet for ETH & ERC-20 that runs on top of Parity Ethereum light client

Signer - mobile app to use smartphone as air-gapped wallet for cold storage

Parity UI (deprecated) - previously accessible as a web app from the Ethereum client

Substrate - framework for building new blockchains

devcon iv

Ethereum Node: parity (additional projects)

devcon iv

Infrastructure Components: Application Server

!37 devcon iv

devcon iv!38

Act as an Ethereum remote client, which relies on a node to provide access to the blockchain

May interact with a light client running locally on the host

May interact with other P2P/crypto applications on the host

devcon iv

Infrastructure Components: Application Server

devcon iv

Infrastructure Components: Reverse Proxy

!39 devcon iv

devcon iv!40

Load balancing improves the performance and reliability of a service by distributing the workload across multiple computing resources, such as hosts or network links.

High availability assures a high level of operational performance for a given period of time by eliminating single points of failure

TLS Termination provides a secure connection to the outside

Implemented using HAProxy, NGINX, or a cloud mechanism

devcon iv

Infrastructure Components: Reverse Proxy

devcon iv!41

Dedicated reverse proxy component

Layer 3 (TCP) and Layer 7 (HTTP) load balancing

Health checking

Rate limiting

SSL/TLS termination

Multiple instances can use a heartbeat for high availability

Very fast and efficient

devcon iv

Infrastructure Components: HAProxy

devcon iv!42

Primarily a web server, can serve static and dynamic content

Easily configured as a reverse proxy

Also able to perform caching duties

Status page only provides 7 metrics (versus 61 for HAProxy)

devcon iv

Infrastructure Components: NGINX

devcon iv

Workshop Setup

!43 devcon iv

devcon iv!44

Create account (use link for $100/60-day trial)https://do.co/devcon4-eth

Add SSH public key for authentication on new droplets

Create a personal access token for API access

QuickStart Guide: https://www.cryptokube.io

devcon iv

Workshop Setup: DigitalOcean account

devcon iv!45

Create a droplet for the Management Host using these options:

Image: Ubuntu 18.04 x64

Size: 2GB/2vCPU standard droplet

Datacenter region: your choice

Additional options: private networking, user data, monitoring

User data: see contents on next slide (docs)

SSH keys: select yours

click Create devcon iv

Workshop Setup: Management Host

devcon iv!46

#cloud-config# from: https://cryptokube.io/devcon4/cloud-config.yaml

package_upgrade: true

packages: - python - python-pip - git - zip - jq

runcmd: - [curl, -o, /tmp/terraform.zip, “https://releases.hashicorp.com/terraform/0.11.10/terraform_0.11.10_linux_amd64.zip"] - [unzip, -d, /usr/local/bin/, /tmp/terraform.zip] - [curl, -L, -o, /tmp/terraform-inventory.zip, "https://github.com/adammck/terraform-inventory/releases/download/v0.7-pre/terraform-inventory_v0.7-pre_linux_amd64.zip"] - [unzip, -d, /usr/local/bin/, /tmp/terraform-inventory.zip] - [pip, install, -U, pip, ansible] - [git, clone, “https://github.com/cryptokube-io/devcon4-workshop.git”]

devcon iv

Workshop Setup: User-Data

devcon iv!47

SSH into the management hostssh root@<ip>

Monitor cloud-init progresstail -f /var/log/cloud-init-output.log

Enter workshop directorycd devcon4-workshop

Run initialization scriptbin/init_config

devcon iv

Workshop Setup: completion

devcon iv

Workshop Exercises

!48 devcon iv

devcon iv!49

01 Light client

02 Full node

03 Reverse proxy

04 Ethereum Application

devcon iv

Workshop Exercises

devcon iv

Exercise 01 Light Client

!50 devcon iv

devcon iv!51

Introduces: digitalocean, parity, light client

Goal: deploy a light client on a DO droplet using Terraform

Terraform config: provisions a single DO droplet

Ansible playbook: installs & configures Parity Ethereum

This is mainly to verify that the cloud admin tools are working. It is more important to observe the commands and output than it is to get the node synced.

devcon iv

Exercise 01: Light Client

devcon iv!52

Image: Ubuntu 18.04 x64Name: devcon4-parity_light

Ports (proto int:ext ip - name) TCP 8545:8545 127.0.0.1 - HTTP RPC TCP 8546:8546 127.0.0.1 - HTTP WS TCP 30303:30303 0.0.0.0 - P2P UDP 30301:30301 0.0.0.0 - node discovery

devcon iv

Exercise 01: Terraform config

devcon iv!53

Enter the exercise directorycd 01_light_client

Initialize the Terraform configurationterraform init

Get Terraform modulesterraform get

View the Terraform execution planterraform plan

Apply the Terraform config to build the infrastructureterraform apply devcon iv

Exercise 01: [1/3] build the infrastructure

devcon iv!54

Run Ansible playbookansible-playbook -i terraform-inventory site.yml

devcon iv

Exercise 01: [2/3] run Ansible playbook

devcon iv!55

Query the node using the JSON-RPC APIip=$(terraform-inventory -list | jq -r .parity_node[0])header="Content-Type: application/json”host="http://ip:8545"queries=“web3_clientVersion net_version net_peerCount”for q in queries; do data="{'jsonrpc':'2.0','method':'$q','params':[],'id':67}" curl -H "$header" -X POST -data “$data”done

devcon iv

Exercise 01: [3/4] query the node

devcon iv!56

Clean up the infrastructure by deleting everythingterraform destroy

devcon iv

Exercise 01: [3/3] clean up

devcon iv

Exercise 02 Full Node

!57 devcon iv

devcon iv!58

Introduces: DigitalOcean volume, full node

Goal: deploy a full Ethereum node on a DO droplet (w/data volume) using Terraform and Ansible

Terraform config:

digitalocean_droplet parity_full

digitalocean_volume parity_data

devcon iv

Exercise 02: Full Node

devcon iv!59

Enter the exercise directorycd 02_full_node

Initialize the Terraform configurationterraform init

View the Terraform execution planterraform plan

Apply the Terraform config to build the infrastructureterraform apply

devcon iv

Exercise 02: [1/4] build the infrastructure

devcon iv!60

Run Ansible playbookansible-playbook -i terraform-inventory site.yml

devcon iv

Exercise 02: [2/4] run Ansible playbook

devcon iv!61

Query the node using the JSON-RPC APIip=$(terraform-inventory -list | jq -r .parity_node[0])header="Content-Type: application/json”host="http://ip:8545"queries=“web3_clientVersion net_version net_peerCount”for q in queries; do data="{'jsonrpc':'2.0','method':'$q','params':[],'id':67}" curl -H "$header" -X POST -data “$data”done

devcon iv

Exercise 02: [3/4] query the node

devcon iv!62

Clean up the infrastructure by deleting everythingterraform destroy

devcon iv

Exercise 02: [4/4] clean up

devcon iv

Exercise 03 Reverse Proxy

!63 devcon iv

devcon iv!64

Demonstrates load balancing & high availability

Simple setup with two backend web servers

Web servers simply display a message with the hostname

devcon iv

Exercise 03: Reverse Proxy

devcon iv!65

Enter the exercise directorycd 03_proxy

Initialize the Terraform configurationterraform init

View the Terraform execution planterraform plan

Apply the Terraform config to build the infrastructureterraform apply

devcon iv

Exercise 03: [1/4] build the infrastructure

devcon iv!66

Run Ansible playbookansible-playbook -i terraform-inventory site.yml

devcon iv

Exercise 03: [2/4] run Ansible playbook

devcon iv!67

Observe the responses from multiple requestsip=$(terraform-inventory -list | jq -r .haproxy[0])for i in `seq 1 10`; do curl -k $ip; sleep 1; done

View the HAProxy statistics page in web browserecho http://$ip/haproxy?stats

devcon iv

Exercise 02: [3/4] observe running infrastructure

devcon iv!68

Clean up the infrastructure by deleting everythingterraform destroy

devcon iv

Exercise 03: [4/4] clean up

devcon iv

Exercise 04 Ethereum Application

!69 devcon iv

devcon iv!70

Combines previous exercises into full application stack

Two Parity Ethereum full nodes behind HAProxy

EthStats

App Server running eth-netstats

Parity nodes running eth-net-intelligence-api

devcon iv

Exercise 04: Ethereum Application

devcon iv!71

Enter the exercise directorycd 04_ethereum_app

Initialize the Terraform configurationterraform init

View the Terraform execution planterraform plan

Apply the Terraform config to build the infrastructureterraform apply

devcon iv

Exercise 04: [1/5] build the infrastructure

devcon iv!72

Run Ansible playbookansible-playbook -i terraform-inventory site.yml

devcon iv

Exercise 04: [2/5] run Ansible playbook

devcon iv!73

View the EthStats web UIip=$(terraform-inventory -list | jq -r .app_node[0])echo http://$ip:3000

View HAProxy statistics web pageip=$(terraform-inventory -list | jq -r .haproxy[0])echo http://$ip:8545/haproxy?stats

devcon iv

Exercise 04: [3/5] observe running infrastructure

devcon iv!74

Define convenience function for web3 queriesfunction web3query() { curl -s -H "Content-Type: application/json" \ -X POST --data \ ”{\"jsonrpc\":\"2.0\",\"method\":\"$1\",\"params\":[],\"id\":74}" \ http://$ip:8545 | jq -r .result}

Send JSON-RPC requests to the proxyweb3query web3_clientVersionweb3query net_versionweb3query net_peerCount

devcon iv

Exercise 04: [4/5] make JSON-RPC requests to proxy

devcon iv!75

Clean up the infrastructure by deleting everythingterraform destroy

devcon iv

Exercise 04: [5/5] clean up

devcon iv!76

Thank you for attending!

Follow the latest updates: https://www.CryptoKube.io

alex@thinkmassive.org | @thinkmassive | Denver, Colorado, USA