On the Edge Systems Administration with Golang

October 29–November 3, 2017 | San Francisco, CAwww.usenix.org/lisa17 #lisa17

Close to the Edge Systems Administration in Go

Chris "mac" McEniry

https://app.strigo.io/event/iEHzHx4rhedZsJyTQ

token: XGVI

slack: #2017-tutorial-t1-go

Administrivia

Goals for Today• Sysadmin Topics

• File systems, Web Servers, One-liners, Containers, SSH

• Go Language Features

• Interfaces, Type Assertions, Function Types, Anonymous Functions, Empty Interface, Anonymous Structs, GOOS, GOARCH

• Go Libraries

• filepath, os, syscall, syscall, syscall, crypto/x509, crypto/tls, net/http, encoding/json, expvar, x/crypto/ssh

• Go Tools (in Go or for use on Go)

• go-bindata, go-bindata-assetfs, h2i, h2c, dep, gorram, expvarmon |

3

Prerequisites• Some Go experience

• Mainly need to be able to read code and follow the basic control flow

4

Schedule

9:00 - 9:10 Administrivia

9:10 - 9:20 Introduction

9:20 - 10:30 Interfaces, Files, Web Servers, TLS, HTTP/2

10:30 - 11:00 Break

11:00 - 12:30 JSON, Package Management, One Liners, Cross compilation, Metrics, Containers, SSH

5

WARNING

Going to cut some corners with our code.

Any errors that arise - just going to panic.

This is not what you want to normally do - this is just for the sake of demonstration.

DO NOT USE ANY OF THIS CODE IN PRODUCTION WITHOUT TAKING A SECOND

LOOK AT IT

6https://pixabay.com/en/animals-black-and-white-bomb-boom-985500/

Working with the Examples

• `go get github.com/cmceniry/cttesa`

• All exercises are meant to work with `go run`

• We'll continue after any example when ~50% of the attendees say to continue (by show of hands)

7

https://app.strigo.io/

Close to the Edge...

What do you remember as your

most ingenious moment?

9https://pixabay.com/p-1872376/?no_redirect

• Built something

• Completed something

• Used something in a way it wasn't intended

• Succeeded even though you were blocked

• Learned something

10https://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/Mr_pipo_Learning.svg/1000px-Mr_pipo_Learning.svg.png

"the quality of being clever, original, and

inventive."

11https://www.google.com/search?q=ingenuity

David Blank-Edelman's

Over the Edge Systems Administration

Homage

12https://c1.staticflickr.com/7/6004/6013286848_c9d1782d7f_b.jpg

• Generalize

• Use a new interface

• Repurpose your tools

• Do the smallest or wrong thing

• Transports and Tunnels

Over the Edge System Administration

13https://pixabay.com/p-1966997

Close to the Edge• Experiment

• Sometimes you do it the hard way

• Look under the hood

• Read the code

• Read the docs

• Have fun

14https://pixabay.com/p-759080

• Look at your tools. What do they tell you about themselves?

• Look at what your tools are operating on. What do the tools tell you about what they operate on?

• What do the tools tell you about how you do work?

15

Let's get started...16

https://pixabay.com/p-1414148/?no_redirect

17https://upload.wikimedia.org/wikipedia/commons/thumb/7/7b/Interface_logo.svg/1280px-Interface_logo.svg.png

Go Types• Most Go Types represent some form of data

• Boolean, Numeric, String, Array, Slice, Struct, Pointer, Map, Channel

• The Interface Type represents behavior

• Bit of a placeholder

• "Any value whose type has these methods can be used here."

• "Any value whose type can do at least what I need of it can be used here."

• Can only use the methods of that interface

• This is used a lot without even knowing it

18

io.Reader• Implements just one method: Read

• https://golang.org/pkg/io/#Reader

• godoc io | grep -A2 -m2 '^type Reader'

• Side note: error is an Interface Type. It implements `Error() string`

19

type Reader interface { Read(p []byte) (n int, err error)}

cttesa/interface

• Can only call `Read([]byte) (int,error)` on `data`

• But that's all we need

func saveFile(filename string, data io.Reader) error { ... nr, err := data.Read(buf) ...}

20

cttesa/interface

• `a` and `b` are different different types, but both satisfy `io.Reader` so can be used for `saveFile`

func main() { a := bytes.NewBuffer([]byte("file1\n")) err := saveFile("file1", a) ... b := strings.NewReader("file2\n") err = saveFile("file2", b) ...}

21

Another io.Reader• Find another io.Reader from the library

• Use saveFile to write it to disk

22

Working with file metadata...23

https://upload.wikimedia.org/wikipedia/commons/c/c2/Logo_X-FILES.png

syscall.Stat_t• Goal: Get the inode for a file

• Go concept: Type Assertion, syscall.Stat_t

24

cttesa/inode

• `s.Sys()` returns a interface which must be asserted to `*syscall.Stat_t`

• `*syscall.Stat_t` struct has actual the `Ino` field

func getInode(path string) (uint64, error) { s, err := os.Stat(path) ... stat, ok := s.Sys().(*syscall.Stat_t) ... return stat.Ino, nil}

25

filepath.Walk• Goal: Locate files with the same Inode

• Go concepts: Function Type, filepath.Walk

26

cttesa/inode-walk

• The `filepath` library defines a `WalkFunc` Function Type to be used to define any custom logic for it. We can define that and provide it to the `Walk` function.

type WalkFunc func(path string, info os.FileInfo, err error) error...func Walk(root string, walkFn WalkFunc) error

27

func walkFunc(path string, info os.FileInfo, err error) error {...fun main() { filepath.Walk(".", walkFunc)

Anonymous Function• You don't have to define the function. It can be used as an expression inline -

this is called an Anonymous Function

• Complete `inode-anon` to use an anonymous function. Copy portions from `inode-walk`

• Place the Println to add a prefix defined in the `main` function

28

func main() { prefix := "FOUND:" filepath.Walk(".", ...) // fmt.Printf("%s %s = %d\n", prefix, path, ino)}

Why Anonymous Functions?• Inline

• Locate code closer to where it is used

• If only used once, no reason to separate it

• Allows for closures

• Wrapping up of variables

• Access to inside of function which wouldn't be available otherwise

29

Returning Anonymous Function• Advanced: Do-on-your-own challenge..

• Write a function which takes in a string, the prefix, and returns a `filepath.WalkFunc`

• Update `inode-anon` to use this function

• Behaves the same as `inode-walk`

30

Need to serve files somehow..31

https://upload.wikimedia.org/wikipedia/commons/thumb/b/b5/Kickstarter_logo.svg/2000px-Kickstarter_logo.svg.png

cttesa/kickstart• Goal: Deploy a web server to handle static files

• Go concepts: net/http, +Function Type, +Interface

32

33

https://golang.org/pkg/net/http/#Handle

https://golang.org/pkg/net/http/#FileServer

https://golang.org/pkg/net/http/#FileSystem

34

https://golang.org/pkg/net/http/#Dir

==> An http.Dir value (acts as http.FileSystem) can be fed to http.FileServer ==> http.FileServer returns an http.Handler which can be fed to http.Handle

Setup the static server• Fill in `kickstart` with the appropriate line so that it will serve the files in the

`data` directory

35

func main() { ... fmt.Println("Listening on :8080") http.ListenAndServe(":8080", nil)}

• Goal: Make the file server stand alone

• Go concepts: go generate, go-bindata-assetfs

36

Single binary• One of Go's strengths is its ability to package up as a single binary.

• What if you could package up resources in that binary as well?

37

38

https://github.com/elazarl/go-bindata-assetfs

go-bindata-assetfs data/...cat bindata_assetfs.go...func assetFS() *assetfs.AssetFS { assetInfo := func(path string) (os.FileInfo, error) { return os.Stat(path) } for k := range _bintree.Children { return &assetfs.AssetFS{Asset: Asset, AssetDir: AssetDir, AssetInfo: assetInfo, Prefix: k} }

cttesa/kickstart-embedded

39

40

https://godoc.org/github.com/elazarl/go-bindata-assetfs#AssetFS

Look familiar?

go generate• Part of build tool which runs prior to build

• Can trigger manually with `go generate`

• Executes what's in a generate comment

• Intention is for any code generation pieces

41

//go:generate echo foo

Setup Generate and assetFS• Update `kickstart-embedded` to use `assetFS`

• Update `kickstart-embedded` to generate automatically

• Build/run stand alone web server

42

Encryption43

https://upload.wikimedia.org/wikipedia/commons/thumb/0/08/TLS_in_TCPIP_stack.svg/500px-TLS_in_TCPIP_stack.svg.png

• Goal: Add encryption to the web server

• Go concept: net/http.Server, crypto/tls, crypto/x509

44

Custom http server configuration• Been using the default http server in net/http

• Can setup custom http servers - different configs, multiple at once, etc

• Define the server as a value and use that instead of the default library functions

45

http.ListenAndServe(":8080", nil)

s := http.Server{Addr: ":8080"}err = s.ListenAndServe()

Generate certificates• Process

• Generate Private Key

• Generate Certificate Information

• Self-sign Certificate with Private Key

• Everything else is just manipulating it so the libraries take it

46

Two Certificate Types• crypto/x509.Certificate: Used for general certificate information

• crypto/tls.Certificate: Used for certificates in the context of TLS

• Support for Cert/Key together

• Support for name/Cert mapping

• Support for Certificate chain required by TLS

47

48

https://golang.org/pkg/crypto/x509/#Certificate

https://golang.org/pkg/crypto/tls/#Certificate

TLS Certificates• Need to generate the certificate using crypto/x509

• Then format it into crypto/tls

49

cttesa/tls• Has a helper function : generateTLSCert

• Returns the tls.Certificate since that is what is used later

• Start by generating the private key

• Next go into certificate information...

50

func generateTLSCert() (*tls.Certificate, error) {

k, err := rsa.GenerateKey(rand.Reader, 2048)

51

t := x509.Certificate{ SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "localhost"}, NotBefore: time.Now().Add(-30 * time.Second), NotAfter: time.Now().Add(86400 * time.Second), KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{ x509.ExtKeyUsageServerAuth, },}

cttesa/tls

• Next step is to sign the certificate (see exercise). Need an crypto/x509 function that returns a certificate format (type or bytes or...)

52

https://golang.org/pkg/crypto/x509/#CreateCertificate

• Then need to format it for tls.Certificate (PEM vs DER). Looking at the crypto/tls library, we look for a function which returns a Certificate.

53

https://golang.org/pkg/crypto/tls/#X509KeyPair

cttesa/tls• We need to format the cert and key into a pem []byte

certPem := pem.EncodeToMemory(&pem.Block{ Type: "CERTIFICATE", Bytes: c,})keyPem := pem.EncodeToMemory(&pem.Block{ Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(k),})

54

cttesa/tls• Setup the handler as before

• Setup the net/http.Server value, with a crypto/tls.Config wired up to it.

• And then can start the listen and serve loop, though in this case, it's ListenAndServerTLS

err = s.ListenAndServeTLS("", "")

55

http.Handle("/", http.FileServer(http.Dir(".")))

s := http.Server{...}

56

https://golang.org/pkg/net/http/#Server

https://golang.org/pkg/crypto/tls/#Config

Generating Certs and Enabling TLS• Complete `tls/main.go` by

• making the appropriate call to generate the x509.Certificate

• making the appropriate call to generate the tls.Certificate

• making the appropriate call to configure TLS

• Can skip hostname verification

• Listen on localhost:8443

57

Can generate as files...• https://golang.org/src/crypto/tls/generate_cert.go

58

http259

• Goal: Setup for HTTP/2

• Go concept: h2i, h2c, net/http

60

HTTP/2• Binary protocol

• Required encryption

• Multiplexing of requests

• Header compression

• Wraps (essentially) HTTP/1.1 requests into frames

61

• Golang's net/http library already has HTTP/2 built in (if you run TLS)

• Several tools in Go to test it

• https://github.com/fstab/h2c

• https://godoc.org/golang.org/x/net/http2/h2i

62

Access web server from h2i

63

shell$ h2i -insecure localhost:8443...h2i> headers(as HTTP/1.1)> GET / HTTP/1.1(as HTTP/1.1)>...

Access web server from h2c

• Go back and restart the other kickstart examples. Try the same with localhost:8080

64

shell$ h2c start &...shell$ h2c connect localhost:8443shell$ h2c get /...shell$ h2c stop...

Universal config files...65

https://c2.staticflickr.com/4/3880/14636491710_d347ce16ae_b.jpg

Reading JSON• Goal: Load configuration data from a JSON structure

• Go concepts: Empty Interface{}, encoding/json

66

Parsing JSON/XML/YAML• Golang's encoding libraries behave the same way

• Use reflection to determine how to convert data

• Or use `map[string]interface{}` to create for key/value with the value being non-specific types

67

cttesa/json-empty• The JSON structure that we're going to work with

• We're going to just try to pull "bar" out.

{ "foo": 123, "bar": "baz", "really": true}

68

cttesa/json-empty• Initialize a map to store data. Note the inline `interface{}` -- essentially an

anonymous interface definition

• Now use `json.Unmarshal` to load data into c from our source. Unmarshal works on a slice of bytes, so have to convert the string to allow it.

c := make(map[string]interface{})

69

json.Unmarshal([]byte(data1), &c)

cttesa/json-empty• Now, safely get the data out

barEmpty, ok := c["bar"]if !ok { panic("bar missing")}bar, ok := barEmpty.(string)if !ok { panic("bar is not a string")}fmt.Printf("bar: %s\n", bar)

70

Reading JSON using a struct• The constant conversion is fairly verbose.

• Can rely on the library to do the work for us.

71

cttesa/json-struct• Need to define a struct type to hold the data and give form

• Then can decode the exact same way, but without the same checks

type config struct { Foo int64 Bar string Really bool}

72

c := &config{}json.Unmarshal([]byte(data1), &c)fmt.Printf("bar: %s\n", c.Bar)

Read XML• Perform the same exercise with XML

• Complete `xml-struct`:

• Fill in the fields for `config` based on the XML input

• Call the right function to convert `data1` to `c`

73

Writing JSON• Goal: Output JSON

• Go concepts: Anonymous Struct, Inline Initialization

74

Writing JSON• Same reflection can be used to output JSON

• But you don't have to do any of the checking

• Becomes dealers choice for what to output from

• Existing or inline map[string]interface{}

• Existing or inline (anonymous) struct

75

Writing JSON• Complete `json-output`

• All blocks should output a `Name` and `Value`

76

Do the same with XML• On your own...

• Do `json-output`, but with encoding/xml instead of encoding/json

77

Struct Tags• On your own...

• Investigate struct tags to map other fields into the structs

• E.g. a JSON field "common_name" needs to map to camelcase "CommonName" field inside of a struct

78

dep79

https://c2.staticflickr.com/8/7165/6514584423_a9b13d6b70_b.jpg

Pulling in Others' Code• So far, everything we used has been a command, so we didn't look at

dependencies, just used them via `go get...`

• Now we want to pull in an actual code dependency.

• In the previous JSON examples, we converted JSON to and from

• map of the empty interfaces

• structs

• What if we want to skip the JSON part?

80

81

https://godoc.org/github.com/mitchellh/mapstructure

Why?

Some functions work with pluggable backends but have a common interface. That interface may or may not be able to use fix types and so have to resort to some more abstract pieces. It's still nice to be able to convert those.

Go Package Management• Has be very limited*

• Several different package managers sprung up to fill the void

• Community finally started to push to a common one

• Still being sorted out, so this might change...

82

https://pixabay.com/p-311665


dep• Command which will handle dependency

• Looks through your code, sees what is dependent (imports)

• Looks at the dependencies, see what they depend on, and repeat

• Takes some input as to if there are any constraints (library versions which you have said you are locked to)

• Solves to pull in the best version of dependencies

83

dep Structure• `vendor` directory in project would keep all of the upstream dependencies

• `Gopkg.toml` has any constraints you want to put into it

• `Gopkg.lock` is the current state of the dependency solver

84

dep Commands• `dep init`: Setup `Gopkg` and `vendor`

• `dep ensure`: Regularly keep dependencies in alignment

• Updates to your imports

• Updates to upstream version

• Updates to constraints

85

Use map structure• Complete `dep/main.go`:

• Fill in the appropriate import statement

• Fill in the appropriate data conversion function

• Use `dep` to manage the dependency

• Expected out is similar to JSON exercises

86

shell$ go run main.goProperty1=100

Golang "one-liners"87

http://www.urbandictionary.com/define.php?term=Gorram

Golang Oneliners• The need for `main.main` makes it hard to do a oneliner in Go.

• However, if we make some assumptions about input and output, we can pull it off...

88

90

Lots of options for

inferring input/

output streams

Play with gorram

91

shell$ gorram crypto/sha1 Sum /etc/hostsshell$ echo 12345 | gorram encoding/base64 StdEncoding.EncodeToStringshell$ # is there an equivalent to decode base64?shell$ gorram net/http Get https://www.usenix.org/conference/lisa17

shell$ gorram math/rand Intshell$ gorman math/rand Int # run this twice - what's up?

Notes for the future...• Think about building library functions so that you can refer to them with

tools like Gorram

92

expvar93

• Goal: Monitoring Go processes

• Go concept: expvar

94

95

https://golang.org/pkg/expvar/

96

https://github.com/divan/expvarmon

Use expvar• Instrument `kickstart` using expvar

• Use a browser or command line web client to get data from expvar endpoint

• Examine the metrics with `expvarmon`

97

Caveats...• This is either a really good approach or a really bad approach - you have

to decide for yourself

• Can use different `net/http.Server` and `expvar.Handler` to move where the metrics are exposed (exercise for yourself)

98

GOOS and GOARCH99

Cross Compilation• The Go tool chain has cross compilation built in

• Can easily build on one platform for many platforms

• There are edges with cgo bindings and syscall specific items

• Use GOOS (pronounced goose) and GOARCH (garch) to control the target of the build

100

cttesa/cc// +build darwin

101

// +build darwin,i386

// +build windows

// +build linux,amd64

Attempt Cross Compile• `cc` has three files

• `main-darwin.go`

• `main-linux.go`

• `main-windows.go`

• Attempt to cross compile with various GOOS settings and see the results

102

shell$ GOOS=... go build .shell$ file cc*...

Make your own containers...103


Containers• Goal: Build a limited container by hand

• Go concepts: os/exec, syscall

104

Containers: The Short Short Version• Processes with:

• Isolation (namespaces) - Visibility

• Resource constraints (cgroups)

• Privileges (capabilities)

105

e.g. chroot• The first partial namespace: chroot

• Would give you a different perspective to the root file system

• Extend this out to the full mnt ("ns") namespace, you can control completely different sets of mounts; not just one as a child of the other.

106

Other Namespace• UTS - UNIX Timesharing System (aka hostname/domainname)

• Net

• PID

• IPC

• User

• Cgroup

107

108

https://golang.org/pkg/os/exec/#Cmd

cttesa/container• Containers are built on processes; new processes handled by os/

exec.Cmd

• Can connect process inputs and outputs (just like pipes and redirection)

• And start the command:

cmd := exec.Command("/bin/bash")

109

cmd.Stdin = os.Stdincmd.Stdout = os.Stdoutcmd.Stderr = os.Stderr

err := cmd.Run()

110

https://golang.org/pkg/syscall/#SysProcAttr

111

https://golang.org/pkg/syscall/#pkg-constants

A couple of isolations...• Finish `container/main.go` by setting the appropriate field on `cmd`.

• Isolate hostname: Change the hostname inside and out and show no cross impact

• Isolate network: Run it and see the difference on `ip addr`

112

ssh113

https://upload.wikimedia.org/wikipedia/commons/thumb/3/3d/SSH_Communications_Security_logo.svg/1280px-SSH_Communications_Security_logo.svg.png

• Goal: Setting up ssh in code

• Go concepts: golang.org/x/crypto/ssh

114

Using the ssh library• Three parts to using it

• Get any authentication credentials

• Setup the configuration and connect to the server

• Run session which execute the commands we use

115

116

https://godoc.org/golang.org/x/crypto/ssh#Client

117

https://godoc.org/golang.org/x/crypto/ssh#ClientConfig

cttesa/ssh

• `getKey` gets the auth key needed for ssh key authentication

func getKey() (ssh.Signer) { keyFile := "/home/got/.ssh/id_rsa" pemKey, err := ioutil.ReadFile(keyFile) ... signer, err := ssh.ParsePrivateKey(pemKey) ... return signer}

118

cttesa/ssh

• `connect` uses the auth key to connect to the localhost

func connect(host string, signer ssh.Signer) *ssh.Client { config := &ssh.ClientConfig{ User: "got", Auth: []ssh.AuthMethod{ssh.PublicKeys(signer)}, HostKeyCallback: ssh.InsecureIgnoreHostKey(), } client, err := ssh.Dial("tcp", "127.0.0.1:22", config) ... return client

119

120

https://godoc.org/golang.org/x/crypto/ssh

cttesa/ssh• Start by opening a new session. This is for a single command to run

• As usual, we will want to clean up when we're done

• Connect our process to the remote command by pulling out a StdinPipe

session, err := client.NewSession()

121

defer session.Close()

sin, err := session.StdinPipe()

cttesa/ssh• Start a command inside of the session

• Perform any needed actions

• Wait till the command finishes

err = session.Start(...)

122

session.Wait()

"Copy" a file over ssh• Simulate the action of "echo input | ssh ..."

• Finish `ssh/main.go` to simulate the action of `echo input | ssh ...` to "copy" a file over ssh

• Fill in the command to execute on the remote side

• Pipe data over the ssh session (`io.Copy` will help)

123

Wrap up124


Some laughter

Some thinking

Goals for today

125https://commons.wikimedia.org/wiki/Emoji#/media/File:Twemoji2_1f914.svghttps://commons.wikimedia.org/wiki/File:Twemoji2_1f602.svg

Questions Inspired by each Exercise• Interface: How much do we rely on concrete aspects instead of behaviors?

• Files: How can we provide a smarter response to `find` results?

• Kickstart: Can we package *everything* that we need up into a nice package? What are the trade offs?

• TLS: Is it really that scary to be secure?

• http2: How much is already built into what we're using?

• JSON: How can we make data transformations easy?

126

Questions Inspired by each Exercise• dep: What if other people wanted to use this?

• gorram: How do we structure our code so it's reusable, not just by other libraries, but also by other tools?

• expvar: Can (should?) we use our application ports for signaling and monitoring as well? What are the tradeoffs?

• GOOS/GOARCH: How portable is our code really?

• Containers: What does this tell us about the underlying system?

• SSH: Just because we're working code, are there other methods (e.g. shell-isms) that we can bring into our code?

127

October 29–November 3, 2017 | San Francisco, CAwww.usenix.org/lisa17 #lisa17

Remember to fill in yourtutorial evaluation!

Thank You!

T1 - Close to the Edge Systems AdministrationChris "mac" McEniry

On the Edge Systems Administration with Golang

Technology

Transcript of On the Edge Systems Administration with Golang