Git with Swift Atlanta 2014

Chmouel Boudjnah!OpenStack Summit 2014 Atlanta

OpenStack Swift as a Backend for Git

Chmouel Boudjnah - @chmouel - eNovance


The mandatory about me


About me

I am not the author, Fabien Boucher aka: @bzho



About me

We both work for eNovance


About me

OpenStacker for quite some time now


So what is Swift?


• Object Storage • Not a file system • Eventually Consistent • HTTP REST API • Very Scalable • Software defined Storage

What is Swift


Why would you want to use GIT with Swift?


Why Swift with GIT

Why not?


Why Swift with GIT

When you use a file system you have to :

Perform backups of your repositories regularly

Perform complex and risky operations to extend the storage capabilities for you repositories.

You have mostly have to manage it yourself

It’s an another piece in your infrastructure

You have to manage the downtime

Not fun


So what are the benefits?


Why Swift with GIT

The advantages of using Swift

Safety: The inherent replication of a data store.

Capability: easily extend storage capacity.

High Availability: no single point of failure.

Maintainability: allow operations teams to maintain the cluster without any downtime.

In an OpenStack environment it’s usually already there.


Let’s see how git internal works


So how did we plug that to Swift?


We did not want to modify the GIT client binaries.


We used a pure python GIT library called Dulwich


• Create, read, manage loose objects (blob, tree, commit, tag).

• Create, read, manage pack files, pack indexes. • Implement the Git smart protocol through git-

upload-pack and git-receive-pack. • Implement the Git, HTTP, SSH listeners to start

Dulwich as a Git server. • Provide a well designed interface to the storage

backend. (Repo, MemoryRepo).

Dulwich


How do we use Dulwhich

We implemented a custom repo called SwiftRepo as a backend to the git listener of Dulwich which communicate to OpenStack Swift


So about that SwiftRepo backend how does it work?


SwiftRepos backend to Dulwich

It doesn’t use the standard way to store Git references (one file by reference).

It doesn’t store loose objects but instead uses the pack format.

In addition to the .index that relate to each pack file we added .info object

It makes the Request to Swift concurrently.


How’s the benchmark then?


Benchmark Setup

Benchmark performed on two architectures : Github : ● Git client from eNovance network <-> GitHub ! Dulwich/Swift : ●· Git client from eNovance network <-> Dulwich (Rax compute) <-> Rax Cloudfiles !Three different repos : ! ●· Swiftsync (873 objects with 293 commits) ●· Edeploy (2246 objects with 618 commits) ●· Swift (23480 objects with 3212 commit)


Full repository push


Clone


That’s it, hope you enjoyed.


Eventual Consistentie

Why Eventual Consistentie window is not an issue

Git with Swift Atlanta 2014

Internet

Transcript of Git with Swift Atlanta 2014