Camlistore.orgYour, Personal Datastore for Life

A side project in GoLang by Brad Fitzpatrick. Maintainer Mathieu Lonjaret. Reprised by Clive Boulton for Google Developer Groups NYC. Jan 9, 2017.https://www.meetup.com/gdgnyc/events/234963670/

Entrepreneurs: personal memories are priceless.

Demo

How Brad got Camlistore going Brad Fitzpatrick @bradfitz is a member of the Go team at Google, working primarily on the standard library.

● Brad wanted to create a system to store all of the content he’s created, which is currently scattered across multiple sites and disks. He started hacking in this directory:

$ Content Addressable Multi Level Indexed Storage = Camlistore

Camlistore Goals● Content-addressable● Searchable● Web UI● Extensibility: FUSE, etc.

Majority of Camlistore is pure Go. (Ed. note: It’s a great repo to read and learn from.)

Content-addressable advantages● Sync any which way, no conflicts● (merge resolution at higher layers)● corruption verification/repair● de-duping

Content Addressable: What things are named depends on their content. Two identical things have the same name. For example, the "name" or "key" for the data is the SHA-1 for the data, ala git.

Multi-level● blob storage● indexing● search● web UI, FUSE, CU, apps are just clients of search + blob storage

Multi-Layer: The whole storage stack is built out of several layers. The blob store sits on the bottom, and only knows about bytes, and access is via the SHA-1 of those bytes. Things that you might store (Files, directories, sets, collections of tweets, social graphs, etc) build on top of the blob store by additional blobs that hold pointers to data blobs. Again, it's sort of like git. A front-end might sit on top of that abstraction.

Indexed ● ask the indexer● tags● search = find

Indexed: blobs of JSON that have a few special attributes are recognized and indexed. So, you might have a bunch of blobs with these special attributes (ie, "tag") and be able to ask the indexer "Give me all blobs with tag equal to foo", rather than having to search through the blobs directly.

OverviewPermanoderandom #

Set-attribute camliContent

Directory JSON

FileBytes

FileBytes

FileBytes

FileBytes

DirectoryJSON

FileJSON

FileJSON

unassigned blobs

object

Set-attribute camliContent

Logically

Features of @camlistore● Personal Datastore for life: Like Git for Social Backup. ● Security is private by default (share if you wish, more sharing in v.10)● FUSE file system● Data versioning, time travel ● Snappy UI (react.js like Instagram moving to gopherjs)● Clients in JavaScript, Java, Obj-C, ● Deploy Web, Phone, Home, S3, GCE (Docker)● Targets Desktop Chrome, Firefox, Safari, Edge (last two stable versions).● Targets Safari and Chrome on Android and iOS tablets (last two stable ver)

Why Brad chose Go for Camlistore● He considered a few other candidate languages:● Perl and Python: too slow and single-threaded● C++ and Java: tedious and not fun (slow builds)● Go? Yeah.

As Brad hacked on Camlistore, he realized he needs to implement support for various formats and protocols (like mime/multipart) and improve stdlib packages (like os/exec nee exec, database/sql, etc.).

Camlistore ⇔ 20 per cent time (now a side project)

East deployment to GCP (now) Makes firing up your own Camlistore server on GCE+MySQL+Cloud Storage a few clicks from a web app, using CoreOS, Docker, and the various Google Cloud Platform APIs..

Visit https://camlistore.org/launch/ which lets you create your own camlistore instance on GCE. The Camlistore website will say:

Step 1) go to https://console.developers.google.com/project

Step 2) click "Create Project". remember its name.

Step 3) Enable billing. (tell Google your credit card info) [Ed note. Dev’s sign up for GCP get $x00 credit for 60 days]

Step 4) Enter your project name: [ ]

[ Create ]

You click Create, and then we send you on an OAuth2 voyage to get access to create VMs & DBs & Buckets on your behalf, and then we create it all, including creating a minimal CoreOS VM that boots right up into a tiny Docker container just running camlistore. [Ed note, the API forms are also Camlistore open source]

Web UI● React → gopherjs● Infinite scroll● Lightbox● HTML history API● Search

…

Camlistore Android app: demo time! https://play.google.com/store/apps/details?id=org.camlistore

Command line tools $● camget (blobs, trees, shares)● camput (blobs, files, dirs, shares, permanodes, claims)● camtool {init, sync, init, describe, claims…)● cammount

Camlistore is pure Go: (no C)● HTTP client/server, TLS, AES, …● JSON, XML● WebSockets● SMTP server (coming)● Rolling checksums● GIF/JPG/PNG en/dec, resizing, thumbnails● FUSE● Notification bus● database/sql, MySQL, Postgres, MongoDB● OpenPGP signing & verification

Ed note: NYC is adoption center for Blockchain technologies, also written in GoLang

More @camlistore

https://camlistore.orgReprised by clive boulton twitter @iC