Lattus REST API User’s Guide - Quantum...Lattus REST API User’s Guide xi Preface Document...

User’s Guide

Lattus REST API

6-67815-09 Rev B

ii Lattus REST API User’s Guide

Lattus REST API User’s Guide, 6-67815-09 Rev B, April 2015, Product of USA.

Quantum Corporation provides this publication “as is” without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability or fitness for a particular purpose. Quantum Corporation may revise this publication from time to time without notice.

COPYRIGHT STATEMENT

© 2015 Quantum Corporation. All rights reserved.

Your right to copy this manual is limited by copyright law. Making copies or adaptations without prior written authorization of Quantum Corporation is prohibited by law and constitutes a punishable violation of the law.

TRADEMARK STATEMENT

Quantum, the Quantum Logo, Backup. Recovery. Archive. It's What We Do., Be Certain, Be Quantum Certain, DLT, the DLT Logo, DLTSage, DLTtape, the DLTtape Logo, DXi, DXi Accent, Dynamic Powerdown, FastSense, FlexLink, GoProtect, GoVault, iLayer, Lattus, MediaShield, Optyon, Pocket-sized., Well-armored., Preserving the World's Most Important Data. Yours., Q-Cloud, Quantum Certain, Quantum Certainty, Quantum vmPRO, Scalar, SDLT, SiteCare, SmartVerify, StorageCare, StorNext, Super DLTtape, SuperLoader, and Vision are either registered trademarks or trademarks of Quantum Corporation and its affiliates in the United States and/or other countries. All other trademarks are the property of their respective owners.

Products mentioned herein are for identification purposes only and may be registered trademarks or trademarks of their respective companies. All other brand names or trademarks are the property of their respective owners.

Quantum specifications are subject to change.

Contents

Preface xi

Chapter 1 Introduction 1

Chapter 2 S3 Integration 3

Starting with Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Query String Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Compatibility Calls. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5GET /?location. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5GET /?logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6GET /?versioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7GET /?acl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Compatibility Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Service-API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Objects-API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Buckets-API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Differences Between Lattus S3 and Amazon S3 . . . . . . . . . . . 10Response Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Other Behavioral Differences. . . . . . . . . . . . . . . . . . . . . . . . . . 14

Lattus REST API User’s Guide iii

Contents

Request Headers for GET and HEAD Object . . . . . . . . . . . . . . . . . . . . . 15Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Example Combined Header Request . . . . . . . . . . . . . . . . . . . . 20

Important Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Virtual Host Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Supported Bucketnames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Interpretation of Query Parameters . . . . . . . . . . . . . . . . . . . . 23Custom Headers You Want to Upload With the Object. . . . . . 24Headers You Would Like the Server to Ignore . . . . . . . . . . . . . 24x-amz-meta-* Headers Can be Uploaded with the Object. . . . 24Query String Authentication is Possible . . . . . . . . . . . . . . . . . 24Bucket Calls Can be Disabled . . . . . . . . . . . . . . . . . . . . . . . . . 25Anonymous GET Feature. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Object Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Bucket Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Service Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Error Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32S3 Error Return Code Examples: . . . . . . . . . . . . . . . . . . . . . . . 35

Supported Bucket Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36PUT Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36GET Request (listing of objects). . . . . . . . . . . . . . . . . . . . . . . . 36HEAD Request. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Supported Multipart Bucket Operations . . . . . . . . . . . . . . . . . . . . . . . 40List Multipart Uploads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Supported Object Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42PUT Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42GET Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43HEAD Request. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44DELETE Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44PUT - Copy Request. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Supported Multipart Object Operations . . . . . . . . . . . . . . . . . . . . . . . 46Initiate Multipart Upload . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Upload Multipart Part . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Complete Multipart Upload . . . . . . . . . . . . . . . . . . . . . . . . . . 49Abort Multipart Upload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51List Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51Multipart Upload (Copy) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

iv Lattus REST API User’s Guide

Contents

Supported Service Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54GET Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Chapter 3 REST Definitions and Configuration 57REST Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

How To Enable REST in Lattus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Using Curl to Demonstrate the REST Functionality . . . . . . . . . . . . . . . 58

Other Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

API Interface Documentation Assumptions . . . . . . . . . . . . . . . . . . . . . 59

Resource Identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Transfer Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Behavior Common to All Requests. . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

What Has Changed Compared to the 3.2.X REST API . . . . . . . . . . . . . 61

Chapter 4 Design Considerations 63

Designing for Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Connection Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Parallelism. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65Partial Reads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65Overwrites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66Caching. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Designing for Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66Namespace Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66Connection Locality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Chapter 5 Policy Operations 69

Create Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69http / text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 xml. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Get Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74http / text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Lattus REST API User’s Guide v

Contents

xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

List Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77http / text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Delete Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Chapter 6 Namespaces 81

List Namespaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81http/text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

List Objects in a Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83http/text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Create Namespace. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86http/text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Update Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91http/text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Get Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95http/text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

List Namespaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98http/text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Delete Namespace. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

GET/HEAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

List Namespace Capacity Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . 101json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

vi Lattus REST API User’s Guide

Contents

List Syncstores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Get Syncstore Capacity Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Chapter 7 Object Operations 113

Create or Overwrite Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Transfer with Content-Length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114http. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

Transfer Without Content-Length . . . . . . . . . . . . . . . . . . . . . . . . . . . 117Request Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117If-Match . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118If-Modified-Since . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 If-None-Match . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119If-Unmodified-Since . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Transfer-Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

Response Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Create Object with Generated Name. . . . . . . . . . . . . . . . . . . . . . . . . 121

Using Curl to Upload a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

Create an Object with Generated Name . . . . . . . . . . . . . . . . . . . . . . 123

Create Directory Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123http. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125Curl Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Custom Metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126http. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129Apply Updates to Existing Custom Metadata . . . . . . . . . . . . 130http. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

Lattus REST API User’s Guide vii

Contents

List Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133http / text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136Using Curl to List the Objects in a Namespace . . . . . . . . . . . 136

Move Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137Extra Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Get Object Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138Complete Content. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138Request Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138If-Match . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138If-Modified-Since . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 If-None-Match . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139If-Unmodified-Since . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140Range . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140Response Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141Using Curl to Get an Object . . . . . . . . . . . . . . . . . . . . . . . . . 142

Get Object Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142http / text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Delete Object. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145Using Curl to Delete an Object . . . . . . . . . . . . . . . . . . . . . . . 145

Adding A user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146http / text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

List Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150http / text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

GET User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152http / text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152json. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

Delete A User. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

viii Lattus REST API User’s Guide

Contents

Chapter 8 Authentication 157

Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

How It Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

REST Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

API Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159Users. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159Paths and Flags. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159Q-Shell Interface for User Management . . . . . . . . . . . . . . . . 160Q-shell Interface for Assignment of Rights to a Specific Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

REST Authentication in Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166WWW-Authenticate header . . . . . . . . . . . . . . . . . . . . . . . . . 166Authorization header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166Curl Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

Getting Started with Authentication . . . . . . . . . . . . . . . . . . . . . . . . . 168Out of the Box Authentication . . . . . . . . . . . . . . . . . . . . . . . 168Before you Start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168Allowing an Admin User to Create Namespaces and Polcies. 168Allowing Another User to Only Read Data from a Specific Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169Inheriting Permissions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169Tips and Tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

Chapter 9 Return Codes and Troubleshooting 173

Return Codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Chapter 10 Microsoft .net Bindings 177

AmplistorClientLib Class Documentation. . . . . . . . . . . . . . . . . . . . . . 177AmplistorClientLib.Types.AmpliObject Class Reference - Public Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177AmplistorClientLib.Types.AmpliPolicy Class Reference - Public Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

Lattus REST API User’s Guide ix

Contents

AmplistorClientLib Member Function Documentation. . . . . . . . . . . . 178Policy Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178Namespace Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179Object Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181Various Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185Getting the .net Bindings . . . . . . . . . . . . . . . . . . . . . . . . . . . 185Create a Policy and Namespace . . . . . . . . . . . . . . . . . . . . . . 185Create a Lattus Connection Object . . . . . . . . . . . . . . . . . . . . 185Upload an Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

Chapter 11 Working With REST 187Starting the q-shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187Setting up a Test Environment . . . . . . . . . . . . . . . . . . . . . . . 187Starting with Development . . . . . . . . . . . . . . . . . . . . . . . . . . 187Resetting the Environment . . . . . . . . . . . . . . . . . . . . . . . . . . 188

Chapter 12 Working with Lattus S3 189

Starting the q-shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

Setting up an S3 Test Environment . . . . . . . . . . . . . . . . . . . . . . . . . . 189

Enabling S3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190Adding a User for S3 Authentication . . . . . . . . . . . . . . . . . . 190Configure the Client Daemon to Listen on its S3 Interface . . 190Updating the HTTPS Proxy . . . . . . . . . . . . . . . . . . . . . . . . . . 190Defining the Policy for S3 Bucket Creation . . . . . . . . . . . . . . 191

Testing Your Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192Create a New Bucket. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193List Buckets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193Put a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193Get a File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193List Files in a Bucket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193List All Objects in All Buckets . . . . . . . . . . . . . . . . . . . . . . . . 194Delete File From a Bucket . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

Rebooting an S3 Test environment . . . . . . . . . . . . . . . . . . . . . . . . . . 194

Tearing Down an S3 Test Environment . . . . . . . . . . . . . . . . . . . . . . . 195

x Lattus REST API User’s Guide

Contents

Getting Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195Output of Following qshell Command . . . . . . . . . . . . . . . . . 195Clientdaemon Log File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Lattus REST API User’s Guide xi

Contents

xii Lattus REST API User’s Guide

Preface

Document Organization Following is a list of chapters in this guide:

• Chapter 1, Introduction

• Chapter 2, S3 Integration

• Chapter 3, REST Definitions and Configuration

• Chapter 4, Design Considerations

• Chapter 5, Policy Operations

• Chapter 6, Namespaces

• Chapter 7, Object Operations

• Chapter 8, Authentication

• Chapter 9, Return Codes and Troubleshooting

• Chapter 10, Microsoft .net Bindings

• Chapter 11, Working With REST

• Chapter 12, Working with Lattus S3

Lattus REST API User’s Guide xi

Preface

Notational Conventions This manual uses the following conventions:

The following formats indicate important information:

Note: Note emphasizes important information related to the main topic.

Caution: Caution indicates potential hazards to equipment or data.

WARNING: Warning indicates potential hazards to personal safety.

Convention Example

User input is shown in bold font. grep next-server /opt/qbase3/cfg/dhcpd/dhcpd.conf# dhcp-helper -s 10.1.0.204.

Computer output and command line examples are shown in monospace font.

GET /manage/namespace/VOL?meta=json HTTP/1.1Host: my.example.comDate: _date_

User input variables are enclosed in angle brackets.

#ip address <ip address>

For UNIX and Linux commands, the command prompt is implied.

hostname <name>is the same as#hostname <name>

File and directory names, menu commands, button names, and window names are shown in bold font.

“In the Add Namespace window, select the number of namespaces you want to add.”

Menu names separated by arrows indicate a sequence of menus to be navigated.

Dashboard > Administration > Storage Management > Storage Policies

xii Lattus REST API User’s Guide

Preface

• Right side of the system — Refers to the right side as you face the component being described.

• Left side of the system — Refers to the left side as you face the component being described.

• b — All binary numbers are succeeded by “b.”

• h — All hexadecimal numbers are succeeded by “h.”

• Error or attention conditions are represented in parenthesis that translate as follows:

(SK=S ASC=AA ASCQ=QQ)

where:

S — hexadecimal sense key value

AA — hexadecimal additional sense code

QQ — hexadecimal additional sense code qualifiers

Product Safety Statements

Quantum will not be held liable for damage arising from unauthorized use of the product. The user assumes all risk in this aspect.

This unit is engineered and manufactured to meet all safety and regulatory requirements. Be aware that improper use may result in bodily injury, damage to the equipment, or interference with other equipment.

WARNING: Before operating this product, read all instructions and warnings in this document and in the system, safety, and regulatory guide.

Lattus REST API User’s Guide xiii

Preface

xiv Lattus REST API User’s Guide

Preface

Contacts Quantum Home Page

Visit the Quantum home page at:

http://www.quantum.com

Getting More Information or Help

• Service and Support Website - Register products, license software, browse Quantum Learning courses, check backup software and operating system support, and locate manuals, FAQs, firmware downloads, product updates and more in one convenient location. Benefit today at:

http://www.quantum.com/ServiceandSupport/Index.aspx

• eSupport - Submit online service requests, update contact information, add attachments, and receive status updates via email. Online Service accounts are free from Quantum. That account can also be used to access Quantum’s Knowledge Base, a comprehensive repository of product support information. Sign up today at:

http://www.quantum.com/osr

For further assistance, or if training is desired, contact the Quantum Customer Support Center:

United States 1-800-284-5101 (toll free)+1-720-249-5700

EMEA +800-7826-8888 (toll free)+49-6131-3241-1164

APAC +800-7826-8887 (toll free)+603-7953-3010

For worldwide support:http://www.quantum.com/ServiceandSupport/Index.aspx

Lattus REST API User’s Guide xv

http://www.quantum.com/osr

http://www.quantum.com/ServiceandSupport/Index.aspx



Preface

Worldwide End-User Product Warranty

For more information on the Quantum Worldwide End-User Standard Limited Product Warranty:

http://www.quantum.com/pdf/QuantumWarranty.pdf

xvi Lattus REST API User’s Guide

http://www.quantum.com/pdf/QuantumWarranty.pdf

Chapter 1Introduction

Quantum Lattus™ Object Storage is disk-based storage that meets the extreme scalability, durability, and access requirements of large-scale Big Data archives.

Lattus object storage offers several different access methods to put data in and get data out of the Lattus storage, including StorNext access, Lattus A10 Access for NFS/CIFS, and HTTP/HTTPS REST access for customers who have ported their applications to use the Lattus REST APIs.

This document is designed for the REST API. You can adopt your applications to talk directly to the Lattus C10 controllers to be able to write objects natively to the Lattus system.

Lattus REST API User’s Guide 1

Chapter 1: Introduction

2 Lattus REST API User’s Guide

Chapter 2S3 Integration

Starting with Development

Quantum supports authentication and authorization on the bucket level (not on the object level).

In the following chapters, take into account the following:

• "bucketname" is usually the Lattus namespace name.

• The username is admin (having sufficient rights to perform the requested operation).

• This client daemon listens by default on port 7070 for S3 calls.

Authentication

The authentication information for S3 is hosted in the header field Authorization. This is of the form:

Authorization: AWSAccessKeyId:<Signature>


Chapter 2: S3 IntegrationAuthentication

For more information, see: http://docs.amazonwebservices.com/AmazonS3/latest/dev/RESTAuthentication.html

There is no anonymous user, every call to Lattus S3 needs to carry authentication information along with it.

Query String Authentication

You can authenticate certain types of requests by passing the required information as query-string parameters instead of using the Authorization HTTP header.

This is useful for enabling direct third-party browser access to your private Amazon S3 data without proxying the request. The idea is to construct a “pre-signed” request and encode it as a pre-signed URL that an end user’s browser can retrieve. Additionally, you can limit a pre-signed request by specifying an expiration time.

Instead of using special HTTP headers, the required authentication elements are specified as query string parameters:

Parameter Name Description Example Value

Expires The time when the signature expires, specified as the number of seconds since the epoch (00:00:00 UTC on January 1, 1970).A request received after this time (according to the server), will be rejected.

1141889120


http://docs.amazonwebservices.com/AmazonS3/latest/dev/RESTAuthentication.html

http://docs.amazonwebservices.com/AmazonS3/latest/dev/RESTAuthentication.html

Chapter 2: S3 IntegrationCompatibility Calls

For more information, see: http://docs.amazonwebservices.com/AmazonS3/latest/dev/RESTAuthentication.html#RESTAuthenticationQueryStringAuth.

Compatibility Calls

To allow compatibility with some applications, the following calls have not been implemented but will return a dummy response. The dummy response eliminates the need for an exception handler in the calling routine.

GET /?location http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETlocation.html

s3cmd info s3://bucketname

AWSAccessKeyId Your AWS Access Key Id. The value is alphanumeric (A-Z, a-z and/or 0-9) and up to 256 characters.This specifies the AWS Secret Access Key used to sign the request, and (indirectly) the identity of the developer making the request.

AKIAIOSFODNN7-EXAMPLE

Signature The URL encoding of the Base64 encoding of the HMAC-SHA1 of StringToSign.

vjbyPxybdZaNmGa%2-ByT272YEAiv4%3D

Parameter Name Description Example Value


http://docs.amazonwebservices.com/AmazonS3/latest/dev/RESTAuthentication.html#RESTAuthenticationQueryStringAuth.

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETlocation.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETlocation.html


Request:

GET /?location HTTP/1.1Date: Mon, 26 Nov 2012 20:31:36 GMTAuthorization: AWS admin:W8QyXloRQVXXGeLkMzDE223iYMg=User-Agent: jclouds/1.5.0-beta.6 java/1.7.0_09Host: testfolder2.s3.amazonaws.com:7070Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2Connection: keep-alive""

Response:

<?xml version=\"1.0\" encoding=\"UTF-8\"?><LocationConstraintxmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"/>

GET /?logging http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETlogging.html

s3cmd accesslog s3://bucketname

Request:

GET /bucketname?logging HTTP/1.1Host: 127.0.0.1:7070Accept: */*x-amz-acl:authenticate-readx-amz-date:Tue, 27 Nov 2012 08:43:34 GMTAuthorization: AWS admin:bAGHIFf98WMPu1FGpAmTAEHeaJM=""

Response:

<?xml version=\"1.0\" encoding=\"UTF-8\"?><BucketLoggingStatus


http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETlogging.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETlogging.html


xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><LoggingEnabled><TargetBucket>mybucketlogs</TargetBucket><TargetPrefix>mybucket-access_log</TargetPrefix></LoggingEnabled></BucketLoggingStatus>

GET /?versioning http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETversioningStatus.html

[no s3cmd call exists]

Request:

GET /bucketname?versioning HTTP/1.1Host: 127.0.0.1:7070Accept: */*x-amz-acl:authenticate-readx-amz-date:Tue, 27 Nov 2012 08:43:34 GMTAuthorization: AWS admin:iC1TR8CvqFYxM+QOoCG9PY0r6cg=""

Response:

<?xml version=\"1.0\" encoding=\"UTF-8\"?><VersioningConfigurationxmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"/>

GET /?acl http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETacl.html

s3cmd info s3://bucketname

Request:

GET /?acl HTTP/1.1Date: Mon, 26 Nov 2012 21:22:36 GMTAuthorization: AWS admin:05MWHGy46KX0gW6mL5DN1v6FfC8=User-Agent: jclouds/1.5.0-beta.6 java/1.7.0_09Host: testfolder5.s3.amazonaws.com:7070


http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETversioningStatus.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETversioningStatus.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETacl.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETacl.html

Chapter 2: S3 IntegrationCompatibility Table

Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2Connection: keep-alive""

Response:

<?xml version=\"1.0\" encoding=\"UTF-8\"?><AccessControlPolicyxmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"/>

Compatibility Table

Service-API The following table shows which Service-API methods are supported and which are not supported.

Objects-API The following table shows which Objects-API methods are supported and which are not supported.

Supported Methods Non-Supported Methods

GET --


DELETE Object Delete Multiple Objects

GET Object (*) GET Object ACL

HEAD Object (*) GET Object torrent

PUT Object POST Object

PUT Object - Copy PUT Object ACL



(*) The GET and HEAD Object requests support “if-*” headers. For more information, refer to see the section Request Headers for GET and HEAD Objects.

Buckets-API The following table shows which Buckets-API methods are supported and which are not supported.

Initiate Multipart Upload

Upload Part

Upload Part - Copy

Complete Multipart Upload

Abort Multipart Upload

List Parts



DELETE Bucket DELETE Bucket lifecycle

GET Bucket (List Objects) DELETE Bucket policy

GET Bucket acl (dummy response)

DELETE Bucket website

GET Bucket location (dummy response)

GET Bucket lifecycle

GET Bucket logging (dummy response)

GET Bucket policy

GET Bucket versioning (dummy response)

GET Bucket notification

HEAD Bucket GET Bucket Object versions

List Multipart Uploads GET Bucket requestPayment

PUT Bucket GET Bucket website



Differences Between Lattus S3 and Amazon S3

List Multipart Uploads

PUT Bucket acl

PUT Bucket lifecycle

PUT Bucket policy

PUT Bucket logging

PUT Bucket notification

PUT Bucket requestPayment

PUT Bucket versioning

PUT Bucket website


Situation Amazon AWS S3 Lattus S3

Number of characters for an object key name

Maximum up to 1024 characters

More than 1024 is allowed

Size of custom metadata perobject

Maximum of 2 KiB More than 2 KiB is allowed (see limitations)

Largest object that can be uploaded in a single PUT

5 GB More than 5 GB (which may result in errors)

Dummy headers HTTP 200 OK Returns errors

Creating an existing bucket HTTP 200 OK Returns an HTTP 409: conflict

Creating bucket with special characters ({, }, <, >, [, ] | ‘, ^, ”)

HTTP 400 Bad Request Returns an HTTP 501: Not Implemented



Object name with [, ], \, and ^ with s3cmd, unverified with other clients

HTTP 200 OK Returns an HTTP 501: Not Implemented

• S3 request with “Expires” parameter• S3 request with “TE” parameter

HTTP 200 OK Returns an HTTP 501: Not Implemented

Deleting non-existent objects HTTP 200 OK Returns errors

Calling GET Bucket (List Objects) with max-keys greater than 1000

Works without problems More than 1000 is not allowed, due to possible stack overflow

Performing a HEAD HTTP operation on a Bucket or key that does not exist

Returns an HTTP 404: page not found

Returns an HTTP 404: page not ound, together with the reason in the body:• NoSuchBucket: The specified

bucket does not exist• NoSuchKey: The specified

key does not exist

Performing a HEAD HTTPoperation on a bucket while the user does not have sufficient permissions to access them

Returns an HTTP 403: forbidden

Returns an HTTP 403: forbidden, together with the reason in the body:• AccessDenied: Access

Denied




Cancelled multipart upload and then retry to upload another part.

Socket timeout HTTP 404: Not Found<?xml version="1.0"encoding="UTF-8"?><Error><Code>NoSuchUpload</Code><Message>The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.</Message> <RequestId></RequestId><Resource>testfile7</Resource></Error>

The timezone string 'CETrandomstring' is interpreted differently

This string is interpreted as 'CET' by Amazon S3

This string is interpreted as 'CETran' by Lattus, which is an unknown timezone

Request with unsupported time zone format

HTTP 403 Forbidden HTTP 500 Internal Server Error

Request without date header HTTP 403 Forbidden HTTP 500 Internal Server Error

Request with unsupported HTTP version

HTTP 505 HTTP Version Not Supported

HTTP 500 Internal Server Error

Request with bad URL encoding

HTTP 400 Invalid URI: EOF HTTP 501 Not Implemented

Request with invalid path HTTP 400 Invalid URI HTTP 500 Internal Server Error

Request with different “host” than the one configured in Lattus

HTTP 301 Moved Permanently HTTP 400 Bad Request

Incorrect md5sum value The Content-MD5 you specified did not match what we received

The Content-MD5 you specified was an invalid

HEAD bucket versioning request

HTTP 405 Method Not Allowed HTTP 501 Not Implemented




Response Differences

Request with Accept-Encoding: Identity

Transfer-Encoding is chunked Lattus ignores header and Transfer-Encoding is not chunked


Lattus Response Difference Request

• Extra period (”.”) mark • Missing element

“BucketName” in response

Delete non-empty bucket

Create too many buckets for the same S3 user

Content-Type in header has extra ”;charset=UTF-8”

HEAD bucket

Extra headers:• Content-Type• Pragma• Cache-Control• Expires• Date (omitted)

GET bucket versioning, dummy response

Missing elements:• CurrentNumberOfBuckets• AllowdNumberOfBuckets• HostId

Extra element:• Resource

Create too many buckets for the same S3 user

Missing elements:• BucketName• HostId

Put object in non-existent bucket



Other Behavioral Differences

Following are other limitations of Lattus S3:

• Using one of the following response-header query parameters in the GET Object and Head request will not result in the correct response:

• response-content-type

• response-content-language

• response-expires

• response-cache-control

• response-content-disposition

• response-content-encoding

• Trying to PUT an object with the following headers may fail:

• x-amz-storage-class

• x-amz-server-side-encryption

• x-amz-website-redirect-location

• x-amz-grant-read

• x-amz-grant-write

• x-amz-grant-read-acp

• x-amz-grant-write-acp

• x-amz-grant-full-control

• The following common response headers are not implemented:

• x-amz-id-2

• x-amz-request-id

• The "x-amz-metadata-directive" header is not implemented.

• When trying to to do a server-side copy, the following x-amz-copy-source-if-* headers are not implemented:

Missing elements:• Key• HostId

GET /* where response is NoSuchKey

Lattus Response Difference Request


Chapter 2: S3 IntegrationRequest Headers for GET and HEAD Object

• x-amz-copy-source-if-match

• x-amz-copy-source-if-none-match

• x-amz-copy-source-if-unmodified-since

• x-amz-copy-source-if-modified-since

• GET SERVICE requests are unable to handle several Query parameters when using a webdrive client. Be sure to specify a bucketname in the request.

Request Headers for GET and HEAD Object

The GET and HEAD Object requests support “if-*” headers. These include:

• If-Match: return the object if its entity tag (ETag) is the same as the one specified, or return HTTP 412 (precondition failed, XML response).

• If-Modified-Since: return the object if it has been modified since the specified time, or return HTTP 304 (object not modified).

• If-None-Match: return the object if its entity tag (ETag) is different from the one specified, or return HTTP 304 (object not modified).

• If-Unmodified-Since: return the object if it has not been modified since the specified time, or return HTTP 412 (precondition failed, XML response).

The time stamps used for If-Modified-Since and If-Unmodified-Since, must comply with the specifications of RFC2616.

The preferred time stamp is the RFC-822 (updated by RFC-1123): ‘Day, DD Mon YYYY HH:MM:SS GMT’.

The time stamp is case sensitive. See the following list:

• Day: Mon, Tue, Wed, Thu, Fri, Sat, Sun

• DD (*): two digits for the day of the month

• Mon: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec

• YYYY: four digits for the year



• HH: two digits for the hour

• MM: two digits for the minutes

• SS: two digits for the seconds

• MT: time stamp must be represented in GMT

Note: If you don’t use two digits for the day of the month with an If-Modified-Since or If-Unmodified-Since request, you should receive respectively an HTTP 304 and HTTP 412 message. However, in Lattus you receive an HTTP 200 message but with warnings in the client daemon log.

The implementation of these headers are according the RFC2626. More information about these header optionscan be found in the RFC2626, section 14.24, 14.25, 14.26, and 14.27. (See http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html for more information.)

You can also combine the following requests:

• If-Match and If-Unmodified-Since

• If-None-Match and If-Modified-Since

If-Match If-Unmodified-Since Result

True True HTTP 200, object returned

True False XML Error 412 (precondition failed)

False True XML Error 412 (precondition failed)

False False XML Error 412 (precondition failed)

If-Match If-Unmodified-Since Result

True True HTTP 200, object returned

True False HTTP 200, object returned

False True HTTP 200, object returned

False False HTTP 304 (not modified)


http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html


Examples Object created/modified on: Mon Jan 20 09:00:00 2014 etag: dbb126075a1d0b4ca64c3e2ae0159bc7

If-Match

• Matching ETag (HTTP 200):curl -X GET "http://monty.s3.amazonws.com:7070/python" -H "if-match:dbb126075a1d0b4ca64c3e2ae0159bc7"

HTTP/1.1 200 OKx-amz-id-2: V+INEFf6K34RtoqgQGfhDBiWo2oUauc8oh9Jpm2aA/XahYK8s3lt5s+pNGztkJh4x-amz-request-id: 8A0EA6052595BDB2Date: Fri, 14 Feb 2014 12:34:01 GMTLast-Modified: Mon, Jan 20 09:00:00 2014 GMTETag: "dbb126075a1d0b4ca64c3e2ae0159bc7"Accept-Ranges: bytesContent-Type: application/octet-streamContent-Length: 4Server: AmazonS3

• Different ETag (HTTP 412 Precondition Failed XML Response):curl -X GET "http://monty.s3.amazonws.com:7070/python" -H "if-match:dbb126075a1d0b4ca64c3e2ae0159xv3"

<Error><Code>PreconditionFailed</Code><Message>At least one of the pre-conditions you specified did not hold</Message><Condition>If-Match</Condition><RequestId>F58D72A68D9CF0C2</RequestId><HostId>xdvNIe0dyKfGOPu9lfwWcMbKKReL3oyO+vqRHtWI591rtqD7dZxHax02uOmr/HWM</HostId></Error>



If-None-Match

• Matching ETag:curl -X GET "http://monty.s3.amazonws.com:7070/python" -H "if-match:dbb126075a1d0b4ca64c3e2ae0159bc7"

–> returns HTTP Error 304

HTTP/1.1 304 Not Modifiedx-amz-id-2: T7kHsSgFbRy+DLcO2Y+jNkC/ZKVJN5KkKsAolS6Lmu9f0siX47sAdUEQsq/gzAr9x-amz-request-id: 58449835A39BB796Date: Fri, 14 Feb 2014 10:55:57 GMTLast-Modified: Mon, Jan 20 09:00:00 2014 GMTETag: "dbb126075a1d0b4ca64c3e2ae0159bc7"Server: AmazonS3

• Different ETag:curl -X GET "http://monty.s3.amazonws.com:7070/python" -H "if-match:dbb126075a1d0b4ca64c3e2ae0159xv3"

HTTP/1.1 200 OKx-amz-id-2: V+INEFf6K34RtoqgQGfhDBiWo2oUauc8oh9Jpm2aA/XahYK8s3lt5s+pNGztkJh4x-amz-request-id: 8A0EA6052595BF2CDate: Fri, 14 Feb 2014 12:34:01 GMTLast-Modified: Mon, Jan 20 09:00:00 2014 GMTETag: "dbb126075a1d0b4ca64c3e2ae0159bc7"Accept-Ranges: bytesContent-Type: application/octet-streamContent-Length: 4Server: AmazonS3

If-Modified-Since

• Object modified since given date:



curl -X GET "http://monty.s3.amazonaws.com:7070/python" -H"if-modified-since: Tue Feb 18 08:38:47 2011"

HTTP/1.1 200 OKx-amz-id-2: V+INEFf6K34RtoqgQGfhDBiWo2oUauc8oh9Jpm2aA/XahYK8s3lt5s+pNGztkJh4x-amz-request-id: 8A0EA605259D345ADate: Fri, 14 Feb 2014 12:34:01 GMTLast-Modified: Mon, Jan 20 09:00:00 2014 GMTETag: "dbb126075a1d0b4ca64c3e2ae0159bc7"Accept-Ranges: bytesContent-Type: application/octet-streamContent-Length: 4Server: AmazonS3

• Object not modified since given date:curl -X GET "http://monty.s3.amazonaws.com:7070/python" -H"if-modified-since: Tue Feb 18 08:38:47 2015HTTP/1.1 304 Not Modifiedx-amz-id-2: T7kHsSgFbRy+DLcO2Y+jNkC/ZKVJN5KkKsAolS6Lmu9f0siX47sAdUEQsq/gzAr9x-amz-request-id: 58449835A39BB796Date: Fri, 14 Feb 2014 10:55:57 GMTLast-Modified: Mon, Jan 20 09:00:00 2014 GMTETag: "dbb126075a1d0b4ca64c3e2ae0159bc7"Server: AmazonS3

If-Unmodified-Since

• Object modified since given date:curl -X GET "http://monty.s3.amazonaws.com:7070/python" -H"if-unmodified-since: Tue Feb 18 09:40:53 2011"

<Error><Code>PreconditionFailed</Code>



<Message>At least one of the pre-conditions you specified did not hold</Message><Condition>If-Unmodified-Since</Condition><RequestId>F58D72A68D9CFDE1</RequestId><HostId>xdvNIe0dyKfGOPu9lfwWcMbKKReL3oyO+vqRHtWI591rtqD7dZxHax02uOmr/HWM</HostId></Error>

• Object not modified since given date:curl -X GET "http://monty.s3.amazonaws.com:7070/python" -H"if-unmodified-since: Tue Feb 18 09:38:47 2015"

HTTP/1.1 200 OKx-amz-id-2: V+INEFf6K34RtoqgQGfhDBiWo2oUauc8oh9Jpm2aA/XahYK8s3lt5s+pNGztkJh4x-amz-request-id: 8A0EA605259DCEF9Date: Fri, 14 Feb 2014 13:34:01 GMTLast-Modified: Mon, Jan 20 09:00:00 2014 GMTETag: "dbb126075a1d0b4ca64c3e2ae0159bc7"Accept-Ranges: bytesContent-Type: application/octet-streamContent-Length: 4Server: AmazonS3

Example Combined Header Request

• If-Match combined with If-Unmodified-Since:curl -X GET "http://monty.s3.amazonaws.com:7070/python" -H"if-unmodified-since: Tue Feb 18 09:40:53 2011" -H "if-match:dbb126075a1d0b4ca64c3e2ae0159bc6"

• If-None-Match combined with If-Modified-Since:curl -X GET "http://monty.s3.amazonaws.com:7070/python" -H"if-modified-since: Tue Feb 18 09:40:53 2011" -H "if-none-match:dbb126075a1d0b4ca64c3e2ae0159bc6"


Chapter 2: S3 IntegrationImportant Information

Important Information

S3 uses a virtual host style. For more information and virtual hosting examples, see: http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/VirtualHosting.html#VirtualHostingExamples

Virtual Host Style If Your Host Header Ends in s3.amazonaws.com

As such, in the header bucket1.s3.amazonaws.com bucket1 is a valid bucketname.

If the host header has this format, the rest of the URL is the key to put in this bucket.

Example:PUT /myobjectname HTTP/1.1Host: mynewbucket.s3.amazonaws.comContent-Length: 0Content-Type: binary/octet-streamx-amz-acl: authenticated-readx-amz-date: Mon, 27 Aug 2012 13:41:44 GMTAuthorization: AWS admin:3gXQTb+Qx5/v2JmE18LlVyiuoc8=Expect: 100-continue""

If Your Host Header Ends in Something Else

When you want to send a host header with the following format:Host: testbucket.s3.mydomain.com:10101

You will need to add a configuration entry to the clientdeamon config file:

[s3]domain=s3.mydomain.com

Restart the clientdaemon for this change to be applied.

Requests using the abovementioned header will now result in a bucketname called testbucket


http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/VirtualHosting.html#VirtualHostingExamples

http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/VirtualHosting.html#VirtualHostingExamples


Virtual Host Style Requests (Bucket Name as Part of the URL)

In this style of requests, the bucketname is part of the Host header field as first part of the URL.

Note: For all examples below, the header field is assumed to be:"Host: s3.amazonaws.com" or "Host: s3.cloudparrot.com" (if configured as such in the [s3] section in the domain field)

Examples:

PUT a bucket:

"PUT http://amplibucket.s3.amazonaws.com/ HTTP/1.1" puts the bucket with name amplibucket.

DELETE a bucket:

"DELETE http://amplibucket.s3.amazonaws.com/ HTTP/1.1" deletes the amplibucket.

PUT a file with the name "fileA" in the amplibucket:

"PUT http://amplibucket.s3.amazonaws.com/fileA HTTP/1.1" puts fileA in the amplibucket.

Path Style Requests

Note: We will assume path style request in all examples below and bucket name in the URL path.

In this style of requests; the host header has one of the following values:

• Either the exact string: "Host: s3.amazonaws.com" or "Host: s3.cloudparrot.com" (if configured as such in the [s3] section in the domain field).

• For bucket names that are not DNS aware: "Host: 192.169.12.148:7070".

• Is omitted completely.

PUT a bucket:

"PUT http://s3.amazonaws.com/amplibucket HTTP/1.1" puts the bucket with name amplibucket.



DELETE a bucket:

"DELETE http://s3.amazonaws.com/amplibucket HTTP/1.1" deletes the amplibucket.

PUT a file named "fileA" in the amplibucket:

"PUT http://s3.amazonaws.com/amplibucket/fileA HTTP/1.1" puts fileA in the amplibucket..

Supported Bucketnames

Use DNS compliant bucket names that comply to following rules:

• Bucket names must be at least 3 and no more than 63 characters long.

• Bucket name must be a series of one or more labels separated by a period (.), where each label:

• Must start with a lowercase letter or a number.

• Must end with a lowercase letter or a number.

• Can contain lowercase letters, numbers and dashes.

• Bucket names must not be formatted as an IP address (e.g., 192.168.5.4).

More information can be found here: http://docs.amazonwebservices.com/AmazonS3/latest/dev/BucketRestrictions.html

Interpretation of Query Parameters

At the server side we process the following 7 query parameters:

• With the GET bucket request:

• prefix

• delimiter

• marker

• max-keys

• With the GET object request:

• Expires

• AWSAccessKeyId

• Signature


http://docs.amazonwebservices.com/AmazonS3/latest/dev/BucketRestrictions.html

http://docs.amazonwebservices.com/AmazonS3/latest/dev/BucketRestrictions.html


All other query parameters are not implemented and will throw an error to the client.

To avoid getting a client error you can do the following:

1 add to the client daemon configuration file:

[s3]

query_params_to_ignore=ignoreme

Custom Headers You Want to Upload With the Object

If you want to upload custom headers, configure the following:

[s3]

custom_headers_to_upload=uploadme

Now the header "uploadme", together with its value, will be stored with the object metadata.

Note: This is only supported for PUT object.

Headers You Would Like the Server to Ignore

If you want the server to ignore some headers, you can do the following:

[s3]headers_to_ignore=Origin

This way the server will not respond with an error if you send the Origin header. It will ignore it.

x-amz-meta-* Headers Can be Uploaded with the Object

For more information, see http://docs.amazonwebservices.com/AmazonS3/latest/dev/UsingMetadata.html.

Query String Authentication is Possible

We now can also do authentication when the GET request of an OBJECT specifies the 3 query parameters:

• Expires


http://docs.amazonwebservices.com/AmazonS3/latest/dev/UsingMetadata.html

http://docs.amazonwebservices.com/AmazonS3/latest/dev/UsingMetadata.html


• AWSAccessKeyId

• Signature

For more information, see: http://docs.amazonwebservices.com/AmazonS3/latest/dev/RESTAuthentication.html#RESTAuthenticationQueryStringAuth.

Bucket Calls Can be Disabled

The following three calls are enabled by default:

• enable_bucket_list

• enable_bucket_create

• enable_bucket_delete

You can disable them by setting the following parameters:[s3]enable_bucket_list = false # disables listing of buckets (does not disable listing of the content of buckets)enable_bucket_create = false # disables creation of bucketsenable_bucket_delete = false # disables deletion of buckets

If these calls are disabled, you will see the following error message: "action is disabled or not allowed on bucket"

You will receive the "Exc.MethodNotAllowed" exception.

Anonymous GET Feature

Anonymous GET Access

If an S3 bucket has READ access enabled for the special "everyone" user, then unauthenticated clients (including web browsers) can read objects within that bucket.

By default, the "everyone" user does not get any access to S3 buckets, but that access can be turned on with Q shell.

When anonymous GET is enabled, regular HTTP clients can fetch objects using URLs like http://s3.amplistore.host.com/bucketname/objectname.


http://docs.amazonwebservices.com/AmazonS3/latest/dev/RESTAuthentication.html#RESTAuthenticationQueryStringAuth



Chapter 2: S3 IntegrationPermissions

Anonymous LIST Permission

If an S3 bucket has LIST access enabled for the "everyone" user, then unauthenticated clients can list the contents of that bucket by fetching a URL like http://s3.amplistore.host.com/bucketname.

The listing is an xml document in the format specified by S3.

Permissions

Permissions use the same paradigms as the AmpliStor REST Permission Schemes.

A bucket is conceptually the same as an AmpliStor Namespace. We map a bucket onto a namespace.

As an authenticated user you can create up to 100 buckets. The authenticated user creating the bucket is the "owner" of the bucket.

Note: Namespaces created using AmpliStor REST are not visible using S3 and vice versa

As an owner of a bucket you can do all below actions on your buckets.

As a non-owner, you need certain permissions to perform actions on these buckets (and you need to be authenticated).

Object Access PUT: update permissions if the object exists or create permissions if the object does not exist on the object path

Request:

"PUT /bucketname/a/b HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""Content-Length: 0"



"Content-Type: binary/octet-stream""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 13:41:44 GMT""Authorization: AWS admin:3gXQTb+Qx5/v2JmE18LlVyiuoc8=""Expect: 100-continue"""

Response:

HTTP/1.1 200 OK\r\nDAV: 1,3\Date: Mon, 27 Aug 2012 13:41:06 GMT\Server:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\Content-Type: text/plain;charset=UTF-8\r\nContent-Length: 0\Accept-Ranges:bytes\Last-Modified: Mon, 27 Aug 2012 13:41:06 GMT\ETag:\"3c46e4cdef1a6f620271991fbb84e7b6\"\

GET and HEAD: read permissions on the object path

Request:

"GET /bucketname/a/b HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 13:41:44 GMT""Authorization: AWS admin:wIVvNlW/VnYY+0aeKW6FSG9xuCc="""

Response:

HTTP/1.1 200 OK\r\nDAV: 1,3\Date: Mon, 27 Aug 2012 13:42:03 GMT\Server:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\Content-



Length: 0\r\nAccept-Ranges: bytes\Last-Modified: Mon, 27 Aug 2012 13:41:06GMT\ETag: \"f0a602cd8cd2d4e01c811daa4ba1b451\"\r\n

DELETE: delete permissions on the object path

Request:

"DELETE /bucketname/a/b/c/d/e HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 13:42:42 GMT""Authorization: AWS admin:b+zPIV1T/pCyic0vJ4WHFNh0sEQ="""

Response:

""DELETE response:HTTP/1.1 204 No Content\DAV: 1,3\Date: Mon, 27 Aug 201213:45:35 GMT\Server:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\Content-Type: text/plain\r\nContent-Length: 0\

Bucket Access PUT: you must be an authenticated user

Request:

"PUT /bucketname HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""Content-Length: 0"



"x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 14:51:57 GMT""Authorization: AWS admin:XYzowiIv5TngegQsSmIRFPpk8mw=""Expect: 100-continue"""

Response:

HTTP/1.1 200 OK\DAV: 1,3\Date: Mon, 27 Aug 2012 14:51:19 GMT\Server:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\Content-Type: text/plain;charset=UTF-8\Content-Length: 0\

GET (listing) and HEAD: you must be the bucket owner or have list permissions on the bucket path: e.g: /namespace/bucketname

Request:

"GET /bucketname HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 13:46:14 GMT""Authorization: AWS admin:GQfdYkzk1UzDc8X3spAh1Wa1YzU="""

Response:

HTTP/1.1 200 OK\DAV: 1,3\Date: Mon, 27 Aug 2012 13:45:35 GMT\Server:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\r\nContent-Type: application/xml;charset=UTF-8\Content-Length: 5903\Pragma:no-cache\r\nCache-Control: no-cache\Expires: Thu, 01 Jan 1970 00:00:00 GMT\



body (in case file x and y are present as keys in bucket mynamespace):<?xml version=\"1.0\" encoding=\"UTF-8\"?><ListBucketResultxmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"> <Name>bucketname</Name><Prefix></Prefix> <Marker></Marker> <MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated> <Contents> <Key>x</Key><LastModified>2012-09-04T11:15:12.000Z</LastModified><ETag>c10be79e50b74fff8c643b9760a33b3e</ETag> <Size>55</Size><StorageClass>STANDARD</StorageClass>\n <Owner>\n <ID></ID><DisplayName></DisplayName> </Owner> </Contents> <Contents><Key>y</Key> <LastModified>2012-09-04T11:15:07.000Z</LastModified><ETag>30cbeaa5e43c4443bc60d24aa6fd8d92</ETag> <Size>55</Size><StorageClass>STANDARD</StorageClass> <Owner> <ID></ID><DisplayName></DisplayName>\n </Owner> </Contents></ListBucketResult>

DELETE: you must be the owner of the bucket.

Request

"DELETE /bucketname HTTP/1.1""Host: s3.amazonaws.com""Accept-Encoding: identity""content-length: 0""Authorization: AWS admin:1KuDFb/uJzQI1+omTzJBU8i+3Ek=""x-amz-date: Tue, 04 Sep 2012 08:03:16 +0000"""



Response

HTTP/1.1 204 No Content\r\nDAV: 1,3\r\nDate: Tue, 04 Sep 2012 08:03:16GMT\Server:Amplidata-AmpliStor/unknown-b55e743f085c9eb5bdea15501a8d60a9519e7ae4\Content-Length: 0\

Service Access GET: The following lists all the buckets owned by the authenticated sender of the request.

Request

"GET / HTTP/1.1""Host: s3.amazonaws.com""Accept-Encoding: identity""content-length: 0""Authorization: AWS admin:1KuDFb/uJzQI1+omTzJBU8i+3Ek=""x-amz-date: Tue, 04 Sep 2012 08:03:16 +0000"""

Response

HTTP/1.1 200 OKDAV: 1,3Date: Fri, 12 Oct 2012 11:48:28 GMTServer: Amplidata-AmpliStor/unknown-259aa153955d15a9f33a81e5992ae8c679c4e7eaContent-Type: application/xml;charset=UTF-8Content-Length: 525Pragma: no-cacheCache-Control: no-cacheExpires: Thu, 01 Jan 1970 00:00:00 GMTThe body for this request is in XML, and looks similar to the following example:<?xml version="1.0" encoding="UTF-8"?>


Chapter 2: S3 IntegrationError Codes

<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>32</ID><DisplayName>admin</DisplayName></Owner><Buckets><Bucket><Name>mybucket</Name><CreationDate>2013-06-13T14:26:06.000Z</CreationDate></Bucket><Bucket><Name>myotherbucket</Name><CreationDate>2013-06-13T14:26:13.000Z</CreationDate></Bucket><Bucket><Name>joriske</Name><CreationDate>2013-06-13T11:56:02.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

Error Codes

For more information about error responses in S3, see: http://docs.amazonwebservices.com/AmazonS3/latest/API/ErrorResponses.html

Error Code Description Http Status Code DSS Exception

InvalidBucketName The specified bucket is not valid.

400 Bad Request Exc.Invalid_namespace_name


http://docs.amazonwebservices.com/AmazonS3/latest/API/ErrorResponses.html




MetadataTooLarge Your metadata headers exceed the maximum allowed metadata size.

400 Bad Request

InvalidArgument Invalid Argument 400 Bad Request

TooManyBuckets You have attempted to create more buckets than allowed.

400 Bad Request

InvalidDigest The Content-MD5 you specified was invalid.

400 Bad Request

EntityTooLarge Your proposed upload exceeds the maximum allowed object size.

400 Bad Request

AccessDenied Access Denied 403 Forbidden None

SignatureDoesNotMatch The calculated request signature does not match the signature you provided. Check your AWS Secret Access Key and signing method.

403 Forbidden

InvalidAccessKeyId The AWS Access Key Id you provided does not exist in our records.

403 Forbidden

NoSuchBucket The specified bucket does not exist.

404 Not Found Exc.Namespace_not_found




NoSuchKey The specified key does not exist.

404 Not Found Exc.Storage_object_not_found

NotSuchBucketPolicy The specified bucket does not have a bucket policy.

404 Not Found

MethodNotAllowed The specified method is not allowed against this resource.

405 Method Not Allowed

BucketAlreadyExists The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

409 Conflict Exc.Namespace_already _exists

BucketNotEmpty The bucket you tried to delete is not empty.

409 Conflict

MissingContentLength You must provide the Content-Length HTTP header.

411 Length Required

InternalError An internal error was encountered. Please try again.

500 Internal Server Error

NotImplemented A header you provided implies functionality that is not implemented.

501 Not Implemented




S3 Error Return Code Examples:

If the bucket does not exist, the following Error Response is generated:<?xml version=\"1.0\"encoding=\"UTF-8\"?><Error>\n<Code>NoSuchBucket</Code>\n<Message>The specified bucket does not exist.</Message>\n<Resource></Resource>\n<RequestId></RequestId>\n</Error>\nIf the key does not exist, the following error response is generated:<?xml version=\"1.0\"encoding=\"UTF-8\"?><Error>\n<Code>NoSuchKey</Code>\n<Message>The specified key does not exist.</Message>\n<Resource>fileX</Resource>\n<RequestId></RequestId>\n</Error>\n

Note: A requestId is not returned.

If a concurrent PUT request is detected:<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>ConcurrentPutDetected</Code><Message>"Concurrent put on this resource detected."</Message><RequestId></RequestId><Resource>/bucket/object</Resource>

</Error>

Concurrent PUT Detected

Concurrent PUT request detected for identical file. First PUT request wins

XML error Concurrent PUT on this resource detected



Chapter 2: S3 IntegrationSupported Bucket Operations

Supported Bucket Operations

PUT Request http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketPUT.html

s3cmd mb s3://bucketname

Request:

"PUT /bucketname HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""Content-Length: 0""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 14:51:57 GMT""Authorization: AWS admin:XYzowiIv5TngegQsSmIRFPpk8mw=""Expect: 100-continue"""

Response:

response: HTTP/1.1 200 OK\r\nDAV: 1,3\r\nDate: Mon, 27 Aug 2012 14:51:19GMT\r\nServer:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\r\nContent-Type: text/plain;charset=UTF-8\r\nContent-Length: 0\r\n

GET Request (listing of objects)

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGET.html

s3cmd ls s3://bucketname




http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketPUT.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketPUT.html


Example with full listing, no prefix:

Request:

"GET /bucketname HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 13:46:14 GMT""Authorization: AWS admin:GQfdYkzk1UzDc8X3spAh1Wa1YzU="""

Response:

HTTP/1.1 200 OK\r\nDAV: 1,3\r\nDate: Mon, 27 Aug 2012 13:45:35 GMT\r\nServer:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\r\nContent-Type: application/xml;charset=UTF-8\r\nContent-Length: 5903\r\nPragma:no-cache\r\nCache-Control: no-cache\r\nExpires: Thu, 01 Jan 1970 00:00:00 GMT\r\n

The body (in case file x and y are present as keys in bucket bucketname) looks like this:

<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<ListBucketResultxmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\">\n <Name>bucketname</Name>\n<Prefix></Prefix>\n <Marker></Marker>\n <MaxKeys>1000</MaxKeys>\n<IsTruncated>false</IsTruncated>\n <Contents>\n <Key>x</Key>\n<LastModified>2012-09-04T11:15:12.000Z</LastModified>\n<ETag>c10be79e50b74fff8c643b9760a33b3e</ETag>\n <Size>55</Size>\n<StorageClass>STANDARD</StorageClass>\n <Owner>\n <ID></ID>\n<DisplayName></DisplayName>\n </Owner>\n </Contents>\n<Contents>\n <Key>y</Key>\n



<LastModified>2012-09-04T11:15:07.000Z</LastModified>\n<ETag>30cbeaa5e43c4443bc60d24aa6fd8d92</ETag>\n <Size>55</Size>\n<StorageClass>STANDARD</StorageClass>\n <Owner>\n <ID></ID>\n<DisplayName></DisplayName>\n </Owner>\n </Contents>\n</ListBucketResult>\n

Example with full listing with prefix:

Request:

"GET /bucketname?prefix=x HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 13:46:14 GMT""Authorization: AWS admin:GQfdYkzk1UzDc8X3spAh1Wa1YzU="""

Response:

HTTP/1.1 200 OK\r\nDAV: 1,3\r\nDate: Mon, 27 Aug 2012 13:45:35 GMT\r\nServer:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\r\nContent-Type: application/xml;charset=UTF-8\r\nContent-Length: 5903\r\nPragma:no-cache\r\nCache-Control: no-cache\r\nExpires: Thu, 01 Jan 1970 00:00:00 GMT\r\n

The body (in case file x and y are present as keys in bucket bucketname) looks like:

<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<ListBucketResultxmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\">\n <Name>bucketname</Name>\n<Prefix></Prefix>\n <Marker></Marker>\n <MaxKeys>1000</MaxKeys>\n



<IsTruncated>false</IsTruncated>\n <Contents>\n <Key>x</Key>\n<LastModified>2012-09-04T11:15:12.000Z</LastModified>\n<ETag>c10be79e50b74fff8c643b9760a33b3e</ETag>\n <Size>55</Size>\n<StorageClass>STANDARD</StorageClass>\n <Owner>\n <ID></ID>\n<DisplayName></DisplayName>\n </Owner>\n </Contents>\n </ListBucketResult>\n

Note how the key with name 'y' is not present in the XML since it did not match the mentioned prefix):

HEAD Request http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketHEAD.html

No corresponding s3cmd call.

Request:

"HEAD /bucketname HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""x-amz-acl: authenticated-read""x-amz-date: Mon, 03 Sep 2012 14:54:17 GMT""Authorization: AWS admin:1oGc8tU8s4EWdcouB9esg6FE5so="""

Response:

HTTP/1.1 200 OK\r\nDAV: 1,3\r\nDate: Mon, 27 Aug 2012 13:45:35 GMT\r\nServer:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\r\nContent-Type: application/xml;charset=UTF-8\r\nContent-Length: 5903\r\nPragma:


http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketHEAD.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketHEAD.html

Chapter 2: S3 IntegrationSupported Multipart Bucket Operations

no-cache\r\nCache-Control: no-cache\r\nExpires: Thu, 01 Jan 1970 00:00:00 GMT\r\n

DELETE Request

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketDELETE.html

s3cmd rb s3://bucketname

Request:

"DELETE /bucketname HTTP/1.1""Host: s3.amazonaws.com""Accept-Encoding: identity""content-length: 0""Authorization: AWS admin:1KuDFb/uJzQI1+omTzJBU8i+3Ek=""x-amz-date: Tue, 04 Sep 2012 08:03:16 +0000"""

Response:

HTTP/1.1 204 No Content\r\nDAV: 1,3\r\nDate: Tue, 04 Sep 2012 08:03:16GMT\r\nServer:Amplidata-AmpliStor/unknown-b55e743f085c9eb5bdea15501a8d60a9519e7ae4\r\nContent-Length: 0\r\n

Supported Multipart Bucket Operations

List Multipart Uploads http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html




http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html

Chapter 2: S3 IntegrationSupported Multipart Bucket Operations

This operation shows all multipart uploads which are in progress. An in-progress multipart upload is a multipart upload that has been initiated, using the Initiate Multipart Upload request, but has not yet been completed or aborted.

Request:

GET /?uploads HTTP/1.1Host: 192.168.12.148:7070Date: Mon, 1 Nov 2013 20:34:56 GMTAuthorization: AWS admin:GQfdYkzk1UzDc8X3spAh1Wa1YzU=

Response:

HTTP/1.1 200 OKx-amz-id-2: Uuag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==x-amz-request-id: 656c76696e6727732072657175657374Date: Mon, 1 Nov 2013 20:34:56 GMTContent-Length: 1330Connection: keep-aliveServer: Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b

The body in case there are in-progress multipart uploads:<?xml version="1.0" encoding="UTF-8"?><ListMultipartUploadsResult xmlns="http://192.168.12.148:7070"><Bucket>testbucket</Bucket><KeyMarker /><UploadIdMarker /><NextKeyMarker /><NextUploadIdMarker /><IsTruncated>false</IsTruncated><Upload><Key>TheHolyGrail.mp4</Key><UploadId>XMgbGlrZSBlbHZpbmcncyBub3QgaGF2aW5nIG11Y2ggbHVjaw</UploadId><Initiator>


Chapter 2: S3 IntegrationSupported Object Operations

<ID>arn:aws:iam::111122223333:user/user1-11111a31-17b5-4fb7-9df5-b111111f13de</ID>

<DisplayName>user1-11111a31-17b5-4fb7-9df5-b111111f13de</DisplayName>

</Initiator><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>OwnerDisplayName</DisplayName>

</Owner><StorageClass>STANDARD</StorageClass><Initiated>2010-11-10T20:48:33.000Z</Initiated>

</Upload></ListMultipartUploadsResult>

Supported Object Operations

PUT Request http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectPUT.html

s3cmd put /tmp/x s3://bucketname/x

Request:

"PUT /bucketname/a/b HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""Content-Length: 0""Content-Type: binary/octet-stream""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 13:41:44 GMT""Authorization: AWS admin:3gXQTb+Qx5/v2JmE18LlVyiuoc8=""Expect: 100-continue"


http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectPUT.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectPUT.html


""

Response:

response: HTTP/1.1 200 OK\r\nDAV: 1,3\r\nDate: Mon, 27 Aug 2012 13:41:06GMT\r\nServer:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\r\nContent-Type: text/plain;charset=UTF-8\r\nContent-Length: 0\r\nAccept-Ranges:bytes\r\nLast-Modified: Mon, 27 Aug 2012 13:41:06 GMT\r\nETag:\"3c46e4cdef1a6f620271991fbb84e7b6\"\r\n

GET Request http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectGET.html

s3cmd get s3://bucketname/x /tmp/x.res

Request:

"GET /bucketname/a/b HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 13:41:44 GMT""Authorization: AWS admin:wIVvNlW/VnYY+0aeKW6FSG9xuCc="""

Response:

HTTP/1.1 200 OK\r\nDAV: 1,3\r\nDate: Mon, 27 Aug 2012 13:42:03 GMT\r\nServer:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\r\nContent-


http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectGET.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectGET.html


Length: 0\r\nAccept-Ranges: bytes\r\nLast-Modified: Mon, 27 Aug 2012 13:41:06GMT\r\nETag: \"f0a602cd8cd2d4e01c811daa4ba1b451\"\r\n

HEAD Request http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectHEAD.html

No corresponding s3cmd call.

Request:

"HEAD /bucketname/a/b/c/d/e HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 13:42:42 GMT""Authorization: AWS admin:b+zPIV1T/pCyic0vJ4WHFNh0sEQ="""

Response:

HTTP/1.1 200 OK\r\nDAV: 1,3\r\nDate: Mon, 27 Aug 2012 13:42:03 GMT\r\nServer:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\r\nContent-Length: 1048576\r\nAccept-Ranges: bytes\r\nLast-Modified: Mon, 27 Aug 201213:42:03 GMT\r\nETag: \"fdf3a80e1ce052e7cd021563d98cb53b\"\r\n

DELETE Request http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectDELETE.html

s3cmd del s3://bucketname/x


http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectHEAD.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectHEAD.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectDELETE.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectDELETE.html


Request:

"DELETE /bucketname/a/b/c/d/e HTTP/1.1""User-Agent: Mozilla/4.0 (Compatible; AMZS3; CVRF 1.0; Windows Server 2008 R2)""Host: 192.168.12.148:7070""Accept: */*""x-amz-acl: authenticated-read""x-amz-date: Mon, 27 Aug 2012 13:42:42 GMT""Authorization: AWS admin:b+zPIV1T/pCyic0vJ4WHFNh0sEQ="""

Response:

HTTP/1.1 204 No Content\r\nDAV: 1,3\r\nDate: Mon, 27 Aug 2012 13:45:35GMT\r\nServer:Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b\r\nContent-Type: text/plain\r\nContent-Length: 0\r\n

PUT - Copy Request http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html.

s3cmd cp s3://bucketname/xs3://bucketname/y

Request:

"PUT bucket/a HTTP/1.1"'content-length': '0''Authorization': 'AWS admin:8f5E54cGlLvt+NUrTZxmoEXAyQI=''x-amz-date': 'Thu, 13 Jun 2013 14:35:25 +0000''x-amz-copy-source': '/joriske/x''x-amz-metadata-directive': 'COPY'

Response:

HTTP/1.1 200 OK


http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html

Chapter 2: S3 IntegrationSupported Multipart Object Operations

Date: Thu, 13 Jun 2013 14:35:25 GMTDav: 1,3content-type: application/xmlcontent-length: 239server: Amplidata-AmpliStor/unknown-1fda372687b59bb934e108292c1d6873f519f37aThe body of this response is in XML:<?xml version="1.0" encoding="UTF-8"?><CopyObjectResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LastModified>Thu, 13 Jun 2013 14:35:25 GMT</LastModified><ETag>"7c909b3e2820c8b47ed418753698a6da"</ETag></CopyObjectResult>

Supported Multipart Object Operations

Most of the S3 clients, (such as s3cmd or Cyberduck,) support multipart by default.

• Initiate Multipart Upload

• Upload Multipart Part

• Complete Multipart Upload

• Abort Multipart Upload

• List Parts

• Multipart Upload (Copy)



Note: There are some default restrictions for the S3 multipart protocol:

• Maximum object size is 5 TB

• Minimum object size is 5 MB

• Minimum part size is 5 MB

• Maximum number of parts is 10,000 (default)

These settings can updated in the client daemon configuration file. The real maximum number of parts that can be used is 65,535.

Initiate Multipart Upload

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadInitiate.html

Initiating a multipart upload does the initialization of the multipart upload and returns an upload ID and a storage object ID. The upload ID is used to associate all the parts in the specific multipart upload. A multipart upload request is stored with a key: mpoI_<bucketID>_<object name>_<upload id>

Request:

POST http://testbucket.mys3service.com/example-object.iso?uploads HTTP/1.1Host: testbucket.mys3service.comAccept-Encoding: identitycontent-length: 0content-type: application/x-iso9660-imageAuthorization: AWS thomas:HvU5l9u6ss3xHQxIPzUt9JzYA6c=x-amz-date: Fri, 13 Dec 2013 08:19:16 +0000content-encoding: UTF-8

Response:

HTTP/1.1 200 OKx-amz-id-2: JBP5qj+DsZcfrawymY/4UDTf2JKdYD0B/rmL+USGtVwg5AZKuu/icNAAiwqfEgX9





x-amz-request-id: 7F200D15AB4013C8Date: Fri, 13 Dec 2013 08:19:03 GMTTransfer-Encoding: chunkedServer: Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b

Body:

<?xml version="1.0" encoding="UTF-8"?><InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>testbucket.mys3service.com</Bucket><Key>c2c4dc85-38ac-4f89-aa29-16a19bbc3468</Key>

<UploadId>BbyOjajJe30TSjDP7Bq4Z.rmTu_pDwVOFBFi7tqHWwe5ZgUFMKGwZje2TQl4Zz_i4mBhjLD1vXxj8_c7rQC_BLwGQplJh_bcOftTq0b.U0o4_F8lyRkYJ8qBKFFmNwhS</UploadId></InitiateMultipartUploadResult>

Upload Multipart Part http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html

A multipart part has a 1-on-1 mapping with a part, which can be considered a subobject. The part is first uploaded and created, and subsequently it is linked to the specific multipart part.

A possible previous "old" part will be overwritten and the "old" part is no longer accessible and will be deleted.

Request:

PUT/c2c4dc85-38ac-4f89-aa29-16a19bbc3468?partNumber=1&uploadId=BbyOjajJe30TSjDP7Bq4Z.rmTu_pDwVOFBFi7tqHWwe5ZgUFMKGwZje2TQl4Zz_i4mBhjLD1vXxj8_c7rQC_BLwGQplJh_bcOftTq0b.U0o4_F8lyRkYJ8qBKFFmNwhS HTTP/1.1Content-Length: 5242880Accept-Encoding: identity


http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html


Accept: */*Content-MD5: p17Vdc2ncwx8FLQDUtoVVQ==Host: testbucket.mys3service.comx-amz-date: Fri, 13 Dec 2013 08:19:18 +0000Content-Type: application/octet-streamAuthorization: AWS thomas:HvU5l9u6ss3xHQxIPzUt9JzYA6c=

Response:

HTTP/1.1 200 OKx-amz-id-2: BD6QQJVqotgk5pzONQl8xQRLfY7XIUF24n0PZScTlnS3CP4fMSmwm5yf958KYDNpx-amz-request-id: F28B7459125E4943Date: Fri, 13 Dec 2013 08:19:05 GMTETag: "a75ed575cda7730c7c14b40352da1555"Content-Length: 0Server: Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b

Complete Multipart Upload

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html

Request:

POST /c2c4dc85-38ac-4f89-aa29-16a19bbc3468?uploadId=BbyOjajJe30TSjDP7Bq4Z.rmTu_\\pDwVOFBFi7tqHWwe5ZgUFMKGwZje2TQl4Zz_i4mBhjLD1vXxj8_c7rQC_BLwGQplJh_\\bcOftTq0b.U0o4_F8lyRkYJ8qBKFFmNwhS HTTP/1.1Content-Length: 243Accept-Encoding: identityAccept: */*Host: testbucket.mys3service.comx-amz-date: Fri, 13 Dec 2013 08:19:30 +0000Content-Type: application/octet-streamAuthorization: AWS thomas:HvU5l9u6ss3xHQxIPzUt9JzYA6c=





<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>a75ed575cda7730c7c14b40352da1555</ETag></Part><Part><PartNumber>2</PartNumber><ETag>1f778a8d73d57b0e162667b635631173</ETag></Part></CompleteMultipartUpload>

Response:

HTTP/1.1 200 OKx-amz-id-2: vjq6W420mzRUGJcTPHSDTCDpYMbjba7RLXdhb0MSJzxaz5/cuLq6cHGFhAgPVfs/x-amz-request-id: CCC7CBBC674808C5Date: Fri, 13 Dec 2013 08:19:17 GMTContent-Type: application/xmlTransfer-Encoding: chunkedServer: Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b

Body:

<?xml version="1.0" encoding="UTF-8"?><CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>http://testbucket.mys3service.com/c2c4dc85-38ac-4f89-aa29-16a19bbc3468</Location><Bucket>testbucket.mys3service.com</Bucket><Key>c2c4dc85-38ac-4f89-aa29-16a19bbc3468</Key><ETag>"329454b9b991ccd9b34e0ba4ae0b5004-2"</ETag>

</CompleteMultipartUploadResult>



Abort Multipart Upload http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadAbort.html

Request:

DELETE/example-object?uploadId=BbyOjajJe30TSjDP7Bq4Z.rmTu_pDwVOFBFi7tqHWwe5ZgUFMKGwZje2TQl4Zz_i4mBhjLD1vXxj8_c7rQC_BLwGQplJh_bcOftTq0b.U0o4_F8lyRkYJ8qBKFFmNwhS HTTP/1.1Host: testbucket.mys3service.comDate: Fri, 13 Dec 2013 08:19:22 GMTAuthorization: AWS thomas:HvU5l9u6ss3xHQxIPzUt9JzYA6c=

Response:

HTTP/1.1 204 OKx-amz-id-2: Weag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==x-amz-request-id: 996c76696e6727732072657175657374Date: Fri, 13 Dec 2013 08:19:22 GMTContent-Length: 0Connection: keep-aliveServer: Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b

List Parts http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListParts.html

List relevant multipart part entries.

Request:

GET/example-object?uploadId=BbyOjajJe30TSjDP7Bq4Z.rmTu_pDwVOFBFi7tqHWwe5ZgUFMKGwZje2TQ


http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadAbort.html

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadAbort.html

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListParts.html

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListParts.html


l4Zz_i4mBhjLD1vXxj8_c7rQC_BLwGQplJh_bcOftTq0b.U0o4_F8lyRkYJ8qBKFFmNwhS&max-parts=2&part-number-marker=1 HTTP/1.1Host: testbucket.mys3service.comDate: Fri, 13 Dec 2013 08:19:32 GMTAuthorization: AWS thomas:HvU5l9u6ss3xHQxIPzUt9JzYA6c=

Response:

HTTP/1.1 200 OKx-amz-id-2: Uuag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==x-amz-request-id: 656c76696e6727732072657175657374Date: Fri, 13 Dec 2013 08:19:19 GMTContent-Length: 985Connection: keep-aliveServer: Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b

Body:

<?xml version="1.0" encoding="UTF-8"?><ListPartsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>testbucket</Bucket><Key>example-object</Key>

<UploadId>BbyOjajJe30TSjDP7Bq4Z.rmTu_pDwVOFBFi7tqHWwe5ZgUFMKGwZje2TQl4Zz_i4mBhjLD1vXxj8_c7rQC_BLwGQplJh_bcOftTq0b.U0o4_F8lyRkYJ8qBKFFmNwhS</UploadId><Initiator><ID>arn:aws:iam::111122223333:user/some-user-11116a31-17b5-4fb7-9df5-b288870f11xx</ID><DisplayName>umat-user-11116a31-17b5-4fb7-9df5-b288870f11xx</DisplayName></Initiator><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>someName</DisplayName>



</Owner><StorageClass>STANDARD</StorageClass><PartNumberMarker>1</PartNumberMarker><NextPartNumberMarker>3</NextPartNumberMarker><MaxParts>2</MaxParts><IsTruncated>true</IsTruncated><Part><PartNumber>2</PartNumber><LastModified>2010-11-10T20:48:34.000Z</LastModified><ETag>"7778aef83f66abc1fa1e8477f296d394"</ETag><Size>10485760</Size>

</Part><Part><PartNumber>3</PartNumber><LastModified>2010-11-10T20:48:33.000Z</LastModified><ETag>"aaaa18db4cc2f85cedef654fccc4a4x8"</ETag><Size>10485760</Size>

</Part></ListPartsResult>

Multipart Upload (Copy)

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html

Request:

PUT/newobject?partNumber=2&uploadId=VCVsb2FkIElEIGZvciBlbZZpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZR HTTP/1.1Host: target-bucket.mys3service.comDate: Fri, 13 Dec 2013 08:25:19 GMTx-amz-copy-source: /source-bucket/sourceobjectx-amz-copy-source-range:bytes=500-6291456Authorization: AWS thomas:HvU5l9u6ss3xHQxIPzUt9JzYA6c=




Chapter 2: S3 IntegrationSupported Service Operations

Response:

HTTP/1.1 200 OKx-amz-id-2: Vvag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==x-amz-request-id: 656c76696e6727732072657175657374Date: Fri, 13 Dec 2013 08:25:19 GMTServer: Amplidata-AmpliStor/unknown-c111f895b629566707818005fa6ff8f66346011b<CopyPartResult><LastModified>2008-01-29T08:22:00</LastModified><ETag>"9b2cf535f27731c974343645a3985328"</ETag></CopyPartResult>

Supported Service Operations

GET Request http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTServiceGET.html

s3cmd ls

Request:

"GET / HTTP/1.1""Host: s3.amazonaws.com""Accept-Encoding: identity""content-length: 0""Authorization: AWS admin:1KuDFb/uJzQI1+omTzJBU8i+3Ek=""x-amz-date: Tue, 04 Sep 2012 08:03:16 +0000"""

Response:

HTTP/1.1 200 OK\r\nDAV: 1,3\r\nDate: Fri, 12 Oct 2012 11:48:28 GMT\r\nServer:


http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTServiceGET.html

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTServiceGET.html


Amplidata-AmpliStor/unknown-259aa153955d15a9f33a81e5992ae8c679c4e7ea\r\nContent-Type:application/xml;charset=UTF-8\r\nContent-Length: 525\r\nPragma:no-cache\r\nCache-Control: no-cache\r\nExpir es: Thu, 01 Jan 1970 00:00:00GMT\r\n

The body for this request is in XML:<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">

<Owner><ID>32</ID><DisplayName>admin</DisplayName>

</Owner><Buckets>

<Bucket><Name>mybucket</Name><CreationDate>2013-06-13T14:26:06.000Z</

CreationDate></Bucket><Bucket>

<Name>myotherbucket</Name><CreationDate>2013-06-13T14:26:13.000Z</

CreationDate></Bucket>

<Bucket><Name>joriske</Name><CreationDate>2013-06-13T11:56:02.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>


Chapter 3REST Definitions and

Configuration

In addition to the S3 implementation, you can also communicate with Lattus through a REST API. Details for using this REST API are provided in subsequent chapters.

REST Overview Any Lattus server can be configured to act as a REST Capable Storage Server. The more Lattus servers are available in the environment, the higher the potential overall throughput performance.

All Lattus servers can view and access the complete storage pool as a global namespace and put or receive data from all namespaces. An object stored on one controller is immediately visible and available for access from any other controller.

The Lattus server can be configured to listen on a dedicated port to receive incoming HTTP requests. The following REST commands are currently supported:

• Create policies

• Create, overwrite, retrieve, delete, and list objects

• Add, list, get, and delete users

• Retrieve policy metadata

• Create and delete namespaces

• Retrieve namespace metadata


Chapter 3: REST Definitions and ConfigurationHow To Enable REST in Lattus

• Store, retrieve, delete, list, set and retrieve object metadata

How To Enable REST in Lattus

In Lattus, REST is enabled by default on every controller and listening on port 8080.

If you are using the REST development virtual machine, a command line option is available to enable REST.

Using Curl to Demonstrate the REST Functionality

As a way to demonstrate the REST API, the program Curl is used in some of the following examples. Curl has the following options:

Note: Do not use wget. The most common versions of wget only provide HTTP 1.0 support, while HTTP 1.1 is required.


Chapter 3: REST Definitions and ConfigurationOther Interfaces

Other Interfaces

Besides the REST API interface, there is also a command line interface (CLI) and the Q-Shell interface, available on any controller. This guide describes mainly the REST API and the Q-Shell interfaces, but all functions described in this guide can also be executed from the CLI.

The CLI is available via /opt/qbase/bin/dss. Call the help function by adding --help to the command.

~# /opt/qbase3/bin/dss --permission-settings-get /namespace

{

"Flags":[],

"User-Permissions":

{

"everyone":["UPDATE","LIST","DELETE","CREATE","READ"]

}

}

API Interface Documentation Assumptions

For readability purposes, some parameters within the HTTP headers are replaced with a named parameter. The following named parameters are used:

• _date_: a date string following the HTTP 1.1 date specification, e.g. Fri, 03 Jun 2011 00:00:00 GMT.

• _length_: the length in bytes of the specified variable, e.g. for Content-length: length. This means that the length in bytes of the passed content must be filled in.

• _epoch_time_: time in seconds since Thu, 01 Jan 1970 00:00:00 GMT.


http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.18

Chapter 3: REST Definitions and ConfigurationResource Identifiers

Resource Identifiers

The Lattus Web Service API uses Internationalized Resource Identifiers (IRI) (RFC 3987), to refer to the entities addressable in the storage system.

An IRI is similar to a Uniform Resource Identifier (URI) but without the limitation of containing only characters from the ASCII character set. As such, an IRI can contain Unicode characters. The RFC also documents how to encode this for systems handling only ASCII character set URIs.

IRI setup: /manage/policy is used for managing policies

Transfer Protocol

The transfer protocol for the Web Service is Hypertext Transfer Protocol v1.1, as documented in RFC 2616.

If the Lattus receives an HTTP/1.0 request, it will always return with an HTTP 505 Version Not Supported return code.

Behavior Common to All Requests

In the remainder of this document, the list of REST Web Service API requests and responses are described in detail.

Irrespective of the different requests, there is some base HTTP 1.1 behavior that an integrator can expect from the Lattus REST Web Service. Such behavior is described in this section.


http://tools.ietf.org/html/rfc3987

http://tools.ietf.org/html/rfc3986

http://www.w3.org/Protocols/rfc2616/rfc2616.html

http://www.w3.org/Protocols/rfc2616/rfc2616.html

Chapter 3: REST Definitions and ConfigurationWhat Has Changed Compared to the 3.2.X REST API

What Has Changed Compared to the 3.2.X REST API

• In its response headers, the client daemon will reply with Quantum-Lattus/3.6.0-<versionGUID>.


Chapter 3: REST Definitions and ConfigurationWhat Has Changed Compared to the 3.2.X REST API


Chapter 4Design Considerations

Integration of your application with Lattus requires a carefully architected integration in order to get the maximum out of Lattus's performance and scalability.

Take the following items into account prior to starting the actual development of the integration code.

Designing for Performance

Connection Management

Use a REST-library that can handle persisting connections. If your library can't handle this, you might encounter performance degradation since the connection will have to be re-established upon every REST operation.

Because of this, keep the following advice in mind, especially when using authentication:

• Do not use the Linux curl executable.

• Use the libcurl library instead.

Do not direct all connections from your application towards a single controller. If you have multiple controllers at your disposal, spread and


Chapter 4: Design ConsiderationsDesigning for Performance

balance your connections over all controllers. Preferably, pool connections to the same controller that are bound to a namespace:

• Example:

• You control 9 namespaces (ns1, ns2...ns9) and 3 controllers.

• A good practice from a performance perspective would be to:

• Send all connections for namespaces ns1, ns2 and ns3 to controller1.



• The advantage of this approach is that the connections are spread evenly over all controllers. Each namespace that needs to be referenced on a controller leads to a set of connections to the back-end.

• A bad practice would be to consult namespaces on random controllers. This would still work, but you will encounter performance degradation and you can encounter connection exhaustion on the back-end.

• If a controller fails, your code should be capable of temporarily rerouting traffic to another controller. If your environment was scaled properly, you should have made provisions for spare capacity.

If your application uses a lot of persistent connections and the data is slowly retrieved by your application (i.e. your application reads the data slower than Lattus can decode it from the back-end), match the amount of memory on your controllers to the expected number of connections. When a connection is open and the encoded data is not entirely retrieved by your application, the controller will still have a reference to a buffer in memory holding the encoded data (which is up to 6x the Superblock size). If your application leads to many of these semi-stale connections, the amount of memory inside the controller should be sufficient.


Chapter 4: Design ConsiderationsDesigning for Performance

Parallelism Reading different objects in multiple parallel threads leads to improved performance. The Lattus Client daemon is designed for parallelism. The amount of parallelism is determined by the number of connections your controllers and environment can handle.

Do not read the object in different parallel threads. Each read operation will lead to a process of fetching content from the storage back-end and decoding the fetched data. If this process is done multiple times for the same object at the same time, performance will adversely be affected as each thread will fighting for the same resource.

Avoid writing the same object from different threads. This is supported, but every object will be encoded and written to the back-end and only 1 object will be retained. For the writes that failed, metadata updates need to happen and data needs to be deleted from the back-end.

Partial Reads Keep in mind that a partial read triggers fetching and decoding of a superblock.

Example:

• Your storage policy uses 64 MiB superblocks.

• You stored a 1 GiB object and want to read 8 kiB from that object.

• This will lead to 64 MiB being fetched from the back-end, which leads to many times that amount of data being read from the back-end (depending on your storage policy).

Consider changing your superblocksize to better match your partial read size. However, keep the following in mind:

• Lowering the superblocksize lowers your overall performance.

• The minimum superblocksize is 2 MiB.

A valid design choice could be: Using a dedicated namespace with a smaller superblocksize for this type of file.

As such, make sure that your reads are superblock aligned in order to avoid having to fetch 2 superblocks for a single read.


Chapter 4: Design ConsiderationsDesigning for Scalability

Overwrites Overwrites are possible but keep in mind that these are costly operations. The object needs to be encoded, stored on the back-end and the original content has to be deleted.

Overwrites are done on an object level and currently do not support superblock aligned overwrites, which would only update a subset of the object (in case it entails multiple superblocks).

As such there is no performance benefit in only changing a couple of bytes in an object since the objects is rewritten in its entirety.

Caching If your application is reading the same (updated) content very frequently, consider deploying a read cache. This cache can be configured to update the content in the cache after it has been stored to the back-end.

Subsequent reads will benefit from the cache: their read latency will significantly go down. First byte read latency will go from hundreds of milliseconds to a few milliseconds.

Designing for Scalability

Namespace Limitations Namespaces typically have 2 limitations:

• Number of concurrent operations that can be done.

• Number of objects they can hold.

Maximum Number of Objects Per Namespace

Every object in a namespace consumes metadata. This metadata is stored in a MetaStore, which lives on a file system on top of an SSD. As such, the maximum number of objects that can be stored is limited by the size of this SSD. Lattus doesn't support spanning a namespace over multiple SSDs.

In order to identify how many objects can be stored in a namespace, one should determine the number of superblocks per object and the



size of the object metadata. Typically every superblock of an object consumes 200 bytes. We recommend measuring MetaStore consumption for your application (when the object name gets longer, more metadata is required).

When designing for the number of objects, take this limitation in consideration. There are 2 approaches:

1 When the namespace is exhausted, the application starts using a new namespace which lives one a different MetaStore (using different SSDs).

2 Your application is capable of handling multiple namespaces and balancing writes into these namespaces based upon their fill-level. In the early phase, when a single MetaStore is sufficient, your application writes to this one namespace. When your application exceeds a specified limit, configure your application so that it will start writing to a second namespace but still uses the first namespace but based upon consumption.

A. Example:

a Start with a single namespace (ns1) with maximum 1,000,000,000 superblocks. Your objects typically are 1 GiB in size. With a SuperBlocksize of 32 MiB, you need 32 superblocks per object. This means you can write 31,250,000 million objects to this namespace.

b After 9 months 400,000,000 superblocks (or 12.5 million objects) have been written to ns1 and you decide to extend your Lattus with an additional MetaStore capable of handling 1,500,000,000 superblocks (commercially available SSDs are still obeying Moore's Law). This implies you can write 46,875,000 objects to a namespace (ns2) managed by this MetaStore.

c Your application will write 600,000,000 / (600,000,000 + 1,500,000,000) = 28.5% of its writes to ns1 and 1,500,000,000 / (1,500,000,000 + 600,000,000) = 71.4% of its writes to ns2.

d 10 months later, you extend your Lattus with an additional MetaStore and now you balance over 3 namespaces. Let's assume that ns1 is filled for 90%, ns2 is filled for 40% and ns3 is still empty, but can take 2,000,000,000 superblocks. Writes will happen according to the following distribution:

1 100,000,000 / (100,000,000 + 900,000,000 + 2,000,000,000)= 3.3% to ns1.



2 900,000,000 / (100,000,000 + 900,000,000 + 2,000,000,000)= 30% to ns2.

3 2,000,000,000 / (100,000,000 + 900,000,000 + 2,000,000,000)= 66.6% to ns3.

B. Other mapping methods which guarantees proportional spreading of metadata over MetaStores can also be considered.

Keep in mind that moving an object from one namespace to another not only requires rewriting the metadata but also the data underneath (which drives data retrieval, decoding and encoding the object and finally storing the encoded data on the back-end).

Concurrent Operations

The MetaStore managing object metadata for a namespace is limited in the number of object operations it can handle. These operations are:

1 put

2 get

3 delete

4 list

5 repair

6 monitoring

When using appropriate hardware (dedicated SSDs and sufficient memory for caching) we recommend to never exceed 1000 of these operations (put, get, delete, list combined together) per second per MetaStore.

Connection Locality When reviewing the previous mentioned performance considerations related to connection management, one can easily see that accessing your data from everywhere introduces sub-optimal use of your Lattus.

For optimal scalability your application should be capable of localizing reads and writes for the same namespace to the same controller, or at least attempt to pursue this goal.


Chapter 5Policy Operations

Create Policy

A POST/PUT request creates a policy with certain parameters and returns the GUID of this policy. As the name of the policy can't be chosen by the user, a POST/PUT on a /policy subdirectory suffices.

On successful creation of a policy, the return status is “201 Created”, and the metadata of the new policy is returned.

If a policy with the same parameters already exists, the return status is “200 OK”, and the metadata of the existing policy is returned.

If for a certain parameter no value is passed, the default is used. The default values for REST are the following:

• Spread-Width: 16

• Safety: 4

• Hierarchy-Rules: true, true, true

• Max-Superblock-Size: 32 MiB

• Safety-Strategy: “RepairSpread”,”DynamicSafety”

• N-Messages: 4096


Chapter 5: Policy OperationsCreate Policy

http / text REQUEST /manage/policy

PUT /manage/policy HTTP/1.1

Host: my.example.com

Date: _date_

Accept: text/plain

Content-Type: text/plain

Content-Length: 0

X-Ampli-Spread-Width: 16

X-Ampli-Safety: 4

X-Ampli-Hierarchy-Rules: [ false , true , true ]

X-Ampli-Max-Superblock-Size: 33554432

X-Ampli-Safety-Strategy: [ "RepairSpread" , "DynamicSafety" ]

X-Ampli-N-Messages: 4096

X-Ampli-Full-Copy: false

or:PUT /manage/policy?meta=http HTTP/1.1Host: my.example.comDate: _date_Content-Length: 0X-Ampli-Spread-Width: 16X-Ampli-Safety: 4X-Ampli-Hierarchy-Rules: [ false , true , true ]X-Ampli-Max-Superblock-Size: 33554432X-Ampli-Safety-Strategy: [ "RepairSpread" , "DynamicSafety" ]X-Ampli-N-Messages: 4096X-Ampli-Full-Copy: false

RESPONSE /manage/policy

HTTP/1.1 201 CreatedDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24



Content-Type: text/plain;charset=UTF-8Content-Length: 0X-Ampli-Id: "d27a3bc66f7e3c8e957cdaa01d5eacc1"X-Ampli-Spread-Width: 16X-Ampli-Safety: 4X-Ampli-Hierarchy-Rules: [ false , true , true ]X-Ampli-Max-Superblock-Size: 33554432X-Ampli-Safety-Strategy: [ "RepairSpread" , "DynamicSafety" ]X-Ampli-N-Messages: 4096X-Ampli-Full-Copy: false

json REQUEST /manage/policy

PUT /manage/policy HTTP/1.1Host: my.example.comDate: _date_Accept: application/jsonContent-Type: application/jsonContent-Length: _length_{"Spread-Width":16,"Safety":4,"Hierarchy-Rules":[ false , true , true ],"Max-Superblock-Size":33554432,"Safety-Strategy":[ "RepairSpread" , "DynamicSafety" ],"N-Messages":4096"Full-Copy":false}

or:PUT /manage/policy?meta=json HTTP/1.1Host: my.example.comDate: _date_Content-Length: _length_{"Spread-Width":16,



"Safety":4,"Hierarchy-Rules":[ false , true , true ],"Max-Superblock-Size":33554432,"Safety-Strategy":[ "RepairSpread" , "DynamicSafety" ],"N-Messages":4096"Full-Copy":false}


HTTP/1.1 201 CreatedDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8Content-Length: _length_{"Id":"9adea14a85004c44883301dca8864b99","Spread-Width":16,"Safety":4,"Hierarchy-Rules":[false,true,true],"Max-Superblock-Size":33554432,"Safety-Strategy":["RepairSpread","DynamicSafety"],"N-Messages":4096"Full-Copy":false}

xml REQUEST /manage/policy

PUT /policy HTTP/1.1Host: my.example.comDate: _date_Accept: application/xmlContent-Type: application/xmlContent-Length: _length_<?xml version="1.0" encoding="UTF-8"?>



<Policy xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Spread-Width>16</Spread-Width><Safety>4</Safety><Hierarchy-Rules>false true true</Hierarchy-Rules><Max-Superblock-Size>33554432</Max-Superblock-Size><Safety-Strategy>RepairSpread DynamicSafety</Safety-Strategy><N-Messages>4096</N-Messages><Full-Copy>false</Full-Copy></Policy>

or:PUT /manage/policy?meta=xml HTTP/1.1Host: my.example.comDate: _date_Content-Length: _length_<?xml version="1.0" encoding="UTF-8"?><Policy xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Spread-Width>16</Spread-Width><Safety>4</Safety><Hierarchy-Rules>false true true</Hierarchy-Rules><Max-Superblock-Size>33554432</Max-Superblock-Size><Safety-Strategy>RepairSpread DynamicSafety</Safety-Strategy><N-Messages>4096</N-Messages><Full-Copy>false</Full-Copy></Policy>


HTTP/1.1 201 CreatedDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8Content-Length: _length_<?xml version="1.0" encoding="UTF-8"?>


Chapter 5: Policy OperationsGet Policy

<Policy xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>9adea14a85004c44883301dca8864b99</Id><Spread-Width>16</Spread-Width><Safety>4</Safety><Hierarchy-Rules>false true true</Hierarchy-Rules><Max-Superblock-Size>33554432</Max-Superblock-Size><Safety-Strategy>RepairSpread DynamicSafety</Safety-Strategy><N-Messages>4096</N-Messages><Full-Copy>false</Full-Copy></Policy>

Get Policy

A GET request on /policy/<policy id> returns the configuration details of the policy with <policy id> as ID. A HEAD request yields the same response as a GET request, except that the server does not return a message-body in the response.

http / text REQUEST /manage/policy/<policy id>

GET /manage/policy/d27a3bc66f7e3c8e957cdaa01d5eacc1 HTTP/1.1Host: my.example.comDate: _date_Accept: text/plain

or:GET /manage/policy/d27a3bc66f7e3c8e957cdaa01d5eacc1?meta=http HTTP/1.1Host: my.example.comDate: _date_



RESPONSE /policy/<policy id>

HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: 0X-Ampli-Spread-Width: 16X-Ampli-Safety: 4X-Ampli-Hierarchy-Rules: [ false , true , true]X-Ampli-Max-Superblock-Size: 33554432X-Ampli-Safety-Strategy: [ "RepairSpread" , "DynamicSafety" ]X-Ampli-N-Messages: 4096X-Ampli-Full-Copy: false

json REQUEST /manage/policy/<policy id>

GET /manage/policy/d27a3bc66f7e3c8e957cdaa01d5eacc1 HTTP/1.1Host: my.example.comDate: _date_Accept: application/json

or:GET /manage/policy/d27a3bc66f7e3c8e957cdaa01d5eacc1?meta=json HTTP/1.1Host: my.example.comDate: _date_

RESPONSE /policy/<policy id>

HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8



Content-Length: _length_{"Id":"9adea14a85004c44883301dca8864b99","Spread-Width":16,"Safety":4,"Hierarchy-Rules":[ false , true , true ],"Max-Superblock-Size":33554432,"Safety-Strategy":[ "RepairSpread" , "DynamicSafety" ],"N-Messages":4096"Full-Copy":false}

xml REQUEST /manage/policy/<policy id>

GET /manage/policy/d27a3bc66f7e3c8e957cdaa01d5eacc1 HTTP/1.1Host: my.example.comDate: _date_Accept: application/xml

or:GET /manage/policy/d27a3bc66f7e3c8e957cdaa01d5eacc1?meta=xml HTTP/1.1Host: my.example.comDate: _date_

RESPONSE /manage/policy/<policy id>

HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Length: _length_Content-Type: application/xml;charset=UTF-8<?xml version="1.0" encoding="UTF-8"?><Policy xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>9adea14a85004c44883301dca8864b99</Id>


Chapter 5: Policy OperationsList Policies

<Spread-Width>16</Spread-Width><Safety>4</Safety><Hierarchy-Rules>false true true</Hierarchy-Rules><Max-Superblock-Size>33554432</Max-Superblock-Size><Safety-Strategy>RepairSpread DynamicSafety</Safety-Strategy><N-Messages>4096</N-Messages><Full-Copy>false</Full-Copy></Policy>

List Policies

A GET performed on /policy directly returns a list of all policy ids.

A HEAD request yields the same response as a GET request, except that the server does not return a message-body in the response.

http / text REQUEST /manage/policy/

GET /manage/policy/ HTTP/1.1Host: my.example.comDate: _date_Accept: text/plain

or:GET /manage/policy/?list=http HTTP/1.1Host: my.example.comDate: _date_

RESPONSE /manage/policy/

HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24


Chapter 5: Policy OperationsList Policies

Content-Type: text/plain;charset=UTF-8Content-Length: _length_"d27a3bc66f7e3c8e957cdaa01d5eacc1""9f5fb1b3de6e314ca093cd613d6b383a"

json REQUEST /manage/policy/

GET /manage/policy/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/json

or:GET /manage/policy/?list=json HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8Content-Length: _length_["d27a3bc66f7e3c8e957cdaa01d5eacc1","9f5fb1b3de6e314ca093cd613d6b383a"]

xml REQUEST /manage/policy/

GET /manage/policy/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/xml


Chapter 5: Policy OperationsDelete Policy

or:GET /manage/policy/?list=xml HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8Content-Length: _length_<Policies xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>d27a3bc66f7e3c8e957cdaa01d5eacc1</Id><Id>9f5fb1b3de6e314ca093cd613d6b383a</Id></Policies>

Delete Policy

Deleting policies is not possible.


Chapter 5: Policy OperationsDelete Policy


Chapter 6Namespaces

User Operations

List Namespaces

A GET can be performed on /namespace/ directly and returns a list of all namespace names the current authenticated user has (any) permissions on. An administrator is able to retrieve the list of all namespace names, regardless of the permissions on the namespaces.


http/text REQUEST /namespace/

GET /namespace/ HTTP/1.1Host: my.example.comDate: _date_ Accept: text/plain

or:GET /namespace/?list=http HTTP/1.1


Chapter 6: NamespacesList Namespaces

Host: my.example.comDate: _date_

RESPONSE /namespace/

HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: _length_"VOL"

json REQUEST /namespace/

GET /namespace/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/json

or:GET /namespace/?list=json HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OK Date: _date_Server: Quantum-Lattus/3.0.3-ba79602e871550668391a673fa57589cf531497eContent-Type: application/json;charset=UTF-8Content-LengHTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8Content-Length: _length_["VOL",]


Chapter 6: NamespacesList Objects in a Namespace

xml REQUEST /namespace/

GET /namespace/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/xml

or:GET /namespace/?list=xml HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8Content-Length: _length_<Namespaces xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Name>VOL</Name></Namespaces>

List Objects in a Namespace

A GET can be performed on /namespace/<namespace name> and returns a list of objects in the root directory of a specific namespace (see the Object API for details).


http/text REQUEST /namespace/<namespace name>

GET /namespace/VOL HTTP/1.1



Host: my.example.comDate: _date_ Accept: text/plain

or:GET /namespace/VOL?list=http HTTP/1.1Host: my.example.comDate: _date_

RESPONSE /namespace/<namespace name>

HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: _length_"file1""file2"

json REQUEST /namespace/<namespace name>

GET /namespace/VOL HTTP/1.1Host: my.example.comDate: _date_Accept: application/json

or:GET /namespace/VOL?list=json HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8Content-Length: _length_["file1","file2"



]

xml REQUEST /namespace/<namespace name>

GET /namespace/VOL HTTP/1.1Host: my.example.comDate: _date_Accept: application/xml

or:GET /namespace/VOL?list=xml HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8Content-Length: _length_<Directory-Entriesxmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Name>file1</Name><Name>file2</Name></Directory-Entries>


Chapter 6: NamespacesCreate Namespace

Admin Operations

Create Namespace

A POST/PUT request creates a NameSpace. Creating a NameSpace requires the following parameters:

• Name: Unique name of the NameSpace (string)

• Policy-Id: Unique policy GUID that represents the protection policy (string)

If the namespace is successfully created, the return status is "201 Created". It then returns the metadata of the created Namespace.

If a namespace with the same name already exists, the return status is "403 Forbidden", and no metadata is returned.

http/text REQUEST /manage/namespace

PUT /manage/namespace HTTP/1.1Host: my.example.comDate: _date_Accept: text/plainContent-Type: text/plainContent-Length: 0X-Ampli-Name: "VOL"X-Ampli-Policy-Id: "d27a3bc66f7e3c8e957cdaa01d5eacc1"X-Ampli-Syncstore-Id:"a5069e6108ed48de933904a1d87eb925"X-Ampli-Master-Node-Id: 4X-Ampli-Verification-Interval: 15552000X-Ampli-Verification-Target-Date: 15552000X-Ampli-Small-Files-Policy-Id: "7c76bb4ca67f4e07a2a47db280c2a32f"X-Ampli-Small-Files-Threshold: 50000

or:PUT /manage/namespace?meta=http HTTP/1.1Host: my.example.com



Date: _date_Content-Length: 0X-Ampli-Name: "VOL"X-Ampli-Policy-Id: "d27a3bc66f7e3c8e957cdaa01d5eacc1"X-Ampli-Syncstore-Id: "a5069e6108ed48de933904a1d87eb925"X-Ampli-Master-Node-Id: 4X-Ampli-Verification-Interval: 15552000X-Ampli-Verification-Target-Date: 15552000X-Ampli-Small-Files-Policy-Id: "7c76bb4ca67f4e07a2a47db280c2a32f"X-Ampli-Small-Files-Threshold: 50000

RESPONSE /manage/namespace

HTTP/1.1 201 CreatedDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: 0X-Ampli-Id: "fbee9730697b43458e705d10bc6b8c94"X-Ampli-Name: "VOL"X-Ampli-Policy-Id: "d27a3bc66f7e3c8e957cdaa01d5eacc1"X-Ampli-Syncstore-Id: "a5069e6108ed48de933904a1d87eb925"X-Ampli-Master-Node-Id: 4X-Ampli-Namespace-Location-Id: 0X-Ampli-Verification-Interval: 15552000X-Ampli-Verification-Target-Date: _epoch_time_X-Ampli-Creation-Date: _epoch_time_X-Ampli-Small-Files-Policy-Id: "7c76bb4ca67f4e07a2a47db280c2a32f"X-Ampli-Small-Files-Threshold: 50000

json REQUEST /manage/namespace

PUT /manage/namespace HTTP/1.1Host: my.example.comDate: _date_Accept: application/json



Content-Type: application/json Content-Length: _length_

{"Name":"VOL""Policy-Id":"d27a3bc66f7e3c8e957cdaa01d5eacc1" "Syncstore-Id":"a5069e6108ed48de933904a1d87eb925","Master-Node-Id":4,"Verification-Interval":15552000,"Verification-Target-Date":15552000,"Small-Files-Policy-Id":"68484ee04c57449abf1dffcb214a322b","Small-Files-Threshold":50000}

or:PUT /manage/namespace?meta=json HTTP/1.1Host: my.example.comDate: _date_Content-Length: _length_

{"Name":"VOL""Policy-Id":"d27a3bc66f7e3c8e957cdaa01d5eacc1""Syncstore-Id":"a5069e6108ed48de933904a1d87eb925","Master-Node-Id":4,"Verification-Interval":15552000,"Verification-Target-Date":15552000,"Small-Files-Policy-Id":"68484ee04c57449abf1dffcb214a322b","Small-Files-Threshold":50000}


HTTP/1.1 201 CreatedDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8Content-Length: _length_{"Id":"fbee9730697b43458e705d10bc6b8c94""Name":"VOL"



"Policy-Id":"d27a3bc66f7e3c8e957cdaa01d5eacc1""Syncstore-Id":"a5069e6108ed48de933904a1d87eb925""Master-Node-Id":4"Namespace-Location-Id":0"Verification-Interval":15552000"Verification-Target-Date":_epoch_time_"Creation-Date":_epoch_time_,"Small-Files-Policy-Id":"68484ee04c57449abf1dffcb214a322b","Small-Files-Threshold":50000}

xml REQUEST /manage/namespace

PUT /manage/namespace HTTP/1.1Host: my.example.comDate: _date_Accept: application/xml Content-Type: application/xml Content-Length: _length_

<?xml version="1.0" encoding="UTF-8"?><Namespace xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Name>VOL</Name><Policy-Id>d27a3bc66f7e3c8e957cdaa01d5eacc1</Policy-Id><Syncstore-Id>a5069e6108ed48de933904a1d87eb925</Syncstore-Id><Master-Node-Id>4</Master-Node-Id><Verification-Interval>15552000</Verification-Interval><Verification-Target-Date>_epoch_time_</Verification-Target-Date><Small-Files-Policy-Id>c81af402571b45c19096617c5a8fe461</Small-Files-Policy-Id><Small-Files-Threshold>50000</Small-Files-Threshold></Namespace>

or:PUT /manage/namespace?meta=xml HTTP/1.1Host: my.example.comDate: _date_



Content-Length: _length_

<?xml version="1.0" encoding="UTF-8"?><Namespace xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Name>VOL</Name><Policy-Id>d27a3bc66f7e3c8e957cdaa01d5eacc1</Policy-Id><Syncstore-Id>a5069e6108ed48de933904a1d87eb925</Syncstore-Id><Master-Node-Id>4</Master-Node-Id><Verification-Interval>15552000</Verification-Interval><Verification-Target-Date>_epoch_time_</Verification-Target-Date><Small-Files-Policy-Id>c81af402571b45c19096617c5a8fe461</Small-Files-Policy-Id><Small-Files-Threshold>50000</Small-Files-Threshold></Namespace>


HTTP/1.1 201 CreatedDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8Content-Length: _length_<?xml version="1.0" encoding="UTF-8"?><Namespace xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>fbee9730697b43458e705d10bc6b8c94</Id><Name>VOL</Name><Policy-Id>d27a3bc66f7e3c8e957cdaa01d5eacc1</Policy-Id><Syncstore-Id>a5069e6108ed48de933904a1d87eb925</Syncstore-Id><Master-Node-Id>4</Master-Node-Id><Namespace-Location-Id>0</Namespace-Location-Id><Verification-Interval>15552000</Verification-Interval><Verification-Target-Date>_epoch_time_</Verification-Target-Date><Creation-Date>_epoch_time_</Creation-Date>


Chapter 6: NamespacesUpdate Namespace

<Small-Files-Policy-Id>c81af402571b45c19096617c5a8fe461</Small-Files-Policy-Id><Small-Files-Threshold>50000</Small-Files-Threshold></Namespace>

Update Namespace

With a PUT request, you can also modify a namespace. However, at this time you can modify only the encryption policy.

http/text To enable encryption for a namespace/bucket using a specific encryption policy, set the desired encryption policy ID.

To disable encryption, set the encryption policy to “none”.

REQUEST /manage/namespace/<namespace name>

PUT /manage/namespace/<namespace name> HTTP/1.1Host: my.example.comDate: _date_Accept: text/plainContent-Type: text/plainContent-Length: 0X-Ampli-Encryption-Policy: "1"

or:PUT /manage/namespace/<namespace name>?meta=http HTTP/1.1Host: my.example.comDate: _date_Content-Length: 0X-Ampli-Encryption-Policy: "1"

RESPONSE /manage/namespace/<namespace name>

HTTP/1.1 200 OK



Date: _date_Server: Quantum-Lattus/3.0.0Content-Type: text/plain;charset=UTF-8Content-Length: 0X-Ampli-Id: "fbee9730697b43458e705d10bc6b8c94"X-Ampli-Name: _namespace name_X-Ampli-Policy-Id: "d27a3bc66f7e3c8e957cdaa01d5eacc1"X-Ampli-Syncstore-Id: "a5069e6108ed48de933904a1d87eb925"X-Ampli-Master-Node-Id: 4X-Ampli-Namespace-Location-Id: 0X-Ampli-Verification-Interval: 15552000X-Ampli-Verification-Target-Date: _epoch_time_X-Ampli-Creation-Date: _epoch_time_X-Ampli-Small-Files-Policy-Id: "7c76bb4ca67f4e07a2a47db280c2a32f"X-Ampli-Small-Files-Threshold: 50000X-Ampli-Encryption-Policy: "1"X-Ampli-Encryption-Scheme: "AES-256-CTR"

Note: The ‘‘X-Ampli-Encryption-Scheme header cannot be modified or user specified. Only one scheme is currently supported: AES-256-CTR.

json To enable encryption for a namespace/bucket using a specific encryption policy, set the desired encryption policy ID.

To disable encryption, set the encryption policy to “null”.

REQUEST /manage/namespace/<namespace>

PUT /manage/namespace/<namespace name> HTTP/1.1Host: my.example.comDate: _date_Accept: application/jsonContent-Type: application/jsonContent-Length: _length_{"Encryption-Policy":"1"}



or:PUT /manage/namespace/<namespace name>?meta=json HTTP/1.1Host: my.example.comDate: _date_Content-Length: _length_{"Encryption-Policy":"1"}


HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0Content-Type: application/json;charset=UTF-8Content-Length: 471{"Id":"8960a887ce4b441ebc22f9843877a355","Name":"VOL","Policy-Id":"b0d8e3dd5b674ef1b3bd09185198c270","Syncstore-Id":"701d5929a46f4bd49e8aa7627ecd62d8","Master-Node-Id":1,"Namespace-Location-Id":0,"Verification-Interval":15552000,"Verification-Target-Date":1385465338,"Creation-Date":1369913338,"Small-Files-Policy-Id":"796e6685ef534a88945071b9b3222d80","Small-Files-Threshold":1048576,"Encryption-Policy":"1","Encryption-Scheme":"AES-256-CTR"}

xml To enable encryption for this namespace/bucket, using a specific encryption policy: <Encryption-Policy>1</Encryption-Policy>

To disable encryption for this name space/bucket: <Encryption-Policy></Encryption-Policy> or simply <Encryption-Policy/>



REQUEST /manage/namespace<name space>

PUT /manage/namespace/<namespace name> HTTP/1.1Host: my.example.comDate: _date_Accept: application/xmlContent-Type: application/xmlContent-Length: _length_<?xml version="1.0" encoding="UTF-8"?><Namespace xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Encryption-Policy>1</Encryption-Policy></Namespace>

or:PUT /manage/namespace/<namespace name>?meta=xml HTTP/1.1Host: my.example.comDate: _date_Content-Length: _length_<?xml version="1.0" encoding="UTF-8"?><Namespace xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Encryption-Policy>1</Encryption-Policy></Namespace>


HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0Content-Type: application/xml;charset=UTF-8Content-Length: _length_<?xml version="1.0" encoding="UTF-8"?><Namespace xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>8960a887ce4b441ebc22f9843877a355</Id><Name>VOL</Name><Policy-Id>b0d8e3dd5b674ef1b3bd09185198c270</Policy-Id><Syncstore-Id>701d5929a46f4bd49e8aa7627ecd62d8</Syncstore-Id><Master-Node-Id>1</Master-Node-Id><Namespace-Location-Id>0</Namespace-Location-Id>


Chapter 6: NamespacesGet Namespace

<Verification-Interval>15552000</Verification-Interval><Verification-Target-Date>1385465338</Verification-Target-Date><Creation-Date>1369913338</Creation-Date><Small-Files-Policy-Id>796e6685ef534a88945071b9b3222d80</Small-Files-Policy-Id><Small-Files-Threshold>1048576</Small-Files-Threshold><Encryption-Policy>1</Encryption-Policy><Encryption-Scheme>AES-256-CTR</Encryption-Scheme></Namespace>

Get Namespace

A GET request on /manage/namespace/<namespace name> returns (depending on the query parameters) a list of objects in the root directory of a specific namespace or the metadata of that specific namespace. See the Object API for details.


http/text REQUEST /manage/namespace/<namespace name>

GET /namespace/VOL?meta= HTTP/1.1Host: my.example.comDate: _date_ Accept: text/plain

or:GET /namespace/VOL?meta=http HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_



Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: 0X-Ampli-Id: "fbee9730697b43458e705d10bc6b8c94"X-Ampli-Name: "VOL"X-Ampli-Policy-Id: "d27a3bc66f7e3c8e957cdaa01d5eacc1"X-Ampli-Syncstore-Id: "a5069e6108ed48de933904a1d87eb925"X-Ampli-Master-Node-Id: 4X-Ampli-Namespace-Location-Id: 0X-Ampli-Verification-Interval: 15552000X-Ampli-Verification-Target-Date: _epoch_time_X-Ampli-Creation-Date: _epoch_time_X-Ampli-Small-Files-Policy-Id: "c81af402571b45c19096617c5a8fe461"X-Ampli-Small-Files-Threshold: 50000

json REQUEST /manage/namespace/<namespace name>

GET /manage/namespace/VOL?meta= HTTP/1.1Host: my.example.comDate: _date_Accept: application/json

or:GET /manage/namespace/VOL?meta=json HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8Content-Length: _length_{"Id":"fbee9730697b43458e705d10bc6b8c94""Name":"VOL""Policy-Id":"d27a3bc66f7e3c8e957cdaa01d5eacc1"



"Syncstore-Id":"a5069e6108ed48de933904a1d87eb925""Master-Node-Id":4"Namespace-Location-Id":0"Verification-Interval":15552000"Verification-Target-Date":_epoch_time_X-Ampli-Creation-Date: _epoch_time_X-Ampli-Small-Files-Policy-Id: "c81af402571b45c19096617c5a8fe461"X-Ampli-Small-Files-Threshold: 50000}

xml REQUEST /manage/namespace/<namespace name>

GET /manage/namespace/VOL?meta= HTTP/1.1Host: my.example.comDate: _date_Accept: application/xml

or:GET /manage/namespace/VOL?meta=xml HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Length: _length_Content-Type: application/xml;charset=UTF-8<?xml version="1.0" encoding="UTF-8"?><Namespace xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>fbee9730697b43458e705d10bc6b8c94</Id><Name>VOL</Name><Policy-Id>d27a3bc66f7e3c8e957cdaa01d5eacc1</Policy-Id><Syncstore-Id>a5069e6108ed48de933904a1d87eb925</Syncstore-Id><Master-Node-Id>4</Master-Node-Id><Namespace-Location-Id>0</Namespace-Location-Id><Verification-Interval>15552000</Verification-Interval>



<Verification-Target-Date>_epoch_time_</Verification-Target-Date>X-Ampli-Creation-Date: _epoch_time_X-Ampli-Small-Files-Policy-Id: "c81af402571b45c19096617c5a8fe461"X-Ampli-Small-Files-Threshold: 50000</Namespace>

List Namespaces

A GET performed on /manage/namespace/ returns a list of all namespaces, regardless of the permissions on the namespaces and whether the client side GET request has a digest user specified.


http/text REQUEST /manage/namespace/

GET /manage/namespace/ HTTP/1.1Host: my.example.comDate: _date_ Accept: text/plain

or:GET /manage/namespace/?list=http HTTP/1.1Host: my.example.comDate: _date_

RESPONSE /manage/namespace/

HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: _length_"VOL""VOL2"



json REQUEST /manage/namespace/

GET /namespace/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/json

or:GET /manage/namespace/?list=json HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8Content-Length: _length_["VOL","VOL2",]

xml REQUEST /manage/namespace/

GET /manage/namespace/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/xml

or:GET /manage/namespace/?list=xml HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_


Chapter 6: NamespacesDelete Namespace

Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8Content-Length: _length_<Namespaces xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Name>VOL</Name><Name>VOL2</Name></Namespaces>

Delete Namespace

REQUEST /manage/namespace/<namespace name>

DELETE /manage/namespace/VOL HTTP/1.1Host: my.example.comDate: _date_

RESPONSE /manage/namespace/<namespace name>μ

HTTP/1.1 204 No ContentDate: _date_Server: Amplidata-AmpliStor/2.5.0-1f206c99fa6c1df2070e69741a6c0c6d52353ec9Content-Length: 0


Chapter 6: NamespacesGET/HEAD

Capacity Reporting

GET/HEAD

To retrieve a batched list of namespace capacity reports, perform a GET on either:

• /manage/capacity/namespace

• /s3manage/capacity/bucket

To retrieve a batched list of syncstore id's, a GET can be performed on either:

• /manage/capacity/syncstore/

• /s3manage/capacity/syncstore/

To retrieve the capacity report of a specific syncstore, perform a GET on either:

• /manage/capacity/syncstore/<syncstore id>

• /s3manage/capacity/syncstore/<syncstore id>

A HEAD request returns the same response as a GET request, but the server does not return a message-body in the response.

List Namespace Capacity Reports

By default, only the first 50 namespace capacity reports are returned. This limit can be adjusted to X by appending the query parameter "limit=X" to the URL.

Note: The maximum value for this limit is 1,023. Higher values will result in an HTTP 500 Error for AXR or in at most 1000 entries for S3.


Chapter 6: NamespacesList Namespace Capacity Reports

The following example url can be used:

/manage/capacity/namespace/?limit=500

In addition, the namespace from which the list should start can be specified using "marker=Y". For example, the following URL could be used to list 4 namespace capacity reports starting from namespace "test": /manage/user?marker=test&limit=4. An additional query parameter, "include_marker=false" can be added, which results in the marker being not included in the list.

json REQUEST /manage/capacity/namespace/

GET /manage/capacity/namespace/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/json

or:GET /manage/capacity/namespace/?list=json HTTP/1.1Host: my.example.comDate: _date_

RESPONSE /manage/capacity/namespace/

HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Type: application/json;charset=UTF-8Content-Length: 1201Pragma: no-cacheCache-Control: no-cacheExpires: Thu, 01 Jan 1970 00:00:00 GMT{"name":"NS1","start_date":1400154451.88,"last_update":1400154452.01,"current_policy_id_stats":[{"Id":"8e8b395a79d941e7b0d9a0e9476babeb","Statistics":{"nr_objects_ok":4,"nr_superblocks_ok":4,"nr_objects_repair":0,"nr_superblocks_repair":0,



"nr_objects_delete":0,"nr_superblocks_delete":0,"nr_objects_unverified":0,"nr_superblocks_unverified":0,"capacity_frontend_ok":30788,"capacity_frontend_repair":0,"capacity_frontend_delete":0,"capacity_frontend_objects_unverified":0,"capacity_frontend_superblocks_unverified":0,"disksafety_objects":[ {"disk safety":4, "count":4} ],"disksafety_superblocks":[ {"disk safety":4, "count":4} ],"disksafety_objects_offline":[ {"disk safety":4, "count":4} ],"disksafety_superblocks_offline":[ {"disk safety":4, "count":4} ],"disksafety_objects_decommissioned":[ {"disk safety":4, "count":4} ],"disksafety_superblocks_decommissioned":[ {"disk safety":4, "count":4} ],"policy_stats_hashtbl":{"nr" : {"object" : {"ok" : 4,"all" : 4,"repair" : 0,"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 4 },"disk safety offline" : { "4" : 4 },"disk safety decommissioned" : { "4" : 4 },"needs conversion" : 0 },"part" : {"ok" : 4,"all" : 4,"repair" : 0,"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 4 },"disk safety offline" : { "4" : 4 },"disk safety decommissioned" : { "4" : 4 },"needs conversion" : 0 },"superblock" : {



"ok" : 4,"all" : 4,"repair" : 0,"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 4 },"disk safety offline" : { "4" : 4 },"disk safety decommissioned" : { "4" : 4 },"needs conversion" : 0 } },"capacity" : {"object" : {"ok" : 30788,"all" : 30788,"repair" : 0,"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 30788 },"disk safety offline" : { "4" : 30788 },"disk safety decommissioned" : { "4" : 30788 },"needs conversion" : 0 },"part" : {"ok" : 30788,"all" : 30788,"repair" : 0,"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 30788 },"disk safety offline" : { "4" : 30788 },"disk safety decommissioned" : { "4" : 30788 },"needs conversion" : 0 },"superblock" : {"ok" : 30788,"all" : 30788,"repair" : 0,"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 30788 },"disk safety offline" : { "4" : 30788 },"disk safety decommissioned" : { "4" : 30788 },"needs conversion" : 0 }}}}



},{"Id":"e10bf033967e46baa88909d521cdd963","Statistics":{"nr_objects_ok":5,"nr_superblocks_ok":201,"nr_objects_repair":0,"nr_superblocks_repair":0,"nr_objects_delete":0,"nr_superblocks_delete":0,"nr_objects_unverified":0,"nr_superblocks_unverified":0,"capacity_frontend_ok":13309621000,"capacity_frontend_repair":0,"capacity_frontend_delete":0,"capacity_frontend_objects_unverified":0,"capacity_frontend_superblocks_unverified":0,"disksafety_objects":[ {"disk safety":4, "count":5} ],"disksafety_superblocks":[ {"disk safety":4, "count":201} ],"disksafety_objects_offline":[ {"disk safety":4, "count":5} ],"disksafety_superblocks_offline":[ {"disk safety":4, "count":201} ],"disksafety_objects_decommissioned":[ {"disk safety":4, "count":5} ],"disksafety_superblocks_decommissioned":[ {"disk safety":4, "count":201} ],"policy_stats_hashtbl":{"nr" : {"object" : {"ok" : 5,"all" : 5,"repair" : 0,"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 5 },"disk safety offline" : { "4" : 5 },"disk safety decommissioned" : { "4" : 5 },"needs conversion" : 0 },"part" : {"ok" : 5,"all" : 5,"repair" : 0,



"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 5 },"disk safety offline" : { "4" : 5 },"disk safety decommissioned" : { "4" : 5 },"needs conversion" : 0 },"superblock" : {"ok" : 201,"all" : 201,"repair" : 0,"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 201 },"disk safety offline" : { "4" : 201 },"disk safety decommissioned" : { "4" : 201 },"needs conversion" : 0 }},"capacity" : {"object" : {"ok" : 13309621000,"all" : 13309621000,"repair" : 0,"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 13309621000 },"disk safety offline" : { "4" : 13309621000 },"disk safety decommissioned" : { "4" : 13309621000 },"needs conversion" : 0 },"part" : {"ok" : 13309621000,"all" : 13309621000,"repair" : 0,"change policy" : 0,"unverified" : 0,"disk safety normal" : { "4" : 13309621000 },"disk safety offline" : { "4" : 13309621000 },"disk safety decommissioned" : { "4" : 13309621000 },"needs conversion" : 0 },"superblock" : {"ok" : 13309621000,"all" : 13309621000,"repair" : 0,"change policy" : 0,



"unverified" : 0,"disk safety normal" : { "4" : 13309621000 },"disk safety offline" : { "4" : 13309621000 },"disk safety decommissioned" : { "4" : 13309621000 },"needs conversion" : 0 }}}}} ],"old_policy_id_stats":[ ],"object_name_length_stats":{"name": "object_name_length_stats","average": 12.777778,"variance": 18.839506,"count": 9,"min": 5.000000,"max": 20.000000 }

• version: version of the format in which the data is stored. This information is of no further use for the user.

• name: name of the monitored name space

• start_date: date and time when the last monitor crawl was started

• last_update: last update of the cached data

• current_policy_id_stats: dict with policy guid as key and value the following information:

• nr_objects_ok: number of healthy objects

• nr_superblocks_ok: number of healthy superblocks

• nr_objects_repair: number of objects that have at least one superblock in repair status

• nr_superblocks_repair: number of superblocks that need repair

• nr_objects_delete: number of objects that need to be deleted

• nr_superblocks_delete: number of superblocks that still need to be deleted

• nr_objects_unverified: number of unverified objects. Every object is verified each 180 days if it is still a healthy object. If this verification would not have taken place within these 180 days, the object is considered as unverified.



• nr_superblocks_unverified: number of unverified superblocks. This is similar as the unverified objects, but for superblocks.

• capacity_frontend_ok: the sum of the sizes of the objects put that are healthy, expressed in bytes

• capacity_frontend_repair: the sum of the sizes of the objects put that need to be repaired, expressed in bytes

• capacity_frontend_delete: the sum of the sizes of the objects put that still need to be deleted, expressed in bytes

• capacity_frontend_objects_unverified: the sum of the sizes of the objects put that is still unverified, expressed in bytes

• capacity_frontend_superblocks_unverified: the sum of the sizes of the superblocks put that is still unverified, expressed in bytes

• disksafety_objects: a dict with keys going from ‘disksafety - spread width’ to disksafety giving for all these values the number of objects that have that disk safety, taking into account the ABANDONED blockstores.

• disksafety_superblocks: a dict with keys going from ‘disksafety - spread width’ to disksafety giving for all these values the number of superblocks that have that disk safety, taking into account the ABANDONED blockstores

• disksafety_objects_offline: a dict with keys going from ‘disksafety - spread width’ to disksafety giving for all these values the number of objects that have that disk safety, taking into account the ABANDONED and OFFLINE blockstores

• disksafety_superblocks_offline: a dict with keys going from ‘disksafety - spread width’ to disksafety giving for all these values the number of superblocks that have that disk safety, taking into account the ABANDONED and OFFLINE blockstores

• disksafety_objects_decommissioned: a dict with keys going from ‘disksafety - spread width’ to disksafety giving for all these values the number of objects that have that disk safety, taking into account the ABANDONED, OFFLINE and DECOMMISSIONED blockstores

• disksafety_superblocks_decommissioned: a dict with keys going from ‘disksafety - spread width’ to disksafety giving for all these values the number of superblocks that have that disk



safety, taking into account the ABANDONED,OFFLINE and DECOMMISSIONED blockstores

• policy_stats_hashtbl: overview with the statistics of all objects, parts, and superblocks with that policy as their target. This is a dict which contains two main sections, “nr” and “capacity”, respectively the number of items and the taken disk space of the items. These two main sections show the information by objects, parts, and superblocks.

Note: The Lattus parts are not exactly the same as the S3 parts. In S3, there is a flat structure of the parts, in Lattus you have a tree structure, where the main object (or storage object) consists of parts and each part can consist of child-parts.

• nr: gives you an overview of the number of items (objects, parts, superblocks)

• capacity: gives you an overview of the taken disk space, split in objects, parts, and superblocks. The indicated disk spaces are expressed in bytes.

• ok: all healthy items, which don’t have either label REPAIR, CHANGE_POLICY, or UNVERIFIED

• all: all items

• repair: all items which need a repair action

• change policy: items for which a new policy is selected, but which are not yet encoded with the new policy

• unverified: items which are unverified. Every item is verified each 180 days if it is still a healthy object. If this verification would not have taken place within these 180 days, the item is considered as unverified.

• disk safety normal: dict which takes the healthy blockstores into account. The disk safety is the key and the number of items/disk space is the value. This dict can contain multiple key/value pairs.

• disk safety offline: dict which takes the offline blockstores into account.

• needs conversion: remaining items/data volume that need conversion. This is a decreasing value and is ideally 0.


Chapter 6: NamespacesList Syncstores

• disk safety decommissioned: dict which takes the decommissioned blockstores into account.

• old_policy_id_stats: is no longer in use as of Lattus 3.4.0, and is replaced by the parameter change policy. It was used to report on the statistics of objects in a name space which were not yet encoded with the target policy. This parameter is still in use for compatibility reasons.

• object_name_length_stats: statistics about the length of the object names in the name space.

• average: average object name length

• count: number of object to calculate the average length

• variance: variance of the length

• max: length of the longest object name

• min: length of the shortest object name

• name: name of the statistic

You can see that there are two policy ID statistices. This occurs when the namespace uses a policy which has also an enabled small files policy.

List Syncstores

json REQUEST /manage/capacity/syncstore/

GET /manage/capacity/syncstore/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/json

or:GET /manage/capacity/syncstore/?list=json HTTP/1.1Host: my.example.comDate: _date_


Chapter 6: NamespacesGet Syncstore Capacity Report

RESPONSE /manage/capacity/syncstore

HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Type: application/json;charset=UTF-8Content-Length: 78Pragma: no-cacheCache-Control: no-cacheExpires: Thu, 01 Jan 1970 00:00:00 GMT["160dfb46b98c481f9d18b3d98041017d","825a2f8627334b3bb8f480f63ef7f663"]

Get Syncstore Capacity Report

json REQUEST /manage/capacity/syncstore/<syncstore id>

GET /manage/capacity/syncstore/<syncstore id> HTTP/1.1Host: my.example.comAccept: application/json

or:GET /manage/capacity/syncstore/<syncstore id>?meta=json HTTP/1.1Host: my.example.com

RESPONSE /manage/capacity/syncstore/<syncstore id>

HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/_revisionContent-Type: application/json;charset=UTF-8Content-Length: 58{"Id":"167da646596f4f83b6cb979cfd4c2afb","Count":5000}


Chapter 6: NamespacesGet Syncstore Capacity Report


Chapter 7Object Operations

Create or Overwrite Object

Use the PUT operation to add objects to a namespace. The Lattusenvironment always stores the complete object. An error code will be returned if a failure is encountered during the upload process.

By default the object is overwritten every time when a PUT action is processed on an existing object ID.

If multiple PUT actions are processed simultaneously on the same object, then the last successful PUT will define the final content of the object.

A POST/PUT request creates an object and returns the metadata of the object. On successful creation of an object, the return status is “201 Created”. On successful overwrite, the return status is “200 OK”.

For HTTP, the default is to allow overwrite. If any existing object should not be overwritten, the header “If-None-Match: *” can be added. See the If-None-Match sections for details.

Lattus also supports the concept of subdirectories. To create a file object in a subdirectory of a namespace, its parent directory must already have been created. Parent directories are not automatically created. See Create Directory Object on page 123 for more information.


Chapter 7: Object OperationsTransfer with Content-Length

Overwriting objects preserves creation date and preserves existing custom metadata.

There are two modes for uploading files:

• Chunked mode

• Non-Chunked mode

Chunked mode is used when the object size is not known up front. Data streaming can be achieved by uploading multiple chunks one after another. Non-chunked mode expects the content length of the object to be known up front.

Transfer with Content-Length

http REQUEST /namespace/<namespace name>/<object name>

PUT /namespace/namespace1/object1 HTTP/1.1Host: my.example.comDate: _date_ Accept: text/plainContent-Type: binary/octet-streamContent-Length: _object_length_

_object_length_bytes_of_data_or:PUT /namespace/namespace1/object1?meta=http HTTP/1.1Host: my.example.comDate: _date_Content-Type: binary/octet-streamContent-Length: _object_length_

_object_length_bytes_of_data_

RESPONSE /namespace/<namespace name>/<object name>

HTTP/1.1 201 CreatedDate: _date_



Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: 0Last-Modified: _date_ETag: "8a6547849714fa7f021a923e391a87a8"X-Ampli-Name: "object1"X-Ampli-Id: "6a4f803b1efc48ad9a93ab5e1ee8380e"X-Ampli-Namespace-Id: "9777e0eef27c44b7929806a8c42f1684"X-Ampli-Kind: "File"X-Ampli-Policy-Id: "d684bbd656004c50a29db47b67ae9666"X-Ampli-Size: _object_length_X-Ampli-Crc32: 3827519356X-Ampli-Status: OKX-Ampli-Creation-Date: _epoch_time_X-Ampli-Modification-Date: _epoch_time_X-Ampli-Verification-Date: _epoch_time_

json REQUEST /namespace/<namespace name>/<object name>

PUT /namespace/namespace1/object1 HTTP/1.1Host: my.example.comDate: _date_Accept: application/jsonContent-Type: binary/octet-streamContent-Length: _object_length_


or:PUT /namespace/namespace1/object1?meta=json HTTP/1.1Host: my.example.comDate: _date_Content-Type: binary/octet-streamContent-Length: _object_length_



HTTP/1.1 201 Created



Date: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8Content-Length: _length_Last-Modified: _date_ETag: "8a6547849714fa7f021a923e391a87a8"{"Name": "object1""Id": "6a4f803b1efc48ad9a93ab5e1ee8380e""Namespace-Id": "9777e0eef27c44b7929806a8c42f1684""Kind":"File","Policy-Id": "d684bbd656004c50a29db47b67ae9666""Size": _object_length_"Crc32": 3827519356"Status": OK"Creation-Date": _epoch_time_"Modification-Date": _epoch_time_"Verification-Date": _epoch_time_}

xml REQUEST /namespace/<namespace name>/<object name>

PUT /namespace/namespace1/object1 HTTP/1.1Host: my.example.comDate: _date_Accept: application/xmlContent-Type: binary/octet-streamContent-Length: _object_length_


or:PUT /namespace/namespace1/object1?meta=xml HTTP/1.1Host: my.example.comDate: _date_Content-Type: binary/octet-streamContent-Length: _object_length_



Chapter 7: Object OperationsTransfer Without Content-Length


HTTP/1.1 201 CreatedDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8Content-Length: _length_Last-Modified: _date_ETag: "8a6547849714fa7f021a923e391a87a8"<Object xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>6a4f803b1efc48ad9a93ab5e1ee8380e"</Id><Name>object1</Name><Namespace-Id>9777e0eef27c44b7929806a8c42f1684"</Namespace-Id><Kind>File</Kind><Policy-Id>d684bbd656004c50a29db47b67ae9666</Policy-Id><Size>_object_length_</Size><Crc32>3827519356</Crc32><Status>OK</Status><Creation-Date>_epoch_time_</Creation-Date><Modification-Date>_epoch_time_</Modification-Date><Verification-Date>_epoch_time_</Verification-Date></Object>

Transfer Without Content-Length

The HTTP/1.1 specification provides a way to to PUT an object of which the size is not known beforehand, by repeatedly sending chunks of a number of bytes of content, and a final chunk of size 0 bytes.

For this purpose, the Transfer-Encoding header exists. See Transfer-Encoding on page 119 below for details.

Request Headers For the PUT of an object several request headers can be specified to:

• make the PUT conditional, or



• modify the way in which the object content is streamed to the server.

Note: The examples below also work for other meta types than "meta=http".

If-Match This header is supported for the PUT of an object, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.24 "If-Match".

See also ETag on page 141.

The subsections contains an example: the ETag we received from the server for object "/namespace/namespace1/object1" in a previous response to a PUT request or GET request was "19d0a529e39347f4823c6e22f1160825". We wish to overwrite the object, but only if it has not been changed in the mean time. In this scenario, the object has been changed in the mean time, so the ETag no longer matches, and the server responds with a 412 error response.

REQUEST /namespace/<namespace name>/<object name>

PUT /namespace/namespace1/object1?meta=http HTTP/1.1Host: my.example.comIf-Match: "19d0a529e39347f4823c6e22f1160825" Content-Length: 19Expect: 100-continue


HTTP/1.1 412 Precondition FailedDate: _date_Server: Quantum-Lattus/3.0.3-ba79602e871550668391a673fa57589cf531497e

If-Modified-Since This header is supported for PUT of an object, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.25 "If-Modified-Since". If this precondition is not satisfied, the response will be "412 Precondition Failed".



See also the Last-Modified response header.

If-None-Match This header is supported for PUT of an object, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.26 "If-None-Match".

See also ETag on page 141.

The subsections contain an example where the object should only be put if no object with the specified name exists, and in this case, the object already exists.


PUT /namespace/namespace1/object1?meta=http HTTP/1.1Host: my.example.com If-None-Match: * Content-Length: 19Expect: 100-continue


HTTP/1.1 412 Precondition FailedDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24

If-Unmodified-Since This header is supported for PUT of an object, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.28 "If-Unmodified-Since". If this precondition is not satisfied, the response will be "412 Precondition Failed".


Transfer-Encoding Please see RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", sections 3.6.1 "Chunked Transfer Coding" and 19.4.6 "Introduction of Transfer-Encoding" for more information. The subsections include an example.


PUT /namespace/namespace1/object1 HTTP/1.1



Host: my.example.comDate: _date_ Accept: text/plainContent-Type: binary/octet-streamTransfer-Encoding: chunked

1e50_7760_bytes_of_data_

a30_2608_bytes_of_data_

0


HTTP/1.1 201 CreatedDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: 0Last-Modified: _date_ETag: "8a6547849714fa7f021a923e391a87a8"X-Ampli-Name: "object1"X-Ampli-Id: "6a4f803b1efc48ad9a93ab5e1ee8380e"X-Ampli-Namespace-Id: "9777e0eef27c44b7929806a8c42f1684"X-Ampli-Kind: "File"X-Ampli-Policy-Id: "d684bbd656004c50a29db47b67ae9666"X-Ampli-Size: _object_length_X-Ampli-Crc32: 3827519356X-Ampli-Status: OKX-Ampli-Creation-Date: _epoch_time_X-Ampli-Modification-Date: _epoch_time_X-Ampli-Verification-Date: _epoch_time_


Chapter 7: Object OperationsResponse Headers

Response Headers

ETag

This header is returned in responses to GET, HEAD, PUT and POST requests for objects, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.19 "ETag".

An ETag uniquely identifies a specific state of an object. Whenever the content or custom metadata of an object is modified, the ETag changes with it. An ETag returned in a response to a request can be used in subsequent requests with an If-Match header or an If-None-Match header to conditionally retrieve or update an object.

Last-Modified

This header is returned in responses to GET, HEAD, PUT and POST requests for objects, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.29 "Last-Modified".

The Last-Modified value returned in a response is equal to the "Modification-Date" property of the object. Whenever the content or custom metadata of an object is modified, the "Modification-Date" property of the object is updated. A Last-Modified date value returned in a response to a request can be used in subsequent requests with an If-Modified-Since header or an If-Unmodified-Since header to conditionally retrieve or update an object.

An ETag is more accurate than the Last-Modified date, because the Last-Modified date is only accurate to the second.

Create Object with Generated Name

It is possible to create an object whose name is generated. The generated name will be the guid of the put object.

To do so, you have to add the X-Ampli-Generate-Name header to a PUT the request.


Chapter 7: Object OperationsUsing Curl to Upload a File

PUT /namespace/<namespace-name> HTTP/1.1...X-Ampli-Generate-Name: GUID...

Currently, only one generation method (GUID) is supported.

The client knows the generated name by looking at the X-Ampli-Name field in the response. The X-Ampli-Id field also contains the same name, however this is only the case for the GUID method so clients should always inspect the X-Ampli-Name field.

The response is a 201 Created, identical to what is shown above for a normal PUT/POST of an object.

Using Curl to Upload a File

$ curl --verbose --fail --upload-file /myfiles/test2.txt "http://myhost.amplidata.com:8080/namespace/Documents/myfiles/myfavoritefile1.txt"* About to connect() to myhost.amplidata.com port 8080 (#0)* Trying www.quantum.com... connected* Connected to myhost.amplidata.com (127.0.0.1) port 8080 (#0)....

The unique ID of the object is set to /Documents/myfiles/myfavoritefile1.txt in this example.

In the response message no other ID is returned as the ID required to receive the object's content is the object's name set during the PUT process.


Chapter 7: Object OperationsCreate an Object with Generated Name

Create an Object with Generated Name

It is possible to have Lattus generate the name automatically when storing a new object.

This is indicated by adding the X-Ampli-Generate-Name header to a PUT request with the namespace directory as base URL.

PUT /namespace/<namespace-name> HTTP/1.1...X-Ampli-Generate-Name: GUID...

Currently only one generation method (GUID) is supported. This generates a Globally Unique Identifier (128 bit) for the object which can be used as a name for that object in the namespace.

You can verify the generated name by checking the X-Ampli-Name field in the response message.

The X-Ampli-Id field contains the same name as the X-Ampli-Name field when using this automated GUID generation method. In case you are not using this automated method always inspect the X-Ampli-Name field.

The response output returns a "201 Created" message, identical to the standard PUT/POST request of an object.

Create Directory Object

To create a directory object in a subdirectory of a namespace, its parent directory must already have been created. Parent directories are not automatically created.

http REQUEST /namespace/<namespace name>/<directory object name>

PUT /namespace/namespace1/directory1?meta=http HTTP/1.1Host: my.example.com


Chapter 7: Object OperationsCreate Directory Object

X-Ampli-Kind: Directory

RESPONSE /namespace/<namespace name>/<directory object name>

HTTP/1.1 201 CreatedDAV: 1,3Date: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: 0Last-Modified: _date_ETag: "16ab1e9936f92a7494e417099937b4f8"X-Ampli-Id: "e45ee50dba0440259575bb04c4217fa3"X-Ampli-Name: "directory1"X-Ampli-Namespace-Id: "b42cf6efc9f94d3585372877828fc4e6"X-Ampli-Kind: "Directory"X-Ampli-Policy-Id: "034daa52fd22402f83be5a760b6a47cd"X-Ampli-Size: 0X-Ampli-Crc32: 4294967295X-Ampli-Status: "OK"X-Ampli-Creation-Date: _epoch_time_X-Ampli-Modification-Date: _epoch_time_X-Ampli-Verification-Date: _epoch_time_

json REQUEST /namespace/<namespace name>/<directory object name>

PUT /namespace/namespace1/directory1?meta=json HTTP/1.1Host: my.example.comX-Ampli-Kind: Directory


HTTP/1.1 201 CreatedDAV: 1,3Date: _date_


Chapter 7: Object OperationsCreate Directory Object

Server: Quantum-Lattus/3.0.3-ba79602e871550668391a673fa57589cf531497eContent-Type: application/json;charset=UTF-8Content-Length: _length_ Last-Modified: _date_ETag: "16ab1e9936f92a7494e417099937b4f8"

{"Id":"e45ee50dba0440259575bb04c4217fa3", "Name":"directory1","Namespace-Id":"b42cf6efc9f94d3585372877828fc4e6", "Kind":"Directory","Policy-Id":"034daa52fd22402f83be5a760b6a47cd", "Size":0,"Crc32":4294967295, "Status":"OK""Creation-Date":_epoch_time_ "Modification-Date":_epoch_time_ "Verification-Date":_epoch_time_}

xml REQUEST /namespace/<namespace name>/<directory object name>

PUT /namespace/namespace1/directory1?meta=xml HTTP/1.1Host: my.example.comX-Ampli-Kind: Directory


HTTP/1.1 201 CreatedDAV: 1,3Date: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8Content-Length: _length_Last-Modified: _date_ETag: "16ab1e9936f92a7494e417099937b4f8"<?xml version="1.0" encoding="UTF-8"?><Object xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>e45ee50dba0440259575bb04c4217fa3</Id>


Chapter 7: Object OperationsCustom Metadata

<Name>directory1</Name><Namespace-Id>b42cf6efc9f94d3585372877828fc4e6</Namespace-Id><Kind>Directory</Kind><Policy-Id>034daa52fd22402f83be5a760b6a47cd</Policy-Id><Size>0</Size><Crc32>4294967295</Crc32><Status>OK</Status><Creation-Date>_epoch_time_</Creation-Date><Modification-Date>_epoch_time_</Modification-Date><Verification-Date>_epoch_time_</Verification-Date></Object>

Curl Example curl -v -X MKCOL http://127.0.0.1/namespace/namespace1/a/b

Custom Metadata

Custom metadata can be associated with existing directory objects and file objects. All custom metadata field names have prefix "Custom-Meta-".

All custom metadata field names are case-insensitive and are converted automatically to lowercase (ASCII byte range 65..90 are converted to 97..122) for internal tracking, e.g "Id" can be referred to as "id", "Id", "iD", or "ID" in HTTP header names, in json object field names and in xml tags. Therefore, a custom metadata field name such as "Custom-Meta-User" will be stored as "user" and show up in the response message as "Custom-Meta-user".

Setting or modifying custom metadata updates the following two attributes of the object:

• the Lattus Modification-Date attribute.

• the Last-Modified and ETag headers.



Note: The recommended maximum size for object metadata is 1000 bytes.Custom metadata is not encrypted in the backend database, nor in the blockstores (tlogs).

Set Custom Metadata

http REQUEST /namespace/<namespace name>/<object name>?update=set

PUT /namespace/namespace1/object1?update=set&meta=http HTTP/1.1Host: my.example.comDate: _date_ Content-Length: 0X-Ampli-Custom-Meta-User: "jdoe"X-Ampli-Custom-Meta-Group: "users"

RESPONSE /namespace/<namespace name>/<object name>?update=set

HTTP/1.1 200 OKDAV: 1,3Date: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: 0Last-Modified: _date_ETag: "2191d1429936ea1aae67962a56110c83"X-Ampli-Id: "f5c104cba97b42139448c80c217f143c"X-Ampli-Name: "namespace1/object1"X-Ampli-Namespace-Id: "011d9d4d92cc456b91d69e4e441be915"X-Ampli-Kind: "File"X-Ampli-Policy-Id: "814d15ed2c4240b2b918c63514718c4c"X-Ampli-Size: 3X-Ampli-Crc32: 3827519356X-Ampli-Status: "OK"



X-Ampli-Creation-Date: _epoch_time_X-Ampli-Modification-Date: _new_epoch_time_X-Ampli-Verification-Date: _epoch_time_X-Ampli-Custom-Meta-group: "users"X-Ampli-Custom-Meta-user: "jdoe"

json REQUEST /namespace/<namespace name>/<object name>?update=set

PUT /namespace/namespace1/object1?update=set&meta=json HTTP/1.1Host: my.example.comDate: _date_Content-Length: _length_

{"Custom-Meta-User":"jdoe", "Custom-Meta-Group:"users"}


HTTP/1.1 200 OKDAV: 1,3Date: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: _length_Last-Modified: _date_ETag: "2191d1429936ea1aae67962a56110c83"{"Id":"f5c104cba97b42139448c80c217f143c","Name":"namespace1/object1","Namespace-Id":"011d9d4d92cc456b91d69e4e441be915","Kind":"File","Policy-Id":"814d15ed2c4240b2b918c63514718c4c","Size":3,"Crc32":3827519356,"Status":"OK","Creation-Date":_epoch_time_,



"Modification-Date":_new_epoch_time_,"Verification-Date":_epoch_time_,"Custom-Meta-group":"users","Custom-Meta-user":"jdoe"}

xml REQUEST /namespace/<namespace name>/<object name>?update=set

PUT /namespace/namespace1/object1?update=set&meta=xml HTTP/1.1Host: my.example.comDate: _date_Content-Length: _length_

<Object xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Custom-Meta-User>jdoe</Custom-Meta-User><Custom-Meta-Group>users</Custom-Meta-Group></Object>


HTTP/1.1 200 OKDAV: 1,3Date: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8Content-Length: _length_Last-Modified: _date_ETag: "2191d1429936ea1aae67962a56110c83"<Object xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>f5c104cba97b42139448c80c217f143c</Id><Name>namespace1/object1</Name><Namespace-Id>011d9d4d92cc456b91d69e4e441be915</Namespace-Id><Kind>File</Kind>



<Policy-Id/>814d15ed2c4240b2b918c63514718c4c</Policy-Id><Size>3</Size><Crc32>3827519356</Crc32><Status>OK</Status><Creation-Date>_epoch_time_</Creation-Date><Modification-Date>_new_epoch_time_</Modification-Date><Verification-Date>_epoch_time_</Verification-Date><Custom-Meta-group>users</Custom-Meta-group><Custom-Meta-user>jdoe</Custom-Meta-user></Object>

Apply Updates to Existing Custom Metadata

In the examples below, we start from an object that has the following custom metadata fields:

Custom-Meta-group: "users" Custom-Meta-user: "jdoe"

http REQUEST /namespace/<namespace name>/<object name>?update=apply

PUT /namespace/namespace1/object1?update=apply&meta=http HTTP/1.1Host: my.example.comDate: _date_ Content-Length: 0X-Ampli-Custom-Meta-User: "john" X-Ampli-Custom-Meta-Color: "red"

RESPONSE /namespace/<namespace name>/<object name>?update=apply

HTTP/1.1 200 OKDAV: 1,3Date: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: 0Last-Modified: _date_ETag: "8a6547849714fa7f021a923e391a87a8"



X-Ampli-Id: "f5c104cba97b42139448c80c217f143c"X-Ampli-Name: "namespace1/object1"X-Ampli-Namespace-Id: "011d9d4d92cc456b91d69e4e441be915"X-Ampli-Kind: "File"X-Ampli-Policy-Id: "814d15ed2c4240b2b918c63514718c4c"X-Ampli-Size: 3X-Ampli-Crc32: 3827519356X-Ampli-Status: "OK"X-Ampli-Creation-Date: _epoch_time_X-Ampli-Modification-Date: _new_epoch_time_X-Ampli-Verification-Date: _epoch_time_X-Ampli-Custom-Meta-color: "red"X-Ampli-Custom-Meta-group: "users"X-Ampli-Custom-Meta-user: "john"

json REQUEST /namespace/<namespace name>/<object name>?update=apply

PUT /namespace/namespace1/object1?update=apply&meta=json HTTP/1.1Host: my.example.comDate: _date_Content-Length: _length_

{"Custom-Meta-User":"john", "Custom-Meta-Color":"red"}


HTTP/1.1 200 OKDAV: 1,3Date: _date_Server: Quatnum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8Content-Length: 0Last-Modified: _date_ETag: "8a6547849714fa7f021a923e391a87a8"



{"Id":"f5c104cba97b42139448c80c217f143c","Name":"namespace1/object1","Namespace-Id:"011d9d4d92cc456b91d69e4e441be915","Kind":"File","Policy-Id":"814d15ed2c4240b2b918c63514718c4c","Size":3,"Crc32":3827519356,"Status":"OK","Creation-Date":_epoch_time_,"Modification-Date":_new_epoch_time_,"Verification-Date":_epoch_time_,"Custom-Meta-color":"red","Custom-Meta-group":"users","Custom-Meta-user":"john"}

xml REQUEST /namespace/<namespace name>/<object name>?update=apply

PUT /namespace/namespace1/object1?update=apply&meta=xml HTTP/1.1Host: my.example.comDate: _date_Content-Length: _length_

<Object xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Custom-Meta-User>john</Custom-Meta-User><Custom-Meta-Color>red</Custom-Meta-Color></Object>


HTTP/1.1 200 OKDAV: 1,3Date: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8


Chapter 7: Object OperationsList Objects

Content-Length: 0Last-Modified: _date_ETag: "8a6547849714fa7f021a923e391a87a8"<Object xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>f5c104cba97b42139448c80c217f143c</Id><Name>namespace1/object1</Name><Namespace-Id>011d9d4d92cc456b91d69e4e441be915</Id><Kind>File</Kind><Policy-Id>814d15ed2c4240b2b918c63514718c4c</Policy-Id><Size>3</Size><Crc32>3827519356</Crc32><Status>OK</Status><Creation-Date>_epoch_time_</Creation-Date><Modification-Date>_new_epoch_time_</Modification-Date><Verification-Date>_epoch_time_</Verification-Date><Custom-Meta-color>red</Custom-Meta-color><Custom-Meta-group>users</Custom-Meta-group><Custom-Meta-user>john</Custom-Meta-user></Object>

List Objects

A GET can be performed on /namespace/<namespace name>/<object name>.


List Objects

The media type of the list to be returned. By default, only the first 50 items are returned.

You can add one of the following parameters to the URL:



http / text REQUEST /namespace/<namespace name>/<directory object name>/

GET /namespace/namespace1/directory1/ HTTP/1.1Host: my.example.comDate: _date_ Accept: text/plain

or:GET /namespace/namespace1/directory1/?list=http HTTP/1.1Host: my.example.com

Parameter Explanation Example URL

"limit=X" Use this query parameter to adjust the limit of the items to X (maximum 1000).Note: Any value higher than

1000 will result in 1000 entries.

"/namespace/namespace1/directory1/?limit=500"

With this URL, the first 500 itemsare returned.

"marker=Y" Use this query parameter to define from which marker onward the list should start.

"/namespace/namespace1/directory1/?marker=objectc&limit=4&meta=xml"

With this URL, the first 4 objects will be returned in xml format, starting with "objectc".

"include_marker=false" Use this parameter, to exclude the marker from the list.

"/namespace/namespace1/directory1/?marker=objectc&limit=4&include_marker=false&meta=xml"

With this URL, the first 4 objects will be returned in xml format, start ing with the first object after "objectc".



Date: _date_

RESPONSE /namespace/<namespace name>/<directory object name>/

HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: text/plain;charset=UTF-8Content-Length: _length_"object1""object2"

json REQUEST /namespace/<namespace name>/<directory object name>/

GET /namespace/namespace1/directory1/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/json

or:GET /namespace/namespace1/directory1/?list=json HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/json;charset=UTF-8Content-Length: _length_["object1","object2"]



xml REQUEST /namespace/<namespace name>/<directory object name>/

GET /namespace/namespace1/directory1/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/xml

or:GET /namespace/namespace1/directory1/?list=xml HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Type: application/xml;charset=UTF-8Content-Length: _length_<Directory-Entriesxmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Name>object1</Name><Name>object2</Name></Directory-Entries>

Using Curl to List the Objects in a Namespace

curl --verbose --fail -H "Accept: application/xml" "http://127.0.0.1:8080/namespace/examplenamespace/"


Chapter 7: Object OperationsMove Object

Move Object

Moving an object, renames the object. This request is a limited implementation of the move command of the WebDAV specification of RFC 4918, section 9.9.

You can only rename objects inside a name space and when the object is a file. When moving a file to another directory, the destination directory must exist upfront.

You can not rename files, nor move them to another name space.

Prerequisites A user can only move objects with the proper permissions: DELETE, READ, and WRITE

Response /namespace/namespace1/source_object

HTTP/1.1 201 CreatedDate: _date_Server: Amplidata-AmpliStor/2.1.0-1f206c99fa6c1df2070e69741a6c0c6d52353ec9

Extra Information The Depth request header is optional. The only value allowed at this time is “infinity”.

The Destination request header is required and should specify the full destination path, including the new object name. The directory component of the destination path to move the object must exist. If it does not exist, the request fails with the HTTP response 409 Conflict.

The Overwrite request header is required and can have value T for true or F for false. At this moment T is not supported and results in the HTTP response 405 Method Not Allowed.


Chapter 7: Object OperationsGet Object Content

Get Object Content

Complete Content REQUEST /namespace/<namespace name>/<object name>

GET /namespace/namespace1/object1 HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Quantum-Lattus/2.1.0-1f206c99fa6c1df2070e69741a6c0c6d52353ec9Content-Type: text/plainContent-Length: _length_Last-Modified: _date_ETag: "c0af7bfb6616414d89fad0552274a722"X-Ampli-Id: "c0af7bfb6616414d89fad0552274a722"X-Ampli-Name: "namespace/namespace1/object1"X-Ampli-Namespace-Id: "9ec81b3cfd714655adb9a9fcf20d277c"X-Ampli-Policy-Id: "31784b34443440a89a0dab38563a5937"X-Ampli-Size: _length_X-Ampli-Crc32: 3827519356X-Ampli-Status: "OK"X-Ampli-Creation-Date: _epoch_time_X-Ampli-Modification-Date: _epoch_time_X-Ampli-Verification-Date: _epoch_time__length_bytes_of_object_content_

Request Headers Please note that the examples below also work for other meta types than "meta=http".

If-Match This header is supported for PUT of an object, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.24 "If-Match". If this



precondition is not satisfied, the response will be "412 Precondition Failed".

See also the ETag response header.

The subsections contain an example: the ETag we received from the server for object "/namespace/namespace1/object1" in a previous response to a range GET request was "19d0a529e39347f4823c6e22f1160825". We wish to GET the entire content of the object, but only if it has not been changed in the mean time. In this scenario, the object has been changed in the mean time, so the ETag no longer matches, and the server responds with a 412 error response.


GET /namespace/namespace1/object1?meta=http HTTP/1.1Host: my.example.comIf-Match: "19d0a529e39347f4823c6e22f1160825"


HTTP/1.1 412 Precondition FailedDate: _date_Server: Quantum-Lattus/3.0.3-ba79602e871550668391a673fa57589cf531497e

If-Modified-Since This header is supported for GET and HEAD of an object, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.25 "If-Modified-Since". If this precondition is not satisfied, the response will be "304 Not Modified".


If-None-Match This header is supported for GET and HEAD of an object, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.26 "If-None-Match". If this precondition is not satisfied, the response will be "304 Not Modified". See also #Last-Modified.

See also the ETag response header.



If-Unmodified-Since This header is supported for GET and HEAD of an object, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.28 "If-Unmodified-Since". If this precondition is not satisfied, the response will be "412 Precondition Failed". See also the Last-Modified response header.

Range The Range header is supported for GET of object content, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.35 "Range".

An example, for a file containing the text "AmplidataAmplistor":

REQUEST

GET /namespace/namespace1/object1 HTTP/1.1Host: my.example.comRange: bytes=10-14,5-9,0-4,15-,-10

REPLY

HTTP/1.1 206 Partial ContentAccept-Ranges: bytesContent-Length: 611Content-Type: multipart/byteranges;boundary=999200ad8891450f8f9aedb1245c7bde768e308403964649a8f16f1bf5e630a0Last-Modified: _date_ETag: "cbd1130013be42d2a9af04bbb2805905"X-Ampli-Id: "cbd1130013be42d2a9af04bbb2805905" X-Ampli-Name: "object1"X-Ampli-Namespace-Id: "2b130b116dc74308a6a40e148dda5e12" X-Ampli-Policy-Id: "b349b3b886b242229305827cfeb2b178"X-Ampli-Size: 19X-Ampli-Crc32: 3768369899X-Ampli-Status: "OK"X-Ampli-Creation-Date: _epoch_time_X-Ampli-Modification-Date: _epoch_time_ X-Ampli-Verification-Date: _epoch_time_



--999200ad8891450f8f9aedb1245c7bde768e308403964649a8f16f1bf5e630a0Content-Range: bytes 10-14/19

Ampli--999200ad8891450f8f9aedb1245c7bde768e308403964649a8f16f1bf5e630a0Content-Range: bytes 5-9/19

data--999200ad8891450f8f9aedb1245c7bde768e308403964649a8f16f1bf5e630a0Content-Range: bytes 0-4/19

Ampli--999200ad8891450f8f9aedb1245c7bde768e308403964649a8f16f1bf5e630a0Content-Range: bytes 15-18/19

Stor--999200ad8891450f8f9aedb1245c7bde768e308403964649a8f16f1bf5e630a0Content-Range: bytes 9-18/19

AmpliStor--999200ad8891450f8f9aedb1245c7bde768e308403964649a8f16f1bf5e630a0--

Response Headers ETag

This header is returned in responses to GET, HEAD, PUT and POST requests for objects, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.19 "ETag".


Chapter 7: Object OperationsGet Object Metadata

An ETag uniquely identifies a specific state of an object. Whenever the content or custom metadata of an object is modified, the ETag changes with it. An ETag returned in a response to a request can be used in subsequent requests with an If-Match header or an If-None-Match header to conditionally retrieve or update an object.

Last-Modified

This header is returned in responses to GET, HEAD, PUT and POST requests for objects, as specified in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1", section 14.29 "Last-Modified".

The Last-Modified value returned in a response is equal to the "Modification-Date" property of the object. Whenever the content or custom metadata of an object is modified, the "Modification-Date" property of the object is updated. A Last-Modified date value returned in a response to a request can be used in subsequent requests with an If-Modified-Since header or an If-Unmodified-Since header to conditionally retrieve or update an object.

An ETag is more accurate than the Last-Modified date, because the Last-Modified date is only accurate to the second.

Using Curl to Get an Object

curl --verbose --fail

"http://127.0.0.1:8080/namespace/examplenamespace/firstobject" >

/root/mydownloadedfile

Get Object Metadata

http / text REQUEST /namespace/<namespace name>/<object name>?meta=http

GET /namespace/namespace1/object1?meta=http HTTP/1.1Host: my.example.com



Date: _date_

RESPONSE /namespace/<namespace name>/<object name>?meta=http

HTTP/1.1 200 OKDAV: 1,3Date: __date__Server: Amplidata-AmpliStor/unknown-2bb0ba7170b333cd4e9f7f9d94d4fb8d450194b9Content-Type: text/plain;charset=UTF-8Content-Length: 0Last-Modified: __date__ETag: "4059eea7457257b679aae2e2e6c55aff"X-Ampli-Id: "38e618a671334338be2b060f414fafee"X-Ampli-Name: "object1"X-Ampli-Namespace-Id: "830a1890c94e46bfa86d82e4cf42123f"X-Ampli-Kind: "File"X-Ampli-Policy-Id: "cfd345302fae4bc6824f15fc6cfd603b"X-Ampli-Size: 4229X-Ampli-Crc32: 3552406836X-Ampli-Status: "OK"X-Ampli-Creation-Date: 1329472463X-Ampli-Modification-Date: 1329472463X-Ampli-Verification-Date: 1329472463

json REQUEST /namespace/<namespace name>?meta=json

GET /namespace/VOL HTTP/1.1Host: my.example.comDate: _date_

RESPONSE /namespace/<namespace name>?meta=json

HTTP/1.1 200 OKDAV: 1,3Date: __date__Server: Amplidata-AmpliStor/unknown-2bb0ba7170b333cd4e9f7f9d94d4fb8d450194b9Content-Type: application/json;charset=UTF-8



Content-Length: 323Last-Modified: __date__ETag: "4059eea7457257b679aae2e2e6c55aff"{"Id":"38e618a671334338be2b060f414fafee","Name":"object1","Namespace-Id":"830a1890c94e46bfa86d82e4cf42123f","Kind":"File","Policy-Id":"cfd345302fae4bc6824f15fc6cfd603b","Size":4229,"Crc32":3552406836,"Status":"OK","Creation-Date":1329472463,"Modification-Date":1329472463,"Verification-Date":1329472463}

xml REQUEST /namespace/<namespace name>/<object name>?meta=xml

GET /namespace/namespace1/object1?meta=xml HTTP/1.1Host: my.example.comDate: _date_

RESPONSE /namespace/<namespace name>/<object name>?meta=xml

HTTP/1.1 200 OKDAV: 1,3Date: __date__Server: Amplidata-AmpliStor/unknown-2bb0ba7170b333cd4e9f7f9d94d4fb8d450194b9Content-Type: application/xml;charset=UTF-8Content-Length: 538Last-Modified: __date__ETag: "4059eea7457257b679aae2e2e6c55aff"<?xml version="1.0" encoding="UTF-8"?><Object xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Id>38e618a671334338be2b060f414fafee</Id><Name>object1</Name>


Chapter 7: Object OperationsDelete Object

<Namespace-Id>830a1890c94e46bfa86d82e4cf42123f</Namespace-Id><Kind>File</Kind><Policy-Id>cfd345302fae4bc6824f15fc6cfd603b</Policy-Id><Size>4229</Size><Crc32>3552406836</Crc32><Status>OK</Status><Creation-Date>1329472463</Creation-Date><Modification-Date>1329472463</Modification-Date><Verification-Date>1329472463</Verification-Date></Object>

Delete Object


DELETE /namespace/namespace1/object1 HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 204 No ContentDate: _date_Server: Quantum-Lattus/3.0.0-1526584800240429ab180380ccfa8aa6ceda7b24Content-Length: 0

Using Curl to Delete an Object

curl -f -v -X DELETE http://127.0.0.1:8080/namespace/test-4-1/test.txt

User Management Operations


Chapter 7: Object OperationsAdding A user

Adding A user

To add a new user, the following fields must be specified:

• name: the user name. Needs to be unique

• secret: the password

The following fields are optional:

• Default-policy: the default policy to use when this user creates a bucket

• Default-small-files-policy: the default small files policy to use when this user creates a bucket

• Default-small-files-threshold: the default threshold to use to select either the regular or the small files policy when this user creates a bucket

Default-small-files-policy and default-small-files-threshold need to be specified together: either both or not at all. If an optional field is not specified, it is set to None.

http / text REQUEST /manage/user

POST /manage/user HTTP/1.1Host: my.example.comX-Ampli-Name: "my_user_name"X-Ampli-Secret: "my_secret"X-Ampli-Default-Policy: "cadb732d2fdd4486be9b05dbe4b51d14"X-Ampli-Default-Small-Files-Policy: "66da5c8f321d4dd28e3da264c42d04b7"X-Ampli-Default-Small-Files-Threshold: 25000Accept:plain/text

or:POST /manage/user?meta=http HTTP/1.1Host: my.example.comX-Ampli-Name: "my_user-name"X-Ampli-Secret: "my_secret"



X-Ampli-Default-Policy: "cadb732d2fdd4486be9b05dbe4b51d14"X-Ampli-Default-Small-Files-Policy: "66da5c8f321d4dd28e3da264c42d04b7"X-Ampli-Default-Small-Files-Threshold: 25000

RESPONSE /manage/user

HTTP/1.1 201 CreatedDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Type: text/plain;charset=UTF-8Content-Length: 0X-Ampli-Name: "my_user_name"X-Ampli-Id: "36"X-Ampli-Status: "ACTIVE"X-Ampli-Seq: 0X-Ampli-Num-S3-Namespaces: 0X-Ampli-Default-Policy: "cadb732d2fdd4486be9b05dbe4b51d14"X-Ampli-Default-Small-Files-Policy: "66da5c8f321d4dd28e3da264c42d04b7"X-Ampli-Default-Small-Files-Threshold: 25000

json REQUEST /manage/user

POST /manage/user HTTP/1.1Host: my.example.comContent-Type: application/jsonAccept: application/jsonContent-Length: 205{"Name":"my_user_name","Secret":"my_secret","Default-Policy":"cadb732d2fdd4486be9b05dbe4b51d14","Default-Small-Files-Policy":"66da5c8f321d4dd28e3da264c42d04b7","Default-Small-Files-Threshold":10000}

or:POST /manage/user?meta=json HTTP/1.1



Host: my.example.comContent-Length: 205{"Name":"my_user_name","Secret":"my_secret","Default-Policy":"cadb732d2fdd4486be9b05dbe4b51d14","Default-Small-Files-Policy":"66da5c8f321d4dd28e3da264c42d04b7","Default-Small-Files-Threshold":10000}

RESPONSE /manage/user

HTTP/1.1 201 CreatedDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Type: application/json;charset=UTF-8Content-Length: 245{"Name":"my_user_name","Id":"41","Status":"ACTIVE","Seq":0,"Num-S3-Namespaces":0,"Default-Policy":"cadb732d2fdd4486be9b05dbe4b51d14","Default-Small-Files-Policy":"66da5c8f321d4dd28e3da264c42d04b7","Default-Small-Files-Threshold":10000}

xml REQUEST /manage/user

POST /manage/user HTTP/1.1Host: my.example.comContent-Type: application/xmlAccept: application/xmlContent-Length: 410<?xml version="1.0" encoding="UTF-8"?><User xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Name>my_user_name</Name>



<Secret>my_secret</Secret><Default-Policy>cadb732d2fdd4486be9b05dbe4b51d14</Default-Policy><Default-Small-Files-Policy>66da5c8f321d4dd28e3da264c42d04b7</Default-Small-Files-Policy><Default-Small-Files-Threshold>50000</Default-Small-Files-Threshold></User>

or:POST /manage/user?meta=xml HTTP/1.1Host: my.example.comContent-Length: 410<?xml version="1.0" encoding="UTF-8"?><User xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Name>my_user_name</Name><Secret>my_secret</Secret><Default-Policy>cadb732d2fdd4486be9b05dbe4b51d14</Default-Policy><Default-Small-Files-Policy>66da5c8f321d4dd28e3da264c42d04b7</Default-Small-Files-Policy><Default-Small-Files-Threshold>50000</Default-Small-Files-Threshold></User>RESPONSE /manage/userHTTP/1.1 201 CreatedDate: _date_Server: Amplidata-AmpliStor/_revision-Content-Type: application/xml;charset=UTF-8Content-Length: 403<?xml version="1.0" encoding="UTF-8"?><User><Name>my_user_name</Name><Id>47</Id><Status>ACTIVE</Status><Seq>0</Seq><Num-S3-Namespaces>0</Num-S3-Namespaces><Default-Policy>cadb732d2fdd4486be9b05dbe4b51d14</Default-Policy>


Chapter 7: Object OperationsList Users

<Default-Small-Files-Policy>66da5c8f321d4dd28e3da264c42d04b7</Default-Small-Files-Policy><Default-Small-Files-Threshold>50000</Default-Small-Files-Threshold></User>

List Users

By default, only the first 50 users are returned. This limit can be adjusted to X by appending the query parameter "limit=X" to the URL. The maximum value for this limit is 1000, higher values will result in at most 1000 entries.

For example, the following URL could be used: /manage/user?limit=500.

In addition, the user from which the list should start can be specified using "marker=Y".

For example, the following URL could be used to list 4 users starting from user "test": /manage/user?marker=test&limit=4.

An additional query parameter, "include_marker=false" can be added, which results in the marker being not included in the list.

http / text REQUEST /manage/user/

GET /manage/user/ HTTP/1.1Host: my.example.comDate: _date_Accept: text/plain

or:GET /manage/user/?list=http HTTP/1.1Host: my.example.comDate: _date_


Chapter 7: Object OperationsList Users

RESPONSE /manage/user/

HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Type: text/plain;charset=UTF-8Content-Length: _length_"fred""erik"

json REQUEST /manage/user/GET /manage/user/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/json

or:GET /manage/user/?list=json HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Type: application/json;charset=UTF-8Content-Length: _length_["fred","erik"]

xml REQUEST /manage/user/

GET /manage/user/ HTTP/1.1Host: my.example.comDate: _date_Accept: application/xml


Chapter 7: Object OperationsGET User

or:GET /manage/user/?list=xml HTTP/1.1Host: my.example.comDate: _date_


HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Type: application/xml;charset=UTF-8Content-Length: _length_<Users xmlns="http://www.amplidata.com/amplistor/api/rest/1/amplistor.xsd"><Name>fred</Name><Name>erik</Name></Users>

GET User

A GET can be performed on /manage/user directly, on /manage/user/<username> or on /manage/user_by_id/<id>. The first returns a list of all users names, the second the details of a specific user, identified by his user name, and the third the details of a specific user, identified by his user id.

Listing by id is not supported.

If a user has certain optional fields which are currently set to None, they will be absent from the http/json/xml output.

A HEAD request yields the same response as a GET request, except that the server does not return a message-body in the response

http / text REQUEST /manage/user/<username>

GET /manage/user/my_user_name HTTP/1.1Host: my.example.com



Accept: text/plain

or:GET /manage/user/my_user_name?meta=http HTTP/1.1Host: my.example.com

RESPONSE /manage/user/<username>

HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Type: text/plain;charset=UTF-8Content-Length: 0X-Ampli-Name: "my_user_name"X-Ampli-Id: "48"X-Ampli-Status: "ACTIVE"X-Ampli-Seq: 0X-Ampli-Num-S3-Namespaces: 0X-Ampli-Default-Policy: "cadb732d2fdd4486be9b05dbe4b51d14"X-Ampli-Default-Small-Files-Policy: "66da5c8f321d4dd28e3da264c42d04b7"X-Ampli-Default-Small-Files-Threshold: 50000

json REQUEST /manage/user/<username>

GET /manage/user/my_user_name HTTP/1.1Host: my.example.comAccept: application/json

or:GET /manage/user/my_user_name?meta=json HTTP/1.1Host: my.example.com


HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Type: application/json;charset=UTF-8Content-Length: 245{



"Name":"my_user_name","Id":"48","Status":"ACTIVE","Seq":0,"Num-S3-Namespaces":0,"Default-Policy":"cadb732d2fdd4486be9b05dbe4b51d14","Default-Small-Files-Policy":"66da5c8f321d4dd28e3da264c42d04b7","Default-Small-Files-Threshold":50000}

xml REQUEST /manage/user/<username>

GET /manage/user/my_user_name HTTP/1.1Host: my.example.comAccept: application/xml

or:GET /manage/user/my_user_name?meta=xml HTTP/1.1Host: my.example.com


HTTP/1.1 200 OKDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Type: application/xml;charset=UTF-8Content-Length: 402<?xml version="1.0" encoding="UTF-8"?><User><Name>my_user_name</Name><Id>48</Id><Status>ACTIVE</Status><Seq>0</Seq><Num-S3-Namespaces>0</Num-S3-Namespaces><Default-Policy>cadb732d2fdd4486be9b05dbe4b51d14</Default-Policy><Default-Small-Files-Policy>66da5c8f321d4dd28e3da264c42d04b7</Default-Small-Files-Policy>


Chapter 7: Object OperationsDelete A User

<Default-Small-Files-Threshold>50000</Default-Small-Files-Threshold></User>

Delete A User

The user to delete can be identified by his user name ( URL /manage/user/<user name> ) or his id ( URL /manage/user_by_id/<user id> ).

REQUEST /manage/user/<username>

DELETE /manage/user/my_user_name HTTP/1.1Host: my.example.com


HTTP/1.1 204 No ContentDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Length: 0orREQUEST /manage/user/<username>DELETE /manage/user_by_id/my_id HTTP/1.1Host: my.example.comRESPONSE /manage/user/<username>HTTP/1.1 204 No ContentDate: _date_Server: Amplidata-AmpliStor/_revision_Content-Length: 0


Chapter 7: Object OperationsDelete A User


Chapter 8Authentication

Purpose

Lattus implements a security model in which an identified and authenticated user has certain user rights (permissions) on a namespace.

The different user rights are:

• Read

• Create

• Update

• Delete

• List


Chapter 8: AuthenticationHow It Works

How It Works

Using the Lattus command line interface, the storage administrator does the following:

• creates users with their credentials (password)

• assigns these users and their respective rights to a specific namespace

When the application, communicating over REST authenticates to the client daemon, it supplies the user and their credentials and a such access to the namespace is controlled.

The security rights are granted on a namespace basis, not on the objects that belong to this namespace

REST Authentication

Authentication is implemented by means of HTTP Digest authentication, as specified in RFC2617

This is how digest authentication is used:

• The request URL is inspected and determined for which namespace the request is. The namespace metadata is inspected to find the relevant credentials. Special care is taken to also map the “upload-form” to the correct namespace secrets.

• A 401 response is sent back.

• The client re-sends the request with an Authentication header.

• The server checks the content of that header against the relevant credentials.


Chapter 8: AuthenticationAPI Description

API Description

Users Users are identified by their unique user id. This int64 value is automatically assigned when creating a new user and cannot be changed. Each user id is linked to a user name (which must also be unique, deployment-wide), a password and a status, which can be ACTIVE, INACTIVE or DELETED.

User id 0 corresponds to a default user with user name “everyone”, which has no password. It corresponds to the unauthenticated user. Setting permissions on a path for this user allows unauthenticated users to perform operations.

Paths and Flags The previuosly mentioned permissions can be assigned, for each individual user, to a number of specific paths:

• root "/"

• manage "/manage"

• namespace root "/namespace"

• a namespace "/namespace/<namespace name>"

In addition to user permissions, the security system allows to enable certain global (i.e. not user specific) flags on each of these paths. These flags are:

• INHERIT: enabling the inherit flag on a path means that the permissions defined on this path are combined with the permission of the parent path.

Example:

If a user has permission READ on /namespace/VOL and permission CREATE on /namespace, this user will, if the INHERIT flag on path /namespace/VOL is not set, not be allowed to create new objects in /namespace/VOL. If the INHERIT flag is enabled on /namespace/VOL, this user will obtain effective rights of READ, CREATE on /namespace/VOL.

INHERIT also works in cascade, e.g. if inherit is enabled on /, /namespace and /namespace/VOL, then the effective permissions on /namespace/



VOL are those combined over all three paths. The cascade works top -> down and stops at the first level where the INHERIT flag is not set.

Q-Shell Interface for User Management

Add a New User with Username, Credentials and Status

The status can be ACTIVE (default) or INACTIVE)In [7]: q.dss.manage.addUser(?Definition: q.dss.manage.addUser(self, userName, credentials, status='ACTIVE', nodeIP='127.0.0.1', port=23510)Documentation:

Add a user.Parameters:- userName: string The username of the user.- credentials: string The secret keyword of the user.- status: string The user status, one of: ACTIVE, INACTIVE.- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact.

Get User Information by Referring to its Username

Setting showDeleted to True will also show the user if it has status DELETED:In [8]: q.dss.manage.showUser(?Definition: q.dss.manage.showUser(self, userName, showDeleted=False, nodeIP='127.0.0.1', port=23510)Documentation:

Show a user by name.Parameters:- userName: string The username of the user.- showDeleted: bool Show deleted users or not.- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact.



Getting User Information by Referring to its ID

In [9]: q.dss.manage.showUserById(?Definition: q.dss.manage.showUserById(self, userId, showDeleted=False, nodeIP='127.0.0.1', port=23510)Documentation:

Show a user by id.

Parameters:

- userId: int The userid of the user.- showDeleted: bool Show deleted users or not.- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact.

List All Users

Setting showDeleted to True will include users that have status DELETED:In [10]: q.dss.manage.listUsers(?Definition: q.dss.manage.listUsers(self, showDeleted=False, nodeIP='127.0.0.1', port=23510)Documentation:

List all users.

Parameters:

- showDeleted: bool Show deleted users or not.- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact.

Modifying User Information with a UserName

In [11]: q.dss.manage.changeUser(?



Definition: q.dss.manage.changeUser(self, userName, credentials=None, status=None, nodeIP='127.0.0.1', port=23510)Documentation:

Change a user.

Parameters:

- userName: string The username of the user.- credentials: string The new secret of the user, None for no change.- status: string The new status of the user, one of: ACTIVE, INACTIVE, None for no change.- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact.

Modify User Information by Referring to its ID

In [13]: q.dss.manage.changeUserById(?

Definition: q.dss.manage.changeUserById(self, userId, userName=None, credentials=None, status=None, nodeIP='127.0.0.1', port=23510) Documentation:

Change a user by id.

Parameters:

- userId: int The userId of the user.- userName: string The new userName of the user, None for no change.- credentials: string The new secret of the user, None for no change.- status: string The new status of the user, one of: ACTIVE, INACTIVE, None for no change.- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact.



Deleting a User

In [14]: q.dss.manage.deleteUser(?Definition: q.dss.manage.deleteUser(self, userName, nodeIP='127.0.0.1', port=23510)Documentation:

Delete a user.

Parameters:

- userName: string The new userName of the user, None for no change.- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact.

Q-shell Interface for Assignment of Rights to a Specific Path

Get the Permission Flags for a Specific Path

In [15]: q.dss.manage.showPermissionFlags(?Definition: q.dss.manage.showPermissionFlags(self, path, nodeIP='127.0.0.1', port=23510)Documentation:

Show the permission flags for a specified path.

Parameters:

- path: string The path for which to show the permission flags: "/", "/manage", "/namespace", or "/namespace/<namespace-name>".- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact. Returns: permission flags



Setting the Permission Flags for a Specific Path

In [21]: q.dss.manage.setPermissionFlags(?

Definition: q.dss.manage.setPermissionFlags(self, path, permissionFlags, nodeIP='127.0.0.1', port=23510)Documentation:

Set the permission flags for a specified path.

Parameters:

- path: string The path for which to set the permission flags: "/", "/manage", "/namespace", or "/namespace/<namespace-name>".- permissionFlags: [string] A list of permission flags. Currently allowed values: "Inherit".- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact. Returns: permission flags

Get the permissions for a specific user on a path

In [18]: q.dss.manage.showPermissions(?

Definition: q.dss.manage.showPermissions(self, path, userName, nodeIP='127.0.0.1', port=23510)Documentation:

Show the user access permissions of a user for a specified path.

Parameters:

- path: string The path for which to show the permissions: "/", "/manage", "/namespace", or "/namespace/<namespace-name>".



- userName: string The username of the user, or "everyone" for the permissions for special user "everyone".- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact. Returns: permissions

Set the permissions for a specific user on a path

In [20]: q.dss.manage.setPermissions(?Definition: q.dss.manage.setPermissions(self, path, userName, permissions, nodeIP='127.0.0.1', port=23510)Documentation:

Set the user access permissions of a user for a specified path.

Parameters:

- path: string The path for which to set the permissions: "/", "/manage", "/namespace", or "/namespace/<namespace-name>".- userName: string The username of the user, or "everyone" for the permissions for special user "everyone".- permissions: [string] A list of permissions. Currently allowed values: "Read", "Create", "Delete", "List", "Update".- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact. Returns: permissions

Getting the permission-settings for a path (combines flags, permissions, paths)

q.dss.manage.showPermissionSettings(self, path, nodeIP='127.0.0.1', port=23510) Documentation:

Show the permission settings for a specified path.


Chapter 8: AuthenticationREST Authentication in Detail

Parameters:

- path: string The path for which to show the permission settings: "/", "/manage", "/namespace", or "/namespace/<namespace-name>".- nodeIP: string The IP of the Storage Daemon to contact.- port: int The port of the Storage Daemon to contact. Returns: permission flags

REST Authentication in Detail

WWW-Authenticate header

This is the header the server sends back to the client in the 401.

An example:WWW-Authenticate: Digestrealm="Amplidata-AmpliStor/unknown-59b0f5a6e4709f566a5e8464153401ff76964cb9",qop=auth, nonce="246a5bf426a77dc28c195ce0373a9d2c",opaque="7e81b3abf2a84373b087cec73aca62c1"

It contains the following parameters:

• realm: an unique string identifying the server

• qop: this tells the client that we want digest authentication

• nonce: a hex string that is calculated as a hash (see the RFC)

• opaque: another hex string chosen at random at the start of the session

Authorization header This is the header the client sends in the subsequent re-send of the request. An example:

WWW-Authenticate: Digest


Chapter 8: AuthenticationREST Authentication in Detail

realm="Amplidata-AmpliStor/unknown-59b0f5a6e4709f566a5e8464153401ff76964cb9", qop=auth, nonce="246a5bf426a77dc28c195ce0373a9d2c",opaque="7e81b3abf2a84373b087cec73aca62c1"

It contains the following parameters:

• username: user name, used for validating the response

• realm: has to be identical as in WWW-Authenticate

• nonce: has to be identical as in WWW-Authenticate

• uri: has to be the request URI

• qop: has to be auth

• nc: 8 digit hex string counting up for each new request within the authorization session, this has to be 1 in the first and incremented each time

• cnonce: a random 8 byte string chosen by the client

• response: The client calculates the response by means of a formula based on request-method uri nonce username secret realm qop nc and cnonce. As all these fields are also available the server can afterwards do the same calculation and see if it matches.

• opaque: has to be identical as in WWW-Authenticate

The server will change the nonce at times during the session, because a nonce has a limited lifetime. The server sends a STALE flag to the client to indicate that the nonce is stale.

Curl Examples Get

curl --digest -u user:rw_1 -v http://localhost:8080/namespace/VOL/whatever.bin -o/tmp/1.bin

Put

curl --digest -u user:rw_1 -v http://localhost:8080/namespace/VOL/1.bin--upload-file /tmp/1.bin


Chapter 8: AuthenticationGetting Started with Authentication

Both commands assume that a user rw_1 with write rights on namespaceVOL was created.

Getting Started with Authentication

Out of the Box Authentication

When freshly installed, following settings are active:

• no users are created, only the "everyone" account is present

• unauthenticated users can LIST on /

• unauthenticated users have no rights on /manage

• unauthenticated users have all (update, read, create, list, delete) rights on /namespace

Before you Start Prevent an Unauthenticated User From Having Rights on all Namespaces

q.dss.manage.setPermissions('/namespace','everyone',[])

Allowing an Admin User to Create Namespaces and Polcies

Create a User to Administer Namespaces and Policies

q.dss.manage.addUser('admin','pass4admin')

Give the Admin User Rights on /manage

q.dss.manage.setPermissions("/manage","admin",['CREATE','LIST','READ','UPDATE','DELETE'])

The admin user will have no rights on the /namespace path and as such no rights on namespaces underneath.


Chapter 8: AuthenticationGetting Started with Authentication

Add a Namespace

You can do this over REST using the admin user or using the q-shell. Let's assume, for the remainder of this guide, that you created a namespace called mynamespace.

Create a User that Will Have All Rights on This Namespace

q.dss.manage.addUser('namespaceadmin','pass4namespaceadmin')

Set the Permissions for the Namespaceadmin User

q.dss.manage.setPermissions("/namespace/mynamespace","namespaceadmin",['CREATE',' LIST','READ','DELETE','UPDATE'])

Allowing Another User to Only Read Data from a Specific Namespace

Create a User That Only Has Read Rights on This Namespace

q.dss.manage.addUser('namespacereader','pass4namespacereader')

Set the Permissions for the Namespacereader User

q.dss.manage.setPermissions("/namespace/mynamespace","namespacereader",['READ'])

Inheriting Permissions Permissions can be set only at specific levels. Those levels are:

• /

• /namespace

• /manage

• /namespace/mynamespace (where “mynamespace” is your own self-created namespace)

The inherit flag dictates that the level where it is set "inherits" all permissions that are set at the level above it.

For example:

• You have three users, one of which is the "everyone" user.


Chapter 8: AuthenticationLimitations

• The users have permissions at "/namespace".

• You set the inherit flag at /namespace/mynamespace.

In this case the permissions set at /namespace will also apply to /namespace/mynamespace.

In [7]: q.dss.manage.showPermissionSettings('/namespace/mynamespace')

Out[7]: {u'Flags': [u'INHERIT'], u'User-Permissions': {}}

Tips and Tricks If you're using a browser to validate authentication, you can switch between users (and force re-authentication) by specifying the username in the URL.

Example: http://[email protected]:8080/manage/namespace/

The logfile of the clientdaemon is your first resource for troubleshooting authentication issues.

The value of both the username and the password of that user can be up to 256 complex characters (A-Z, a-z, 0-9 and/or special characters like #, @, !, etc.).

Limitations

• At most 5,000,000 users can be created (as many as the number of supported namespaces). This limitation applies only if you are upgrading from a clean installation of Lattus 3.2.0 or higher. If you upgraded from Lattus 3.1.x or earlier, there is a limit of 5000 users.

Note: With the five million user limitation, a new directory structure for the blockstores is introduced. After the upgrade to 3.4.0 the new directory structure is in place, the existing data is automatically moved into the new structure. This process is periodic and is spread over 180 days.



• The maximum size of the username and credentials is 256 bytes

• We recommend limiting the number of users that have rights on a single namespace to 64

• Authentication is currently not implemented in the Microsoft .net interface


Chapter 9Return Codes and

Troubleshooting

Return Codes

Lattus follows the standard HTTP/1.1 status code definitions. The following table describes a list of potential status codes:

TypeHTTP Description Message HTTP/1.1 Status Code Description

SUCCESS Continue 100 The client SHOULD continue with its request.

SUCCESS OK 200 The request was successfully executed.

SUCCESS Object Created 201 The request was successfully executed and the object is created.

REDIRECTION Not Modified 304 The precondition is not satisfied.


Chapter 9: Return Codes and TroubleshootingReturn Codes

ERROR Bad Request 400 The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.

ERROR Unauthorized 401 The request failed because of lack of permissions.

ERROR Forbidden 403 The Server understood the request, but refuses to fulfill it.

ERROR Object Not Found 404 The server has not found anything matching the Request-URI.

ERROR Conflict 409 Operation aborted because of a conflict.

ERROR Precondition Failed 412 The precondition given in the request header field evaluated to false when tested on the server.

ERROR Request Entity Too Large

413 The requested data (object, customer data, etc.) is larger than the server is able to process.

ERROR Metastore <ID> is full 500 The server encountered an internal error.




ERROR Internal Server Error 501 The server encountered an internal error.

FAILURE HTTP Version Not Supported

505 The server does not support the HTTP protocol version that was used in the request message.

ERROR Insufficient Storage 507 No more free space on the blockstores.



Chapter 10Microsoft .net Bindings

Note: There is currently no support for authentication when using the .net bindings

AmplistorClientLib Class Documentation

AmplistorClientLib.Types.AmpliObject Class Reference - Public Attributes

• string Id

• string Name• string NamespaceId• ObjectKind Kind • string PolicyId• long Size • long Crc32• string Status• long CreationDate • long ModificationDate• long VerificationDate


Chapter 10: Microsoft .net BindingsAmplistorClientLib Member Function Documentation

AmplistorClientLib.Types.AmpliPolicy Class Reference - Public Attributes

• const string REPAIR_SPREAD = "RepairSpread" • const string DYNAMIC_SAFETY = "DynamicSafety"• string Id• long SpreadWidth• bool SpreadWidthSpecified• long Safety• bool SafetySpecified • string HierarchyRules • long MaxSuperblockSize• bool MaxSuperblockSizeSpecified• string SafetyStrategy• long NMessages• bool NMessagesSpecified

AmplistorClientLib Member Function Documentation

Policy Operations Policy Creation

AmpliPolicy AmplistorClientLib.AmplistorClient.CreatePolicy (AmpliPolicy policyConfiguration)

Creates a new policy in the Lattus StoragePool.

Parameters:

• policyConfiguration: Configuration for the new policy. All fields of the configuration object are optional.

Returns:

The full policy configuration as stored during the creation.



Getting Policy Information

AmpliPolicy AmplistorClientLib.AmplistorClient.GetPolicy (string id)

Gets the policy configuration with id id from the LattusStoragePool. Parameters:

• id: id of the desired namespace.

Returns:

The full policy configuration as stored during the creation.

Listing Policies

IList<string> AmplistorClientLib.AmplistorClient.ListPolicies ()

Gets the policies on the LattusStoragePool.

Returns:

A list of policies on the Lattus StoragePool.

Namespace Operations Namespace Creation

AmpliNamespace AmplistorClientLib.AmplistorClient.CreateNamespace (AmpliNamespace namespaceConfig)

Creates a new namespace in the Lattus StoragePool.

Parameters:

• namespaceConfig: Configuration for the new namespace. Except for the Name & PolicyId fields, all fields of the configuration object are optional.

Returns:

The full namespace configuration as stored during the creation.



Listing Namespaces

IList<string> AmplistorClientLib.AmplistorClient.ListNamespaces ()

Gets the namespaces on the LattusStoragePool.

Returns:

A list of namespaces on the Lattus.

Getting Namespace Information

AmpliNamespace AmplistorClientLib.AmplistorClient.GetNamespace (string name)

Gets the namespace configuration with name name from the LattusStoragePool.

Parameters:

Name: name of the desired namespace.

Returns:

The full namespace configuration as stored during the creation.

Deleting Namespaces

void AmplistorClientLib.AmplistorClient.DeleteNamespace (string name)

Removes a complete namespace and its content from the LattusStoragePool.

Parameters:

Name: name of the namespace to be removed.



Object Operations Streambased Operations

Getting an object

Stream AmplistorClientLib.AmplistorClient.GetObject (string ampliNamespace, IList< string > pathEntries, string objectName)

Gets a read-only stream to a stored object in the Lattusin the specified namespace in the specified directory.

Parameters:

• ampliNamespace: Lattus namespace in which the specified directory needs to be located.

• pathEntries: The different sub-directory entries concatendated as a full path to the desired directory.

• objectName: The name of the saved object.

Returns:

A read-only stream to the saved object.

Uploading an Object

AmpliObject AmplistorClientLib.AmplistorClient.StoreObject (Stream sourceStream, string ampliNamespace, string ampliFilename)

Creates or updates an object in the LattusStoragePool using a stream.

Parameters:

• sourceStream: Stream to upload to the Lattus StoragePool.

• ampliNamespace: Lattus namespace in which the streamed object will be stored.

• ampliFilename: Lattus object name

Returns:

Detailed information about the stored stream.



Non-Streambased Operations

Uploading an Object

AmpliObject AmplistorClientLib.AmplistorClient.StoreObject (string sourceFilename, string ampliNamespace, string ampliFilename)

Creates or updates an object in the Lattus StoragePool using the locally accessible sourceFilename.

Parameters:

• sourceFilename: Local file to store on the Lattus StoragePool.

• ampliNamespace: Lattus namespace in which the file will be stored.

• ampliFilename: Lattus object name

Returns:

Detailed information about the stored object.

Creating a Directory

AmpliObject AmplistorClientLib.AmplistorClient.CreateDirectory (string ampliNamespace, IList< string > pathEntries)

Creates the specified directory path on the Lattus StoragePool in the specified namespace. Parameters:



Returns:

Detailed information about the created directory.

Deleting an object

void AmplistorClientLib.AmplistorClient.DeleteObject (string ampliNamespace, IList< string > pathEntries, string objectName)

Removes a stored object with name objectNme from the LattusStoragePool in the specified namespace in the specified directory.



Parameters:




Downloading an object

void AmplistorClientLib.AmplistorClient.DownloadObject (string ampliNamespace, IList< string > pathEntries, string objectName, string targetFilename)

Downloads a stored object from the Lattus in the specified namespace in the specified directory to a targetFilename.

Parameters:




• targetFilename: The local accessible path to the file to which the object needs to be saved.

Listing the Contents of a Directory

IList<string> AmplistorClientLib.AmplistorClient.ListObjects (string ampliNamespace)

Convenience method to get up to 50 object names in the root directory of the specified namespace.

Parameters:

AmpliNamespace: Lattus namespace in which the root directory needs to be located.



Returns:

A list of directory entries.IList<string> AmplistorClientLib.AmplistorClient.ListObjects (string ampliNamespace, IList< string > pathEntries, int? limit, string marker)

Gets the first limit object names found after marker in the specified directory of the specified namespace.

Parameters:



• limit: A value between 1 and 1000 indicating the number of items to fetch. If null is passed, the limit parameter is not passed through to the Lattus REST interface and which will make the Amplistor® return a default (50) maximum amount of entries.

• marker: Sets the start position at marker + 1. If null is passed, the marker parameter is not passed through to the Lattus REST interface which will make the Lattus start at the beginning of the supplied directory.

Returns:

a list of directory entries.

Getting Object information

AmpliObject AmplistorClientLib.AmplistorClient.ObjectDetails (string ampliNamespace, IList< string > pathEntries, string objectName)

Gets detailed information about a stored object in the Lattusin the specified namespace in the specified directory.

Parameters:



Chapter 10: Microsoft .net BindingsGetting Started



Returns:

Detailed information about a stored object.

Various Operations Managing the Streaming Buffersize

int AmplistorClientLib.AmplistorClient.BufferSize [get, set]

Gets or Sets the buffersize when streaming objects to and from the Lattus StoragePool. The default is 1 MiB.

Getting Started

Getting the .net Bindings

The latest version for 3.3.0 can be downloaded at https://confluence.amplidata.com/login.action;jsessionid=74BDF8F5058500D6B241D445944287EB?os_destination=%2Fnotpermitted.action%3Fversion%3D1%26modificationDate%3D1329472414000.

The md5 fingerprint of this library is A840B8E750846F59F926A4C46961DF66

Create a Policy and Namespace

Create both a default policy and default namespace to use with Lattus.

Create a Lattus Connection Object

AmplistorClientLib.AmplistorClient.AmplistorClient (string host, int port)

Creates a connection object to a Lattus host.


https://confluence.amplidata.com/login.action;jsessionid=74BDF8F5058500D6B241D445944287EB?os_destination=%2Fnotpermitted.action%3Fversion%3D1%26modificationDate%3D1329472414000



Chapter 10: Microsoft .net BindingsGetting Started

Parameters:

• host:Lattus hostname or ipaddress.

• port: Port on which the Lattus REST interface is listening.

Upload an Object The following examples assume you have created a form with an uploadbutton:

using System;using System.Collections.Generic;using System.ComponentModel;using System.Data; using System.Drawing; using System.IO;using System.Linq;using System.Text;using System.Windows.Forms;

using AmplistorClientLib;

namespace ampli_upload_app{public partial class FormUploadApp : Form{public FormUploadApp(){InitializeComponent();}

private void uploadbutton_Click(object sender, EventArgs e){string filename = textBox.Text;Stream filestream = File.OpenRead(filename);AmplistorClient client = new AmplistorClient("192.168.12.206", 8080);client.StoreObject(filestream, "testnamespace", "uploadedobject.txt");}}


Chapter 11Working With REST

This chapter describes how to start the q-shell, set up a test environment, begin developing, and resetting the environment.

Starting the q-shell sudo /opt/qbase3/qshell

Setting up a Test Environment

q.dsstestframework.setUp(enableAXR=True)

Expected output of the command:In [1]: q.dsstestframework.setUp(enableAXR=True)Out[1]: <_pm__factory._LocalEnvironment object at 0x31a59d

Starting with Development

When you setup the test-environment, a default 'test' namespace is created, which you can use without any authentication settings.

Example listing all the namespaces in the system using curl:curl http://[VM_IP_ADDRESS]:8080/namespace -H "Accept: text/plain"

Output on a freshly installed system will be:

"test"


Chapter 11: Working With REST

Resetting the Environment

q.dsstestframework.tearDown()

q.dsstestframework.setUp(enableAXR=True)

Caution: A reset will restart all services and remove all data from your environment.


Chapter 12Working with Lattus S3

The S3 Web services interface can be used to store and retrieve data from locations on the Web. This interface gives developers access to a data storage infrastructure that is scalable, reliable, fast, and inexpensive.

Starting the q-shell

sudo /opt/qbase3/qshell

Setting up an S3 Test Environment

q.dsstestframework.setUp(enableS3=True)

Expected output of the command:In [1]: q.dsstestframework.setUp()Out[1]: <_pm__factory._LocalEnvironment object at 0x31a59d

Lattus REST API Guide 189

Chapter 12: Working with Lattus S3Enabling S3

The memory address will differ from environment to environment.

Enabling S3

Adding a User for S3 Authentication

You need to create a user that will be used to authenticate:q.dss.manage.addUser('s3user','s3pass')q.dss.manage.setPermissions("/manage","s3user",["Read", "Create", "Delete","List", "Update"])

Configure the Client Daemon to Listen on its S3 Interface

1 Open the configuration file located at /opt/qbase3/cfg/dss/clientdaemons/0.cfg (where “0” is the GUID for the machine).

2 Change following entry to the config section using an editor: (Large S becomes lowercase s)

[S3]

address = 127.0.0.1:7070

into

[s3]

address = 127.0.0.1:7070

3 Save the cfg file and restart the client daemon using Q-shell:

q.dss.clientdaemons.restart()

Updating the HTTPS Proxy

AmpliStor is using a standard Web Proxy Server (Pound) to offer secure transmission services between the client and the S3 rest server. Incoming HTTPS S3 requests are decrypted and proxied onto the client daemon listening on localhost for incoming S3 requests.

You will need to tell the Web Proxy Server where it can find its pem file. A file with the .pem extension is used to create the private key. Pem stands for privacy enhanced mail.

190 Lattus REST API Guide

Chapter 12: Working with Lattus S3Enabling S3

Store your pem file in: /opt/qbase3/cfg/pound/server.pem (after creating the appropriate subdirectory)

Start the HTTPS proxy:

sudo /etc/init.d/pound restart

Note: By default, HTTPS is disabled and we will connect to the S3 service on localhost on port 7070 (by using the s3cmd proxy settings)

Defining the Policy for S3 Bucket Creation

Since S3 doesn't have the concept of storage policy, we need to provide a method to define what policy to use when creating an S3 bucket. There are 2 options.

When the Test environment has been created, a policy is created by default. You need to identify the unique policy-GUID in order to enable the S3 interface. This is done with following command:

In Q-shell:

mypolicy = q.dss.manage.listPolicies().keys()[0]

Setting a Storage Pool-Wide Policy for Creating S3 Buckets

When you select this option, you can define a single storage policy for creating ALL your S3 buckets.

Execute following command:

q.dss.manage.changeEnvironment(policyID=mypolicy)

Note: Once you associate a default S3 policy with your storage pool, you cannot remove this default policy anymore. The only thing you can do is change the default S3 policy.

Setting a Per-User Policy for Creating S3 Buckets

When you select this option, you can define a storage policy per user. When the authenticated user creates a bucket, the default policy assigned to that user will be used.


Chapter 12: Working with Lattus S3Testing Your Setup

q.dss.manage.changeUser(userName='s3user', policyID=mypolicy)

Testing Your Setup

We installed the tool s3cmd by default on the machine and we created in the Amplidata home directory a configfile for it, called .s3cfg

Note: Quantum installed s3cmd version 1.1.0 beta3 since the 0.9 GA version contains several bugs.

Open this file with an editor and supply following fields:

• access_key (the user that you created)

• secret_key (the password you supplied for that user)

Note: The cfg file already has some default values that match these examples but modify them to match your setup. Keep in mind that s3cmd is very sensitive about spaces in the configuration file. It will show authentication or broken pipe errors if there is an additional space after access_key or secret_key.In these examples, we are assuming that the commands will be executed as the Amplidata user. As such, the s3cmd tool will use the configuration file in the home directory of the Lattus user account.

s3cmd will by default connect to s3.amazonaws.com. To prevent needing to update your hosts file for every bucket you add, we preinstalled and preconfigured dsnmasq (see http://www.thekelleys.org.uk/dnsmasq/doc.html)

Make sure that you have modified the resolv.conf file of this VM, so that a dns-lookup uses localhost (on which this dns server is running).

sudo vim /etc/resolv.conf

Make sure that your resolv.conf file starts with following line:

nameserver 127.0.0.1


http://www.thekelleys.org.uk/dnsmasq/doc.html

http://www.thekelleys.org.uk/dnsmasq/doc.html

Chapter 12: Working with Lattus S3Testing Your Setup

Check that the s3.amazonaws.com FQDN resolves to the correct IP:

ping <bucketname>.s3.amazonaws.com

You can now execute following commands to create, list and delete buckets and to put, get and delete objects in these buckets:

Create a New Bucket s3cmd mb s3://mynewbucket

Bucket 's3://mynewbucket/' created

List Buckets s3cmd ls

2012-09-14 12:45 s3://mynewbucket

Put a File dd if=/dev/zero of=/home/amplidata/1MB bs=1M count=1

s3cmd put /home/amplidata/1MB s3://mynewbucket/file_s3_1

/home/amplidata/1MB -> s3://mynewbucket/file_s3_1 [1 of 1] 1048576 of 1048576 100% in 0s 3.85 MB/s done

Get a File s3cmd get s3://mynewbucket/file_s3_1 1MB.get.13

s3://mynewbucket/file_s3_1 -> 1MB.get.13 [1 of 1] 1048576 of 1048576 100% in 0s 11.85 MB/s done

List Files in a Bucket s3cmd ls s3://mynewbucket2012-09-14 09:20 99258469 s3://mynewbucket/file12012-09-14 09:15 99258469 s3://mynewbucket/file1-u2012-09-14 09:21 99258469 s3://mynewbucket/file22012-09-14 09:22 99258469 s3://mynewbucket/file32012-09-14 09:22 99258469 s3://mynewbucket/file42012-09-14 09:23 99258469 s3://mynewbucket/file52012-09-14 09:23 99258469 s3://mynewbucket/file62012-09-14 09:26 99258469 s3://mynewbucket/file7


Chapter 12: Working with Lattus S3Rebooting an S3 Test environment

List All Objects in All Buckets

s3cmd la2012-09-14 09:20 99258469 s3://mynewbucket/file12012-09-14 09:15 99258469 s3://mynewbucket/file1-u2012-09-14 09:21 99258469 s3://mynewbucket/file22012-09-14 09:22 99258469 s3://mynewbucket/file32012-09-14 09:22 99258469 s3://mynewbucket/file42012-09-14 09:23 99258469 s3://mynewbucket/file52012-09-14 09:23 99258469 s3://mynewbucket/file62012-09-14 09:26 99258469 s3://mynewbucket/file72012-09-14 09:20 99258469 s3://mynewbucket/file12012-09-14 09:15 99258469 s3://s3test/testfile12012-09-14 09:21 99258469 s3://s3test/testfile2

Delete File From a Bucket

s3cmd del s3://mynewbucket/file_s3_1

File s3://mynewbucket/file_s3_1 deleted

Rebooting an S3 Test environment

When an S3 Test environment is rebooted, the data is persisted but the services are not automatically started. These should be started in the following sequence:q.manage.arakoon.start()q.dss.storagedaemons.start()q.dss.maintenanceagents.start()q.dss.clientdaemons.start()


Chapter 12: Working with Lattus S3Tearing Down an S3 Test Environment

Tearing Down an S3 Test Environment

You can start from scratch by issuing following command from the q-shell (sudo /opt/qbase3/qshell):

q.dsstestframework.tearDown()

This command will remove all your data and stop all services.

Note: As a result of this action, you will have to reconfigure S3.

Getting Support

When escalating issues to your Quantum contact, make sure to provide following information:

Output of Following qshell Command

q.dss.manage.getRevisionInformation()

This will look like this:

version: 3.2.0, branch: default, revision:

379a00490bde08f5a1a456853aaa176df16cb5fd, compile_time: 19/06/2013 14:00:44 UTC'

Clientdaemon Log File This file is located under:

/opt/qbase3/var/log/dss/clientdaemons/


Chapter 12: Working with Lattus S3Getting Support


Lattus REST API User’s Guide - Quantum...Lattus REST API User’s Guide xi Preface Document...

Documents

Transcript of Lattus REST API User’s Guide - Quantum...Lattus REST API User’s Guide xi Preface Document...