2017-04 Campaign Storage 30

CampaignStorage

PeterBraam 2017-04Co-founder&CEOCampaignStorage

Contents

• Memoryclassstorage&Campaignstorage• ObjectStorage• CampaignStorage• SearchandPolicyManagement• DataMovers&Servers• RoadAhead

4/1/17 CampaignStorageLLC 2

CampaignStorage

• CampaignStoragewasinventedatLosAlamosNationalLaboratory• 2014-

• PeterBraam &NathanThompsonfoundedCampaignStorage,LLCinMarch2016• deliverproductsinthisspace• SoftwareDefinedStorage– wewillpartnerwithintegrators

• OthercompaniesareaddressingpartsofCampaignStoragealso


StorageTiers&CampaignStorage


CPUorGPUpackages

CPUcores

HighBandwidthMemory

NVRAMe.g.XPOINT,PCM,STTRAM

RAM

FLASH DISK

NodeBW (GB/sec) 1TB/s 100GB/s 20 GB/s 5GB/s 350MB/s

ClusterBW(TB/sec)

1 PB/s 100TB/s 5TB/s 100GB/s 10’sGB/s

Software Languagelevel Languagelevel,NVMlibsHDF5&DAOS

HDF5DAOS

ParallelFS&CampaignStorage

Archive&Campaign

Keyfeatures transparentcomputation transparentcomputationultra-fast storageapps

namespacescientificformatsFS stylecontainer

bulk datamovement- manyfiles- subtreesofMD

BWCost $/(GB/s) $10(CPUincluded!) $10 $300 $2K $30K

CapacityCost$/GB $ $8 $0.3 $0.05 $0.01


BurstBuffers–DDNIME,CrayDataWarp

TAPE

Roleofcontainers

Fundamentallyunlikely:differenttiersperformdatamovementatsimilargranularity

Containersareamust-have


TiersandNVRAMConsiderationsTiering

RAMtiersareforcomputationèmigratepointers,pages

Flashstorageis5xfasterwithlargeIODisksimilarlyisveryIOsizesensitive:

èRetrieve&storecontainers (distributed?)èShowinternalstructureonfastersideèStreamandserializedatatoslowerside

Internalprogramdataformatsnotre-usableè computingformattonamespace

Persistence

DistinguishingNVMfeatureisthatdatastaysifpowerisoff.

NVRAMwillbethefasteststoragedeviceè formostdemandingstorageapplications

NVRAM:whatotherbenefitstocomputing?

Currentlibraries– transactions,persistentheaps(notsonovel– seeCamelot&RVMfrom1980’s)


ExampleContainerFunctionality- lowertiers


Tiers&TransparencyRAM

- Demoteinfrequentlyusedpointers- Promotefrequentlyusedpointers

Ifpointersarenotfirstclassobjects- Promoteuponaccess- Demotefindinglessusedones

Lowlevellanguages– HWorOSsupport

Storage

Sameprinciple– transparencyrequiresaccessingdatathroughahandle

Onehandlesystemwithlocationdatabaseallowsotherobjectstomove

ExpectdistributedtieredKVstore• Keyvaluelookup• Callbacksforinvalidation


UsingTiers


EnterpriseDataServices

- Groupsoffileservers- Availability- Performance



DataCenterComputeNodes

DataCenter


CampaignStorage

ServiceLayer- SMB,NFS,other

DataMovers- Parallelingest&restore

Storagelayer- ZFS&object- Integratedsearch- AnalyticsSupport- Datamanagement- Massive,low$





Containersforglobalnamespace

Identityandnamespacemanagementwithe.g.ADorLDAP

FutureExa-ScaleStorageArchitecture


ObjectStorage


Cloudobjectstores– pros&cons

promassivescalabilityverygoodstoragemanagementwidelyagreedS3RESTAPIrunsoncheapesthardware

condatalacksorganizationAPI’sdon’tallowdistributedconcurrentaccessorrandomwritesperformancecanbedisappointingdifficulttore-useasacomponentofotherstoragesystems


Toomuchchoice?• CaringoSwarm (formerlyCAStor)• CleversafedsNet

• Cloudian

• DataDirectNetworksWebObjectScaler (WOS)

• EMCAtmos• EMCCentera

• EMCElasticCloudStorage(ECS)

• HPStoreAll

• HGSTHimalaya• HGSTActiveArchive

• HitachiDataSystemsHCP

• NetAppStorageGridWebscale

• QuantumLattus• ScalityRing

• SwiftStackSwift

Tomentionafew…….(othersS3,CEPH,SNIAT10,SeagateA200,DDNWOZ….)

- Normalread/writeIOperobject- NonoverlappingIOfrommultipleclients- 3tierhierarchicalredundancy(box,rack,datacenter)- Transactionprotocoltosnapshotconsistentstate

Whatisneededoffers:


CampaignStorage


CampaignStorage- anewtier


ParallelFileSystem

Archive

ParallelFileSystem

Archive

BurstBuffer

CampaignStorage

Cloud

decreasingemphasis

HighBW,high$$$Decreasingcapacities

OldWorld NewWorld

new

CampaignStorage

Itis…AfilesystemFocus:stagingandarchivingBuiltfrom

• Industrystandardobjectstores• Existingmetadatastores

LowestcostHWHighcapacity,ultrascalableNothighestBWorlowestlatency

• 10-100xhigherthanarchives• 10xlowerthanPFS

Itisnot…Generalpurposefilesystem• Wait… thesedon’texistactually

Usingobjectstoreshasproblems• Limitedsetofdatamoverssupported


Implementation


OSwithVFSandFuse

MarFS

ObjectStorage MetadataFS

HPCClusterA

SimulationCluster20PF

BurstBuffer5PB&5TB/s

HPCD&Viz Cluster HDFS

HPCClusterB

HPCDCluster20PF

BurstBuffer5PB&5TB/s

Lustre

FS1TB/s

CampaignStorage

CampaignStorageMoverNodes

CampaignStorageMetadata Repository

CampaignStorageMoverNodes

ParallelStaging&Archiving

CampaignStorageObject Repository

Search&DataManagement

CampaignStorage


FileSystemInterface

Optionalothertools:• Policymanagers(e.g.Robinhood)• Workflowmanagers(e.g.Irods)

customerinfrastructure

CampaignStorageDataLayout


Search&PolicyManagement


Histogramsforsubtreesearch

EverydirectoryhashistogramDBrecordingpropertiesofitssubtree:• i.e.#files,#bytesinthesubtreehaveaproperty?• Limitedgranularity,limitedrelationalalgebra• Storeperhaps~100,000propertiesinmultiplehistograms

Examples:• Quotainsubtree?• Whatfileserverscontainfiles?• Geospatialinformationinfile?• (filetype,size,accesstime)tuples

• Allowslimitedrelationalalgebra• Userdatabaseforsubtree– eliminatesrelianceonexternalidentitymanagement

Notanewidea.CanbeaddedtoZFS&Lustre


DataMovers&Services


DataMovers

DataMovement

Today• LANL“parallelrsync”– pftool• Lustre HSMmover• Packingsmallfiles&stripingbigfiles

Candidates• DMAPIHSMmover• Gridftp• FullPOSIXinterface

MetadataMovement

Today• TraditionalmetadataAPI• Multiplenamespaces

Coming• Bulkintegrationofcontainers• Accompanyingmetadata


pftool internals


LoadBalancerScheduler

dirs queue

statqueue

cp/S/Vqueue

readdir

stat

copy/syncvalidate

DONE

QUEUE

REPORTER

FeaturesofDS3archivaldatamover

• Objectstoremovesbatchesoffiles• Newconcept:filelevelI/Ovectorization• Includesserverdrivenordering• Packingsmallfilesintooneobject


int copy_file_range_fv(copy_range *r, uint count, int flags)

struct copy_range {int source_fd;int dest_fd;off_t source_offset;off_t dest_offset;size_t length;

}

Services

CampaignStoragealwaysexportstheMarFS filesystem

Enterpriseservicesasfurtherexportedprotocols:- SMB,NFS,HTTP- Datamovementcanbeoutofband

Integrationofnamespaces,userdatabases,otherplugins


CampaignStorageUseCases


Workflows- HPC

Staging&De-staging• Schedulemigrationwithpftool

HSM• Copymetadatafirst• Usesubtreesearchindex• Executepolicies• Specializeddatamovers

• Fortransparentretrieval&attributes

Singleprojectextraction• UseZFSnamespaceandobjectbucketperproject

HotvscoldCampaignLocations• Selectdestinationobjectstores• Migrationoncampaignstorage

Multisite• Leverageobjectbucketreplication• LeverageZFSpoolreplication

Cloud• MigratepoolandbucketstoS3

• UseSnowball?


Workflows– DataCenter

Staging&archive• Schedulemigrationwithpftool

ServiceoffloadtoCampaign• Dataavailablewithoutstaging

Singleprojectextraction• UseZFSnamespaceandobjectbucketperproject

Hotvscoldlocations• Selectdestinationobjectstores• Migrationwithincampaignstorage• Automaticmovementwhenservicesneedthedata

Multisite• Leverageobjectbucketreplication• LeverageZFSpoolreplication

Cloud• MigratepoolandbucketstoS3

• UseSnowball?


RoadForward

Uniqueopportunitytoinnovatedatamanagement

LANLandCampaignStoragecreatedan“IndustrySteeringGroup”

Seekagreementon• Datalayouthandling• Attributesusedinconnectionwithlongtermstorage• Interfacesforworkflows


Conclusions


Conclusions

Hardwarediversificationè SoftwareSpecialization

Expectarichhighspeedexa-scaleI/Oplatformtousecontainers

Similarcontainerswillorganizeenterprisetiersofstorage

CampaignStorage:bulkdatastore,archive&datamovement


Thankyou


2017-04 Campaign Storage 30

Documents

Transcript of 2017-04 Campaign Storage 30