How to transfer a BigFile

Presenter: Nguyễn Văn TiếnBarcamp, RMIT, 2013tipuder@gmail.com

Architecture

Upload

Download

Applications

Architecture

SBFSession

SBFProfiles

SBFMeta

SBFUserItems

IDZen

Frontend

SBFUpload

FilePoolSBFWork

er

SBFChunks

SBFZenTok

SBFDownload

Java, C++LibEvent, Poco,KyotoCabinet

Upload

SBFSession

SBFProfiles

SBFMeta

SBFUserItems

IDZen

Frontend

SBFUpload

FilePoolSBFWork

er

SBFChunks

1: reqPageUp

SBFZenTok

2b:genTok 3:tok2a:checkUser

SBFDownload

4:pageUp4:up(tok)

5a:check(tok)

6a:genId()

6b:writeFile(id)

7: putMeta

9:notify

8: putUsersFile

10: readFile

12: putChunks

11: genIds

6c: fileId

Download

SBFSession

SBFProfiles

SBFMeta

SBFUserItems

IDZen

Frontend

SBFUpload

FilePoolSBFWork

er

SBFChunks

1: reqPageDown

SBFZenTok

SBFDownload

4:pageDown

2a:checkUser2b:genTok 3:tok

4:down(tok)

5:getMeta

6:getChunks

loop

sendChunks

One more thing

SBFSession

SBFProfiles

SBFMeta

SBFUserItems

IDZen

Frontend

SBFUpload

FilePoolSBFWork

er

SBFChunks

SBFZenTok

SBFDownload

SBFChunks

SBFDownload

Node 1

SBFUpload

FilePoolSBFWork

er

SBFDownload

SBFChunks

Node 2

SBFNodeInfo

SBFBroker

Applications

Issues

Disk Bottleneck

Chunk Size, Sector Size, Block Size

Speed process file slow when concurrent upload high

Bottleneck disk