Million Book Project @ Bibliotheca Alexandrina Noha Adly 20 November 2006.

Million Book Project @ Bibliotheca Alexandrina

Noha Adly20 November 2006

Bibliotheca Alexandrina 2


BA Digitization Workflow


Statistics - November 2006

Arabic Latin Total

Scanned

Books 22,023 4,646 26,669

Pages 7,003,185 1,350,688 8,353,873

ProcessedBooks 21,947 4,642 26,589

Pages 6,987,392 1,348,900 8,336,292

OCRedBooks 16,652 4,600 21,252

Pages 5,248,337 1,327,385 6,575,722

Total Archived Data 1,500 GB


Statistics (Contd) Daily Rates

– Scan: ≈ 1800 pages/person

– Process: ≈ 1800 pages/person

– Latin OCR: ≈ 4000 pages/person

– Arabic OCR: ≈ 1500 pages/person

Five Minolta scanners 2 shifts – 7 days a week

OCR

Image to Text


OCR - Arabic Poses unique challenges

– Written cursively, with blocks of connected characters

– a ‘block of characters’ can have more than one base line.

– Uses external objects such as dots, 'Hamza' and 'Madda'.

– Diacritization

– Characters can have more than one shape according to their position

– Overlapping makes it difficult to determine the spacing

Sakhr Automatic reader is used Tricky with old books Requires learning


Arabic Script Is Cursive


Old, Smudgy, and Sticked Together


Use of Diacritics


Font Low Bound High Point % BooksAR-H1 97.70% 99.50% 0.24%AR-H2 97.60% 99.50% 2.66%AR-H3 97.04% 99.10% 8.01%AR-H4AR-L4 92.70% 96.70% 6.62%DT-M1DT-L2 88.40% 96.80% 7.24%TA-H1 97.30% 99.10% 1.26%TA-H2 97.60% 99.20% 11.89%TA-H3TA-H4 96.50% 97.74% 2.99%TA-L1 94.00% 97.70% 1.65%TA-L4 94.00% 97.90% 8.68%TA-M2 95.80% 98.80% 23.47%TA-M4 94.50% 97.50% 15.57%X 9.72%

Under construction

Under construction

Under construction

16 Font Groups


Evaluation of VERUS and AR

Challenge Set

95%

5%

VERUS AR

Normal Set

38.67%

61.33%

VERUS AR

Research agreement with NovoDynamics Preliminary evaluation on two data sets is promising

– Challenge: difficult to OCR, degraded images

– Normal: known to return acceptable accuracy

Encoding

Image on Text


Image-on-Text

Multilayered:– Visible page image– Hidden OCR text

View exact original layout while searching and highlighting

Supported with some OCR suites only

Supported format: DJVU and PDF


Quality Assurance

No missing cover or pages

All pages are in order

Text quality

Images quality

PDF quality

DAR

Digital Assets Repository


System Architecture

DAF/DAK APIs

Digital Assets Keeper(DAK)

RepositoryDatabase

Authentication and Authorization Subsystem

Users/groups/permissionsDatabase

Storage Subsystem

OfflineStorage

OnlineStorage

Integrated LibrarySystem

CatalogDatabase

User Interface

AdministrationTool

DigitizationClient

ArchivingTool

CatalogingTool

PublishingInterface

OAIGateway

Digital Assets Factory(DAF)

DigitizationDatabase

EncodingTool


DAK Publishing Module


Show notes


Transfer of Digitized Books

Challenges

– Storage: CD vs Online

– Bandwidth: 10 Mbps vs 155 Mbps

– Copyright: not published Actions:

– Transferred 8,500+ books to the Internet Archive

– Process is still going on

Books From India

Towards better collaboration


Books From India

Language Number Books

Arabic 832

Arabic + French 3

Arabic + German 1

Persian 101

French 2

English 1

Spanish 1

German 1

Total 942


ProgressPhase Name Done as of

November 1, 2006

Expected to finished by

Comments

Cataloging 801 - 35 have metadata problems

Processing 742 November 20, 2006

OCRing 200 March 1, 2007

Encoding 171 - -

Publishing 171 - -


Metadata Problems


Processing


OCR Using VERUS or AR?

Calculated accuracy for a small sample

– Images processed once with darkening effect and once without

– VERUS likes darkening, AR does not

– Overall, AR won 70% of cases

30%

70%

VERUS AR


Thank You

Million Book Project @ Bibliotheca Alexandrina Noha Adly 20 November 2006.

Documents

Transcript of Million Book Project @ Bibliotheca Alexandrina Noha Adly 20 November 2006.