NYARC 2014: Frick/Zeri Results

89
Photo Archive Image Analysis John Resig March 2014 http://ejohn.org/research/

description

Talk given at the 2014 NYARC annual meeting.

Transcript of NYARC 2014: Frick/Zeri Results

Page 1: NYARC 2014: Frick/Zeri Results

Photo ArchiveImage Analysis

John Resig March 2014

http://ejohn.org/research/

Page 2: NYARC 2014: Frick/Zeri Results

Ukiyo-e.org

Page 3: NYARC 2014: Frick/Zeri Results
Page 4: NYARC 2014: Frick/Zeri Results
Page 5: NYARC 2014: Frick/Zeri Results
Page 6: NYARC 2014: Frick/Zeri Results
Page 7: NYARC 2014: Frick/Zeri Results
Page 8: NYARC 2014: Frick/Zeri Results
Page 9: NYARC 2014: Frick/Zeri Results
Page 10: NYARC 2014: Frick/Zeri Results
Page 11: NYARC 2014: Frick/Zeri Results
Page 12: NYARC 2014: Frick/Zeri Results

Frick Art Reference Library Photoarchive

• Frick Art Reference Library Photoarchive established in 1920.

• Collects photographs of works of art from around the world, including sponsoring own photography.

• Traditionally image analysis has not been applied to photo archives.

• The Frick Photoarchive was interested in exploring image analysis more after seeingUkiyo-e.org.

Page 13: NYARC 2014: Frick/Zeri Results

Test case: Anonymous Italian Art Photographs• Anonymous Italian Art photographs are

small part (~18,000) of over 1.2 million images in the Frick Photoarchive.

• Very few color photographs (makes most image analysis difficult, good test case).

• Analysis was performed by myself in collaboration with the Frick Photoarchive.

Page 14: NYARC 2014: Frick/Zeri Results

Digital Anonymous Italian Art

• Early digitization project.

• At the time there was no in-house digital lab at the Frick Photoarchive.

• Digitization self-funded and executed by an outside institution.

• When new images were added they were not “merged” -- too hard with no attribution.

Page 15: NYARC 2014: Frick/Zeri Results

Image Analysis

• I wrote custom software to automate matches using MatchEngine (a TinEye service).

• Extremely effective image matching technology.

• Matches same images and portions of images, even in black-and-white.

Page 16: NYARC 2014: Frick/Zeri Results
Page 17: NYARC 2014: Frick/Zeri Results

MatchEngine

image1.jpg

image2.jpg

image3.jpg

imageN.jpg

Upload Images to MatchEngine

.

.

.

.

.

.

Page 18: NYARC 2014: Frick/Zeri Results

MatchEngine

image1.jpg

image2.jpg

Retrieve Image Matches from MatchEngine

image9.jpg

Matches

image2.jpg

image1.jpg

Matches

Match Database

Page 19: NYARC 2014: Frick/Zeri Results

Archive (e.g. Frick Art Reference Library)

ArtworkImage

Image

Image

ArtworkImage

Image

Image

ArtworkImage

Image

Image

ArtworkImage

Image

Image

ArtworkImage

Image

Image

ArtworkImage

Image

Image

Page 20: NYARC 2014: Frick/Zeri Results

Artwork

Confirming a Known Match

Page 21: NYARC 2014: Frick/Zeri Results

ArtworkArtwork

Discovering a New Match

Page 22: NYARC 2014: Frick/Zeri Results

5%

3%

92%

Artworks with no match Confirms a known match Matches a new work

Frick Artworks That Have A Match

Page 23: NYARC 2014: Frick/Zeri Results

Similar Images

Different photo, same work of art.

Page 24: NYARC 2014: Frick/Zeri Results

Similar Images

Different photo, slightly different cropping.

Page 25: NYARC 2014: Frick/Zeri Results

Similar Images

Different photo, dramatically different lighting.

Page 26: NYARC 2014: Frick/Zeri Results

Alternate Images

Partial Image vs. Full Image

Page 27: NYARC 2014: Frick/Zeri Results

Alternate Images

Partial Image vs. Much Larger Image

Page 28: NYARC 2014: Frick/Zeri Results

Alternate Images

Color vs. Black-and-White

Page 29: NYARC 2014: Frick/Zeri Results

Conservation

Page 30: NYARC 2014: Frick/Zeri Results

Conservation

Repairs and possibly removal of later additions.

Page 31: NYARC 2014: Frick/Zeri Results

Conservation

Page 32: NYARC 2014: Frick/Zeri Results

Conservation

Possibly just a drastic difference in lighting?

Page 33: NYARC 2014: Frick/Zeri Results

Conservation

Analysis even spots dramatic conservation work.

Page 34: NYARC 2014: Frick/Zeri Results

Copies

Page 35: NYARC 2014: Frick/Zeri Results

Copies

Page 36: NYARC 2014: Frick/Zeri Results

Copies

Page 37: NYARC 2014: Frick/Zeri Results

Copies

Page 38: NYARC 2014: Frick/Zeri Results

Copies

Page 39: NYARC 2014: Frick/Zeri Results

Copies

Page 40: NYARC 2014: Frick/Zeri Results

Digitization Errors

Arms with Folded Hands Castello sforzesco, Milan.

Female HeadGabinetto disegni e stampe

degli Uffizi, Florence.

Work doesn’t match description, wrong cataloging.

Page 41: NYARC 2014: Frick/Zeri Results

Still Life with a Bottle, a Plate, a Mortar and Pestle...

Virgin Enthroned Nursing Christ, Between Two Saints

S. Biovenale, Orvieto.

Work doesn’t match description, wrong cataloging.

Page 42: NYARC 2014: Frick/Zeri Results

2%4%4%

26%

31%

34%

Similar Image Wrong Image Alternate Image Ambiguous ImageDifferent Works Conservation

Types of New Matches Discovered

Page 43: NYARC 2014: Frick/Zeri Results

Measuring Image Analysis Efficacy

Page 44: NYARC 2014: Frick/Zeri Results

Artwork

Confirming a Known Match

Page 45: NYARC 2014: Frick/Zeri Results

Artwork

Impossible to Confirm Match

Page 46: NYARC 2014: Frick/Zeri Results

Artworks With Multiple Photos

47%

33%

20%

Failed Match Successful Match No Possible Matches

Page 47: NYARC 2014: Frick/Zeri Results

# of Match Successes and Failures by Type

0

80

160

240

320

Same Similar Alternate Conservation 3D Negative

Success Failure

Page 48: NYARC 2014: Frick/Zeri Results

Similar Images

Page 49: NYARC 2014: Frick/Zeri Results

Similar Images

Page 50: NYARC 2014: Frick/Zeri Results

Alternate Images

Page 51: NYARC 2014: Frick/Zeri Results

Alternate Images

Page 52: NYARC 2014: Frick/Zeri Results

Likelihood of a match by % of image overlap

0%

25%

50%

75%

100%

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Failed Alternate Matches Successful Alternate Matches Successful Other Matches

% of Image Overlapping

Page 53: NYARC 2014: Frick/Zeri Results

Likelihood of an ‘Alternate Image’ match by % of image overlap

9%

91%

Failure Success

86%

14%

Failure Success

Under 30% Overlap Over 30% Overlap

Page 54: NYARC 2014: Frick/Zeri Results

Conservation

Page 55: NYARC 2014: Frick/Zeri Results

3D Works

Page 56: NYARC 2014: Frick/Zeri Results

3D Works

Page 57: NYARC 2014: Frick/Zeri Results

Negative Images

Page 58: NYARC 2014: Frick/Zeri Results

Overall MatchEngine Quality

62%

38%

Failure Success

88%

12%

Failure Success

All Potential Matches No: Under 30% Overlap, 3D, or Negative

Page 59: NYARC 2014: Frick/Zeri Results

Frederico Zeri Foundation

• University of Bologna

• 290,000 photos of artworks from national archives, private archives, museums, etc.

Page 60: NYARC 2014: Frick/Zeri Results

Zeri 15thC Italian Art

• Digitized 15th century Italian art

• Persumably some overlap with the Frick’s Anonymous Italian Art archive

Page 61: NYARC 2014: Frick/Zeri Results

Image-to-Artwork Ratio

0

10000

20000

30000

40000

Frick Zeri

17,929

14,284

38,511

18,548

Images Artworks

Page 62: NYARC 2014: Frick/Zeri Results

3%

36%

61%

Artworks with no match Confirms a known match Matches a new work

Zeri Artworks That Have A Match

Page 63: NYARC 2014: Frick/Zeri Results

Image Analysis Matches

0

5500

11000

16500

22000

Frick Zeri

738815

20,051

1,053

Same Artwork Different Artwork

Page 64: NYARC 2014: Frick/Zeri Results

Image Detail

Page 65: NYARC 2014: Frick/Zeri Results

Image Detail

Page 66: NYARC 2014: Frick/Zeri Results

Same Piece, Similar Style

Page 67: NYARC 2014: Frick/Zeri Results

Same Piece, Similar Style

Page 68: NYARC 2014: Frick/Zeri Results

Similar/Copies

Page 69: NYARC 2014: Frick/Zeri Results
Page 70: NYARC 2014: Frick/Zeri Results

What Happens When We Combine Archives?

• Discover new relationships with the other archive

• Discover new relationships inside our own archive

Page 71: NYARC 2014: Frick/Zeri Results

Frick Photoarchive

ArtworkImage

Image

Image

ArtworkImage

Image

Image

Zeri Archive

ArtworkImage

Image

Image

ArtworkImage

Image

Image

Page 72: NYARC 2014: Frick/Zeri Results

Frick-Zeri Image Matches

18,548 Images

38,511 Images

1,117 Images 1,442 Images

Page 73: NYARC 2014: Frick/Zeri Results

Frick-Zeri Artwork Matches

14,284 Artworks 17,929 Artworks

855 Artworks (6%) 938 Artworks (5%)

Page 74: NYARC 2014: Frick/Zeri Results

Merge Verification

• Previously, 77 artworks were manually matched by a researcher at the Zeri foundation

• Can use to verify the quality of the matches

Page 75: NYARC 2014: Frick/Zeri Results

Merge Verification

16%

84%

Same Artwork Failed to Match

Page 76: NYARC 2014: Frick/Zeri Results

Lighting Differences

Page 77: NYARC 2014: Frick/Zeri Results

Tiny Details

Page 78: NYARC 2014: Frick/Zeri Results

Merging Works!

• 84% success confirming known relationships (with a small sample)

• Only 6% of computer-discovered matches were also found by a human.

• (55 human matches vs. 938 computer matches)

Page 79: NYARC 2014: Frick/Zeri Results

Graph Analysis

Page 80: NYARC 2014: Frick/Zeri Results

Frick 420

420

Zeri 15836

42091

42090

Frick 417

417

42092

?

Page 81: NYARC 2014: Frick/Zeri Results
Page 82: NYARC 2014: Frick/Zeri Results

Finding new Links

• Found 14 new Frick links via Zeri

• Found 236 new Zeri links via Frick

Page 83: NYARC 2014: Frick/Zeri Results

Frick 347

347

Zeri 12227

33526 33527

Frick 348

348

33525 33528

?

Page 84: NYARC 2014: Frick/Zeri Results
Page 85: NYARC 2014: Frick/Zeri Results

Frick 8131

Zeri 23402

57136

Frick 81328132a 8132

57129

57134

Zeri 23405

5713057137

57138

8131a 8131

?

Page 86: NYARC 2014: Frick/Zeri Results
Page 87: NYARC 2014: Frick/Zeri Results
Page 88: NYARC 2014: Frick/Zeri Results

Conclusion

• It becomes possible to merge photo archives, even though they’re in different langauges.

• New relationships are discovered in the process.

• The synergy of combining multiple archives reveals even more information.

• All of this is completely automated!

Page 89: NYARC 2014: Frick/Zeri Results

Contact

• John Resig

[email protected]

• http://ejohn.org/research/

• http://ukiyo-e.org/