Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training...
Transcript of Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training...
![Page 1: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/1.jpg)
Efficient Image Search and Identification: The Making of WINE-O.AI
Michelle L. Gill, Ph.D. Senior Data Scientist, Metis
@modernscientist SciPy 2017
link.mlgill.co/scipy2017
![Page 2: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/2.jpg)
Metis Data Science Training
• Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago, Seattle
• Corporate Training • Python for Data Science • Machine Learning • Natural Language Processing • Spark
• Evening Professional Development Courses
• Explore Data Science Online Training
thisismetis.com
![Page 3: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/3.jpg)
Motivation for WINE-O.AI
• Facilitate discovery and exploration of new wine
• Open source, computer vision project to share with the community
![Page 4: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/4.jpg)
Ingredients of WINE-O.AI
Recommendation System Web Application
Wine Label IdentificationWine Label Identification
Web Application
Content Based Image Retrieval
![Page 5: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/5.jpg)
Content Based Image Retrieval
Image Database
Query Image
Extracted Features
Extracted Features
Image Comparison
Candidate Images
Database Match
![Page 6: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/6.jpg)
Image Comparison Challenges
• Must be robust to differences in size, rotation, occlusion, and illumination
• And search must remain fast!
Query Image Database Image
![Page 7: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/7.jpg)
Web ApplicationImage Selection with RANSAC
Creation of Bag of Visual Words
Scraped ~50,000 Wine Labels
Identification of Image Features with SIFT
2 3
4
1 2
Content Based Image Retrieval in WINE-O.AI
![Page 8: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/8.jpg)
SIFT Feature Detection
• Scale Invariant Feature Transformation (SIFT)
• Blur images using a Gaussian function of increasing width (σ)
Lowe, D., IJCV, 2004
Gaussian Width (σ)
![Page 9: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/9.jpg)
SIFT Feature Detection
• Align images in 3D scale-space (x, y, σ)
• Subtract adjacent images
• Local extrema evaluated as potential features
x
Gaussian Blurred Images
Difference of Gaussians
Gaussian kernel size (σ)
y
–
![Page 10: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/10.jpg)
––
SIFT Feature Detection
• Align images in 3D scale-space (x, y, σ)
• Subtract adjacent images
• Local extrema evaluated as potential features
x
Gaussian Blurred Images
Difference of Gaussians
Gaussian kernel size (σ)
y
–
![Page 11: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/11.jpg)
SIFT Feature Descriptor
• Descriptor calculated in 16 regions around key point
• Changes in intensity calculated and binned
• Produces 128 dimension descriptor for each key point
![Page 12: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/12.jpg)
SIFT Feature Descriptor
• Descriptor calculated in 16 regions around key point
• Changes in intensity calculated and binned
• Produces 128 dimension descriptor for each key point
![Page 13: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/13.jpg)
SIFT Feature Descriptor
• Descriptor calculated in 16 regions around key point
• Changes in intensity calculated and binned
• Produces 128 dimension descriptor for each key point
![Page 14: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/14.jpg)
# image = array of grayscale, resized image pixels
# Detect featuresdetector = cv2.FeatureDetector_create('SIFT')keypoints = detector.detect(image)
# Get feature descriptorsdescriptor = cv2.DescriptorExtractor_create('SIFT')keypoints, features = descriptor.compute(image, keypoints)
# RootSIFT uses L1 norm (absolute value)descriptors /= np.sqrt(descriptors.sum())
SIFT in Practice
Arandjelovic, R. and Zisserman, A., IEEE CCVPR, 2012 PyImageSearch RootSIFT Discussion
OpenCV2 code has been streamlined for presentation
Load and Process Image
Feature Detection
Get Feature Descriptors
Convert to RootSIFT
![Page 15: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/15.jpg)
Detection of Wine Label Features
• Image sizes: 100 x 100 to 1000 x 1200 • Features from high resolution images did
not encode well
![Page 16: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/16.jpg)
Creation of Bag of Visual Words
Scraped ~50,000 Wine Labels
Detection of Image Features with SIFT
2 31 3
Wine Label Recognition: Bag of Visual Words
Web ApplicationImage Selection with RANSAC
4
![Page 17: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/17.jpg)
• K-means clustering on combined features from data set
• Map features for each label to nearest cluster
• Creates a histogram "fingerprint" for each label
Bag of Visual WordsWine Label
Cluster Histogram
0 1 2 3 4
![Page 18: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/18.jpg)
Bag of Visual WordsWine Label
Cluster Histogram
Inverted Index
0
2
3
1
Wine Labels
Cluster ID
0 1 2 3 4
![Page 19: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/19.jpg)
Bag of Visual WordsWine Label
Cluster Histogram
Inverted Index
0
2
3
1
Wine Labels
Cluster ID
0 1 2 3 4
![Page 20: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/20.jpg)
Creation of Bag of Visual Words
Scraped ~50,000 Wine Labels
Identification of Image Features with SIFT
2 31
Wine Label Selection
Web ApplicationImage Selection with RANSAC
44
![Page 21: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/21.jpg)
• Random sample consensus (RANSAC) used to choose best candidate image
• Random subsets of data fit to model
• Robust to outliers but very slow
Image Retrieval with RANSAC
![Page 22: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/22.jpg)
• Random sample consensus (RANSAC) used to choose best candidate image
• Random subsets of data fit to model
• Robust to outliers but very slow
Image Retrieval with RANSAC
![Page 23: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/23.jpg)
# keypointsA, featuresA from query image# keypointsB, featuresB from candidate image
# Pair similar features from each imaged_matcher = cv2.DescriptorMatcher_create('BruteForce')matches = d_matcher.knnMatch(featuresB, featuresA, 2)
# Select matched keypointsmatch_kpsA = keypointsA[matches[0]]match_kpsB = keypointsB[matches[1]]
# Run RANSAC to calculate transformation matrixmatrix, status = cv2.findHomography(match_kpsA, match_kpsB, cv2.RANSAC, 4.0)
Running RANSAC
OpenCV2 code has been streamlined for presentation
Features from Comparison Images
Brute Force Feature Matching
Find Projection
![Page 24: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/24.jpg)
# Run SIFT & calculate histogram on query imagekeypointsA, featuresA = run_SIFT(image)histogramA = map_features_to_clusters(featuresA)
# Get candidate images with similar histogramscandidate_images = get_similar_images(histogramA)
for cimage in candidate_images:
# Load keypointsB, featuresB for cimage # Run RANSAC on candidate image score = get_ransac_matches(keypointsA, featuresA, keypointsB, featuresB)
# Matched image has best score
Putting It All Together
OpenCV2 code has been streamlined for presentation
Calculate Histogram from SIFT Features
Identify Candidate Images
Use RANSAC to Choose Image Match
![Page 25: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/25.jpg)
WINE-O.AI DemonstrationCandidate Image Database Image
![Page 26: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/26.jpg)
WINE-O.AI Demonstration
![Page 27: Efficient Image Search and Identification: The Making of WINE-O · Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago,](https://reader036.fdocuments.in/reader036/viewer/2022070808/5f068e747e708231d418936f/html5/thumbnails/27.jpg)
• Open source wine application
• Educational use
• GitHub repo: mlgill/wine-o.ai • Website: wine-o.ai
Future of WINE-O.AI
WINE-O.AI: Imbibe Intelligently