CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping...
Transcript of CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping...
![Page 1: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/1.jpg)
Automatic Image Cropping based on Aesthetics Prediction using CNNs
Perception-inspired tools for the webAbraia Software SL
https://abraia.me
![Page 2: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/2.jpg)
2
▫ > 1.25 billion websites
▫ >51% mobile web visits
▫ 69% mobile web traffic is images
Some figures
![Page 3: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/3.jpg)
3
The goal is to provide an optimal viewing experience for any display format.
▫ Adapts the layout to the viewing environment▫ Multiple layouts for a range of devices▫ Different aspect ratios
▪ Lots of images must be transformed
Problem: Adapting pictures to different formats is hard
Responsive web design
![Page 4: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/4.jpg)
4
Cloud services to transform and optimize images on-the-fly using simple URLs.
▫ Smart image retargeting driven by perception▫ Retarget image to the many screen formats that enter your
site▪ On the fly▪ In your stock
▫ Easy-to-use API to make a site responsive▫ A perfect fit to media and ecommerce sites
Abraia cloud services
![Page 5: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/5.jpg)
5
Problem
▫ Adapting images to different displays▫ How to crop lots of images to different sizes?
Automatic image cropping
Solution
▫ Crop and resize▫ Smart cropping algorithms
![Page 6: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/6.jpg)
Attention-based approaches
Saliency prediction to detect the most relevant content.
Image Content
Automatic cropping schemes
Aesthetics-based approaches
Photographic rules to balance the composition.
Image Composition
6
Content & Aesthetics
![Page 7: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/7.jpg)
7
1. Visual attention analysis
A saliency heatmap is generated
2. Cropping regions proposal
3. Content preservation ranking
Best regions are selected by content
Content
![Page 8: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/8.jpg)
8
1. Aesthetics scoring
Sorts crops by their aesthetics prediction
2. Final selection and resize
Aesthetics6.68
6.09
6.31
Best score
Intermediate score
Worst score
![Page 9: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/9.jpg)
▫ Regression model▪ Predict the average aesthetic score▪ Based on MobileNet architecture
▫ WMSE as loss function:▪ Weighted Mean Squared Error (WMSE)▪ To reduce the bias introduced by the unbalanced
training dataset▫ AVA dataset
▪ 70% train▪ 10% validation▪ 20% test
▫ Optimizer▪ RMSprop
MSE: 0.45 RMSE: 0.67
9
Aesthetics prediction with CNNsfirst
last
second
![Page 10: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/10.jpg)
Abraia Smart crop
10
![Page 11: CNNs on Aesthetics Prediction using Automatic Image ...€¦ · Image Content Automatic cropping schemes Aesthetics-based approaches Photographic rules to balance the composition.](https://reader034.fdocuments.in/reader034/viewer/2022051907/5ff9f42705b379356377fd31/html5/thumbnails/11.jpg)
11
▫ Aesthetics is a subjective attribute▫ Data are heavily unbalanced and biased▫ Regression enables aesthetics image cropping
Next steps
▫ Prediction of aesthetics attributes▫ Automatic image enhancement
Conclusions