Subjective Image Quality Measurement

Schuyler SmithDept. of Computer Science

Stanford Universityskysmith@stanford.edu

EE 368 Final Project Writeup

Introduction

Estimating the subjective, human-perceived quality ofan image can be very useful. For example, with an ac-curate measure of how interesting and attractive eachphoto from an event is we could create high quality, au-tomated slideshows and summaries. Often people takemultiple pictures of the same scene, and these qualitymetrics could also be used for deduplication: once du-plicate images have been identified, we can choose tokeep only the best one.

In general, this is a hard problem, because it’s notimmediately obvious what makes a particular picturegood or bad. While humans can make snap judgementsabout photos in a couple seconds, they use countlessfactors to do so, and draw upon years of experienceand training.

There has been a reasonable amount of research doneon this topic before. [2] is an example of an early se-ries of experiments, while [1] was in the news only veryrecently. Modern methods have become very complex,for example [4] uses a complex layering of many featureextractors and multiple classifiers to categorize photos.Such complex methods are clearly outside the scope ofa final project. Rather than attempting to match thepower of these approaches, in this project I’ll try toanswer a simpler question: how well can an algorithmperform this task while minimizing complexity and max-imizing speed. While smartphones today are faster thanever, they are still far slower than computers, and analgorithm suitable for a mobile photo organization appwill need to be able to process and categorize imagesfairly quickly to be practical.

The two-stage algorithm I developed uses a variety ofimage processing techniques to measure features of animage, then passes these features into a random forestclassifier to make predictions about the quality of theimage. The classifier was trained on images from red-dit.com, and the resulting algorithm is nearly as goodas a human at predicting the quality of test images.

Method

Due to the complexity of the problem, I decided touse machine learning techniques to construct a systemthat can learn the qualities of a good image itself. Asa result, the algorithm can be broken into two stages.Both stages are fairly simple, and could also be easilyparallelized by splitting different features and decisiontrees across cores.

Stage 1: Image Analysis

The average image is far too large to pass into a classi-fier directly. Additionally, images can be similar in manyhigh-level ways that can’t be recognized by looking onlyat the pixels, at least without massively large trainingdatasets. Since making a large, high quality dataset isdifficult and for all these other reasons, the images arepreprocessed before the classifier is run on the result.

The goal of the preprocessing is to extract measure-ments from the images that are salient to image qualityeither alone or in combination. Individual features don’thave to have direct correlation to quality to be useful.For example perhaps image sharpness is more importantfor photos of people than for landscapes. Then whilea feature that can distinguish people from landscapesmay not be directly useful, it could improve the per-formance of the algorithm by increasing the usefulnessof a sharpness feature. Deducing these relationshipsis left to the classifier. In all, I developed 36 features,which can be separated into six broad categories. In-dividual features will be explained in more detail in theimplementation section.

1. Technical Quality includes features to measurethings such as sharpness, contrast, and noise.Broadly, these are intended to measure the ob-jective quality of the image, and identify mistakessuch as photos that are out of focus, or underex-posed.

2. Blur includes features to measure blur in an im-age. This is distinct from a measure of sharpnessin two important ways: First, images with shal-low depth of field, that isolate the subject andblur the background, are often considered highquality, and a heuristic feature to help distinguish“artistic” images from images that are just blurryis useful. Second, a measure of local contrast orhigh frequency detail in an image will be fooled

1

by a photo that is blurred along only one axis,due for example to camera shake. The algorithmuses an additional feature to detect blur alongdifferent axes to address this.

3. Color includes features to analyse the hue andsaturation of an image. The effect of bright coloron image quality is intuitive and borne out by theresults (below). This section also includes a mea-sure of the level of complementary color in animage.

4. Composition includes several features that at-tempt to extract structural information about theshapes in an image, including simple measures ofimage symmetry, and image alignment (the con-centration and orientation of edges and lines inthe image).

5. Face detection is self explanatory. People mayprefer photos of other people to photos withoutpeople in them.

6. Spatial envelope is a collection of several fea-tures that attempt to classify scene types, basedon [5]. These include “naturalness”, which mea-sures the distribution of edge orientations: pre-dominantly horizontal and vertical is less naturalthan an even mix, and “roughness”, which mea-sures the overall complexity of the image, amongothers.

Stage 2: Machine Learning

The result of analyzing an image is a vector with 36values. Most of these are continuous, with a few ex-ceptions (for example, the number of faces is integral).This vector is fed through a random forest classifier,which predicts whether the image is “good” or “bad”.

Random forest classifiers, which use the consensus ofmany decision trees and were first described in [7], havea few significant advantages for this application overother types of supervised learning algorithms. First,decision trees are very flexible, and perform well onthe mixed datatypes they receive here. Second, as anensemble method, random forest on average does agood job of identifying complex, nonlinear relationshipswithin the data. This also makes the classifier eas-ily parallelizable, but in practice it’s much faster thanimage analysis so this isn’t important. Finally, randomforest makes it easy to estimate feature importance, toanalyze the contribution of each feature to predictionaccuracy. For comparison, SVM (even nonlinear SVM

with the kernel trick) performed several times worse onthe same training data.

The classifier is trained on a dataset of 1800 imagescollected from reddit.com/r/itookapicture. This is farfrom an ideal dataset, but was far less work to collectthan creating a suitable one from scratch would be.I did not find a suitable preexisting dataset available.Taking pictures from Reddit does have a few advan-tages: it results in photos of a wide variety of subjectsand styles, and thanks to Reddit’s up/down voting sys-tem each image conveniently is already scored by Red-dit users. These scores are also highly subjective, whichmakes classification difficult but is more representativeof the goals of the algorithm. The itookapicture sub-reddit in particular was chosen for its focus on pho-tographs (rather than memes and other images), itsgenerality, and due to its reasonable size.

However, using Reddit as a datasource also has someunavoidable disadvantages. First, poor quality photosare inherently underrepresented: in general people onlyshare their better photos, and as a result the photosthat are uploaded are on average good. This makesclassification significantly harder, because the differ-ence between a good and a great image is in absoluteterms smaller and more subjective than the differencebetween a good and a poor image. Second, the vot-ing data is very noisy. For example, some good imagesjust get unlucky and are never upvoted, while somemediocre images are upvoted more for the story behindthem than for their quality. I avoided going through thedataset myself and manually removing images I thoughtwere mislabeled because I didn’t want to introduce myown bias, but when I tried this did dramatically im-crease performance. Ultimately I only removed imagesthat were clearly not photos, and left all others.

Figure 1: An example of 10 images from the dataset.

I decided to stick to a binary classification task for sim-plicity. Due to the noisiness of data in this dataset andconsequent inherent limits on potential accuracy, re-gression or classification with more than two categoriesended up being significantly more complicated to rea-son about and negligibly more informative. I selectedimages such that the log of the number of upvotes

2

was approximately uniformly distributed, and I chose50 upvotes as the threshold for a “good” image, whichroughly split the dataset in half: 52% “bad” photos and48% good photos. Hence, in practice my algorithm ispredicting whether or not an image is likely to receive50 upvotes, rather than quality directly. The algorithmcan be trivially adapted to learn and predict continuousquality scores if desired.

ImplementationAll code was written in Python, with the help of a num-ber of libraries, including PIL (the Python Imaging Li-brary), Numpy/Scipy, OpenCV, and scikit-learn.

Stage 1: Image Analysis

The image is first downscaled to a maximum of 800pixels in either dimension, while preserving aspect ra-tio. This is done primarily for efficiency, but some mea-sures like sharpness are also sensitive to image scaleand benefit from uniform image size. I assume thatfeatures too small to detect on the downscaled imageare unimportant, and experiments adjusting the scalingshow little classification improvement at larger sizes.800 pixels wide is also roughly the size that an imagewould appear on a computer screen.

Then, each feature extractor is run on the image inturn. All but the color features are computed froma grayscale version of the image. In the implementa-tion of features, speed and efficiency were emphasized.Many of these features could be much more accuratelymeasured with subclassifiers of their own, as used in forexample [4], but that is slower and in any case beyondthe possible scope of this project.

• Technical Quality

– Sharpness is extracted by applying a highpass filter to the image (subtracting ablurred version from the original), then com-puting the 98th-percentile pixel in the result-ing image. The 98th-percentile was chosenempiricaly as a good compromise betweenreliability and the goal of measuring peakimage sharpness. An image is blurry if theentire image is blurry, but an image is sharpeven if it contains a small amount of highfrequency data - the rest of the image maynaturally be smooth gradients.

– Contrast is measured as the range and stan-dard deviations of the image’s brightnesshistogram.

– Noise is estimated as the difference be-tween the image and a median filtered ver-sion of the image. This is obviously crude,but is trivial to compute and evidently use-ful.

• Blur

– Depth of field is estimated by breaking theimage into 25 segments (5 per side), andcomparing the sharpnesses of the segments.A large difference between maximum andminimum sharpness presumably indicates ashallow depth of field. This is complicatedby several common false positive, such assolid blue skies and high-key lighting. Ifound no pattern in the locations of thesharpest regions, so to combat this the al-gorithm uses the ratio between the sharpestand median regions in the image.

– Motion blur is estimated by convolvingthe image with a number of different one-dimensional gaussian kernels at different ori-entations, and comparing the resulting im-age sharpness. An image blurred in onlyone direction will be sharper when convolvedwith a kernel in that direction than with akernel in a perpendicular direction.

• Color

– Saturation is simply measured as the meanof the saturation channel after the image isconverted to HSV space.

– Hue computes a rough histogram of huevalues in the image. This leads to 6-12 fea-tures measuring the amount of blue, green,yellow, orange, etc. in the image.

– Complementary colors are measured bytaking the dot product of the hue histogramand the hues rotated 180 degrees. As aresult, the blue bin is multiplied by the or-ange bin, red by green, etc. so a higherscore corresponds to a higher fraction of theimage composed of complementary colorpairs. Partial credit is given for near misses,for example red and blue.

• Composition

– Symmetry is estimated by measuring thedifferences between the image and reflectedcopies of it. The image is reflected hori-zontally and vertically around the centerline,

3

and around the four one-third lines (two ofeach orientation). Far more complex, accu-rate, and expensive methods are describedin [6], but in practice this method is simple,fast, and still meaningful.

– Lines are computed by running a Houghtransform on the image binarized by theCanny edge detector. The mean and spreadof line orientations is reported.

• Face detection

– Face detection is implemented usingthe Viola-Jones face detector built intoOpenCV.

• Spatial Envelope

– These features are loosely based on the onesdescribed in [5], but significantly simplified.

– Naturalness computes the gradient of theimage, then investigates the distribution ofedge orientations. Predominantly horizontaland vertical edges scores low, while a moreeven distribution scores higher.

– Roughness Measures the density of edgesin the image. The Canny edge detector isrun on the image, and returned result is pro-portional to the number and length of edgesdetected.

– Complexity is similar to roughness, butmeasures the amount of high frequency de-tail in a manner closer to how sharpness ismeasured.

– Expansion counts long diagonal edges,which presumably correlate with imageswith strong perspective and vanishingpoints.

Stage 2: Machine Learning

For the random forest classifier I used scikit-learn, apopular Python machine learning library. The parame-ters were tuned by hand.

ResultsI tested the algorithm by performing randomized 10-fold cross validation of my 1800 image dataset (thatis, tested on a randomly selected 10% of the dataset,trained on 90%, and repeated this 10 times, then av-eraged the results). I measured classification accuracy,the percent of images correctly classified. The resultscan be seen below, with two other reference points:

Algorithm Accuracy

Random Forest 62%

Always guess “bad” 52%

Human 67%

The algorithm performed 10% better than random,compared to my 15% better when I attempted to ratethe images myself. Hence, by this measure the algo-rithm is about 23 as good as I am at predicting thequality of the images in the dataset. Note that my67% was the best after several tries (each on a differ-ent subset of the images), and trying very hard not tobias my guesses: I ended up guessing “good” 49% ofthe time, which compares well with the expected 48%proportion, so I believe this is a fair estimate of humanperformance.

These human and algorithm results are both disap-pointingly low. I believe this is almost entirely a re-sult of the dataset, which is particularly difficult for theseveral reasons described in the Method section.

For a more subjective test, I also ran the algorithm ontwelve of my own images from a trip to an air museum.The results are shown in Figures 2 and 3. We can seethat the algorithm does a good job of identifying theobviously boring images as bad, and the images identi-fied as good are certainly more interesting on average.

Figure 2: The six test images that the algorithm clas-sified as “bad”.

4

Figure 3: The six test images that the algorithm clas-sified as “good”.

Finally, I also empirically analyzed the contribution ofeach class of features to the overall classifier perfor-mance. This is based on the frequency of the features’selection in the decision trees, and is calculated au-tomatically by scikit-learn. The results can be seenin Figure 4. The most important features by far arethe color features, followed by technical quality, com-position, and spatial envelope features in that order.Among individual features, the most important are thehue features, then symmetry, complexity, depth of field,and contrast.

Figure 4: The computed importance of the six classesof features.

Surprisingly and conspicuously, face detection was al-most useless on this dataset. There are a couple likelyexplanations for this. First, faces are not very commonin the dataset, and likely far less common than theywould be in a real sample of someone’s photos. Sec-ond, I did not spend much time tuning the OpenCVface detector, and in the limited time I had to exper-iment I could not keep the detection rate reasonablewhile totally eliminating false positives. Even with a

false positive rate of 5%, if faces are only really presentin 5% of the dataset then about half of detected faceswill be incorrect. In actuality faces were detected inabout 15% of images in the dataset.

Future WorkThere are many ways that this work could be expanded.First, many more features could probably be added tothe image analysis stage of the algorithm, and the ac-curacy of many of the existing features could definitelybe improved, in particular face detections and noise.

Second, this project is almost certainly being held backby its dataset. A larger training dataset, with morelow-quality images and more careful curation to removeoutliers, would almost certainly instantly improve theperformance of the algorithm. The feature extractorsand the dataset are both extremely important to thequality of the algorithm’s results.

References[1] Khosla, A., A. Das Sarma and R. Hamid. “Whatmakes an image popular?” International World WideWeb Conference (WWW), 2014.

[2] Savakis, Andreas E., Stephen P. Etz, Alexander C.Loui. “Evaluation of image appeal in consumer photog-raphy.” Proc. SPIE Vol. 3959, p. 111-120, HumanVision and Electronic Imaging V, 2000.

[3] Datta, R., Jia Li, J.Z. Wang. “Algorithmic infer-encing of aesthetics and emotion in natural images: Anexposition.” Image Processing, 2008. ICIP 2008. 15thIEEE International Conference on

[4] Dhar, S., V. Ordonez, T.L. Berg. “High level de-scribable attributes for predicting aesthetics and inter-estingness.” Computer Vision and Pattern Recognition(CVPR), 2011 IEEE Conference on

[5] Oliva, Aude, A. Torralba. “Modeling the Shape ofthe Scene: A Holistic Representation of the SpatialEnvelope.” International Journal of Computer Vision42(3), 145–175, 2001.

[6] Liu, Yanxi, H. Hel-Or, C. Kaplan, L. Van Gool.“Computational Symmetry in Computer Vision andComputer Graphics.” Foundations and TrendsR inComputer Graphics and Vision Vol. 5, Nos. 1–2 (2009)1–195.

[7] Breiman, Leo "Random Forests". Machine Learn-ing 45 (1): 5–32 (2001).

5