What dress fits me best? Fashion Recommendation on theClothing Style for Personal Body Shape

Shintami Chusnul HidayatiResearch Center for IT Innovation

Academia SinicaTaipei, Taiwan

shintami@citi.sinica.edu.tw

Cheng-Chun HsuDept. of CSIE

Nat’l Taiwan Univ. of Sci. and Tech.Taipei, Taiwan

b10415009@mail.ntust.edu.tw

Yu-Ting ChangResearch Center for IT Innovation

Academia SinicaTaipei, Taiwan

juliachang@citi.sinica.edu.tw

Kai-Lung HuaDept. of CSIE

Nat’l Taiwan Univ. of Sci. and Tech.Taipei, Taiwan

hua@mail.ntust.edu.tw

Jianlong FuMicrosoft Research

Beijing, Chinajianf@microsoft.com

Wen-Huang ChengCITI, Academia Sinica

EE, National Chiao Tung UniversityTaipei/Hsinchu, Taiwanwhcheng@nctu.edu.tw

ABSTRACTClothing is an integral part of life. Also, it is always an uneasy taskfor people to make decisions on what to wear. An essential styletip is to dress for the body shape, i.e., knowing one’s own bodyshape (e.g., hourglass, rectangle, round and inverted triangle) andselecting the types of clothes that will accentuate the body’s goodfeatures. In the literature, although various fashion recommenda-tion systems for clothing items have been developed, none of themhad explicitly taken the user’s basic body shape into consideration.In this paper, therefore, we proposed a first framework for learningthe compatibility of clothing styles and body shapes from social bigdata, with the goal to recommend a user about what to wear betterin relation to his/her essential body attributes. The experimental re-sults demonstrate the superiority of our proposed approach, leadingto a new aspect for research into fashion recommendation.

CCS CONCEPTS• Information systems→Personalization;Recommender sys-tems; Information extraction;

KEYWORDSFashion analysis; recommender system; human body shape; cloth-ing style; correlation

ACM Reference Format:Shintami Chusnul Hidayati, Cheng-Chun Hsu, Yu-Ting Chang, Kai-LungHua, Jianlong Fu, and Wen-Huang Cheng. 2018. What dress fits me best?Fashion Recommendation on the Clothing Style for Personal Body Shape.In 2018 ACMMultimedia Conference (MM ’18), October 22–26, 2018, Seoul, Re-public of Korea. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3240508.3240546

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.MM ’18, October 22–26, 2018, Seoul, Republic of Korea© 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM.ACM ISBN 978-1-4503-5665-7/18/10. . . $15.00https://doi.org/10.1145/3240508.3240546

Figure 1: Visual illustration of five different body shapes.(Image courtesy of styleangel.com1)

1 INTRODUCTION“The dress must follow the body of a woman, notthe body following the shape of the dress.”

— Hubert de Givenchy, fashion designerNobody is perfect. Everyone has flaws and strengths, without

exception to their body parts. Accordingly, everyone may haveareas on their body they love to highlight and areas they feel likecovering up [1]. By understanding the type of body shapes andwhat styles flatter and complement particular body shape, everyonesurely can work with their body to give off amazing looks.

To the above challenge, body shape is the first thing to considerbefore choosing the perfect clothing styles. In particular, determin-ing the type of body shape is all about the proportions betweenbody measurements. There are several methods to determine bodyshapes, which are known as body shape calculators. They use differ-ent representations to describe the type of body shapes, such as interms of alphabetical letters (e.g., the S-line for a body shape withample breasts and buttocks when viewed from the side), fruit shapes(e.g., the apple for a body shape with a wide torso, broad shoulders,and a full bust, waist, and upper back), and geometric objects (e.g.,

1https://styleangel.com/discover-your-body-shape/

Figure 2: An example of guideline for choosing weddingdress. Finding the right wedding dress is probably one ofthe most crucial things for any bride to be. To find a gownwith the perfect fit, it is essential to choose one which suitsbody shape and flatters the curves. (Image courtesy of vira-lyfeeds.com2)

the top hourglass for a body shape with well defined waist andlarger bust than hips). Most common are the five basic body shapes,as shown in Figure 1. The existing body shape calculators, however,are based on subjective measures, which are susceptible to multiplebiases. To our best knowledge, there is no standardized method fordetermining the type of body shapes. Therefore, determining thetype of body shapes is still a challenging analytical task.

After knowing the type of body shape, we can find out whatkind of style fits best. The style of clothes affects appearance greatlyas it would affect the look and tidiness. For example, as illustratedin Figure 2, the sheath dress, whose the style fits very closely tothe contours of the body from head-to-toe, is best suited to thosewith an hourglass or rectangle body shape, but not good for othershapes since it will accentuate extra inches and can be unflattering.With so many rules, limitations and potential style disasters ahead,knowing how to best dress body shape and balance the physicalcharacteristics can be tricky business.

Currently, intelligent fashion analysis has been conducted in-tensively due to the huge profit potential in the fashion industry.Though guide on how to dressing for body shapes is importantin fashion styling and receiving much attention from fashionistas,this issue has been ignored in multimedia science. We, therefore,propose a novel intelligent fashion analysis framework to modelthe correlation between human body shapes and their most suit-able clothing styles, which further can be applied to solve fashionrecommendation tasks. In literature, there are two kinds of fashionrecommendation studies. One is recommending outfits that usersmay be interested in [33, 44]. The other is recommending fashionitems that suit to a user-provided fashion item (e.g., boots, cardigan,skirt) [15, 19]. Thus, our proposed work creates a new space inmultimedia mining and recommendation.

There are three basic research problems studied with this frame-work – first, a dataset that can reflect the correlations underlyingbody shapes and clothing styles well; second, a learned model thatcan be used to determine the type of body shapes well; and third, a

2https://viralyfeeds.com/wedding-dress-fit-body-type-best/

reasonable statistical model that can capture the correlations be-tween body shapes and clothing styles well. Therefore, we exploittwo kinds of data, i.e., human body measurements and clothingstyles. In particular, following the styles of our favorite stylish peo-ple, regardless of whether they have the same body shape as ours,will cause a low chance of these styles to suit our body shape; whilefollowing the styles of people that we consider having similar bodyshapes to ours requires an understanding of whether their stylesare fashionable or not. By incorporating both clothing style andbody shape information, we mine the auxiliary clothing featuresto discover semantically important styles for each body shape. Forthis purpose, we construct graphs of images by visual and bodyshape information, respectively. Then, we automatically propagatethe semantic and select the relevant styles across the visual andbody shape graphs.

We summarize our main contributions as follows:• To the best of our knowledge, this is the first study in learningthe golden style rules to how to best flatter each body typeby exploiting big social data.

• To obtain the relevant fashion knowledge rules, we presenta mechanism of style propagation for discovering semanticrelations between clothing styles by considering body figuresof people wearing it.

• We construct a body-style map to model the correlation be-tween clothing styles and body shapes. Based on the body-style map, we can describe which clothing styles that suita particular body appearance, and vice versa, to provide apersonalized style suggestion.

• We design a novel body shape calculator to determine femalebody figure. Empirical results demonstrate that our proposedmethod significantly outperforms existing methods.

• We construct a benchmark dataset for body shape style rec-ommendations. This dataset contains body measurementsof 3,150 female celebrities annotated with the correspond-ing types of body shapes and 349,298 images of 270 stylishcelebrities annotated with the types of clothing items.

The rest of the paper is organized as follows. First, we discussrelated work (Section 2). We then describe our new dataset in detail(Section 3), followed by the overview of our proposed framework(Section 4). Next, we explain our proposed approach in identifyinghuman body shape (Section 5), modeling the correlation betweenfashion styles and human body shapes (Section 6), and selecting therepresentative styles that best suit body shape (Section 7). Finally,we provide experimental results (Section 8) and give conclusionsand an outlook on further work (Section 9).

2 RELATEDWORKIn this section, we review the related work in terms of (1) Fashionitem analysis, (2) Fashion style understanding, and (3) Personalizedstyle suggestions.

Fashion item analysis. Extensive previous research has beenfocused on object-based clothing image analysis, such as clothingrecognition, annotation, segmentation, and retrieval [3]. In recentyears, a number of models have been introduced to learn more dis-criminative representation in order to handle cross-scenario varia-tions [8, 38]. Zhao et al. [45] proposed a novel memory-augmented

Attribute Manipulation Network which to manipulate image rep-resentation at the attribute level. For clothing annotation, Liu etal. [27] proposed a clothes dataset with comprehensive annotationsand a new deep model which learns clothing features by jointly pre-dicting clothing attributes and landmarks. Sun et al. [37] explored apart-based clothing image annotation approach which takes into ac-count tag relevance and tag saliency. Both works of Zhao et al. [46]and Liu et al. [28] addressed the problem of clothing segmentationand clothing alignment. The result of their works can predict thepositions of functional key points defined on the fashion items. Theworks [17, 19, 20] focused on clothing item retrieval research whichcan allow a user to upload a daily human photo captured in thegeneral environment and find similar clothes in online shops.

Fashion style understanding. In addition to some of the tra-ditional problems, interest in high-level fashion understanding hasbeen growing in the computer vision community recently. Theworks [12, 14, 20, 34] explored recognizing and estimating the de-gree of fashion styles. These methods allow learning features formore specific types of images that may be very costly and com-plicated to annotate. Moreover, clothing style understanding canbenefit visual analytics of big social data. By modeling the appear-ance of human clothing and surrounding context, the occupationof the person can be predicted in [36]. Chang et al. [2] depict thestreet fashion of a city by discovering fashion items that are mosticonic for the city. Attempting to directly predict more esoteric mea-surements, such as popularity [11, 41, 42] has also been recentlystudied. For style compatibility discovering, Han et al. [9] proposedto jointly learn a visual-semantic embedding and the compatibilityrelationships among fashion items in an end-to-end fashion.

Personalized style suggestions. User profiling helps person-alization and has received much attention in the social multimediaresearch fields. Simo-Serral et al. [35] analyzed how fashionable aperson looks at a photo whereby advising the user to improve theappeal. Wie et al. [40] intended to explore the inherent relationshipbetweenwearers’ personality type and expressivewearing. Sanchez-Riera et al. [33] proposed a personalized clothing recommendationsystem through the analysis of user’s personal images in his/hersocial networks to predict the most likable items. Liu et al. [26] de-veloped an occasion oriented clothing recommendation and pairingsystem. The system automatically recommends the most suitableclothing by considering the wearing properly and wearing aesthet-ically principles. Though there are several works focus on clothingrecommendation system, to the best of our knowledge, our work isthe first work devoted to considering body measurement to buildthe personal clothing recommendation system.

3 DATASET CONSTRUCTIONWe collected a novel dataset, the Style4BodyShape Dataset3, toenable personalized style suggestion applications that make use ofcelebrities’ style as the knowledge resource. This dataset containsthree types of data: (1) a list of the most stylish female celebrities,who are known for their sophisticated sense of style, (2) body mea-surements of female celebrities, and (3) stylish celebrities photos.In the following subsections, we describe our dataset in detail.

3http://bit.ly/Style4BodyShape.

Figure 3: Example of a celebrity profile page.

3.1 A List of Stylish Celebrity CollectionIn particular, the celebrities’ styles are usually regarded as fashionreferences as they hire fashion stylist(s) to help them get dressedto visually alter their actual body figure. In this work, we thereforepropose to exploit the style of stylish female celebrities to learningthe compatibility of clothing styles and body shapes. We collected alist of the top stylish female celebrities from a popular crowdsourcedpolling website, i.e., Ranker4, and six popular fashion magazinewebsites, i.e., Vogue5, Harper’s Bazaar6, Marie Claire7, Glamour8,and PopSugar9. The top stylish celebrities listed on Ranker are basedon polling of more than 6,300 voters, while the top stylish celebritieslisted on fashionmagazines are selected by fashion editors who haveexpert knowledge in fashion domain. By utilizing these websites asa source of information, we can provide an in-depth understandingof the concept of correlation between body shapes and clothingstyles from the perspective of society as well as fashion experts. Wethen recorded all of the names of stylish celebrities listed on thesewebsites and eliminated the duplicate ones. Finally, we obtained270 names as the top stylish female celebrities for our experiment.

3.2 Body Measurement CollectionWe crawled a collection of human body measurements from acelebritymeasurementswebsite, http://www.bodymeasurements.org.We obtained body measurements of 3,150 female celebrities, includ-ing actress, singers, models, politicians, etc. Each record consists ofthe following attributes: name, short bio, type of body shape (i.e.,hourglass, rectangle, round, triangle, or inverted triangle), dresssize, bust circumstance, waist circumstance, hip circumstance, shoesize, bra size, cup size, height, weight, and information about breasts(natural breasts or implants). Note that body shape informationprovided on this website is merely based on visual judgment. Fig-ure 3 shows the profile of one of the celebrities on the website asan example. We further discarded the data about bio and breastsinformation, as they are not meant to characterize features fordetermining the type of body shape.

4https://www.ranker.com/crowdranked-list/most-stylish-female-celebrities5https://www.vogue.com/tag/franchise/10-best-dressed6https://www.harpersbazaar.com/celebrity/red-carpet-dresses/g8366/female-fashion-icons/7http://www.marieclaire.co.uk/fashion/best-dressed-2016-4577568http://www.glamourmagazine.co.uk/gallery/best-dressed-women-20179https://www.popsugar.com/fashion/Most-Stylish-Celebrities-2017-44071762

Table 1: List of clothing items considered in this work.Category ItemsDress DressOuterwear Blazer, cape, cardigan, coat, jacket, vestPants Jeans, legging, pants, shorts, stockings, tightsSkirt SkirtTop Blouse, shirt, sweater, sweatshirt, T-shirt, top

Table 2: Statistics of stylish celebrity image dataset.Total Minimuma Maximumb Averagec

Dress 67,617 204 278 249.43Outerwear 66,452 95 277 245.12Pants 69,178 205 295 255.21Skirt 66,962 218 291 256.51Top 79,089 249 322 291.92

a Minimum means the minimum number of images per celebrity.b Maximum means the maximum number of images per celebrity.c Average means the average number of images per celebrity.

3.3 Stylish Celebrity Image CollectionWe crawled a large collection of images of stylish female celebritiesvia Google search engine, by issuing each stylish celebrity namecombined with clothing category to be collected as a search query.In particular, we used a list of stylish female celebrities retrievedin Section 3.1 and a list of clothing categories presented in Table 1to generate queries. For example, the keyword “Gigi Hadid skirt”is used to retrieve images of a stylish celebrity named Gigi Hadidwearing a skirt. For each query, we downloaded the first 300 re-turned images. After removing duplicate images, a total of 347,948images of stylish female celebrities were collected. Table 2 showsthe statistics of our collected images.

4 FRAMEWORKKnowing the type of body shape is the foundation for determiningclothing styles that flatter silhouette. Therefore, in this workwe firstbuild a body shape model using our collected body measurementsdata and then use this model to classify the type of body shape fora given body measurements data (cf. Section 5).

A naïve way to figure out what to wear for a body shape is toconsider the outfits of stylish celebrities having the same body shapeas the given query. However, the simple method may not capturethe important features that make up a suitable style. With a largeamount of the collection of styles, it could be a bit overwhelmingat times and hard to know the right things to do when there is somuch information. Therefore, analysis of the data yields valuableinformation that deepens understanding of styling tips.

Having body shape information and image collections of stylishcelebrities, we propose a data mining mechanism that leveragesboth clothing styles and body shape of celebrities wearing them toapproximate the semantic representations of the two modalities (cf.Section 6). In particular, we augment each clothing image in theimage collections with relevant semantic features. We constructgraphs of images by the visual appearance of clothing styles andbody shape information, respectively. Then, we automatically prop-agate and select the informative semantic features across the graphsby exploiting an Auxiliary Visual Words Discovery technique [24].

The discovered semantic features reveal visual features of clothingstyles that are correlated with specific body shapes. Afterward, weutilize the discovered semantic features to select the representativeclothing styles for each body shape by adopting the concept ofstylistic coherence and uniqueness (cf. Section 7). We illustrate theproposed framework in Figure 4.

5 BODY SHAPE CALCULATORAs mentioned previously, determining body shape is the first stepin learning how to dress in a way that makes a body look its best.However, existing body shape calculators are very subjective whichmeans influenced by personal feelings. One promising idea fordetermining the type of body shape is to use unsupervised learningalgorithms to discover the “natural” grouping(s) of a set of bodymeasurements. We see that clustering can be thought of as a viableoption to solve this problem.

Formally, given a collection of body shapes each of which isdescribed by a set of body measurement attributes, clustering aimsto derive a useful division of the n body shapes into a number oftypes. Although there are a large number of methods available toperform clustering, determining different types of body shapes isrelated to the need of a clustering method that does not requireto select the number of clusters. Thus, Affinity Propagation [7],a clustering algorithm based on the concept of “message passing”between data points, is suitable for our intended purpose.

LetV = {v1,v2, ...,vN } be a set of data points representing bodymeasurement features, and also let σ (vi ) be the index of the nearestcommunity center associated to vi . Affinity Propagation aims tofind the mapping σ by minimizing the cost function defined as:

E[σ ] =N∑i=1

s(vi ,vσ (vi )), (1)

where s(vi ,vj ) is the similarity between the pairs vi and vj . In thiswork, we set s(vi ,vj ) to be the negative squared error (Euclideandistance) between vi and vj . The initialized values of s(vi ,vi ) is setequally each other as the median of the input similarities.

In particular, we take advantage of parameters introduced byexisting body shape calculators to construct features for body shapeclassification task. These parameters are: (1) body height; (2) bodymass; (3) bra size; (4) cup size; (5) bust circumference; (6) waistcircumference; (7) hip circumference; (8) the ratio between bustand hip circumferences; (9) the ratio between waist and hip cir-cumferences; (10) the ratio between bust and waist circumferences;(11) the difference between bust and hip circumferences; (12) thedifference between waist and hip circumferences; (13) the differ-ence between bust and waist circumferences; (14) body mass index,which is defined as the body mass divided by the square of the bodyheight; and (15) body shape index, which is calculated by dividingwaist circumference by its estimate obtained from allometric re-gression of body mass and height. Body mass is in kilograms; bodyheight, bust, waist, and hip circumferences are in meters; and brasize and cup size are in US bra sizing. Since bra size and cup size arecategorical data, we then convert them into numerical data usingone-hot encoding scheme. As a result, body measurements of eachperson is represented by 1 × 77 dimensional feature vectors.

Figure 4: An overview of the proposed framework for generating the proposed body-style map. For simplicity in this illustra-tion, we assume that there are four stylish celebrities and a total of ten clothing items in the same category.

6 STYLE AND BODY SHAPE CORRELATIONMODELING

In this section, we focus on modeling the correlations between bodyshapes and clothing styles. We formulate this problem as follows.Each image sample is represented by (b, c,x), where b is the typeof body shape, c = (c1, c2, ..., c5) denotes the clothing categories,and x = (x1,x2, ...,x5) denotes the features describing the visualappearance of clothing styles.

Definition 1. Body shape type b. Each image sample contains astylish female celebrity having specific body measurements. Thus,b is the body shape type associated with body measurements.

Definition 2. Clothing categories c = (c1, c2, ..., c5). c with |c | =5 is the clothing category vector, whose each dimension representsthe presence or absence of a specific clothing category in the image.The details of clothing categories are summarized in Table 1.

Definition 3. Features describing visual appearances of clothingstyles x = (x1,x2, ...,x5). Each dimension of x is associated to aspecific clothing category in c; that is, xi is the visual features ofclothing item(s) in category ci .

Mining Task. We aim to augment each clothing image withrelevant semantic features in order to capture the intrinsic correla-tions between body shapes and clothing styles. Note that this taskis performed on each clothing category separately.

In the following subsections, we provide a detailed descriptionof the methods we employ to model the correlations between bodyshapes and clothing styles.

6.1 Representation of Clothing ItemsAccording to [32], there are hundreds of different types of clothingitems can be categorized into seven basic categories: shirt, outer-wear, skirt, dress, pants, underwear, and swimsuit. In this work,we only focus on investigating shirt, outerwear, skirt, dress, pants.We neglect on analyzing swimsuit and underwear due to privacyissues of some celebrities.

In [43], Yamaguchi et al. suggested there are 56 basic componentsthat compose a fashion photograph. Of these 56 components, 53 ofthem are fashion items. We further adopt 20 fashion items definedin their work since these items are in the taxonomy of clothingcategories being analyzed in this study. A list of our adopted fashionitems and associated categories is presented in Table 1.

Let f represent the outfit of a person in image sample. We definef as a collection of T random variables { f1, ..., fT } representingthe adopted 20 fashion items (T = 20). If the t-th item is not in-cluded in the outfit, then fn = ∅. To localize the region of a specificclothing category ci , we combine the pixels of its correspondingfashion items. After that, we adopt HSV color histogram [29] andBag-of-Visual-Word (BoVW) histogram [31] to extract the visualfeatures. We use these low-level image features since we observedthat both the color and local structures are important. We then con-catenate both a color histogram histColori and a BoVW histogramhistStructurei into a single feature vector to form xi

xi ≡ (histColori ,histStructurei ). (2)

6.2 Style Feature PropagationIn order to obtain more semantically relevant style features for eachclothing image, we propose to augment each clothing image withadditional features propagated from the body shape and clothingstyle clusters based on an Auxiliary Visual Words Discovery tech-nique [24]. Such auxiliary features can enrich the style descriptionof the clothing images. For example, it is promising to derive moresemantic features for a clothing style by simply exchanging the fea-tures among images of the same body shape cluster. We will furtherremove irrelevant or noisy features and preserve representativeones by selection operation (described later in Section 6.3).

Assume there are N clothing images denotes as {I1, I2, ..., IN }

in a category C . Each image In is represented by 1 × D visualfeature vector aswe defined in Equation (2). Formining the auxiliaryfeatures, we start by constructing body shape graph and clothingstyle graph with N nodes each. The body shape graph consistsof body shape clusters that group clothing images of top stylishcelebrities with the same body shape in the same cluster, whilethe clothing style graph consists of clothing style clusters thatgroup visually similar clothing images in the same cluster. We applyaffinity propagation (AP) [7] to cluster images on the clothing stylegraph, making it unnecessary to determine the number of clusters.

Next, we augment each image with additional auxiliary featurespropagated from the body shape and clothing style clusters. Thispropagation is conducted on each extended clothing style cluster,containing the images in a clothing style cluster and those additionalones co-occurring with these images in specific body shape clusters.

Let the matrix X ∈ RZ×D represent the features of Z imagesin the extended clothing style cluster, and each row represent fea-ture vector of each image. Assume M among Z images are fromthe same clothing style cluster. The style feature propagation isconducted by the propagation matrix P ∈ RM×Z , which controlsthe contributions from different images in the extended clothingstyle cluster. We then define the auxiliary style feature Xaux as

Xaux ≡ PX . (3)

Given the initial propagation matrix P0 (i.e., P0(i, j) is the simi-larity score between image i and image j), the operation to find abetter propagation matrix P is formulated as

fP = minP

α∥ PX ∥2F∥ P0X ∥2F

+ (1 − α)∥ P − P0 ∥2F∥ P0 ∥2F

, (4)

where ∥ · ∥F stands for the Frobenius norm, and α stands for theinfluence between the first and the second terms. The objective ofthe first term is to prevent from propagating too many features (i.e.,propagating conservatively). The second term is to maintain thesimilarity to the original propagation matrix P0. In [24], Kuo et al.has a proof that Equation (4) is a strictly convex, unconstrainedquadratic optimization problem. Analytic solver, a quadratic pro-gramming solver, thus can be used to find the optimal solution.By adopting analytic solver, the final propagation matrix P can bederived as

P = α2P0(α1XXT + α2IM

)−1, (5)

where α1 = (α)/(∥ P0X ∥2F ), α2 = (1 − α)/(∥ P0 ∥2F ), and IM is anidentity matrix of sizeM . In our experiment, we set α = 0.5. Havingthe propagation matrix P , we can thus obtain the auxiliary stylefeature Xaux by Equation (3).

6.3 Common Style Feature SelectionAfter the previous style feature propagation step, each image can ob-tain more different features. However, it is possible to obtain someless representative ones that may decrease the precision. Thus, inorder to preserve important (representative) features and suppressthe irrelevant ones, we propose representative style feature selec-tion. We select representative features in each clothing style clustersince clothing images in the same clothing style cluster are visuallysimilar to each other.

Given the initial selection matrix S0 (i.e., S0 is a matrix of ones,which means we select all the dimensions), the operation to find abetter selection matrix S is formulated as

fS = minS

β∥ XS0 −XS ∥2F

∥ XS0 ∥2F+ (1 − β)

∥ S ∥2F∥ S0 ∥2F

, (6)

where β modulates the importance between the first and the secondterms, and S0 and S indicate the weight on each dimension beforeand after selection process, respectively. In particular, the objectiveof the first term is to avoid excessive distortions, while the secondterm is to reduce the number of the selected features in the clothingstyle clusters. Previous work [24] proves that this equation is astrictly convex, unconstrained quadratic problem. Thus, by usinganalytic solver, the final selection matrix S can be derived as

S = β1XT(β1XXT + β2IZ

)−1XS0, (7)

where β1 = (β)/(∥ XS0 ∥2F ), β2 = (1 − β)/(∥ S0 ∥2F ), and IZ isan identity matrix of size Z . In our experiment, we set β = 0.5.After we solve the selection matrix S , the total number of featuresretained after the selection operation can be calculated as

Ssel ≡ XS . (8)

7 REPRESENTATIVE STYLES SELECTIONIn this section, we focus on identifying the most representativeclothing styles that are suitable for a specific body type. As wementioned previously, all body shapes are unique. There are bestfeatures that need to draw attention, as well as less positive fea-tures that need to play down. Given the representative of the mostsuitable clothing styles for each body shape, we can understand thecharacteristics of clothing styles that flatter a specific body shape.

Let X̂ = {x̂1, x̂2, ..., x̂N } be the auxiliary features of N clothingimages in category C obtained from Section 6, and let yj denote atype of body shape. To find clothing images portraying the repre-sentative styles for a specific body shape, we adopt the concept ofstylistic coherence and uniqueness of fashion items [13]. Specifically,clothing styles are considered as the representative styles for acertain body shape if they are both coherent (frequently worn bypeople with a body shape) and unique (worn much more often bypeople with a body shape than with other body shapes).

In this work, we propose to model the aforementioned conceptas a class-conditional-probability density [5]. Assume the elementsin X̂ are mutually independent and obey the Gaussian distribution.Then, for each body shape yj , the class-conditional-probabilitydensity for a clothing image having auxiliary feature x̂n can beformulated as

P(x̂n |yj ) =1√

(2π )D |Σj |exp

[−12(x̂n − µ j

)TΣ−1j

(x̂n − µ j

) ], (9)

where πj and Σj are computed using maximum similarity estima-tion. Thus, the probability of clothing image x̂n portraying repre-sentative style of body shape yj can be obtained by

P(yj |x̂n ) =P(x̂n |yj )P(yj )

P(x̂n ). (10)

Clothing image x̂n is considered as a representative style ofbody shape yj if it satisfies two criteria: (1) if P(yj |x̂n ) is higherthan a threshold T1, which means this style is frequently worn bytop stylish celebrities with body shape yj ; and (2) if the differencebetween the probability of clothing image x̂n to class yj and classyk is higher than a threshold T2, which means this style is wornmuch more often by top stylish celebrities with body shape yj thanwith other body shapes. In our experiments, T1 and T2 are set 50%and 15%, respectively.

8 EXPERIMENTSTo evaluate the efficacy and performance of our proposed method,we conduct extensive experiments and compare the performancewith some baselinemethods – body shape classification (Section 8.1),body-shape-based style ranking (Section 8.2), and style-based bodyshape ranking (Section 8.3). Next, we provide examples of clothingitems depicting suitable styles for specific body shape in Section 8.4.Finally, in Section 8.5, the subjective evaluation results are reported.

Table 3: Body shape classification accuracy (in percentage).Existing body shape calculators Ours

[6] [18] [4] [21] [22] [25] [30]31.84 5.97 37.87 16.41 1.59 28.63 20.38 76.83

8.1 Body Shape ClassificationIn this section, we investigate the effectiveness of existing bodyshape calculators. Seven different methods widely used to deter-mine the type of body shapes are compared. For this evaluation,we used our collected body measurements (cf. Section 3.2) andperformed five-fold cross-validation. We report the accuracy ofrecognizing body shapes using existing methods in the first sevencolumns of Table 3. From these results, we can see that existingbody shape calculators do not perform well. The highest accuracyrate is achieved when using the method implemented in [4]10.

Inspired by the huge success of convolutional neural networks(CNNs) in the tasks of classification [23], we adopted a CNN-basedmethod to perform body shape classification task. Specifically, weused CNN architecture based on Dense Convolutional Network(DenseNet) [16], which uses a densely connected path to concate-nate the input features with the output features, enabling eachmicro-block to receive raw information from all previous micro-blocks. The body shape features described in Section 5 are employed.Before feeding the CNN, we reshaped the normalized body shapefeatures into a square matrix of size 77 × 77 by filling the miss-ing entries in rows with zeroes. By transforming our data into animage-like structure, the height of feature matrix won’t reduce tooquick after multiple downsampling of pooling process.

We report the performance of our proposed body shape calcu-lator obtained from 5-fold cross-validation in the last column ofTable 3. We can see that our proposed body shape calculator out-performs existing methods. These experiments further confirm thecontribution of the proposed method in determining body shapesusing the existing definition of body shape types.

However, since body shape calculator plays an important rolein the decision on what to wear, the performance of our bodyshape calculator still does not meet the desired accuracy target. Theproblem can be due to confusing definition of body shape types.Therefore, we propose to discover the different types of body shapesin an unsupervised manner, as described in Section 5. Seven typesof body shapes are identified using this approach.

8.2 Body-shape-based Style RecommendationThe purpose of this experiment is to evaluate the overall correlationbetween body shapes and clothing styles as described in Section 6.In this task, we seek suitable clothing style for the given bodyshape. We cinducted this evaluation on stylish celebrity imagesfrom our collection database. For each stylish celebrity, 80% ofimages (samples) are used for training, 10% for validation, and 10%for testing, in 9-fold cross-validation.

We measure performance using the average precision, a per-formance metric commonly used to evaluate the ranked retrievalresults. For a single query, the average precision approximates the10Note that the cited references here are representative of several body shape calcula-tors that use the same methods.

Figure 5: Comparison ofAverage Precision for clothing stylerecommendation by body shape.

area under the uninterpolated precision-recall curve. Therefore,we also compute the mean average precision over a set of queriesto evaluate the overall system performance. Intuitively, a higheraverage precision indicates a higher quality.

We compared our proposed method with Adversarial Cross-Modal Retrieval (ACMR) method [10, 39], a cross-media retrievalmethod built around the concept of adversarial learning. Thismethod involves two processes, i.e., a feature projector and a modal-ity classifier, conducted as a minimax game. A feature projectorgenerating modality-invariant and discriminative representationsand aiming to confuse the modality classifier, while a modalityclassifier distinguishing the items in terms of their modalities.

On average, our proposed method outperformed ACMR methodby over 11.26-17.96% in terms of mAP when evaluating style recom-mendations by body shape. We show a comparison of our proposedmethod with ACMR method for each of the five clothing categoriesin Figure 5. Different from ACMRmethod, our method approximatethe semantic representations of two modalities, namely by leverag-ing both the clothing style and associated body shape information.Therefore, it can retrieve more semantically related results and,thus, it achieves higher accuracy.

8.3 Style-based Body Shape RecommendationTo provide another view for evaluating the overall correlation be-tween body shapes and clothing styles constructed using our pro-posed framework, we conduct body shape recommendations byclothing style. Different from the previous evaluation that seeks suit-able clothing styles for a given body shape query, in this evaluationwe seek body shape that is best-suited for a given clothing imagequery. For this purpose, we used the same training and testing dataas in experiment described in Section 8.2. We also compared ourresults with ACMR method, and evaluated their performances interms of the average precision.

Figure 6 shows comparisons of our proposed method with ACMRmethod for each of the seven types of body shapes. On average, ourproposed method outperformed ACMRmethod by over 3.84-34.74%.Thus, based on the results of our evaluations, we confirm that thereare strong correlations between body shape and clothing style, andour proposed framework can model this correlation better thanACMR method, a state-of-the-art cross-media retrieval method.

8.4 Exemplar ResultsIn this section, we present some examples of clothing styles for cer-tain body shapes obtained using the method described in Section 7.As shown in Figure 7, each body shape has unique clothing styles.

Figure 6: Comparison of Average Precision for body shaperecommendation by clothing style shape.

Figure 7: Representative clothing styles for three differentbody shapes.

For example, the celebrity who has body shape type 1 tends to usea belt to emphasize her waistline, while the celebrity who has bodyshape type 7 tends to wear a strapless dress that shows off the bareshoulders.

8.5 User Study EvaluationWe conducted a user study to measure users’ preference towardsrecommended styles generated by the proposed and comparisonmethods. After finding the recommended styles of each body shapes,we recruited twenty female students from our campus as the par-ticipants. The study process is divided into two phases: pre-studyphase and on-study phase.

The main objective of pre-study phase is to give the participantssome general idea of the correlation between fashion styles andbody shapes. We firstly collected a number of online fashion tips ondressing to flatter body shapes (e.g., https://blog.stitchfix.com/fashion-tips/ and https://www.wikihow.com/Dress-for-Your-Body-Type),and then showed them to the participants. Each participant wasallowed to freely explore the web pages with no time limit. Afterthat, the participants started to perform the next phase.

In the on-study phase, we use body measurements of each partic-ipant as query to retrieve the top five most suitable clothing stylesgenerated by our proposed method and the comparison method.Since we focus on five clothing items, a total of 50 images wereshown to each participant. The generating method of each imagewas blind to the participants and the display order was randomlydetermined. We further asked the participants to give a rating from

Table 4: User evaluation results.

MethodSatisfaction score Preference

Average Standard deviationACMR [39] 2.878 1.240 25.00%

Ours 3.574 1.154 54.60%

1 to 5 to indicate how suitable each recommended style for theirbody shape, with 5 being the highest suitability.

We report the user evaluation results in Table 4. It shows that theproposed method achieved the highest average satisfaction scoreof more than 3. Most users prefer the styles recommended by ourproposed method. Thus, it verifies that our proposed recommen-dation framework can help people find the best clothing styles fortheir body shape.

9 CONCLUSIONS AND FUTUREWORKA visual summary of clothing designs is essential to gain a deeperunderstanding of fashion style rules. In this paper, we built scalablemodels to capture the semantic correlation between fashion stylesand body appearances. Based on the created models of data analysis,two questions related to fashion styling tips can be addressed: (1)what are the most suitable clothing styles for a given human bodymeasurement? and (2) which human body shape is best-suited to agiven clothing item? To the best of our knowledge, this is the firstwork devoted to investigating the semantics concepts of fashiondesigns through the visual analytics of big data. Moreover, weshowed how the rich amount of stylish celebrities data availableonline can be exploited, modeled, and analyzed to provide fashionstyling tips.

Several research topics are open for future investigation. Sinceour final goal is to help users find the most suitable clothing whichcan perfectly flatter their figure and make them look gorgeous, in-vestigating multiple personal factors will be helpful for enhancingthe recommendation system performance. Currently, our analysisfocuses on the body measurements, the incorporation of other con-textual factors, such as age or skin color might be useful to improvethe system. Creative applications like personal recommendationcan be realized. The user can provide more personal data for thesystem to recommend the most trendy and well fitted daily outfit.Besides, we can also use our recommendation result as the refer-ence to realize online product retrieval, which can greatly enhancecustomers’ shopping experience.

ACKNOWLEDGEMENTThis work was supported in part by Microsoft Research Asia andMinistry of Science and Technology of Taiwan under Grants MOST-105-2628-E-001-003-MY3, MOST-107-2218-E-002-054, MOST-107-2218-E-011-014, MOST-106-2221-E-011-154-MY2, and MOST-106-2218-E-011-009-MY2.

REFERENCES[1] David Bainbridge. 2015. Curvology: The Origins and Power of Female Body Shape.

Portobello Books Ltd.[2] Yu-Ting Chang, Wen-Huang Cheng, Bo Wu, and Kai-Lung Hua. 2017. Fashion

World Map: Understanding Cities Through Streetwear Fashion. In Proc. ACM Int.Conf. Multimedia. 91–99.

[3] Wen-Huang Cheng, Jia Jia, Si Liu, Jianlong Fu, Johnny Tseng, and Jau Huang. 2018.Perfect Corp. Challenge 2018: Half Million Beauty Product Image Recognition.https://challenge2018.perfectcorp.com/index.html.

[4] Kat Collings. 2018. The Foolproof Way to Find Out Your Real Body Type. http://www.whowhatwear.com/how-to-find-body-shape-calculator/. Accessed:2018-02-15.

[5] Yingying Deng, Fan Tang, Weiming Dong, Hanxing Yao, and Bao-Gang Hu. 2017.Style-oriented Representative Paintings Selection. In SIGGRAPH Asia 2017 Posters.12:1–12:2.

[6] Cherene Francis. [n. d.]. Body Shape Calculator. https://auraimageconsulting.com/body-shape-calculator/. Accessed: 2017-08-21.

[7] Brendan J. Frey andDelbert Dueck. 2007. Clustering by passingmessages betweendata points. Science 315, 5814 (2007), 972–976.

[8] Jianlong Fu, Heliang Zheng, and Tao Mei. 2017. Look Closer to See Better:Recurrent Attention Convolutional Neural Network for Fine-Grained ImageRecognition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition. 4438–4446.

[9] Xintong Han, ZuxuanWu, Yu-Gang Jiang, and Larry S Davis. 2017. Learning Fash-ion Compatibility with Bidirectional LSTMs. In Proc. ACM Int. Conf. Multimedia.1078–1086.

[10] Li He, Xing Xu, Huimin Lu, Yang Yang, Fumin Shen, and Heng Tao Shen. 2017.Unsupervised Cross-modal Retrieval through Adversarial Learning. In Proc. IEEEInt. Conf. Multimedia and Expo. 1153–1158.

[11] Shintami Chusnul Hidayati, Yi-Ling Chen, Chao-Lung Yang, and Kai-Lung Hua.2017. Popularity Meter: An Influence- and Aesthetics-aware Social Media Popu-larity Predictor. In Proc. ACM Int. Conf. Multimedia. 1918–1923.

[12] Shintami Chusnul Hidayati, Wen-Huang Cheng, and Kai-Lung Hua. 2012. Cloth-ing Genre Classification by Exploiting the Style Elements. In Proc. ACM Int. Conf.Multimedia. 1137–1140.

[13] Shintami Chusnul Hidayati, Kai-Lung Hua, Wen-Huang Cheng, and Shih-WeiSun. 2014. What Are the Fashion Trends in New York?. In Proc. ACM Int. Conf.Multimedia. 197–200.

[14] Shintami Chusnul Hidayati, Chuang-Wen You, Wen-Huang Cheng, and Kai-LungHua. 2017. Learning and Recognition of Clothing Genres From Full-Body Images.IEEE Trans. Cybernetics 48, 5 (2017), 1647–1659.

[15] Yang Hu, Xi Yi, and Larry S. Davis. 2015. Collaborative Fashion Recommendation:A Functional Tensor Factorization Approach. In Proc. ACM Int. Conf. Multimedia.129–138.

[16] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.2017. Densely Connected Convolutional Networks. In Proc. IEEE Conf. ComputerVision and Pattern Recognition. 4700–4708.

[17] Shuhui Jiang, Yue Wu, and Yun Fu. 2016. Deep Bi-directional Cross-triplet Em-bedding for Cross-Domain Clothing Retrieval. In Proc. ACM Int. Conf. Multimedia.52–56.

[18] Kelly. 2011. Petite Body Types. http://www.alterationsneeded.com/2011/01/petite-body-types.html. Accessed: 2017-08-21.

[19] M. Hadi Kiapour, Xufeng Han, Svetlana Lazebnik, Alexander C. Berg, andTamara L. Berg. 2015. Where to Buy It: Matching Street Clothing Photos inOnline Shops. In Proc. IEEE Int. Conf. Computer Vision. 3343–3351.

[20] M. Hadi Kiapour, Kota Yamaguchi, Alexander C. Berg, and Tamara L. Berg. 2014.Hipster Wars: Discovering Elements of Fashion Styles. In Proc. European Conf.Computer Vision. 472–488.

[21] Marlene V. Kindes. 2006. Body Image: New Research. Nova Publishers.[22] Nir Y. Krakauer and Jesse C. Krakauer. 2012. A New Body Shape Index Predicts

Mortality Hazard Independently of Body Mass Index. PLOS ONE 7, 7 (2012).[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Clas-

sification with Deep Convolutional Neural Networks. In Proc. Int. Conf. NeuralInformation Processing Systems. 1097–1105.

[24] Yin-Hsi Kuo, Wen-Huang Cheng, Hsuan-Tien Lin, and Winston H. Hsu. 2012.Unsupervised Semantic Feature Discovery for Image Object Retrieval and TagRefinement. IEEE Trans. Multimedia 14, 4 (2012), 1079–1090.

[25] Jeong Yim Lee, Cynthia L. Istook, Yun Ja Nam, and SunMi Park. 2007. Comparisonof body shape between USA and Korean women. Int. Jrnl. of Clothing Scienceand Technology 19, 5 (2007), 374–391.

[26] Si Liu, Jiashi Feng, Zheng Song, Tianzhu Zhang, Hanqing Lu, Changsheng Xu,and Shuicheng Yan. 2012. Hi, Magic Closet, Tell Me What to Wear!. In Proc. ACMInt. Conf. Multimedia. 619–628.

[27] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. 2016. Deepfash-ion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations.In Proc. IEEE Conf. Computer Vision and Pattern Recognition. 1096–1104.

[28] Ziwei Liu, Sijie Yan, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2016. FashionLandmark Detection in the Wild. In Proc. European Conf. Computer Vision. 229–245.

[29] B. S. Manjunath, J.-R. Ohm, V. V. Vasudevan, and A. Yamada. 2001. Color andtexture descriptors. IEEE Trans. Circuits and Systems for Video Technology 11, 6(2001), 703–715.

[30] Helen McCormack. 2005. The shape of things to wear: scientists identify howwomen’s figures have changed in 50 years. https://www.independent.co.uk/

news/uk/this-britain/the-shape-of-things-to-wear-scientists-identify-how-womens-figures-have-changed-in-50-years-516259.html. Accessed: 2017-08-21.

[31] D. Nister and H. Stewenius. 2006. Scalable recognition with a vocabulary tree. InProc. IEEE Computer Vision and Pattern Recognition. 2161–2168.

[32] Mary Brooks Picken. 2013. A Dictionary of Costume and Fashion: Historic andModern. Courier Corporation.

[33] Jordi Sanchez-Riera, Jun-Ming Lin, Kai-Lung Hua, Wen-Huang Cheng, andArvin Wen Tsui. 2017. i-Stylist: Finding the Right Dress Through Your SocialNetworks. In Proc. Int. Conf. Multimedia Modeling. 662–673.

[34] Edgar Simo-Serra and Hiroshi Ishikawa. 2016. Fashion Style in 128 Floats: JointRanking and Classification Using Weak Data for Feature Extraction. In Proc. IEEEConf. Computer Vision and Pattern Recognition. 298–307.

[35] Edgar Simo-Serral, Sanja Fidler, Francese Moreno-Noguerl, and Raquel Urtasun.2015. Neuroaesthetics in Fashion: Modeling the Perception of Fashionability. InProc. IEEE Conf. Computer Vision and Pattern Recognition. 869–877.

[36] Zheng Song, Meng Wang, Xian-sheng Hua, and Shuicheng Yan. 2011. PredictingOccupation via Human Clothing and Contexts. In Proc. IEEE Int. Conf. ComputerVision. 1084–1091.

[37] Guang-Lu Sun, Xiao Wu, and Qiang Peng. 2016. Part-based Clothing ImageAnnotation by Visual Neighbor Retrieval. Neurocomputing 213 (2016), 115–124.

[38] Tsung-Hung Tsai, Wen-Huang Cheng, Chuang-Wen You, Min-Chun Hu,Arvin Wen Tsui, and Heng-Yu Chi. 2014. Learning and Recognition of On-Premise Signs (OPSs) from Weakly Labeled Street View Images. IEEE Trans.Image Processing 23, 3 (2014), 1047–1059.

[39] Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, , and Heng Tao Shen. 2017.Adversarial Cross-Modal Retrieval. In Proc. ACM Int. Conf. Multimedia. 154–162.

[40] Zhiqiang Wei, Yan Yan, Lei Huang, and Jie Nie. 2017. Inferring Intrinsic Corre-lation between Clothing Style and Wearers? personality. Multimedia Tools andApplications 76, 19 (2017), 20273–20285.

[41] BoWu, Tao Mei, Wen-Huang Cheng, and Yongdong Zhang. 2016. Unfolding Tem-poral Dynamics: Predicting Social Media Popularity Using Multi-scale TemporalDecomposition. In Proc. AAAI Conf. Artificial Intelligence. 272–278.

[42] Kota Yamaguchi, Tamara L Berg, and Luis E Ortiz. 2014. Chic or Social: Vi-sual Popularity Analysis in Online Fashion Networks. In Proc. ACM Int. Conf.Multimedia. 773–776.

[43] Kota Yamaguchi, M. Hadi Kiapour, Luis E. Ortiz, and Tamara L. Berg. 2014.Retrieving Similar Styles to Parse Clothing. IEEE Trans. Pattern Analysis andMachine Intelligence 37, 5 (2014), 1028–1040.

[44] Xishan Zhang, Jia Jia, Ke Gao, Yongdong Zhang, Dongming Zhang, Jintao Li, andQi Tian. 2016. Trip Outfits Advisor: Location-oriented Clothing Recommendation.IEEE Trans. Multimedia 19, 11 (2016), 2533–2544.

[45] Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan. 2017. Memory-AugmentedAttribute Manipulation Networks for Interactive Fashion Search. In Proc. IEEEConf. Computer Vision and Pattern Recognition. 1520–1528.

[46] Bo Zhao, Xiao Wu, Qiang Peng, and Shuicheng Yan. 2016. Clothing Cosegmenta-tion for Shopping Images With Cluttered Background. IEEE Trans. Multimedia18, 6 (2016), 1111–1123.