A hands-on introduction tooptimization

Mathematical Programming— a formal language foroptimization

Leo Liberti, CNRS LIX Ecole Polytechniqueliberti@lix.polytechnique.fr

MOMI17

1 / 69

Tutorial

I Python (install it yourselves)I Somemodules:

I OptLang / GLPK, matplotlib, networkxI NLTK, MITIE, pattern, sklearn

I AMPL& solversI Win64: http://tinyurl.com/hpwgbf3I MacOsX 64: http://tinyurl.com/zf2oygbI Linux 64: http://tinyurl.com/gqzdf9m

I If it’s slow, it’s that your all downloading together — wait sometime before retrying

I AMPL licenses valid until AprilAMPL is commercial

I My code: http://tinyurl.com/z6nvhj5

2 / 69

Section 1

What industry o�cially says aboutoptimization

3 / 69

C∗Os

What you hear

I We optimized our strategy (CEO)

I We optimized our revenues (CFO)

I We optimized our processes (CTO)

I We optimized our operation (COO)

I Oh yes we can do big data (CIO)

What they meanWe keep changing everything so that investors will mistakeour wasteful dynamism for growth

4 / 69

C∗Os

What you hear

I We optimized our strategy (CEO)

I We optimized our revenues (CFO)

I We optimized our processes (CTO)

I We optimized our operation (COO)

I Oh yes we can do big data (CIO)

What they meanWe keep changing everything so that investors will mistakeour wasteful dynamism for growth

4 / 69

C∗Os

What you hear

I We optimized our strategy (CEO)

I We optimized our revenues (CFO)

I We optimized our processes (CTO)

I We optimized our operation (COO)

I Oh yes we can do big data (CIO)

What they meanWe keep changing everything so that investors will mistakeour wasteful dynamism for growth

4 / 69

Something else they meanI Departments are compartimentalizedso they can blame each other when they fuck up

I Every division is a separate legal entityso complaining customers must address divisions separately

I Customers must pay to contact the �rmso no-one will complain

I The �rm heavily invests in IBMWatson Technologiesso insistent customers will only ever talk to a computer

5 / 69

And their reasonsI Departments are compartimentalizedso they can blame each other when they fuck up

I Every division is a separate legal entityso complaining customers must address divisions separately

I Customers must pay to contact the �rmso no-one will complain

I The �rm heavily invests in IBMWatson Technologiesso insistent customers will only ever talk to a computer

6 / 69

I may be overly optimisticI Departments are compartimentalizedas a result of internal �ghts

I Every division is a separate legal entitybecause of a recent merge

I Customers must pay to contact the �rmas the �rm has no other revenue

I The �rm heavily invests in IBMWatson Technologiesbecause the CIO heard it’s fashionable

7 / 69

Section 2

What job o�ers currently say aboutoptimization

8 / 69

Job o�ers

9 / 69

An exampleUnder the responsibility of the Commercial Director, the Optimisation / Operations Senior Managerwill have the responsibility to optimise and develop operational aspects for VINCI Airports currentand future portfolio of airports. They will also be responsible for driving forward and managing keyoptimisation projects that assist the Commercial Director in delivering the objectives of the TechnicalServices Agreements activities of VINCI Airports. The Optimisation Manager will support the CommercialDirector in the development and implementation of plans, strategies and reporting processes. As partof the exercise of its function, the Optimisation Manager will undertake the following: Identificationand development of cross asset synergies with a specific focus on the operations and processing functionsof the airport. Definition and implementation of the Optimisation Strategy in line with the objectivesof the various technical services agreements, the strategy of the individual airports and the Group.This function will include: Participation in the definition of airport strategy. Definition of thisairport strategy into the Optimisation Strategy. Regularly evaluate the impact of the OptimisationStrategy. Ensure accurate implementation of this strategy at all airports. Management of the varioustechnical services agreements with our airports by developing specific technical competences from theHead Office level. Oversee the management and definition of all optimisation projects. Identification,overview and management of the project managers responsible for the delivery of the various optimizationprojects at each asset. Construction of good relationships with the key stakeholders, in order tocontribute to the success of each optimization project. Development and implementation of the Groupoptimisation plan. Definition of economic and quality of service criteria, in order to define performancegoals. Evaluation of the performance of the Group operations in terms of processing efficiency, servicelevels, passenger convenience and harmonization of the non-aeronautical activities. Monitoring thestrategies, trends and best practices of the airport industry and other reference industries in terms ofthe applicability to the optimization plan. Study of the needs and preferences of the passengers,through a continuous process of marketing research at all of the airports within the VINCI Airportsportfolio. Development of benchmarking studies in order to evaluate the trends, in internationalairports or in the local market. Development and participation in the expansion or refurbishment projectsof the airports, to assure a correct configuration and positioning of the operational and commercialarea that can allow the optimization of the revenues and operational efficiency. Support the DirectorBusiness Development through the analysis and opportunity assessment of areas of optimization for alltarget assets in all bids and the preparation and implementation of the strategic plan once theassets are acquired. Maintain up to date knowledge of market trends and key initiatives relatedto the operational and commercial aspects of international airports [...]

. . .and blah blah blah: IS THIS APPROPRIATE FORMYCV?10 / 69

Try Natural Language Processing

I Automated summaryI Relation ExtractionI Named Entity Recognition (NER)I Keywords

11 / 69

Automated summary./summarize.py job01.txt

They will also be responsible for driving forward and managing keyoptimisation projects that assist the Commercial Director in deliveringthe objectives of the Technical Services Agreements activities of VINCIAirports. The Optimisation Manager will support the Commercial Directorin the development and implementation of plans, strategies and reportingprocesses. Identification and development of cross asset synergies witha specific focus on the operations and processing functions of the airport.Construction of good relationships with the key stakeholders, in order tocontribute to the success of each optimization project. Definition ofeconomic and quality of service criteria, in order to define performancegoals. Evaluation of the performance of the Group operations in terms ofprocessing efficiency, service levels, passenger convenience andharmonization of the non-aeronautical activities. Development ofbenchmarking studies in order to evaluate the trends, in internationalairports or in the local market. Maintain up to date knowledge of markettrends and key initiatives related to the operational and commercialaspects of international airports. You have a diverse range ofexperiences working at or with airports across various disciplines suchas operations, ground handling, commercial, etc. Demonstrated highlevel conceptual thinking, creativity and analytical skills.

Does it help? hard to say12 / 69

Relation Extraction./relextr-mitie.py job01.txt

======= RELATIONS =======Optimisation Strategy [ INCLUDES_EVENT ] VINCI AirportsSelf [ INCLUDES_EVENT ] Head OfficeHead Office [ INFLUENCED_BY ] SelfHead Office [ INTERRED_HERE ] SelfVINCI Airports [ INTERRED_HERE ] Optimisation StrategyHead Office [ INVENTIONS ] SelfOptimisation Strategy [ LOCATIONS ] VINCI AirportsSelf [ LOCATIONS ] Head OfficeSelf [ ORGANIZATIONS_WITH_THIS_SCOPE ] Head OfficeSelf [ PEOPLE_INVOLVED ] Head OfficeSelf [ PLACE_OF_DEATH ] Head OfficeHead Office [ RELIGION ] SelfVINCI Airports [ RELIGION ] Optimisation Strategy

Does it help? hardly

13 / 69

Named Entity Recognition

./ner-mitie.py job01.txt

==== NAMED ENTITIES =====English MISCFrench MISCHead Office ORGANIZATIONOptimisation / Operations ORGANIZATIONOptimisation Strategy ORGANIZATIONSelf PERSONTechnical Services Agreements MISCVINCI Airports ORGANIZATION

Does it help? . . .maybe

For a documentD, let NER(D) = named entity words

14 / 69

Idea

1. Recognize named entities from all documents2. Use them to compute distances among documents3. Usemodularity clustering

15 / 69

The named entities1. Operations Head Airports O�ce VINCI Technical Self French / Strategy Agreements English Services Optimisation2. Europe and P&CWork Optimization Head He/she of Price Global PhDs Direct Asia Earnix AGD AXA Innovation Coordinate

International English3. Scientist Product Analyze Java Features & Statistics Science PHP Pig/Hive/Spark Optimization Crunch/analyze Team Press

Performance Deezer Data Computer4. Lean6Sigma Lean-type O�ce Banking Paris CDI France RPAMiddle Accenture English Front Benelux5. Partners Management Monitor BC Provide Support Sites Regions Mtiers Program Performance market develop Finance & IS&T

Saint-Ouen Region Control Followings VP Sourcing external Corporate Sector and Alstom Tax Directors Strategic Committee6. Customer Specialist Expedia Service Interact Paris Travel Airline French France Management Egencia English Fares with Company

Inc7. Paris Integration France Automation Automotive French . Linux/Genivi HMI UI Software EB Architecture Elektrobit technologies

GUIDE Engineers German Technology SWwell-structured Experts Tools8. Product Google Managers Python JavaScript AWS JSONBigQuery Java Platform Engineering HTMLMySQL Services Professional

Googles Ruby Cloud OAuth9. EHR Aledades Provide Wellness Perform ACO Visits EHR-system-speci�c Coordinator AledadeMedicare Greenway Allscripts10. Global Java EXCEL Research Statistics Mathematics Analyze Smart Teradata & Python Company GDIA Ford Visa SPARKData

Applied Science Work C++ RUnix/Linux Physics Microsoft Operations Monte JAVAMobility Insight Analytics Engineering ComputerMotor SQLOperation Carlo PowerPoint

11. Management Java CANDIDATE Application Statistics Gurobi Provides Provider Mathematics Service Maintains Deliver SM&GSAS/HPF SAS Data Science Economics Marriott PROFILE Providers OR Engineering Computer SQL Education

12. Alto Statistics Java Sunnyvale ResearchML Learning Science Operational Machine Amazon Computer C++ Palo Internet R Seattle13. LLamasoft Work Fortune Chain Supply C# Top GuruWhat Impactful Team LLamasofts Makes Gartner Gain14. Worldwide Customer JavaMosel Service Python Energy Familiarity CPLEX Research Partnering Amazon R SQL CSOperations15. Operations Science Research Engineering Computer Systems or Build16. Statistics Italy Broad Coins France Australia Python Amazon Germany SAS Appstore Spain Economics Experience R Research US

Scientist UK SQL Japan Economist17. Competency Statistics Knowledge Employer communication ResearchMachine EEOUnited ORMAWay OFCCP CorporationMining

&C# Python Visual Studio Opportunity Excellent Modeling Data Jacksonville Arena Talent Skills Science Florida Life EqualAnyLogic Facebook CSX Oracle The Strategy Vision Operations Industrial Stream of States Analytics Engineering ComputerFramework Technology

18. Java Asia Research Safety in Europe Activities North CompanyWestRocks Sustainability AmericaMasters WRKC++NorcrossOptimization GA ILOG South NYSEOperations AMPL CPLEX Identify Participate OPLWestRock

19. Management Federal Administration SystemNAS Development JMP Tra�c Aviation FAA AdvancedMcLean Center CAASD Flow AirTableau Oracle MITRE TFM Airspace National SQL Campus

20. Abilities & Skills 9001-Quality SManagement ISOGED21. Statistics Group RDBMSResearchMathematics Teradata ORSA Greenplum Java SAS U.S. Solution Time Oracle Military Strategy

Physics Linear/Non-Linear Operations both Industrial Series Econometrics Engineering Clarity Regression 16 / 69

Word similarity: WordNet

17 / 69

WordNet: hyponyms of “boat”

18 / 69

Wu-Palmer word similaritySemantic WordNet distance between wordsw1, w2

wup(w1, w2) =2 depth(lcs(w1, w2))

len(shortest_path(w1, w2)) + 2 depth(lcs(w1, w2))

I lcs: lowest common subsumerearliest common word in paths from both words toWordNet root

I depth: length of path from root to word

Example: wup(dog,boat)?depth( whole ) = 418 -> dog -> canine -> carnivore -> placental -> mammal -> vertebrate

-> chordate -> animal -> organism -> living_thing -> whole -> artifact-> instrumentality -> conveyance -> vehicle -> craft -> vessel -> boat

13 -> dog -> domestic_animal -> animal -> organism -> living_thing-> whole -> artifact -> instrumentality -> conveyance -> vehicle-> craft -> vessel -> boat

wup(dog,boat) = 8/21 = 0.38095238095219 / 69

Extensions of Wu-Palmer similarity

I to lists of wordsH,L:

wup(H,L) =1

|H| |L|∑v∈H

∑w∈L

wup(v, w)

I to pairs of documentsD1, D2:

wup(D1, D2) = wup(NER(D1),NER(D2))

I wup and its extensions are always in [0, 1]

20 / 69

The similarity matrix

Too uniform! Try zeroing values belowmedian

1.00 0.63 0.51 0.51 0.66 0.45 0.46 0.47 0.72 0.58 0.54 0.50 0.72 0.38 0.49 0.47 0.47 0.44 0.54 0.31 0.440.63 1.00 0.45 0.45 0.54 0.40 0.42 0.42 0.57 0.49 0.46 0.45 0.59 0.35 0.43 0.42 0.42 0.41 0.47 0.32 0.400.51 0.45 1.00 0.40 0.53 0.35 0.37 0.37 0.58 0.47 0.43 0.40 0.59 0.28 0.39 0.37 0.38 0.35 0.43 0.24 0.350.51 0.45 0.40 1.00 0.63 0.45 0.46 0.46 0.67 0.56 0.52 0.49 0.68 0.38 0.48 0.47 0.47 0.45 0.53 0.33 0.440.66 0.54 0.53 0.63 1.00 0.34 0.35 0.35 0.49 0.42 0.39 0.37 0.50 0.29 0.36 0.35 0.35 0.34 0.40 0.26 0.340.45 0.40 0.35 0.45 0.34 1.00 0.42 0.43 0.66 0.54 0.49 0.45 0.67 0.34 0.44 0.43 0.43 0.40 0.49 0.28 0.400.46 0.42 0.37 0.46 0.35 0.42 1.00 0.44 0.66 0.54 0.49 0.47 0.67 0.34 0.45 0.45 0.44 0.42 0.50 0.28 0.400.47 0.42 0.37 0.46 0.35 0.43 0.44 1.00 0.67 0.55 0.51 0.48 0.68 0.36 0.47 0.45 0.45 0.43 0.51 0.30 0.420.72 0.57 0.58 0.67 0.49 0.66 0.66 0.67 1.00 0.33 0.31 0.29 0.40 0.23 0.28 0.27 0.28 0.26 0.31 0.21 0.260.58 0.49 0.47 0.56 0.42 0.54 0.54 0.55 0.33 1.00 0.46 0.43 0.59 0.34 0.42 0.41 0.41 0.39 0.46 0.31 0.390.54 0.46 0.43 0.52 0.39 0.49 0.49 0.51 0.31 0.46 1.00 0.39 0.56 0.29 0.38 0.36 0.36 0.34 0.41 0.24 0.350.50 0.45 0.40 0.49 0.37 0.45 0.47 0.48 0.29 0.43 0.39 1.00 0.70 0.40 0.50 0.49 0.48 0.46 0.54 0.35 0.460.72 0.59 0.59 0.68 0.50 0.67 0.67 0.68 0.40 0.59 0.56 0.70 1.00 0.23 0.29 0.29 0.29 0.28 0.33 0.20 0.270.38 0.35 0.28 0.38 0.29 0.34 0.34 0.36 0.23 0.34 0.29 0.40 0.23 1.00 0.48 0.45 0.46 0.42 0.52 0.30 0.430.49 0.43 0.39 0.48 0.36 0.44 0.45 0.47 0.28 0.42 0.38 0.50 0.29 0.48 1.00 0.39 0.39 0.36 0.45 0.26 0.370.47 0.42 0.37 0.47 0.35 0.43 0.45 0.45 0.27 0.41 0.36 0.49 0.29 0.45 0.39 1.00 0.48 0.46 0.54 0.33 0.440.47 0.42 0.38 0.47 0.35 0.43 0.44 0.45 0.28 0.41 0.36 0.48 0.29 0.46 0.39 0.48 1.00 0.43 0.51 0.32 0.430.44 0.41 0.35 0.45 0.34 0.40 0.42 0.43 0.26 0.39 0.34 0.46 0.28 0.42 0.36 0.46 0.43 1.00 0.53 0.31 0.430.54 0.47 0.43 0.53 0.40 0.49 0.50 0.51 0.31 0.46 0.41 0.54 0.33 0.52 0.45 0.54 0.51 0.53 1.00 0.36 0.460.31 0.32 0.24 0.33 0.26 0.28 0.28 0.30 0.21 0.31 0.24 0.35 0.20 0.30 0.26 0.33 0.32 0.31 0.36 1.00 0.470.44 0.40 0.35 0.44 0.34 0.40 0.40 0.42 0.26 0.39 0.35 0.46 0.27 0.43 0.37 0.44 0.43 0.43 0.46 0.47 1.00

21 / 69

The similarity matrix

Too uniform! Try zeroing values belowmedian

1.00 0.63 0.51 0.51 0.66 0.45 0.46 0.47 0.72 0.58 0.54 0.50 0.72 0.00 0.49 0.47 0.47 0.44 0.54 0.00 0.440.63 1.00 0.45 0.45 0.54 0.00 0.00 0.00 0.57 0.49 0.46 0.45 0.59 0.00 0.00 0.00 0.00 0.00 0.47 0.00 0.000.51 0.45 1.00 0.00 0.53 0.00 0.00 0.00 0.58 0.47 0.00 0.00 0.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.51 0.45 0.00 1.00 0.63 0.45 0.46 0.46 0.67 0.56 0.52 0.49 0.68 0.00 0.48 0.47 0.47 0.45 0.53 0.00 0.440.66 0.54 0.53 0.63 1.00 0.00 0.00 0.00 0.49 0.00 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.45 0.00 0.00 0.45 0.00 1.00 0.00 0.00 0.66 0.54 0.49 0.45 0.67 0.00 0.44 0.00 0.00 0.00 0.49 0.00 0.000.46 0.00 0.00 0.46 0.00 0.00 1.00 0.44 0.66 0.54 0.49 0.47 0.67 0.00 0.45 0.45 0.44 0.00 0.50 0.00 0.000.47 0.00 0.00 0.46 0.00 0.00 0.44 1.00 0.67 0.55 0.51 0.48 0.68 0.00 0.47 0.45 0.45 0.00 0.51 0.00 0.000.72 0.57 0.58 0.67 0.49 0.66 0.66 0.67 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.58 0.49 0.47 0.56 0.00 0.54 0.54 0.55 0.00 1.00 0.46 0.43 0.59 0.00 0.00 0.00 0.00 0.00 0.46 0.00 0.000.54 0.46 0.00 0.52 0.00 0.49 0.49 0.51 0.00 0.46 1.00 0.00 0.56 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.50 0.45 0.00 0.49 0.00 0.45 0.47 0.48 0.00 0.43 0.00 1.00 0.70 0.00 0.50 0.49 0.48 0.46 0.54 0.00 0.460.72 0.59 0.59 0.68 0.50 0.67 0.67 0.68 0.00 0.59 0.56 0.70 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.48 0.45 0.46 0.00 0.52 0.00 0.430.49 0.00 0.00 0.48 0.00 0.44 0.45 0.47 0.00 0.00 0.00 0.50 0.00 0.48 1.00 0.00 0.00 0.00 0.45 0.00 0.000.47 0.00 0.00 0.47 0.00 0.00 0.45 0.45 0.00 0.00 0.00 0.49 0.00 0.45 0.00 1.00 0.48 0.46 0.54 0.00 0.440.47 0.00 0.00 0.47 0.00 0.00 0.44 0.45 0.00 0.00 0.00 0.48 0.00 0.46 0.00 0.48 1.00 0.00 0.51 0.00 0.000.44 0.00 0.00 0.45 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.46 0.00 0.00 0.00 0.46 0.00 1.00 0.53 0.00 0.000.54 0.47 0.00 0.53 0.00 0.49 0.50 0.51 0.00 0.46 0.00 0.54 0.00 0.52 0.45 0.54 0.51 0.53 1.00 0.00 0.460.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.470.44 0.00 0.00 0.44 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.46 0.00 0.43 0.00 0.44 0.00 0.00 0.46 0.47 1.00

22 / 69

The graph

G = (V,E), weighted adjacencymatrixAA is likeB with zeroed low components

23 / 69

Modularity clustering“Modularity is the fraction of the edges that fall within a cluster minusthe expected fraction if edges were distributed at random.”

I “at random”= random graphs over same degree sequence

I degree sequence= (k1, . . . , kn) where ki = |N(i)|I “expected”= all possible “half-edge” recombinations

I expected edges between u, v: kukv/(2m) wherem = |E|I mod(u, v) = (Auv − kukv/(2m))

I mod(G) =∑

{u,v}∈E

mod(u, v)xuv

xuv = 1 if u, v in the same cluster and 0 otherwise

I “Natural extension” to weighted graphs: ku =∑v Auv ,m =

∑uv Auv

24 / 69

Usemodularity to de�ne clusteringI What is the “best clustering”?

I Maximize discrepancy between actual and expected“as far away as possible from average”

max∑

{u,v}∈Emod(u, v)xuv

∀u ∈ V, v ∈ V xuv ∈ {0, 1}

I Issue: trivial solution x = 1 “one big cluster”

I Idea: treat clusters as cliques (even if zero weight)then clique partitioning constraints for transitivity

∀i < j < k xij + xjk − xik ≤ 1

∀i < j < k xij − xjk + xik ≤ 1

∀i < j < k − xij + xjk + xik ≤ 1

∀{i, j} 6∈ E xij = 0

if i, j ∈ C and j, k ∈ C then i, k ∈ C25 / 69

The resulting clustering

cluster 1: job01, job02, job03, job05, job10

cluster 2: job04, job06, job22

cluster 3: job07, job08, job11, job12, job20

cluster 4: job13, job21, job23, job24, job25, job26, job27, job2826 / 69

Is it good?

Vinci Accenture Elektrobit Amazon 1-3Axa Expedia Google CSXDeezer fragment1 Ford WestrockAlstom Marriott MitreAledade Llamasoft Clarity

fragment2

I ?— named entities rarely appear in WordNetI Desirable property: chooses number of clusters

27 / 69

KeywordsMost frequent words w over collection C of documents d./keywords.pyglobal environment customers strategic processes teams sql job industry usejava developing project process engineering field models opportunity driveresults statistical based operational performance using mathematical computernew technical highly market company science role dynamic background productslevel methods design looking modeling manage learning service customereffectively technology requirements build mathematics problems plan servicestime scientist implementation large analytical techniques lead available plustechnologies sas provide machine product functions organization algorithmsposition model order identify activities innovation key appropriate differentcomplex best decision simulation strategy meet client assist quantitativefinance commercial language mining travel chain amazon pricing practicescloud supply

tfidfC(w, d) =|{t ∈ d | t = w}| |C||{e ∈ C | w ∈ e}|

keywordC(i, d) = wordw having ith best tfidfC(w, d)valuevecmC (d) = (tfidfC(keywordC(i, d), d) | i ≤ m)

Transforms documents to vectors28 / 69

Minimum sum-of-squares clustering

I MSSC, a.k.a. the k-means problemI Given points p1, . . . , pn ∈ Rm, �nd clustersC1, . . . , Ck

min∑j≤k

∑i∈Cj

‖pi − centroid(Cj)‖22

where centroid(Cj) = 1|Cj |

∑i∈Cj

pi

I k-means alg.: given initial clusteringC1, . . . , Ck

1: ∀j ≤ k compute yj = centroid(Cj)2: ∀i ≤ n, j ≤ k if yj is the closest centr. to pi let xij = 1 else 03: ∀j ≤ k updateCj ← {pi | xij = 1 ∧ i ≤ n}4: repeat until stability

29 / 69

k-means with k = 2

Vinci AXADeezer AlstomAccenture ElektrobitExpedia FordGoogle MarriottAledade Amazon 1-3Llamasoft CSX

WestRockMITREClarity

fragments 1-2

30 / 69

k-means with k = 2: another run

Deezer VinciElektrobit AXAGoogle AccentureAledade Alstom

ExpediaFord

MarriottLlamasoftAmazon 1-3

CSXWestRockMITREClarity

fragments 1-2

31 / 69

k-means with k = 2: third run!

AXA VinciDeezer AccentureExpedia AlstomFord ElektrobitMarriott GoogleLlamasoft AledadeAmazon 1-3CSXWestRockMITREClarityfragments 1-2

A �ckle algorithm

32 / 69

We can’t trust k-means

33 / 69

Let’s �ndMSSC’s global optimum!

minx,y,s

∑i≤n

∑j≤k‖pi − yj‖22 xij

∀j ≤ k 1sj

∑i≤n

pixij = yj

∀i ≤ n∑j≤k

xij = 1

∀j ≤ k∑i≤n

xij = sj

∀j ≤ k yj ∈ Rd

x ∈ {0, 1}nks ∈ Nk

(MSSC)

Nonconvex terms; continuous, binary and integer variables:sounds very di�cult!

34 / 69

ReformulationsThe (MSSC) formulation has the same optima as:

minx,y,P

∑i≤n

∑j≤k

Pij xij

∀i ≤ n, j ≤ k ‖pi − yj‖22 ≤ Pij∀j ≤ k

∑i≤n

pixij =∑i≤n

yjxij

∀i ≤ n∑j≤k

xij = 1

∀j ≤ k yj ∈ [mini≤n

pia,maxi≤n

pia | a ≤ d]

x ∈ {0, 1}nkP ∈ [0, PU ]nk

I Only nonconvexities:products of bounded by binary variables

I Caveat: cannot have empty clusters

35 / 69

Products of binary and continuous vars.

I Suppose term xy appears in a formulation

I Assume x ∈ {0, 1} and y ∈ [0, 1] is boundedI means “either z = 0 or z = y”I Replace xy by a new variable zI Adjoin the following constraints:

z ∈ [0, 1]

y − (1− x) ≤ z ≤ y + (1− x)−x ≤ z ≤ x

I ⇒Everything’s linear now!

36 / 69

Products of binary and continuous vars.

I Suppose term xy appears in a formulation

I Assume x ∈ {0, 1} and y ∈ [yL, yU ] is boundedI means “either z = 0 or z = y”I Replace xy by a new variable zI Adjoin the following constraints:

z ∈ [min(yL, 0),max(yU , 0)]

y − (1− x)max(|yL|, |yU |) ≤ z ≤ y + (1− x)max(|yL|, |yU |)−xmax(|yL|, |yU |) ≤ z ≤ xmax(|yL|, |yU |)

I ⇒Everything’s linear now!

37 / 69

MSSC is a convexMINLPmin

x,y,P,χ,ξ

∑i≤n

∑j≤k

χij

∀i ≤ n, j ≤ k 0 ≤ χij ≤ Pij∀i ≤ n, j ≤ k Pij − (1− xij)PU ≤ χij ≤ xijPU

∀i ≤ n, j ≤ k ‖pi − yj‖22 ≤ Pij ⇐ convex

∀j ≤ k∑i≤n

pixij =∑i≤n

ξij

∀i ≤ n, j ≤ k yj − (1− xij)max(|yL|, |yU |) ≤ ξij ≤ yj + (1− xij)max(|yL|, |yU |)

∀i ≤ n, j ≤ k − xij max(|yL|, |yU |) ≤ ξij ≤ xij max(|yL|, |yU |)

∀i ≤ n∑j≤k

xij = 1

∀j ≤ k yj ∈ [yL, yU ]

x ∈ {0, 1}nk

P ∈ [0, PU ]nk

χ ∈ [0, PU ]nk

∀i ≤ n, j ≤ k ξij ∈ [min(yL, 0),max(yU , 0)]

yj , ξij , yL, yU are vectors inRd38 / 69

How to solve it

I Encoding the problem: AMPLI cMINLP isNP-hard, no e�cient algorithmI Technologically advanced: Branch-and-BoundI Best (open source) solver: BonminI With k = 2, unfortunately. . .

Cbc0010I After 8300 nodes, 3546 on tree, 14.864345 best solution,best possible 6.1855969 (32142.17 seconds)

I Interesting feature: the boundguarantees we can’t to better than boundall BB algorithms provide it

39 / 69

Bonmin’s �rst solution

Alstom VinciElektrobit AXAFord DeezerLlamasoft AccentureAmazon 2 ExpediaCSX GoogleMITRE AledadeClarity Marriottfragment 2 Amazon 1 & 3

WestRockfragment 1

40 / 69

Couple of things left to try

I Approximate `2 by `1 norm`1 is a linearizable norm

I Randomly project the datalose dimensions but keep approximate shape

41 / 69

Linearizing convexityI Replace ‖pi − yj‖22 by ‖pi − yj‖1I Warning: optima will change

but still within “clustering by distance” principle

∀i ≤ n, j ≤ k ‖pi − yj‖1 =∑a≤d|pia − yja|

I Replace each | · | term by new vars.Qija ∈ [0, PU ]Adjust PU in terms of ‖ · ‖1

I Adjoin constraints

∀i ≤ n, j ≤ k∑a≤d

Qija ≤ Pij

∀i ≤ n, j ≤ k, a ≤ d −Qija ≤ pia − yja ≤ Qija

I Obtain aMILPMost advancedMILP solver: CPLEX

42 / 69

CPLEX’s �rst solution

objective 112.24, bound 39.92, in 44.74s

AXA VinciDeezer AccentureFord AlstomMarriott ExpediaAmazon 1-3 ElektrobitLlamasoft GoogleCSX AledadeWestRockMITREClarityfragments 1-2

Interrupted after 281s with bound 59.68

43 / 69

Themagic of random projections

I Very advanced theoretical frameworkI Truly a piece of “mathematics of big data”I In a nutshell

I Clustering on q rather than pyields approx. same results with high probability

44 / 69

Themagic of random projections

I Very advanced theoretical frameworkI Truly a piece of “mathematics of big data”I In a nutshell

1. Given points pi, . . . , pn ∈ Rd with d large and ε ∈ (0, 1)

2. Pick “appropriate” k ≈ O( 1ε2

log n)

3. Sample k × dmatrixA (each comp. i.i.d.N (0, 1√k))

4. Consider projected points qi = Api ∈ Rk for i ≤ n

5. With prob→ 1 exponentially fast as k →∞

∀i, j ≤ n (1−ε)‖pi−pj‖2 ≤ ‖qi−qj‖2 ≤ (1+ε)‖pi−pj‖2

45 / 69

Bonmin on randomly proj. dataobjective 5.07, bound 0.48, stopped at 180s

Deezer VinciFord AXAAmazon 1-3 AccentureCSX AlstomMITRE Expediafragment 1 Elektrobit

GoogleAledadeMarriottLlamasoftWestRockClarity

fragment 2

46 / 69

CPLEX on randomly proj. data

objective 53.19, bound 20.68, stopped at 180s

Vinci AXADeezer AccentureExpedia AlstomGoogle ElektrobitAledade MarriottFord LlamasoftAmazon 1-3 WestRockCSX MITREClarity fragment 1-2

47 / 69

Many clusterings

This ain’t �nished. . .I We obtainedmany di�erent clusteringsI Is there any common sense?I How do we compare them?I Can we extract useful information from thecomparison?

I Howmany clusters should we look for? Is k = 2OK?I Did we just turn the issue of “I have too many data” into“I have too many solutions”?

48 / 69

Mix all clustersI Clustering Ck = {Cki | i ≤ nk}I For (h, i) < (k, j) compute cluster similarity

σhkij =|Chi ∩ Ckj||Chi ∪ Ckj|

I Solve a bunch of assignment problems:

∀h < k

max∑i≤nh

∑j≤nk

σhkij xhkij

∀i ≤ nh∑j≤nk

xhkij = 1

∀j ≤ nk∑i≤nh

xhkij = 1

∀i ≤ nh, j ≤ nk xhkij ∈ [0, 1]

I Get integral solutions because of total unimodularity49 / 69

Our clusterings# clustering 1set C[1,1] := "Vinci" "Axa" "Deezer" "Alstom" "Aledade";set C[1,2] := "Accenture" "Expedia" "fragment1";set C[1,3] := "Elektrobit" "Google" "Ford" "Marriott" "Llamasoft";set C[1,4] := "Amazon1" "Amazon2" "Amazon3" "CSX" "Westrock" "Mitre" "Clarity" "fragment2";# clustering 2set C[2,1] := "Vinci" "Deezer" "Accenture" "Expedia" "Google" "Aledade" "Llamasoft";set C[2,2] := "Axa" "Alstom" "Elektrobit" "Ford" "Marriott" "Amazon1" "Amazon2" "Amazon3" "CSX" "Westrock"

"Mitre" "Clarity" "fragment1" "fragment2";# clustering 3set C[3,1] := "Deezer" "Elektrobit" "Google" "Aledade";set C[3,2] := "Vinci" "Axa" "Accenture" "Alstom" "Expedia" "Ford" "Marriott" "Llamasoft" "Amazon1" "Amazon2"

"Amazon3" "CSX" "Westrock" "Mitre" "Clarity" "fragment1" "fragment2";# clustering 4set C[4,1] := "Axa" "Deezer" "Expedia" "Ford" "Marriott" "Llamasoft" "Amazon1" "Amazon2" "Amazon3" "CSX"

"Westrock" "Mitre" "Clarity" "fragment1" "fragment2";set C[4,2] := "Vinci" "Accenture" "Alstom" "Elektrobit" "Google" "Aledade";# clustering 5set C[5,1] := "Alstom" "Elektrobit" "Ford" "Llamasoft" "CSX" "Mitre" "Clarity" "fragment2";set C[5,2] := "Vinci" "Axa" "Deezer" "Accenture" "Expedia" "Google" "Aledade" "Marriott" "Amazon1" "Amazon2"

"Amazon3" "Westrock";# clustering 6set C[6,1] := "Axa" "Deezer" "Ford" "Marriott" "Amazon1" "Amazon2" "Amazon3" "Llamasoft" "CSX" "Westrock"

"Mitre" "Clarity" "fragment1" "fragment2";set C[6,2] := "Vinci" "Accenture" "Alstom" "Expedia" "Elektrobit" "Google" "Aledade";# clustering 7set C[7,1] := "Deezer" "Ford" "Amazon1" "Amazon2" "Amazon3" "CSX" "Mitre" "fragment1";set C[7,2] := "Vinci" "Axa" "Accenture" "Alstom" "Expedia" "Elektrobit" "Google" "Aledade" "Marriott"

"Llamasoft" "Westrock" "Clarity" "fragment2";# clustering 8set C[8,1] := "Vinci" "Deezer" "Expedia" "Google" "Aledade" "Ford" "Amazon1" "Amazon2" "Amazon3" "CSX" "Clarity";set C[8,2] := "Axa" "Accenture" "Alstom" "Elektrobit" "Marriott" "Westrock" "Mitre" "fragment1" "fragment2";

50 / 69

The similarity graph

Some similar clusters from di�erent clusterings51 / 69

Send your CVs to. . .

C22 ∩ C32 ∩ C41 ∩ C61 =

{ Axa, Marriott, Amazon1-3, CSX,Mitre, Ford, Westrock, Clarity }

52 / 69

Section 3

Once you’re hired

53 / 69

The twomost useful problems of ’em all

I Transportation problemI Flow problem

54 / 69

The transportation problem

Transport produce up to supply s to satisfy demand dI You’ve seen it before with all unit supplies and demand

the assignment problemI Formulation: xij = quantity on arc (i, j)

minx≥0

∑ij cijxij

∀i∑

j xij ≤ si∀j

∑i xij ≥ dj

55 / 69

ExampleI Youmanage software licenses for a bunch of clientsI Savings idea: if clientA not using license L,B can use itI Exploitation: “transport” license supply to client demandI It seemed like a good idea, but. . .

. . . this really happened: no cij ’s!I Acquisitionmanager had negotiated for a “huge discount”

but unit prices were being kept secret by sales

Real corporate life is more irrational and stupid than Dilbert56 / 69

The �ow problem

Given a digraph G = (V,A) with an arc capacity functionc : A → R+ and two distinct nodes s, t ∈ V , �nd the �owfrom s to t havingmaximum value

I GivenG = (V,A, c, s, t) a �ow from s to t is a functionf : A→ R+ s.t.:

∀v ∈ V r {s, t}∑

u∈N−(v)

fuv =∑

w∈N+(v)

fvw

∀(u, v) ∈ A fuv ≤ cuv

I The value of a �ow f is given by∑

v∈N+(s)

fsv

Defn.:N−(v) = {u ∈ V | (u, v) ∈ A},N+(v) = {w ∈ V | (v, w) ∈ A}

57 / 69

Multicommodity �owI Many di�erent �ows on the same networkI GivenN = (V,A, c, s, t) where:

I G = (V,A) is a digraphI c : A→ R+ is an arc capacity functionI s, t ∈ V r s.t.∀k ≤ r (sk 6= tk)

I Find a set of �ows {fk | k ≤ r} from sk to tkI havingmax. total valueI satisfying arc capacity

58 / 69

LP FormulationI Maximize total value:

maxf≥0

∑k≤r

∑v∈N+(sk)

fkskv

I Satisfy �ow equations:

∀k ≤ r, v ∈ V r {sk, tk}∑

u∈N−(v)

fkuv =∑

w∈N+(v)

fkvw

I Satisfy arc capacity:

∀(u, v) ∈ A∑k≤r

fkuv ≤ cuv

LP: we can solve very large instances59 / 69

Minimum cost �ows

I Flow equations de�ne connected subgraphs:G connected⇒∀u 6= v ∈ V (G) a unit of �ow entering uwill exit uas long as “demand”= 0 at intermediate nodes.Conversely: if there is a �ow from u to v thenGmust be connected

I E.g. a SP s→ t is the connected subgraph of minimum costcontaining s, t:

minx:A→R

∑(u,v)∈A

cuvxuv

∀v ∈ V∑

(u,v)∈A

xuv −∑

(v,u)∈A

xvu =

−1 u = s1 u = t0 othw.

∀(u, v) ∈ A xuv ∈ [0, 1]

[SP]

60 / 69

Telecom

An internetproviderusedhistoricaldata to estimatea traf-�c matrix T = (Tij), such that Tij is the typical demand be-tween two nodes i, j of its network digraphG = (V,A). It hasa contract with the backbone provider that limits the capac-ity (inGbs) on each arc (i, j) ∈ A to cij ; the same contract alsoregulates the cost per Gbs, set to γij

Model theproblemof�nding the feasiblemulti�owofmin-imumcost that satis�es eachdemandbetween sourceanddes-tination.

61 / 69

LogisticsA truck-based transportation company needs to plan the

routes for the incomingweek. The demands are given as a list((sk, tk, dk) | k ≤ r) where dk trucks have to be dispatchedfromnode sk to node tk . The capacities cuv on the arcs (u, v) ∈A are estimated using tra�c data, and the operations cost areestimated to 100$ per Km.

1. Model the problem, assuming the company has enoughtrucks to cover every demand

2. Adjust the problem to the situation where the companyhas su�cient trucks to satisfy half of the total demand,and has to rent the others: the operations costs for therented trucks are 200$ per Km.

3. Suggest a way to e�ciently compute a lower bound onthe total cost.

62 / 69

Air courierThe air branch of a shipping company uses a �eet of Boeing 777s

and 747s cargo to serve the EMEA demands. A 777 can carry 653m3 in volume and 103 tonnes (t) in weight. A 747 can carry 854.5m3 and 134.2 t. Each freighter is dedicated to a single segment (ori-gin to destination airport and back once a day: both �ights hap-pen within the same 24 hours). The demand matrix is extremely�ne-grained, and consists of all order IDs (packages) for the week,with origin and destination airports, weight and volume. The net-work consists of airports, linked by the segments that are actually�own. Theper-mile cost of �ying is a linearly increasing functionofthe loaded weight (the two functions are di�erent for 777 and 747).Flights can leave empty (in which case the company subcontractsthe �ight); company policy states that, if loaded, the loaded volumehas to �ll at least half the capacity. Model the corresponding vari-ant of the multicommodity �ow problem.

63 / 69

Interest of the Air courier problem

I Need binary vars to to model conditionsNo longer an LP⇒ can’t solve huge instances

I Variables for each demand in thematrix:too large to be solved

I Aggregate by origin/destination:waste mass or volume capacity

64 / 69

This slide intentionally left blank

65 / 69

Take-awaymessages

I Optimization is at the heart of most problems todayLearnMathematical Programming modelling

I Learn techniques you can deploy in practice1. AMPL or GAMS + CPLEX / Baron (commercial) or Bonmin / Couenne2. GLPK (free, with “AMPL”-like LP/MILP dialect) / SCIP3. Python + cvxopt / optlang / “raw interfaces” (useful for SDP)4. Julia + JuMP

I Trymany con�gurations/techniques/solversI Get out of your comfort zone, never stop learningI’ve had to learn Natural Language Processing to write this tutorial!

66 / 69

Lastly,

I Steer clear of banks!I They’ll listen to your pitch. . .. . .understand your ideas. . .

I . . .and then they’ll hire a fresh PhD just out of graduateschool, who costs ε, to implement the ideas you gavethem for free!

. . .and they’ll also rub it in your face

67 / 69

Again, a blank slide, just to breathe at the end

68 / 69

Tutorial

I Python (install it yourselves)I Somemodules:

I OptLang / GLPK, matplotlib, networkxI NLTK, MITIE, pattern, sklearn

I AMPL& solversI Win64: http://tinyurl.com/hpwgbf3I MacOsX 64: http://tinyurl.com/zf2oygbI Linux 64: http://tinyurl.com/gqzdf9m

I If it’s slow, it’s that your all downloading together — wait sometime before retrying

I AMPL licenses valid until AprilAMPL is commercial

I My code: http://tinyurl.com/z6nvhj5

69 / 69