1.International Series in Operationswww.springer.com/series/6161Research & Management ScienceVolume 170Stanford University, A, USASeries Editor:Frederick S. HillierCFor further volumesS ditorpecial E ial Consultant:Camille C. PriceStephen F. Austin State University, TX, USA

2. John N. HookerIntegrated Methodsfor OptimizationSecond Edition 3. with any form of information storage and retrieval, electronic adaptation, computer software, or by similaror dissimilar methodology now known or hereafter developed is forbidden.The use in this publication of trade names, trademarks, service marks, and similar terms, even if they arenot identied as such, is not to be taken as an expression of opinion as to whether or not they are subjectto proprietary rights.Printed on acid-free paperpermission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NYSpringer is part of Springer Science+Business Media (www.springer.com)10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connectionISSN 0884-8289ISBN 978-1-4614-1899-3 e-ISBN 978-1-4614-1900-6DOI 10.1007/978-1-4614-1900-6Springer New York Dordrecht Heidelberg LondonLibrary of Congress Control Number: 2011940223John N. HookerTepper School of BusinessCarnegie Mellon UniversityPittsburgh PennsylvaniaUSAjohn@hooker.tepper.cmu.edu Springer Science+Business Media, LLC 2012All rights reserved. This work may not be translated or copied in whole or in part without the written, 4. PrefaceOptimization has become a versatile tool in a wide array of applicationareas, ranging from manufacturing and information technology to thesocial sciences. Methods for solving optimization problems are equallynumerous and provide a large reservoir of problem-solving technology.In fact, there is such a variety of methods that it is dicult to takefull advantage of them. They are described in dierent technical lan-guages and are implemented in dierent software packages. Many arenot implemented at all. It is hard to tell which one is best for a givenproblem, and there is too seldom an opportunity to combine techniquesthat have complementary strengths.The ideal would be to bring these methods under one roof, so thatthey and their combinations are all available to solve a problem. Asit turns out, many of them share, at some level, a common problem-solving strategy. This opens the door to integrationto the design of amodeling and algorithmic framework within which dierent techniquescan work together in a principled way.This book undertakes such a project. It deals primarily with the uni-cation of mathematical programming and constraint programming,since this has been the focus of most recent research on integratedmethods. Mathematical programming brings to the table its sophisti-cated relaxation techniques and concepts of duality. Constraint pro-gramming contributes its inference and propagation methods, alongwith a powerful modeling approach. It is possible to have all of theseadvantages at once, rather than being forced to choose between them.Continuous global optimization and heuristic methods can also bebrought into the framework.v 5. vi PrefaceThe book is intended for those who wish to learn about optimizationfrom an integrated point of view, including researchers, software devel-opers, and practitioners. It is also for postgraduate students interestedin a unied treatment of the eld. It is written as an advanced text-book, with exercises, that develops optimization concepts from theground up. It takes an interdisciplinary approach that presupposesmathematical sophistication but no specic knowledge of either math-ematical programming or constraint programming.The choice of topics is guided by what is relevant to understand-ing the principles behind popular linear, mixed integer, and constraintprogramming solversand more importantly, integrated solvers of thepresent and foreseeable future. On the mathematical programmingside, it presents the basic theory of linear and integer programming,cutting planes, Lagrangean and other types of duality, mixed integermodeling, and polyhedral relaxations for a wide range of combinato-rial constraints. On the constraint programming side it discusses con-straint propagation, domain ltering, consistency, global constraints,and modeling techniques. The material ranges from the classical to thevery recent, with some results presented here for the rst time.The ideas are tied together by a search-infer-and-relax algorithmicframework, an underlying theory of inference and relaxation duality,and the use of metaconstraints (a generalization of global constraints)for modeling.The rst edition of the book was published only four years ago, butthe eld has moved ahead. This second edition expands, reorganizes,and updates the earlier edition in several ways. The examples thatbegan the rst book now occupy a separate chapter, followed by twonew chapters. A chapter on optimization basics makes the book morenearly self-contained. A second new chapter on duality presents astronger case for its centrality and provides conceptual background forthe search chapter that follows, which is much expanded. The chapteron inference covers additional global constraints, and the dictionaryof metaconstraints in the nal chapter has been enlarged. The mate-rial throughout has been updated and elaborated where appropriate,resulting in some 170 new references.I would like to acknowledge the many collaborators and formerstudents from whom I have learned much about integrated problemsolving. They include Henrik Andersen, Kim Allen Andersen, Ionut.Aron, David Bergman, Alexander Bockmayr, Endre Boros, SrinivasBollapragada, Jonathan Cagan, Vijay Chandru, Andre Cire, Elvin 6. Preface viiCoban, Milind Dawande, Giorgio Gallo, Latife Genc Kaya, OmarGhattas, Ignacio Grossmann, Tarik Hadzic, Peter Hammer, SamidHoda, Willem-Jan van Hoeve, Hak-Jin Kim, Maria Auxilio Osorio,Barry OSullivan, Greger Ottosson, Gabriella Rago, Ramesh Raman,Erlendur Thorsteinsson, Peter Tiedemann, H. Paul Williams, HongYan, and Tallys Yunes. 7. Contents1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 A Unifying Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Modeling to Reveal Problem Structure . . . . . . . . . . . . . . . 61.3 The Role of Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.4 Advantages of Integrated Methods . . . . . . . . . . . . . . . . . . 101.5 Some Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.6 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131.7 Plan of the Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151.8 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.1 Basic Denitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202.2 The Solution Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.3 Freight Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.3.1 Formulating the Problem . . . . . . . . . . . . . . . . . . . . . 232.3.2 Inference: Bounds Propagation . . . . . . . . . . . . . . . . 232.3.3 Inference: Valid Inequalities . . . . . . . . . . . . . . . . . . . 242.3.4 Relaxation: Linear Programming . . . . . . . . . . . . . . 252.3.5 Branching Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.4 Production Planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.4.1 Formulating the Problem . . . . . . . . . . . . . . . . . . . . . 302.4.2 Relaxation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.4.3 Branching Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.4.4 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342.5 Employee Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34ix 8. x Contents2.5.1 Formulating the Problem . . . . . . . . . . . . . . . . . . . . . 352.5.2 Inference: Domain Filtering . . . . . . . . . . . . . . . . . . . 382.5.3 Inference for Variable Indices . . . . . . . . . . . . . . . . . . 39Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412.6 Continuous Global Optimization . . . . . . . . . . . . . . . . . . . . 422.6.1 Inference: Bounds Propagation . . . . . . . . . . . . . . . . 442.6.2 Relaxation: Factored Functions . . . . . . . . . . . . . . . . 442.6.3 Branching Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472.7 Product Conguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472.7.1 Formulating the Problem . . . . . . . . . . . . . . . . . . . . . 482.7.2 Inference: Indexed Linear Constraint . . . . . . . . . . . 502.7.3 Relaxation: Indexed Linear Constraint . . . . . . . . . 512.7.4 Branching Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532.8 Planning and Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . 532.8.1 Formulating the Problem . . . . . . . . . . . . . . . . . . . . . 542.8.2 Relaxation: The Master Problem . . . . . . . . . . . . . . 562.8.3 Inference: Benders Cuts . . . . . . . . . . . . . . . . . . . . . . 58Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602.9 Routing and Frequency Assignment . . . . . . . . . . . . . . . . . 612.10 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643 Optimization Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653.1 Linear Programming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653.1.1 Optimality Conditions . . . . . . . . . . . . . . . . . . . . . . . 663.1.2 Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 693.1.3 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 733.1.4 Feasibility Conditions . . . . . . . . . . . . . . . . . . . . . . . . 74Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 753.2 Network Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 773.2.1 Basis Tree Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 793.2.2 Optimality Conditions . . . . . . . . . . . . . . . . . . . . . . . 813.2.3 Network Simplex Method . . . . . . . . . . . . . . . . . . . . . 833.2.4 Maximum Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 853.2.5 Bipartite Matching . . . . . . . . . . . . . . . . . . . . . . . . . . 903.2.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 923.3 Nonlinear Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 933.3.1 Local Optimality Conditions . . . . . . . . . . . . . . . . . . 943.3.2 Global Optimality Conditions . . . . . . . . . . . . . . . . . 97 9. Contents xiExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1003.4 Dynamic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1003.4.1 State Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1013.4.2 Recursive Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . 1043.4.3 Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1063.5 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1074 Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1094.1 Inference Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1104.1.1 Weak and Strong Duality . . . . . . . . . . . . . . . . . . . . . 1114.1.2 Certicates and Problem Complexity . . . . . . . . . . . 1124.1.3 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 1134.1.4 Constraint-Directed Search . . . . . . . . . . . . . . . . . . . 114Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1154.2 Relaxation Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1184.3 Linear Programming Duality . . . . . . . . . . . . . . . . . . . . . . . 1184.3.1 Inference Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1184.3.2 Dual Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . 1214.3.3 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 1234.3.4 Relaxation Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1264.4 Surrogate Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1274.4.1 Inference Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1284.4.2 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 1294.4.3 Relaxation Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1304.5 Lagrangean Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1304.5.1 Inference Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1314.5.2 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 1334.5.3 Relaxation Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1344.5.4 Lagrangean Relaxation for LP . . . . . . . . . . . . . . . . . 1364.5.5 Example: Generalized Assignment Problem . . . . . 1384.5.6 Solving the Lagrangean Dual . . . . . . . . . . . . . . . . . . 139Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1404.6 Subadditive Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1414.6.1 Inference Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1424.6.2 Subadditive Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1444.6.3 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 146 10. xii Contents4.6.4 Relaxation Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1484.7 Branching Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1484.7.1 Inference Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1494.7.2 A Value-Bounding Function . . . . . . . . . . . . . . . . . . . 1514.7.3 General Sensitivity Analysis. . . . . . . . . . . . . . . . . . . 1544.7.4 Relaxation Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1584.8 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1595 Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1615.1 Branching Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1625.1.1 Branch-Infer-and-Relax Methods . . . . . . . . . . . . . . 1635.1.2 Node Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1655.1.3 Variable and Branch Point Selection . . . . . . . . . . . 1685.1.4 Cost-Based Branching . . . . . . . . . . . . . . . . . . . . . . . . 1705.1.5 Primal Heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1725.1.6 Branch and Price . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1745.1.7 Example: Airline Crew Scheduling . . . . . . . . . . . . . 175Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1825.2 Constraint-Directed Search . . . . . . . . . . . . . . . . . . . . . . . . . 1835.2.1 The Search Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 1845.2.2 Logic-Based Benders Decomposition . . . . . . . . . . . 1875.2.3 Constraint-Directed Branching . . . . . . . . . . . . . . . . 1895.2.4 Example: Propositional Satisability . . . . . . . . . . . 1935.2.5 Implication Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . 1975.2.6 Partial-Order Dynamic Backtracking . . . . . . . . . . . 2005.2.7 Example: Propositional Satisability . . . . . . . . . . . 202Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2035.3 Local Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2055.3.1 Some Popular Metaheuristics. . . . . . . . . . . . . . . . . . 2065.3.2 Local Search Conceived as Branching . . . . . . . . . . 2085.3.3 Relaxation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2115.3.4 Example: Single-Vehicle Routing . . . . . . . . . . . . . . 2125.3.5 Constraint-Directed Local Search . . . . . . . . . . . . . . 2165.3.6 Example: Single-Vehicle Routing . . . . . . . . . . . . . . 217Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2185.4 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 11. Contents xiii6 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2236.1 Completeness and Consistency . . . . . . . . . . . . . . . . . . . . . . 2246.1.1 Basic Denitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2246.1.2 Domain Consistency . . . . . . . . . . . . . . . . . . . . . . . . . 2256.1.3 Bounds Consistency. . . . . . . . . . . . . . . . . . . . . . . . . . 2276.1.4 k-Completeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2286.1.5 k-Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2306.1.6 Backtracking and Width . . . . . . . . . . . . . . . . . . . . . . 230Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2326.2 Linear Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2336.2.1 Domain and Bounds Consistency . . . . . . . . . . . . . . 2336.2.2 k-Completeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2346.2.3 Domain Reduction with Dual Multipliers . . . . . . . 2366.2.4 Classical Benders Cuts . . . . . . . . . . . . . . . . . . . . . . . 238Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2406.3 General Inequality Constraints . . . . . . . . . . . . . . . . . . . . . . 2416.4 Propositional Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2436.4.1 Logical Clauses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2446.4.2 A Complete Inference Method . . . . . . . . . . . . . . . . . 2456.4.3 Unit Resolution and Horn Clauses . . . . . . . . . . . . . 2476.4.4 Domain Consistency and k-Completeness . . . . . . . 2476.4.5 Strong k-Consistency. . . . . . . . . . . . . . . . . . . . . . . . . 2496.4.6 Parallel Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . 250Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2526.5 0-1 Linear Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2546.5.1 Implication between Inequalities . . . . . . . . . . . . . . . 2546.5.2 Implication of Logical Clauses . . . . . . . . . . . . . . . . . 2576.5.3 Implication of Cardinality Clauses . . . . . . . . . . . . . 2596.5.4 0-1 Resolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2616.5.5 Domain Consistency and k-Completeness . . . . . . . 2636.5.6 Strong k-Consistency. . . . . . . . . . . . . . . . . . . . . . . . . 265Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2666.6 Integer Linear Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . 2676.6.1 Benders Cuts from the Subadditive Dual . . . . . . . 2676.6.2 Benders Cuts from the Branching Dual . . . . . . . . . 2686.6.3 Constraint-Directed Branching . . . . . . . . . . . . . . . . 272Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2756.7 The Element Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . 2766.7.1 Domain Consistency . . . . . . . . . . . . . . . . . . . . . . . . . 2776.7.2 Bounds Consistency. . . . . . . . . . . . . . . . . . . . . . . . . . 279 12. xiv ContentsExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2816.8 The All-Dierent Constraint . . . . . . . . . . . . . . . . . . . . . . . . 2816.8.1 Bipartite Matching Formulation . . . . . . . . . . . . . . . 2826.8.2 Domain Consistency . . . . . . . . . . . . . . . . . . . . . . . . . 2836.8.3 Bounds Consistency. . . . . . . . . . . . . . . . . . . . . . . . . . 286Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2896.9 The Cardinality and Nvalues Constraints . . . . . . . . . . . . . 2896.9.1 The Cardinality Constraint . . . . . . . . . . . . . . . . . . . 2896.9.2 Network Flow Model for Cardinality . . . . . . . . . . . 2906.9.3 Domain Consistency . . . . . . . . . . . . . . . . . . . . . . . . . 2916.9.4 The Nvalues Constraint . . . . . . . . . . . . . . . . . . . . . . 292Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2936.10 The Among and Sequence Constraints . . . . . . . . . . . . . . . 2946.10.1 The Among Constraint . . . . . . . . . . . . . . . . . . . . . . . 2946.10.2 The Sequence Constraint . . . . . . . . . . . . . . . . . . . . . 2956.10.3 Filtering Based on Cumulative Sums . . . . . . . . . . . 2966.10.4 Flow-Based Filtering . . . . . . . . . . . . . . . . . . . . . . . . . 300Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3036.11 The Stretch Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3046.11.1 Dynamic Programming Model . . . . . . . . . . . . . . . . . 3056.11.2 Domain Consistency . . . . . . . . . . . . . . . . . . . . . . . . . 306Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3116.12 The Regular Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3116.12.1 Determistic Finite Automata . . . . . . . . . . . . . . . . . . 3126.12.2 Domain Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3146.12.3 Filtering by Decomposition . . . . . . . . . . . . . . . . . . . 3156.12.4 Nondeterministic Finite Automata . . . . . . . . . . . . . 3186.12.5 Cyclic Regular Constraint . . . . . . . . . . . . . . . . . . . . 3196.12.6 A Dynamic Programming Constraint . . . . . . . . . . . 320Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3216.13 The Circuit Constraint. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3216.13.1 Modeling with Circuit . . . . . . . . . . . . . . . . . . . . . . . . 3226.13.2 Elementary Filtering Methods . . . . . . . . . . . . . . . . . 3246.13.3 Filtering Based on Separators . . . . . . . . . . . . . . . . . 3256.13.4 Network Flow Model . . . . . . . . . . . . . . . . . . . . . . . . . 328Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3296.14 Disjunctive Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3306.14.1 Edge Finding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3316.14.2 Not-First/Not-Last Rules . . . . . . . . . . . . . . . . . . . . . 3356.14.3 Benders Cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 13. Contents xvExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3476.15 Cumulative Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3496.15.1 Edge Finding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3506.15.2 Extended Edge Finding . . . . . . . . . . . . . . . . . . . . . . 3566.15.3 Not-First/Not-Last Rules . . . . . . . . . . . . . . . . . . . . . 3586.15.4 Energetic Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . 3596.15.5 Benders Cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3646.16 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3657 Relaxation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3717.1 Piecewise Linear Functions . . . . . . . . . . . . . . . . . . . . . . . . . 373Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3757.2 0-1 Linear Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3767.2.1 ChvatalGomory Cuts . . . . . . . . . . . . . . . . . . . . . . . 3787.2.2 0-1 Knapsack Cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . 3817.2.3 Sequential Lifting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3827.2.4 Sequence-Independent Lifting . . . . . . . . . . . . . . . . . 3857.2.5 Set-Packing Inequalities . . . . . . . . . . . . . . . . . . . . . . 387Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3897.3 Integer Linear Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . 3917.3.1 ChvatalGomory Cuts . . . . . . . . . . . . . . . . . . . . . . . 3927.3.2 Gomory Cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3947.3.3 Mixed-Integer Rounding Cuts . . . . . . . . . . . . . . . . . 3997.3.4 Separating Mixed-Integer Rounding Cuts . . . . . . . 4017.3.5 Integral Polyhedra . . . . . . . . . . . . . . . . . . . . . . . . . . . 403Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4067.4 Disjunctions of Linear Systems . . . . . . . . . . . . . . . . . . . . . . 4087.4.1 Convex Hull Relaxation . . . . . . . . . . . . . . . . . . . . . . 4087.4.2 Big-M Relaxation . . . . . . . . . . . . . . . . . . . . . . . . . . . 4117.4.3 Disjunctions of Linear Inequalities . . . . . . . . . . . . . 4137.4.4 Disjunctions of Linear Equations . . . . . . . . . . . . . . 4167.4.5 Separating Disjunctive Cuts. . . . . . . . . . . . . . . . . . . 417Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4217.5 Disjunctions of Nonlinear Systems . . . . . . . . . . . . . . . . . . . 4237.5.1 Convex Hull Relaxation . . . . . . . . . . . . . . . . . . . . . . 4247.5.2 Big-M Relaxation . . . . . . . . . . . . . . . . . . . . . . . . . . . 428Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4287.6 Mixed-Integer Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4297.6.1 Mixed-Integer Representability . . . . . . . . . . . . . . . . 430 14. xvi Contents7.6.2 Example: Fixed-Charge Function . . . . . . . . . . . . . . 4347.6.3 Disjunctive Models . . . . . . . . . . . . . . . . . . . . . . . . . . 4387.6.4 Knapsack Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4477.7 Propositional Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4507.7.1 Common Logical Formulas . . . . . . . . . . . . . . . . . . . . 4507.7.2 Resolution as a Tightening Technique . . . . . . . . . . 4557.7.3 Refutation by Linear Relaxation . . . . . . . . . . . . . . . 4587.7.4 Input Resolution and Rank 1 Cuts . . . . . . . . . . . . . 4597.7.5 Separating Resolvents . . . . . . . . . . . . . . . . . . . . . . . . 464Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4667.8 The Element Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . 4687.8.1 Convex Hull Relaxations . . . . . . . . . . . . . . . . . . . . . 4697.8.2 Big-M Relaxations . . . . . . . . . . . . . . . . . . . . . . . . . . 4737.8.3 Vector-Valued Element . . . . . . . . . . . . . . . . . . . . . . . 474Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4777.9 The All-Dierent Constraint . . . . . . . . . . . . . . . . . . . . . . . . 4787.9.1 Convex Hull Relaxation . . . . . . . . . . . . . . . . . . . . . . 4797.9.2 Convex Hull Mixed-Integer Formulation . . . . . . . . 4847.9.3 Relaxing the Objective Function . . . . . . . . . . . . . . . 4867.9.4 Alldi Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4917.10 The Cardinality Constraint . . . . . . . . . . . . . . . . . . . . . . . . . 4927.10.1 Convex Hull Relaxation . . . . . . . . . . . . . . . . . . . . . . 4927.10.2 Convex Hull Mixed-Integer Formulation . . . . . . . . 494Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4957.11 The Circuit Constraint. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4967.11.1 0-1 Programming Model . . . . . . . . . . . . . . . . . . . . . . 4977.11.2 Continuous Relaxations . . . . . . . . . . . . . . . . . . . . . . 4977.11.3 Comb Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . 499Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5027.12 Disjunctive Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5037.12.1 Disjunctive Relaxations . . . . . . . . . . . . . . . . . . . . . . 5047.12.2 Mixed-Integer Relaxations . . . . . . . . . . . . . . . . . . . . 5067.12.3 A Class of Valid Inequalities . . . . . . . . . . . . . . . . . . 508Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5117.13 Cumulative Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5127.13.1 Mixed-Integer Models . . . . . . . . . . . . . . . . . . . . . . . . 5127.13.2 A Class of Valid Inequalities . . . . . . . . . . . . . . . . . . 5177.13.3 Relaxation of Benders Subproblems . . . . . . . . . . . . 520 15. Contents xviiExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5307.14 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5318 Dictionary of Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . 5350-1 linear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536All-dierent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537Among. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538Bin Packing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539Cardinality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540Cardinality Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541Cardinality Conditional . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543Circuit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543Clique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544Conditional . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544Cumulative Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545Cutset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547Din . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547Disjunctive Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550Indexed Linear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551Integer Linear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552Knapsack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553Lex Greater . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554Linear Disjunction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555Lot Sizing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556Mixed Integer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557Min-n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558Network Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558Nonlinear Disjunction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559Nvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 560Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561Piecewise Linear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562Range . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562Regular . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563Roots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564Same . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565 16. xviii ContentsSequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565Set Covering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566Set Packing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567Soft Alldi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568Sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569Spanning tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569Spread . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571Stretch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572Symmetric Alldi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573Symmetric Cardinality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573Value Precedence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621 17. Chapter 1IntroductionOptimization calls for a multifaceted approach. Some classes of opti-mization problems, such as linear programming models, can be solvedby an all-purpose method. However, most problems require individualattention. The secret to solving a problem is to take advantage of itsparticular structure.The result is a proliferation of optimization methods. Thousands ofjournal articles address narrowly dened problems, as they must if theproblems are to be solved in a reasonable amount of time. Not only this,but the articles are distributed across several literatures that speakdierent technical languages. Chief among these are the literaturesof mathematical programming, constraint programming, continuousglobal optimization, and heuristic methods. This imposes an obviousburden on anyone who seeks the right method and software to solve agiven problem.Some of the individual disciplines have made signicant progresstoward developing robust, general-purpose solvers. Each generation ofmathematical programming software solves a wider range of problems,and similarly for constraint programming and global optimizationsoftware. Yet even these solvers fail to exploit most problem-specicmethodswith the partial exception of constraint programming, whoseconcept of a global constraint provides a clue to how to overcome thisweakness. Also, the four disciplines continue to move in largely sep-arate orbits. This not only imposes the inconvenience of becomingfamiliar with multiple solvers, but it also passes up the advantages ofintegrated problem solving.Recent research has shown that there is much to be gained byexploiting the complementary strengths of dierent approaches tooptimization. Mathematical programmers are expert at relaxation- _1, Springer Science+Business Media, LLC 20121,J.N. Hooker Integrated Methods for Optimization International Series in Operationsanagement Science DOI 10.1007/978-1-4614-1900Research M 6,& 170, 18. 2 1 Introductiontechniques and polyhedral analysis. Constraint programming is dis-tinguished by its inference techniques and modeling power. Contin-uous global optimization is known for its convexication methods,and heuristic methods for their search strategies. Rather than choosebetween these, one would like to have them all available to attack agiven problem. Some problems submit to a single approach, but othersbenet from the more exible modeling and orders-of-magnitude com-putational speedups that can result when ideas from dierent eldsare combined.The advantages of integrated methods are being demonstrated in agrowing literature but, as a result, they themselves are multiplying. Ifthere are many solution methods that might be combined, there areeven more ways to combine them. The problem of proliferation seemsonly to be compounded by eorts to integrate.A change of perspective can bring order into this chaos. Rather thanlook for ways to combine methods, one can look for what the methodsalready have in common. Perhaps there is a general problem-solvingstrategy that the various communities have arrived at independently,albeit from dierent directions and with dierent emphases, becauseit is a strategy that works.This book takes such a perspective. It develops an algorithmicframework in which the dierent optimization methods, and moreimportantly their many combinations, are variations on a theme. Itproposes a modeling practice that can bring problem-specic methodsinto the scheme in a natural way. In short, it seeks an underlying unityin optimization methods.The book emphasizes the integration of mathematical programmingand constraint programming in particular, since this is where most ofthe research on integrated methods has been focused to date. Nonethe-less, some attempt is made to show how global optimization and heuris-tic methods t into the same framework.Unication is good for learning as well as practice. Students ofoperations research or computer science who conne themselves totheir own eld miss the insights of the other, as well as an overarchingperspective on optimization that helps make sense of it all. This bookis therefore designed as a graduate-level optimization text that belongsto neither eld but tries to construct a coherent body of material fromboth.Neither this nor any other text covers all concepts related tooptimization. There is nothing here about stochastic optimization, 19. 1.1 A Unifying Framework 3multiobjective programming, semidenite programming, or approx-imation methods, and only a little about nonlinear programming.Some important ideas of combinatorial analysis are left out, as arewhole areas of constraint programming. A broad selection of topicsis nonetheless presented, guided primarily by what is relevant to themajor general-purpose solvers of linear and mixed-integer program-ming, constraint programming, and to some extent global optimiza-tion. The emphasis, of course, is on how these topics form more thanjust a miscellaneous collection but are parts of an integrated approachto optimization.Although not comprehensive, the material presented here is morethan adequate to provide a substantial grounding in optimization. Itis a starting point from which the student can explore other subdisci-plines from an ecumenical point of view.1.1 A Unifying FrameworkOptimization methods tend to employ three strategies that interactin specic ways: search, inference, and relaxation. Search is the enu-meration of problem restrictions, while inference and relaxation makethe search more intelligent. Inference draws out implicit informationabout where the solution might lie, so that less search is necessary.Relaxation replaces the problem with a simpler one whose solutionmay point the way to a solution of the original one.Search is fundamental because the optimal solution of a problem liessomewhere in a solution space, and one must nd it. The solution spaceis the set of solutions that satisfy the constraints of the problem. Somesolutions are more desirable than others, and the objective is to ndthe best one, or at least a good one. In many practical problems, thesolution space tends to be huge and multidimensional. Its boundariesmay be highly irregular, and if there are discrete as well as continuousvariables, it is full of gaps and holes.Whether searching for lost keys or a fugitive from the law, a com-mon strategy is to divide up the territory and focus on one region at atime. Optimization methods almost invariably do the same. A regionis the solution set for a restriction of the problem, or a problem towhich additional constraints have been added. A sequence of restric- 20. 4 1 Introductiontions are solved, and the best solution found is selected. If the searchis exhaustive, meaning that the entire search space is covered, the bestsolution found is optimal.The most prevalent scheme for exhaustive search is branching: split-ting the territory (say) in half, splitting it again if either half is stilltoo large to manage, and so forth. Another basic strategy is constraint-directed search: whenever a region has been searched, a constraint ornogood is created that excludes that part of the search space, and per-haps other parts that are unpromising for the same reason. The nextexamined region must satisfy the constraints so far accumulated.Branching methods include the popular branch-and-cut methodsof mixed-integer programming and branch-and-infer methods of con-straint programming, on which all the major commercial solvers arebased. Constraint-directed methods include Benders decompositionin mathematical programming and such nogood-based methods asbranching with clause learning for propositional satisability problems,and partial-order dynamic backtracking. Continuous global optimizersuse a branching scheme to divide space into multidimensional boxes.Local search or heuristic methods likewise enumerate a sequence ofproblem restrictions, represented by a sequence of neighborhoods. Thedistinction of branching and constraint-directed search carries over toheuristic methods: a greedy adaptive search procedure, for example, isan incomplete form of branching search, and tabu search is an incom-plete constraint-directed search.Inference is a way of learning more about the search space, so asnot to waste time looking in the wrong places. Police might deducefrom a street map that a suspect would never frequent certain neigh-borhoods, just as a problem solver might deduce from a constraintthat certain variables would never take certain values in an optimalsolution. This not only shrinks the region to be searched, but also,sometimes more importantly, reduces its dimensionality. Mathemati-cal programming systems use inference methods in the form of cuttingplanes and preprocessing techniques, while constraint programmingsystems rely heavily on domain ltering and bounds propagation.Relaxation enlarges the search space in a way that makes it easier toexamine. A common strategy is to replace the current problem restric-tion with a continuous, linear relaxation. This replaces the correspond-ing region of the search space with a polyhedron that contains it, thussmoothing out the boundary and lling in the holes, and simplifyingthe search for an optimal solution. Relaxation can help in several ways. 21. 1.1 A Unifying Framework 5The solution of the relaxation may happen to lie inside the originalsearch space, in which case it solves the current problem restriction. Ifnot, the optimal solution of the relaxation may be no better than thebest feasible solution found so far, in which case one can immediatelymove on to another region. Even if this does not occur, the optimalsolution of the relaxation may provide a clue as to where the optimalsolution of the original problem lies.Search, inference, and relaxation reinforce one another. Restrictingthe problem in a search process allows one to draw more inferences andformulate relaxations that are closer to the original. Inference acceler-ates the search by excluding part of the search space, as when lteringreduces domains in constraint programming, when logic-based Ben-ders cuts (a form of nogood) are generated in Benders decomposition,or when items are added to the tabu list in local search. It deducesconstraints that can strengthen the relaxation, such as cutting planesin integer programming.Relaxation abbreviates the search by providing bounds, as in thebranch-and-relax and branch-and-cut methods of integer program-ming. The solution of the relaxation, even when infeasible in the orig-inal problem, can provide information about which problem restrictionto examine next. In branch-and-cut methods, for example, one brancheson a variable that has a fractional value in the solution of the relax-ation. In constraint-directed search, the set of accumulated nogoodsis, in eect, a relaxation whose solution denes the next problemrestriction to be solved. Less obviously, the solution of the relaxationcan direct inference, since one can give priority to deducing constraintsthat exclude this solution. The separating cuts of integer programmingprovide an example of this.This search-infer-and-relax scheme provides a framework withinwhich one can mix elements from dierent traditions. As a searchscheme one might select branching or constraint-based search. Forinference one might apply domain ltering to some constraints, gen-erate cutting planes for others, or for others use some of the intervalpropagation and variable xing methods characteristic of global opti-mization. Some constraints might be given relaxations based on integerprogramming models, others given relaxations that are tailor-made forglobal constraints, others relaxed with the factorization methods usedin continuous global solvers, and still others left out of the relaxationaltogether. 22. 6 1 Introduction1.2 Modeling to Reveal Problem StructureSearch-infer-and-relax methods can succeed only to the extent thatthey exploit problem-specic structure. Experience teaches that infer-ence and relaxation can be blunt instruments unless sharpened withspecic knowledge of the problem class being solved. Yet, it is imprac-tical to invent specialized techniques for every new type of problemthat comes along.The answer is to analyze constraints rather than problems. Althoughevery problem is dierent, certain patterns tend to recur in the con-straints. Many scheduling problems require, for example, that jobs runsequentially without overlapping. Other problems require that employ-ees be assigned to work a specied number of days in a row. It isnot hard to identify structured subsets of constraints that keep com-ing up. Each subset can be represented by a single metaconstraint,and specialized inference and relaxation methods can be designed foreach metaconstraint. The modelers choice of metaconstraints can thencommunicate much about the problem structure to the solver. In par-ticular, it can dictate which inference and relaxation techniques areused in the solution process.This might be called constraint-based control, which can extend tothe search procedure as well as the choice of inference and relax-ation methods. The user begins by choosing the overall search algo-rithm, perhaps branching or constraint-directed search, and perhapsan exhaustive or heuristic version of it. The choice of metaconstraintsdetermines the rest. In a branching framework, for example, the searchbranches when the solution of the current relaxation violates one ormore constraints. A priority list designates the violated constraint onwhich to branch. The constraint knows how to branch when it isviolated, and the search proceeds accordingly.This scheme can work only if the modeler uses constraints that arerich enough to capture substantial islands of structure in the problem.This requires a change from the traditional practice of mathemati-cal programming, which is to build models with a small vocabularyof primitive constraints such as inequalities and equations. It recom-mends something closer to the constraint programmers practice ofusing global constraints, so-named because each constraint stands fora collection of more elementary constraints whose global structure isexploited by the solver. 23. 1.2 Modeling to Reveal Problem Structure 7This book therefore advocates modeling with metaconstraints, whichgeneralize the idea of global constraints. A metaconstraint may consistof a set of inequalities of a certain type, a set of constraints to be ac-tivated under certain conditions, or a global constraint familiar to theconstraint programming worldto mention only a few possibilities.The advantages of metaconstraints are twofold. They not only revealthe problem structure to the solver, which may lead to faster solution,but they also allow one to write more concise models that are easierto follow and easier to debug.Modeling with metaconstraints immediately raises the issue of whatto do when the metaconstraints begin to proliferate, much as thespecial-purpose algorithms that were mentioned earlier. New problemsoften require new metaconstraints to capture a substructure that didnot occur in previous problems. Yet, this is not the stumbling blockthat it may seem to be.The lexicon of metaconstraints can grow large, but there are stillmany fewer constraints than problems, just as there are many fewerwords than sentences. In any eld of endeavor, people tend to settleon a limited number of terms that prove adequate over time for ex-pressing the key ideas. There is no alternative, since most of us canmaster only a limited vocabulary. This is true of technical domains inparticular, since a limited number of technical terms tend to evolveand prove adequate for most situations. Sailors must know about hal-yards, booms, mizzens, and much else, but the nautical vocabulary isnite and learnable.The same applies to modeling. In any given domain, practitionersare likely to develop a limited stock of metaconstraints that frequentlyarise. There might be one stock for project management, one for pro-cess scheduling, one for supply chains, and so forth, with much overlapbetween them. In fact, this has already happened in some domains forwhich specialized software has developed, such as project scheduling.There will be many metaconstraints overall, just as there are manytechnical terms in the world. But no one is obliged to know more thana small fraction of them.Computer-based modeling systems can ease the task further. Thereis no need to write models in a formal modeling language in whichone must get the syntax right or generate error messages. An intel-ligent user interface can provide menus of constraints, convenientlyorganized by application domain or along other dimensions. Selectinga constraint activates a window that allows one to import data, set 24. 8 1 Introductionparameters, and choose options for inference, relaxation, and search.The window contains links to related constraints that may be moresuitable. The system prompts the user with checklists or queries toguide the modeling process. The solver keeps updating the solution ofa small problem as the modeling proceeds, so that the modeler can seewhen the solution begins to look reasonable.1.3 The Role of DualityDuality is a perennial theme of optimization. It occurs in such formsas linear programming duality, Lagrangean duality, surrogate duality,and superadditive duality. It is also a unifying theme for this book fortwo reasons. These various duals turn out to be closely related, a factthat helps to unify optimization theory. They can all be classied asinference duals or relaxation duals, and in most cases as both.Secondly, the two types of duals help elucidate how search, infer-ence, and relaxation relate to one another: inference duality is a dualityof search and inference, while relaxation duality is a duality of searchand relaxation. Successful solution methods for combinatorial prob-lems tend to be primaldual methods, which move toward a solution ofthe original (primal) problem and a dual problem simultaneously. Thisbook sees integrated methods as primaldualdual methods, which atleast partially solve both the inference and relaxation duals as theysolve the primal.Inference duality arises as follows. An optimization problem can beseen as the problem of nding a set of values for the problem vari-ables that minimize the objective function. But it can also be seen asthe problem of inferring from the constraint set the tightest possiblelower bound on the value of the objective function. In the rst case,one searches over values of the variables, and in the second case, onesearches over proofs. The problem of nding the proof that yields thebest bound is the inference dual.The precise nature of the inference dual depends on what infer-ence method one uses to derive bounds. If the inference method iscomplete for the problem class in question, the inference dual has thesame optimal value as the original problem. One particular inferencemethodnonnegative linear combinationyields the classical linear 25. 1.3 The Role of Duality 9programming dual for linear problems and the surrogate dual forgeneral inequality-constrained problems. A slightly dierent inferencemethod gives rise to the all-important Lagrangean dual. In fact, theclose connection between surrogate and Lagrangean duals, which aresupercially unrelated, becomes evident when one regards them as in-ference duals. Still other inference methods yield superadditive andbranching duals, which arise in integer programming and other com-binatorial problems.Inference duality is a unifying concept because, rst of all, it canbe dened for any optimization problem, not just the inequality-constrained problems traditionally studied in mathematical program-ming. Secondly, it can serve as a general basis for sensitivity analysis,which examines the sensitivity of the optimal solution to perturbationsin the problem data, thus revealing which data must be accurate toget a meaningful solution. Most importantly, the proof that solves theinference dual is the source of nogoods or logic-based Benders cutsin a constraint-directed search method. Methods as disparate as Ben-ders decompositiona classic technique of operations researchandstate-of-the-art propositional satisability methods with clause learn-ing are closely related because they both generate nogoods by solvingan inference dual. In principle, any inference dual can give rise to anogood-based algorithm. The use of new inference duals in decomposi-tion methods has resulted in computational speedups of several ordersof magnitude.Relaxation duality is a duality of search and relaxation, or more pre-cisely a duality of restriction and relaxation. A motivation for solving asequence of restrictions is that the restrictions are easier to solve thanthe original. Since relaxations are also designed to be easier than theoriginal, one might ask whether a problem can be addressed by solvinga sequence of relaxations. It can, if the relaxations are parameterizedby dual variables, which allow one to search the space of relaxationsby enumerating values of the dual variables. The solution of eachrelaxation provides a bound on the optimal value, and the problem ofnding the best bound is the relaxation dual. In general, an enumera-tion of relaxations does not solve the problem, as does an enumerationof restrictions, because the best bound may not be equal to the optimalvalue of the original problem. The bound may nonetheless be useful,as in the surrogate and particularly Lagrangean duals, which wereoriginally conceived as relaxation duals rather than inference duals. 26. 10 1 Introduction1.4 Advantages of Integrated MethodsThe academic literature tends to emphasize computation speed whenevaluating a new approach to problem solving, perhaps because it iseasily measured. Practitioners know, however, that model developmenttime is often at least as important as solution time. This argues forthe convenience of having all the modeling and algorithmic resourcesavailable in a single integrated system. One can try several approachesto a problem without having to learn several systems and port databetween them. Metaconstraints can also be a signicant time saver, asthey lead to simpler models that are easier to build and maintain.The computational advantages are there as well. Certainly, oneneed never pay a computational price for using an integrated system,because the traditional techniques can always be available as one ofthe options. However, experience conrms that a more broad-basedstrategy can substantially speed computation. This is borne out by asampling of results from the literature. The focus here is on methodsthat integrate constraint programming (CP) and mixed-integer/linearprogramming (MILP), which can be classied roughly by the type ofintegration they use.Many integrated methods combine CP with linear relaxations fromMILP. One study, for example, combined a CP algorithm with anassignment problem relaxation (Section 7.11.2) and reduced-cost vari-able xing (Sections 6.2.3, 6.3) to solve lesson timetabling problems2 to 50 times faster than CP [202]. Another study, based on [413]and reimplemented in the integrated solver SIMPL [517], combinedCP with convex hull relaxations to solve production planning prob-lems with piecewise linear costs (Section 2.4) 20 to 120 times fasterthan state-of-the-art MILP software, generating search trees that were10008000 times smaller. Linear relaxations combined with logic pro-cessing (Section 6.4) solved a boat party scheduling problem in veminutes that MILP could not solve in twelve hours, and solved owshop instances four times faster than MILP [295].Experiments have been conducted with other kinds of relaxationsas well. A combination of CP and Lagrangean relaxation (Section 4.5)solved automatic digital recording problems one to ten times fasterthan MILP, which was faster than CP [448]. Another CPLagrangeanhybrid achieved order-of-magnitude speedups on a radiation therapyproblem [115]. CP assisted by semidenite programming relaxations 27. 1.4 Advantages of Integrated Methods 11has been successfuly used for stable set problems and clustering prob-lems on bipartite graphs [250, 485]. Logic processing and linear quasi-relaxations solved nonlinear structural design problems up to 600 timesfaster than MILP and solved two of the problems in less than six min-utes when MILP could not solve them in 20 hours [104].Branch-and-price integer programming methods (Section 5.1.6) havealso been combined with CP processing, particularly in the area of air-line and transit crew scheduling (such a problem is discussed in Sec-tion 5.1.7). In one study [519], this approach solved signicantly largerurban transit crew management problems than traditional branch andprice could solve. A CP-based branch-and-price method was the rstto solve the eight-team traveling tournament problem [184].Perhaps the greatest speedups have been achieved by integratingCP and MILP through generalized forms of Benders decomposition(Section 5.2.2). One study [300] solved minimum-cost machine allo-cation and scheduling problems 20 to 1000 times faster than CP orMILP. A subsequent study [473] improved upon these results by anadditional factor of ten. SIMPL [517] brought further improvements,solving some problems in less than a second that were intractablefor CP and MILP. A logic-based Benders approach solved single-machine scheduling problems with much longer time horizons thanwere tractable for either CP or MILP, even though the problem doesnot obviously decompose (the time horizon was broken into segments)[138, 139].Other work has extended the applicability of logic-based Bendersmethods. One industrial implementation [475] solved, in ten min-utes, polypropylene batch scheduling problems at BASF that werepreviously insoluble. A CP/MILP hybrid solved twice as many callcenter scheduling problems as traditional Benders [77]. A dierentCP/MILP hybrid solved planning and scheduling problems, withresource-constrained scheduling, 100 to 1000 times faster than CP orMILP when minimizing cost or makespan [283], 10 to 1000 times fasterwhen minimizing the number of late jobs, and about 10 times faster(with much better solutions when optimality was not obtained) whenminimizing total tardiness [285] (Section 2.8 shows how to solve a sim-plied minimum-makespan problem with a logic-based Benders tech-nique). Finally, a hybrid Benders approach was applied [410] to obtainspeedups of several orders of magnitude relative to the state of the artin sports scheduling. 28. 12 1 IntroductionIt is important to bear in mind that none of these results wereachieved with the full resources of integration. They are also pre-liminary results obtained with experimental codes. Integrated solu-tion software will doubtless improve over time. The chief advantage ofintegrated methods, however, may be that they encourage a broaderperspective on problem solving. This may inspire developments thatwould not have been possible inside individual disciplines.1.5 Some ApplicationsIntegrated methods have been successfully applied in a wide variety ofcontexts. The literature can again be roughly organized according tothe type of integration used.Applications that combine MILP/cutting planes with CP or logic-based methods include the orthogonal Latin squares problem [18], trussstructure design [104], processing network design [245, 295], single-vehicle routing [432], resource-constrained scheduling [83, 170], multi-ple machine scheduling [102], shuttle transit routing [402], boat partyscheduling [295], and the multidimensional knapsack problem [379].Cutting planes for disjunctions of linear systems (Section 7.4.5) havebeen applied to factory retrot planning, strip packing, and zero-waitjob shop scheduling [435].Convex hull relaxations for disjunctions of linear and nonlinear sys-tems (Sections 7.4.1 and 7.5.1) have been used to solve several chemi-cal process engineering problems [336, 337, 409, 436, 492]. Convex hullrelaxations of piecewise linear constraints have been used in a CP con-text to solve xed-charge problems and transportation problems withpiecewise linear costs [413], as well as production planning problemswith piecewise linear costs [380, 381].Applications that combine CP with reduced-cost variable xinginclude the traveling salesman problem with time windows [358], prod-uct conguration [358], xed-charge network ows [317], and lessontimetabling [202].CP-based Lagrangean methods have been applied to network design[153], automatic digital recording [448], traveling tournament problems[78], the resource-constrained shortest-path problem [219], the delivery 29. 1.6 Software 13of radiation therapy [115], and the general problem of ltering domains[313].The most popular application of CP-based branch-and-price meth-ods is to airline crew assignment and crew rostering [125, 190, 308,321, 450]. Other applications include transit bus crew scheduling[519], aircraft scheduling [244], vehicle routing [431], network design[126], employee timetabling [171], physician scheduling [222], radia-tion therapy delivery [115], and the traveling tournament problem[183, 184, 481].Benders methods that combine MILP with CP or logic-based meth-ods have been developed for circuit verication problems [298], integerprogramming [131, 296] and the propositional satisability problem[279, 296]. A series of papers have described applications to plan-ning and scheduling [132, 258, 279, 283, 285, 300]. Other applicationsinclude dispatching of automated guided vehicles [149], steel produc-tion scheduling [257], batch scheduling in a chemical plant [345], andpolypropylene batch scheduling in particular [475]. CP-based Bendersmethods have also been applied to scheduling of computer proces-sors [75, 76, 114], location-allocation problems [194], and trac diver-sion problems [512]. Other applications of logic-based Benders includetransportation network design [394], queuing design and control [471],and sports scheduling [129, 411].1.6 SoftwareMethods that combine constraint programming and optimization havefound their way into a number of software packages. Because softwarerapidly evolves, only a brief summary is given here. A more detailedsurvey can be found in [516].The earliest integrated solver was the Prolog-based constraint logicprogramming system ECLiPSe [19, 428, 497]. It is now an open-sourceproject, and the software continues to be used and maintained. It isorganized around the concept of cooperating solvers. Constraint andLP solvers, for example, exchange information that reduce variabledomains. In addition, column generation, Benders decomposition, andLagrangean relaxation have been implemented in ECLiPSe.IBM/ILOGs OPL Studio [327, 328, 382] provides a modeling lan-guage that invokes CP and MILP solvers. Models can be written for 30. 14 1 Introductioneither type of solver, and a versatile script language allows users to runsolvers iteratively and transfer information between them. This is con-venient for logic-based Benders methods, for example. The CP model-ing language is enhanced for scheduling applications and includes suchnovel features as interval-valued and conditional variables to representoperations that can be present or absent from the schedule.Xpress-Mosel [142, 143] is a modeling language that allows low-level control of cooperating solvers in FICOs Xpress suite of math-ematical modeling and optimization tools. These include CP, MILP,and nonlinear programming solvers. Mosel can implement a varietyof hybrid methods (e.g., [102, 261, 432]) but, like OPL Studio, mayrequire detailed coding to do so.NICTAs G12 system [466] includes a solver-independent modelinglanguage, Zinc; a mapping language, Cadmium; and an algorithmiclanguage, Mercury (all elements of group 12 in the Periodic Table).Cadmium is used to generate a Mercury script from the Zinc model.The solution strategy can invoke a variety of solvers, including con-straint programming, linear programming, and satisability (SAT)solvers. The user can experiment with dierent complete, local, orhybrid methods to solve the Zinc model.The noncommercial system SIMPL [23, 517] provides a high-levelmodeling language that invokes integrated methods at the micro level,using the idea of constraint-based control. The modeling language con-tains meta-constraints that allow the solver to exploit problem sub-structure. The solver is organized around a single search-infer-and-relax loop that integrates inference and relaxation methods from CPand OR to suit the problem at hand, in an approach similar to thatdescribed in this book. The system therefore supports both branchingand constraint-based search, including nogood generation and logic-based Benders methods.SCIP [4, 5], developed at Zuse-Institut Berlin, accommodates con-straint propagation within an ecient MILP solver (constraint integerprogramming). Its constraint handler and plugin-based architec-ture allows the user to supply ltering algorithms for specic constrainttypes. The system also supports nogood generation, with built-in con-ict analysis for MILP.There has been some investigation of integrated problem solvingbeyond the CP/MILP interface. The global optimization packageBARON [468, 469] combines nonlinear (as well as linear) integer pro-gramming with CP-style domain reduction, although it uses a model- 31. 1.7 Plan of the Book 15ing system (AIMMS) that does not support CP-style constraints. Themodeling language in the Comet system [267], which evolved from anearlier system, Localizer [357], allows CP and MILP constraints as wellas high-level constraint-based specications of local search. SALSA[329] is a language that can be used to design hybrids of global andlocal search algorithms. ToOLS [166] is a search specication languagewith a facility to invoke and combine existing algorithms. Numberjack[260] is a Python-based language for embedding CP, MILP and SATtechnology into larger applications.None of the existing systems fully integrate solution methods. Yetthe necessary concepts and technology have reached a stage where aseamlessly integrated system is within reach. Perhaps the discussionto follow will help encourage eorts in this direction.1.7 Plan of the BookThe remainder of the book begins by introducing the main ideas ofthe book in a series of examples (Chapter 2). Seven problems illustratehow models are constructed and how search, inference, and relaxationinteract to solve them. One can get a very good idea of what integratedproblem solving is all about by reading this chapter alone.Chapter 3 provides necessary background in the basics of opti-mization, with brief treatments of linear programming, network ows,matching problems, optimality conditions for nonlinear programming,and deterministic dynamic programming. Chapter 4 then moves intothe theory of integrated methods with an introduction to inference andrelaxation duality, and their relation to problem solving and sensitivityanalysis. Several specic types of duality are presented, including lin-ear programming, surrogate, Lagrangean, subadditive, and branchingduality.The three longest chapters of the bookChapters 5, 6, and 7correspond to the three parts of the search-infer-and-relax frame-work. Chapter 5, on search, focuses successively on branching search,constraint-directed search, and local search. The section on branchingsearch covers node and variable selection, cost-based branching, primalheuristics, and branch-and-price methods. Constraint-directed searchincludes logic-based Benders decomposition, nogood-directed branch- 32. 16 1 Introductioning in MILP and elsewhere, and partial-order dynamic backtracking.Particular attention is paid to conict clause generation in algorithmsfor the propositional satisability problem.Chapter 6, on inference, begins with some basic concepts of con-straint programming, including k-consistency, domain consistency, andbounds consistency. It develops a theory of inference for inequality-constrained problems and propositional logic. The remainder of thechapter presents ltering methods for some popular global constraints,specically element, all-dierent, cardinality, nvalues, among, sequence,stretch, regular, and circuit constraints. It concludes with boundsreduction algorithms for disjunctive and cumulative scheduling, whichhave contributed much to the success of constraint programming.Chapter 7, on relaxation, has a stronger avor of mathematical pro-gramming due to its use of mixed-integer modeling and cutting-planetheory. The chapter provides continuous relaxations for disjunctionsof linear and nonlinear inequality systems, as well as for several globalconstraints. There is a certain parallelism between Chapters 6 and 7,in that both take the reader through linear and integer inequalities,propositional logic, and roughly the same set of global constraints.The main dierence, of course, is that Chapter 6 presents inferencemethods for each of these constraint types, and Chapter 7 presentsrelaxation methods.Chapter 8 is something of an appendix. It lists 48 metaconstraints,including a number of global constraints from the CP community, as astarting point for a menu of constraints in an integrated solver. Whenpossible, it says something about usage, inference methods, relaxationmethods, and related constraints, providing pointers to the literatureand to relevant sections of the book.1.8 Bibliographic NotesIntegrated methods have developed over the last 20 years or so inboth the constraint programming (CP) and operations research (OR)communities. While a fuller history can be found elsewhere [281], avery brief synopsis might go as follows.On the OR side, it is interesting that implicit enumeration [216], anearly 1960s technique for integer programming, can be seen as antic- 33. 1.8 Bibliographic Notes 17ipating the use of constraint propagation in an integer programmingcontext. Constraint programming is explicitly mentioned in the ORliterature as early as 1989 [111], but integrated methods were yet todevelop.The CP community began to investigate integrated methods in aserious way during the 1990s. They were initially conceived as double-modeling approaches, in which some constraints receive both CP andMILP formulations that exchange domain reduction and/or infeasi-bility information [340]. This mechanism was implemented in the con-straint logic programming system ECLiPSe [428, 497]. The constraintscommunity also began to recognize the parallel between constraintsolvers and mixed-integer solvers, as evidenced by [101].In later work, such OR ideas as reduced-cost variable xing, lin-ear relaxations of global constraints, and convex hull relaxations ofpiecewise linear functions were brought into CP-based algorithms[202, 204, 205, 380, 413, 414]. ILOGs OPL Studio [382] provided amodeling language that invokes CP and MILP solvers.While this research was underway in CP, the OR community intro-duced hybrid methods as generalizations of branch and cut or a logic-based form of Benders decomposition. Integer variables were replacedwith logical disjunctions and their relaxations as early as 1990 [50].A series of papers appearing the 1990s integrated CP and logic-basedmethods with branch and cut [245, 276, 279, 295]. The logic-basedBenders approach was developed during the same period, initially forcircuit verication [298] and later as a general method [279, 296]. ABenders method that joins MILP and CP was proposed [279] andsuccessfully implemented [300]. CP-based branch and price, a verydierent approach, was also developed [308, 518].The First International Joint Workshop on AI and OR was orga-nized in 1995 to provide an early forum for discussion of integratedmethods. The idea was revived in 1999 with the annual CP-AI-ORworkshop (Integration of AI and OR Techniques in CP for Combina-torial Optimization), now an annual conference series with publishedproceedings. Papers on hybrid methods regularly appear in CP andOR conferences. 34. Chapter 2ExamplesThis chapter shows how an integrated approach to optimization playsitself out in some concrete cases. It presents several small examples thatcover a wide range of application areas. It each case, it formulates amodel that is appropriate for integrate1d solution, and in most casesit carries the solution procedure to completion.The rst examplea simple freight transfer problemdemonstrateshow inference methods from CP and relaxation methods from MILPcan work together to accelerate branching search. A production plan-ning example then shows how discrete choices can be formulated inan integrated modeling environment to result in much faster solution.An employee scheduling example demonstrates the modeling power ofmetaconstraints.A fourth example shows how inference and relaxation can be prof-itably combined in continuous global optimization. A product cong-uration problem illustrates how high-level modeling can tell the solverhow to combine inference and relaxation. A machine scheduling prob-lem shows how integer programming and constraint programming canbe integrated in a framework of logic-based Benders decomposition. Anal and more complex example deals with communications networkrouting and frequency assignment. It combines elements of networkow programming and constraint programming using another form ofdecomposition.The chapter begins with some basic denitions, followed a generaldescription of the search-infer-and-relax solution process that is illus-trated by the examples.- _2, Springer Science+Business Media, LLC 2012,J.N. Hooker Integrated Methods for Optimization International Series in Operationsanagement Science DOI 10.1007/978-1-4614-1900Research M 6,& 170,19 35. 20 2 Examples2.1 Basic DenitionsFor the purposes of this book, an optimization problem can be writtenmin (or max) f(x)C(x)x D(2.1)where f(x) is a real-valued function of variable x and D is the domainof x. The function f(x) is to be minimized (or maximized) subject to aset C of constraints, each of which is either satised or violated by anygiven x D. Generally, x is a tuple (x1, . . . , xn) and D is a Cartesianproduct D1 Dn, where each xj Dj. The notation C(x) meansthat x satises all the constraints in C.Adopting terminology from mathematical programming, any x Dis a solution of (2.1). Solution x is a feasible solution if C(x), and thefeasible set of (2.1) is the set of feasible solutions. A feasible solutionx is optimal if f(x) f(x) for all feasible x. An infeasible problemis one with no feasible solution. If (2.1) is infeasible, it is convenient tosay that it has optimal value (or in the case of a maximizationproblem). The problem is unbounded if there is no lower bound on f(x)for feasible values of x, in which case the optimal value is (or for maximization).It is assumed throughout this book that (2.1) is either infeasible,unbounded, or has a nite optimal value. Thus, such problems asminimizing x subject to x > 0 are not considered. An optimizationproblem is considered to be solved when an optimal solution is found,or when the problem is shown to be unbounded or infeasible. In incom-plete search methods that do not guarantee an optimal solution, theproblem is solved when a solution is found that is acceptable in somesense, or when the problem is shown to be unbounded or infeasible.A constraint G can be inferred from C if any x D that satisesC(x) also satises G. Equivalently, C implies G, or G is valid for C. Arelaxation R of the minimization problem (2.1) is obtained by droppingconstraints and/or replacing the objective function f(x) with a lowerbounding function f (x). That is, any x D that is feasible in (2.1) isfeasible in R and satises f (x) f(x). When one is minimizing, theoptimal value of a relaxation is always a lower bound on the optimalvalue of the original problem. 36. 2.2 The Solution Process 212.2 The Solution ProcessSearch, inference, and relaxation interact to provide a general schemefor solving (2.1). The search procedure solves a series of restrictionsor special cases of the problem. The best solution of a restriction isaccepted as a solution of the original problem. Inference and relaxationprovide opportunities to exploit problem structure, a key element ofany successful approach to solving combinatorial problems. These ideasare more formally developed in Chapters 5, 6, and 7. Only a briefoverview is given here.Restrictions are obtained by adding constraints to the problem. Therationale for searching over restrictions is that they may be easier tosolve than the original problem, even when there are many of them.Branching search methods, for example, divide the feasible set intosmaller and smaller subsets by branching on alternatives. Each sub-set corresponds to a restriction of the original problem. Well-knownexamples of branching search include the branch-and-cut algorithmsof MILP solvers and the branch-and-infer methods of CP solvers. Ben-ders decomposition also enumerates restrictions by solving a sequenceof subproblems in which certain variables are xed. Local search meth-ods examine a sequence of neighborhoods, each of which denes a re-striction of the problem.Search can often be accelerated by inferencethat is, by inferringnew constraints from the constraint set. The new constraints are thenadded to the problem, which may ease solution by describing the feasi-ble set more explicitly. Common forms of inference are domain lteringin CP, cutting-plane generation in MILP, and Benders cuts in Bendersdecomposition. Inferred constraints may also result in a stronger re-laxation, as in the case of cutting planes and Benders cuts.Inference is most eective when it exploits problem structure. Whena group of constraints has special characteristics, the model can indi-cate this by combining the constraints into a single metaconstraint,known as a global constraint in constraint programming. This allowsinference algorithms to exploit the global structure of the group whenltering domains or generating valid cuts. Developing special-purposeinference procedures has been a major theme of research in CP.Relaxation is the third element of the solution scheme. Like arestriction, a relaxation of the problem may be easier to solve thanthe original. For instance, an MILP problem becomes much easier to 37. 22 2 Examplessolve if one relaxes it by allowing the integer-valued variables to takeany real value.Solving a relaxation can be useful in several ways. Its solution mayhappen to be a solution of the original problem. Even if not, its solutionmay provide a clue to where one might nd a solution of the originalproblem and therefore help guide the search. In addition, the optimalvalue of the relaxation provides a lower bound on the optimal valueof the original problem (when one is minimizing). This is particularlyuseful in branching search, where bounds can often help prune thesearch tree.Relaxation also provides a valuable opportunity to exploit specialstructure in individual constraints or groups of constraints, and thishas been a perennial theme of the optimization literature. For example,strong cutting planes can be inferred from sets of inequalities thatdene certain types of polyhedra. One can use metaconstraints toindicate which inequalities have special structure.2.3 Freight TransferA simple freight transfer problem illustrates, at an elementary level,how inference and relaxation can interact with branching search. Forty-two tons of freight must be conveyed overland. The shipper has a eetof trucks in four sizes, with three vehicles of each size (Table 2.1). Theeight available loading docks must accommodate all the trucks used,because the trucks must be loaded simultaneously. Due to the shape ofthe loading area, three loading docks must be allocated to the largesttrucks even if only one or two of them are loaded. The problem is toselect trucks to carry the freight at minimum cost.Table 2.1 Data for a small instance of a freight transfer problem.Truck Number Capacity Cost pertype available (tons) truck1 3 7 902 3 5 603 3 4 504 3 3 40 38. 2.3 Freight Transfer 232.3.1 Formulating the ProblemIf variable xi indicates the number of trucks of type i loaded, therequirement that 42 tons be transported can be written7x1 + 5x2 + 4x3 + 3x4 42 (2.2)The loading dock constraints can be writtenx1 + x2 + x3 + x4 8(1 x1 2) (x2 + x3 + x4 5)(2.3)where means implies. The problem can therefore be formulatedintegerLinear:min 90x1 + 60x2 + 50x3 + 40x47x1 + 5x2 + 4x3 + 3x4 42 (a)x1 + x2 + x3 + x4 8 (b)conditional: (1 x1 2) (x2 + x3 + x4 5)domains: xi {0, 1, 2, 3}, i = 1, . . . , 4(2.4)In accord with the spirit of revealing problem structure, the constraintsare grouped by type to provide the solver guidance as to how to processthem. Since the objective function becomes an integer linear inequal-ity whenever it is bounded, it is grouped with the integer linear con-straints to form an integer linear metaconstraint. An optimal solutionis (x1, . . . , x4) = (3, 2, 2, 1), with a minimum cost of 530.The problem may be solved by a branching algorithm that usesinference and relaxation. The following sections rst show how to carryout the inference and relaxation steps and then how to conduct thesearch.2.3.2 Inference: Bounds PropagationDomain ltering removes values from a variable domain when the vari-able cannot take those values in any feasible solution. The reduceddomains inferred from one constraint can be used as a starting pointfor domain ltering in another constraint (a form of constraint prop-agation). One can, in principle, cycle through the constraints in thisfashion until no further ltering is possible. This allows one to draw 39. 24 2 Examplessome inferences that do not follow from any single constraint, eventhough the constraints are processed individually.A type of domain ltering known as bounds propagation is useful forthe freight transport problem. Focusing rst on constraint (2.2), onecan solve it for variable x1, for example, to obtainx1 42 5x2 4x3 3x4742 5 3 4 3 3 37=67where the second inequality is due to the fact that each xj {0, 1, 2, 3}.Because x1 must be integral, one can round up the bound to obtainx1 1. The domain of x1 is therefore reduced from {0, 1, 2, 3} to{1, 2, 3}. The same procedure can be applied to the other variables,but as it happens, no further domain reductions are possible. Thesmaller domain for x1 can now be propagated to the second constraint(2.3), which by similar reasoning implies that x1 8 and xj 7for j = 2, 3, 4. Unfortunately, this does not further reduce any of thedomains. If it did, one could cycle back to the rst constraint andrepeat the procedure.If the reduced domain of x1 were a subset of {1, 2}, one could inferfrom the implication constraint that x2 + x3 + x4 5 and perhapsreduce domains further. This condition is not initially satised, but itmay become satised during the search.2.3.3 Inference: Valid InequalitiesIn addition to deducing smaller domains, one can deduce valid in-equalities or cutting planes from the knapsack constraints. The inferredinequalities are added to the original constraint set in order to producea stronger continuous relaxation.One type of cutting plane, known as a general integer knapsack cut,can be inferred from constraints like (2.2). Note that the rst twoterms 7x1, 5x2 of (2.2) cannot by themselves satisfy the inequality,even if x1 and x2 are set to their largest possible value of 3. To satisfythe inequality, one must have4x3 + 3x4 42 7 3 5 3which impliesx3 + x4 42 7 3 5 34=64= 2 40. 2.3 Freight Transfer 25The inequality x3 + x4 2 is an integer knapsack cut.Because the rst two terms of (2.2) cannot satisfy the inequality bythemselves, the index set {1, 2} is a packing. In fact, it is a maximalpacking, because no proper superset is a packing. There are four max-imal packings for (2.2), each of which gives rise to an integer knapsackcut:{1, 2} : x3 + x4 2{1, 3} : x2 + x4 2{1, 4} : x2 + x3 3{2, 3, 4} : x1 1(2.5)These cuts are implied by (2.2) because any integral x = (x1, . . . , x4)that satises (2.2) must satisfy (2.5). Nonmaximal packings also giverise to integer knapsack cuts, and they may be nonredundant. Forexample, the packing {2, 3} produces the cut x1 +x4 3, which is notredundant of (2.5) or any other knapsack cut.Knapsack cuts can sometimes be strengethened using domains. Forexample, the packing {2} gives rise to the cut x1 + x3 + x4 4, whichcan be strengthened to x1 +x3 +x4 5. This is because x1 3, and asa result the largest possible left-hand side of (2.2) when x1+x3+x4 = 4is 40. It is therefore neessary to have x1 + x3 + x4 5.The fourth cut in (2.5) duplicates the bound x1 1 already obtainedfrom bounds propagation. In fact, it is easy to see that any propagatedbound xj L can be obtained from the knapsack cut correspond-ing to the maximal packing that consists of the remaining variables.Knapsack cuts corresponding to maximal packings therefore dominatepropagated bounds. However, it is much more costly to generate allmaximal packings than to propagate bounds.2.3.4 Relaxation: Linear ProgrammingA continuous relaxation of the problem instance (2.4) can be obtainedfrom the inequality constraints by allowing each variable xi to takeany real value in the range between its lowest and highest value. Onecan also add the knapsack cuts (2.5) before relaxing the problem. Thisyields the relaxation below: 41. 26 2 Exampleslinear:min 90x1 + 60x2 + 50x3 + 40x47x1 + 5x2 + 4x3 + 3x4 42 (a)x1 + x2 + x3 + x4 8 (b)x3 + x4 2 (c)x2 + x4 2 (d)x2 + x3 3 (e)Li xi Ui, i = 1, . . . , 4domains: xi R, i = 1, . . . , 4(2.6)where initially (L1, U1) = (1, 3) and (Li, Ui) = (0, 3) for i = 2, 3, 4.This relaxed problem can be easily solved by linear programming. Anoptimal solution is x = (x1, . . . , x4) = (213 , 3, 223 , 0), which has cost52313 . This solution is not feasible in the