Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that...
Transcript of Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that...
![Page 1: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/1.jpg)
Procrastinating with ConfidenceNear-Optimal, Anytime, Adaptive Algorithm Configuration
BRENDAN LUCIER, MICROSOFT RESEARCH
BOBBY KLEINBERG, CORNELL
KEVIN LEYTON-BROWN, UBCDEVON GRAHAM, UBC
TTIC WORKSHOP ON AUTOMATED ALGORITHM DESIGN, AUGUST 7 2019
![Page 2: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/2.jpg)
Algorithm Configuration
Automating Algorithm Design- Encode design choices as parameters- Search for good configurations via learning- Can tailor to specific contexts (input distribs)
Examples: solvers for SAT, MIPs, TSP instances, …
Goal: find good configurations quicklyGood = fast algorithms for relevant problem instances
ParamILS [Hutter, Hoos, Leyton-Brown, Stützle 2009], SMAC [Hutter, Hoos, Leyton-Brown 2011], Hyperband [Li, Jamieson, DeSalvo, Rostamizadeh, Talwalkar 2016], …
![Page 3: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/3.jpg)
Algorithm Configuration
Two parts of the algorithm configuration problem:
1. Which configurations should we test?• Predict promising new configurations• Bayesian methods, structural assumptions, …
2. How should we efficiently test configurations?• Test by running algorithms on random inputs• How many inputs to try on each configuration?• Goal: don’t waste time on duds
This work
![Page 4: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/4.jpg)
Structured Procrastination [2017]: algorithm configuration procedure with guaranteed worst-case running time.
• Find approx. optimal config. in time proportional to [# configs] x [OPT running time] x [error terms].
• Nearly matches worst-case lower bounds (up to logs)
Structured Procrastination with Confidence [2019]: Bounds can be made adaptive, better for “easier” instances
• See also: Leaps & Bounds, Caps & Runs [Weisz, György, Szepesvári 2018, 2019]
Anytime procedures: user stops search procedure at any point, guarantee tightens over time.
• User does not need to pre-specify error bounds
This Work (informal):
![Page 5: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/5.jpg)
Model
![Page 6: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/6.jpg)
Model
Example:
![Page 7: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/7.jpg)
Model
OPT
![Page 8: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/8.jpg)
Structured Procrastination[Kleinberg, Leyton-Brown, L 2017], [Kleinberg, Leyton-Brown, L, Graham 2019]
![Page 9: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/9.jpg)
Toy Example
![Page 10: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/10.jpg)
Toy Example
Bad example:
![Page 11: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/11.jpg)
Toy Example
![Page 12: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/12.jpg)
A time management scheme due to Stanford philosopher John Perry [2011 Ig Nobel prize, Literature]• Keep a set of hard tasks that you procrastinate to avoid, thereby
accomplishing other tasks.• Eventually replace each daunting task with a new task that is even
more daunting, and so complete the former.
Structured Procrastination Algorithm Configuration:• Maintain sets of tasks (for each config., a queue of runs)• Start with the easiest tasks (shortest captimes)• Procrastinate when these tasks prove daunting
(put capped runs back on the queue)
Structured Procrastination
![Page 13: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/13.jpg)
Implementation
![Page 14: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/14.jpg)
Implementation
![Page 15: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/15.jpg)
Implementation
![Page 16: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/16.jpg)
Implementation
![Page 17: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/17.jpg)
Implementation
![Page 18: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/18.jpg)
Implementation
![Page 19: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/19.jpg)
Implementation (Anytime)
5.5. Grow the chosen configuration’s queue (if necessary)
![Page 20: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/20.jpg)
Performance Guarantee
Configurations
![Page 21: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/21.jpg)
Performance Guarantee
Configurations
![Page 22: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/22.jpg)
Performance Guarantee
Configurations
![Page 23: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/23.jpg)
Performance Guarantee
![Page 24: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/24.jpg)
Lower Bound
![Page 25: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/25.jpg)
Beating the Lower Bound
![Page 26: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/26.jpg)
Structured Procrastination with Confidence
5.5. Grow the chosen configuration’s queue (if necessary)
lower confidence bound
lowest confidence bound
ran most often
![Page 27: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/27.jpg)
Details: Confidence Bounds
![Page 28: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/28.jpg)
Analysis
![Page 29: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/29.jpg)
Performance Guarantee
otherwise
![Page 30: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/30.jpg)
Evaluation (I)
![Page 31: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/31.jpg)
Evaluation (II)
![Page 32: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/32.jpg)
Extension: many configurations
![Page 33: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/33.jpg)
Extension: many configurations
![Page 34: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/34.jpg)
Extension: many configurations
![Page 35: Procrastinating with Confidenceckingsf/AutoAlg2019/slides... · • Keep a set of hard tasks that you procrastinate to avoid, thereby accomplishing other tasks. • Eventually replace](https://reader035.fdocuments.in/reader035/viewer/2022071011/5fc97644059d5613dd4d9bf6/html5/thumbnails/35.jpg)
SummaryStructured Procrastination: approach to algorithm configuration.
• Procrastinates on potentially hard inputs rather than solving them to completion when first encountered
Anytime procedure, guaranteed to find an approx. optimal algorithm configuration in nearly optimal worst-case time.
Extension: adaptively better performance on “easy” instances• E.g., presence of bad configurations that can be rejected quickly
Future directions:• Combining with Bayesian optimization, other methods• Thorough empirical evaluations, comparisons
Thanks!