© J. Christopher Beck 2007

Commentary on Session ACommentary on Session A

J. Christopher BeckDepartment of Mechanical & Industrial Engineering

University of Toronto, Canada

jcb@mie.utoronto.ca

Scheduling a Scheduling Competition, Providence, Sept 2007

2© J. Christopher Beck 2007

1+3 Papers1+3 Papers

Ghersi et al. focuses onan operational question– how do we judge the results?– important, technical issue– valuable for whoever runs the competition

Other 3 papers focus ona more strategic question– what problem types should the competition

address?

3© J. Christopher Beck 2007

Some Operational IssuesSome Operational Issues

Automated verification of resultsEntries need to run on the same platformNot just source code vs. binary

– License issues?

4© J. Christopher Beck 2007

Integration vs. FocusIntegration vs. Focus

Rich problems (Le Pape; Guerri et al.) vs. a single fundamental issue (Cicirello)

Things to consider:– industry impact– potential for research progress/breakthroughs

• will we understand the results?

– barriers to entry• easier to participate in Cicirello-style track

– is this an OR vs. AI issue?

5© J. Christopher Beck 2007

Multiple Tracks?Multiple Tracks?

Perhaps of increasing difficultyThings to consider:

– spreading the competition too much• one entry per track is not very interesting

– “granularity” of tracks– creating a challenge– barriers to entry

6© J. Christopher Beck 2007

Robustness as a CriteriaRobustness as a Criteria

Bias evaluation toward good performance on all instances (Le Pape)

Best all-round single machine scheduler across different opt. funcs (Cicirello)

Things to consider:– “jack of all trades, master of none”

• give up on being “the” best on a given problem• marketing

– industry vs. research– another OR vs. AI issue?

7© J. Christopher Beck 2007

Competition or “Challenge”Competition or “Challenge”

Competition– like SAT or Planning competition– multiple tracks

Challenge– one problem type (e.g., one of Guerri et al.’s, one of

Le Pape’s)– long work horizon (e.g., 6 months – 1 year)– like the CP Modeling Challenge (2005)

8© J. Christopher Beck 2007

Competition or “Challenge”Competition or “Challenge”

Things to consider:– marketing– (end user) industry interest and commitment– barriers to entry– potential for lack of community interest– organizational overhead