Slide 1

The New Age of Commonsense Reasoning Henry Kautz University of Rochester Slide 2 The Idea of Commonsense Reasoning Programs with Common Sense (McCarthy 1959): Use declarative representations of knowledge and general reasoning methods to Understand the physical world Understand human behavior Motivation Tasks: advice taker, story understanding Step towards true AI Slide 3 Commonsense Reasoning, circa 1985 Declarative and general Missing: grounding, learning, probabilities, scalable inference Slide 4 Commonsense Reasoning, circa 2008 Slide 5 Methods for grounding theories in sensor data Graphical models for learning & probabilistic inference Practical (low-exponent) reasoning algorithms Slide 6 New Applications Caregiving systems AI-based assistive technology Assisted cognition Automated planning Supra-expert level Verification Natural language processing... Slide 7 This Talk Monitoring activities of daily living Learning and guiding users through transportation plans Planning as satisfiability Slide 8 Object-Based Activity Recognition Activities of daily living involve the manipulation of many physical objects Kitchen: stove, pans, dishes, Bathroom: toothbrush, shampoo, towel, Bedroom: linen, dresser, clock, clothing, We can recognize activities from a time- sequence of object touches (Philipose et al. 2004) Slide 9 Applications ADL (Activity of Daily Living) monitoring for the disabled and/or elderly Changes in routine often precursor to illness, accidents Human monitoring intrusive & inaccurate Image Courtesy Intel Research Slide 10 Sensing Object Manipulation RFID: Radio- frequency identification tags Small Durable Cheap Easy to tag objects Near future: use products own tags IRS bracelet reader Slide 11 Experiment: Morning Activities 10 days of data from the morning routine in an experimenters home (Patterson, Fox, & Kautz 2005) 11 activities, 61 tagged objects Often interleaved and interrupted Many shared objects Use bathroomMake coffeeSet table Make oatmealMake teaEat breakfast Make eggsUse telephoneClear table Prepare OJTake out trash Slide 12 Most Likely Single-State HMM Each activity modeled as an independent single- state Hidden Markov Model Captures relationship between activities and objects Compute probabilities from labeled training data Compute most likely model at each time point 68% accuracy Slide 13 HMM with One State per Activity Single HMM where each activity is a state Captures probability of transitioning between activities Compute most state sequence 88% accuracy Slide 14 HMM with Multiple States per Activity HMM with (# objects) states per activity Idea: model internal structure of activities as well as transitions between activies 87% accuracy Performance degrades Slide 15 Model Complexity Tradeoff Too few parameters: cannot fit data Too many parameters: poor generalization Leta = # different activities b = # objects per activity Number of parameters: HMM with 1 state per activity: O(a 2 + ab) Too simple HMM with b states per activity: O(a 2 b 2 ) Too complex Slide 16 Interleaved HMM Idea: explicitly distinguish progress within an activity from switching between activities (Bai & Kautz 2008) Each activity modeled by an individual HMM User modeled by a set of activities One activity is active, others are suspended When an observation is made, credit it to the current activity, or switch to a different activity Frontier Slide 17 Interleaved HMM Filtering Slide 18 Results Model# ParametersAccuracy Most likely single-state HMM ab68% HMM with one state per activity a 2 + ab88% HMM with (# objects) states per activity a2b2a2b2 87% Interleaved HMM with (# objects) states per activity a 2 +ab 2 98% Slide 19 Interleaved HMM Confusion Matrix Slide 20 Current Work: From Activity Recognition to ADL Prompting Collaboration with Attention Control Systems to integrate activity monitoring into PEAT (Planning and Execution Assistant and Trainer) DARPA Small Business Innovative Research Slide 21 This Talk Monitoring activities of daily living Learning and guiding users through transportation plans Planning as satisfiability Slide 22 Motivation: Community Access for Persons with Cognitive Disabilities Slide 23 Problems Using public transit cognitively challenging Learning bus routes and transfers Recovering from mistakes Point to point shuttle service impractical Slow Expensive Current GPS units hard to use Require extensive user input No help with transfers, timing Slide 24 Solution: Opportunity Knocks User carries GPS-enabled cell phone System infers transportation use System learns model of typical user behavior Novel events = possible user errors Slide 25 Modeling Challenge Create a model of a users Significant places Modes of transportation Transportation plans Approach: Encode a commonsense theory of transportation use as a Dynamic Bayesian Network (DBN) Learn the parameters of the DBN from raw GPS data using Expectation-Maximization Slide 26 Transportation Plans BA Goal: ultimate destination Home, work, friends, stores, doctors, Places: goals or intermediate waypoints Trip segments: Plan = sequence of trip segments Home to Bus stop A on Foot Bus stop A to Bus stop B on Bus Bus stop B to workplace on Foot WorkHome Slide 27 Detecting User Errors Learned model represents typical correct behavior Model is a poor fit to user errors We can use this fact to detect errors! Novelty Normal: model functions as before Abnormal: switch in prior (untrained) parameters for mode and edge transition Slide 28 GPS reading Edge, velocity, position GPS / pathway association Transportation mode Trip segment Goal Novelty Dynamic Bayesian Net z k-1 zkzk x k-1 xkxk k-1 kk Time k-1 Time k m k-1 mkmk t k-1 tktk g k-1 gkgk c k-1 ckck Slide 29 Predict Goal and Path Slide 30 Detecting Novel Events Slide 31 Current Work Modeling multi-person interactions from GPS data Chasing, meeting, hiding, Capture the flag game Strategic understanding Generating prompts using a Markov Decision Process model Slide 32 This Talk Monitoring activities of daily living Learning and guiding users through transportation plans Planning as satisfiability Slide 33 Reasoning about the Physical World The planning problem: Given representations of An initial state of affairs A desired (goal) state of affairs A set of actions defined by preconditions and effects Find a (shortest) sequence of actions that transforms the initial state into a goal state All useful formalizations of planning are NP- complete or worse In practice: desire low-exponential algorithms Slide 34 Planning as Inference Advice taker (McCarthy 1959) Proof checker Planning as first-order theorem proving (Green 1969) Computationally infeasible STRIPS (Fikes & Nilsson 1971) Limited theorem proving + state-space search Very hard UCPOP (Weld 1992) Specialized theorem prover for partial-order planning Starting to scale up SATPLAN (Kautz & Selman 1996, 1999, 2000, 2006, 2007) Planning as general propositional reasoning Solves many hard problems (60+ steps) optimally Slide 35 Planning as Satisfiability Time = bounded sequence of integers Translate planning problem to a Boolean formula Find a satisfying truth assignment using a SAT engine Translate truth assignment to a plan Slide 36 Benchmark Results International Planning Competitions (2004 & 2006) 1 st place for deterministic optimal planning Slide 37 Key Improvements Stochastic local search Clause learning Restart strategies Slide 38 Future Work Extend SATPLAN approach to Planning with metric costs & resources Planning under uncertainty Integrated planning & plan recognition Slide 39 Supra-Human Planning International planning competition benchmarks ; Time 177.13 ; ParsingTime 0.00 ; MakeSpan 8 0: (TURN_TO SATELLITE0 STAR2 STAR8) [1] 0: (TURN_TO SATELLITE1 STAR0 GROUNDSTATION3) [1] 0: (TURN_TO SATELLITE2 STAR2 STAR4) [1] 0: (TURN_TO SATELLITE3 STAR2 PHENOMENON9) [1] 0: (TURN_TO SATELLITE4 STAR2 PHENOMENON9) [1] 0: (SWITCH_ON INSTRUMENT0 SATELLITE0) [1] 0: (SWITCH_ON INSTRUMENT3 SATELLITE1) [1] 0: (SWITCH_ON INSTRUMENT4 SATELLITE2) [1] 0: (SWITCH_ON INSTRUMENT7 SATELLITE3) [1] 0: (SWITCH_ON INSTRUMENT8 SATELLITE4) [1] 1: (CALIBRATE SATELLITE1 INSTRUMENT3 STAR0) [1] 1: (CALIBRATE SATELLITE2 INSTRUMENT4 STAR2) [1] 1: (CALIBRATE SATELLITE3 INSTRUMENT7 STAR2) [1] 1: (CALIBRATE SATELLITE4 INSTRUMENT8 STAR2) [1] 1: (TURN_TO SATELLITE0 STAR0 STAR2) [1] 2: (TURN_TO SATELLITE1 STAR10 STAR0) [1] 2: (TURN_TO SATELLITE2 PHENOMENON7 STAR2) [1] 2: (TURN_TO SATELLITE3 STAR5 STAR2) [1] 2: (TURN_TO SATELLITE4 PLANET6 STAR2) [1] 2: (CALIBRATE SATELLITE0 INSTRUMENT0 STAR0) [1] 3: (TURN_TO SATELLITE0 PLANET6 STAR0) [1] 3: (TAKE_IMAGE SATELLITE1 STAR10 INSTRUMENT3 THERMOGRAPH2) [1] 3: (TAKE_IMAGE SATELLITE2 PHENOMENON7 INSTRUMENT4 INFRARED1) [1] 3: (TAKE_IMAGE SATELLITE2 PHENOMENON7 INSTRUMENT4 IMAGE3) [1] 3: (TAKE_IMAGE SATELLITE2 PHENOMENON7 INSTRUMENT4 INFRARED0) [1] 3: (TAKE_IMAGE SATELLITE3 STAR5 INSTRUMENT7 IMAGE3) [1] 3: (TAKE_IMAGE SATELLITE4 PLANET6 INSTRUMENT8 INFRARED0) [1] 3: (TAKE_IMAGE SATELLITE4 PLANET6 INSTRUMENT8 INFRARED1) [1] 4: (TURN_TO SATELLITE2 PHENOMENON15 PHENOMENON7) [1] 4: (TURN_TO SATELLITE3 STAR8 STAR5) [1] 4: (TURN_TO SATELLITE1 STAR1 STAR10) [1] 4: (TURN_TO SATELLITE4 GROUNDSTATION3 PLANET6) [1] 5: (TURN_TO SATELLITE0 PLANET13 PLANET6) [1] 5: (TURN_TO SATELLITE1 PLANET14 STAR1) [1] 5: (TURN_TO SATELLITE4 PHENOMENON7 GROUNDSTATION3) [1] 5: (TAKE_IMAGE SATELLITE2 PHENOMENON15 INSTRUMENT4 INFRARED1) [1] 5: (TAKE_IMAGE SATELLITE2 PHENOMENON15 INSTRUMENT4 INFRARED0) [1] 5: (TAKE_IMAGE SATELLITE3 STAR8 INSTRUMENT7 IMAGE3) [1] 6: (TURN_TO SATELLITE2 STAR17 PHENOMENON15) [1] 6: (TURN_TO SATELLITE3 PLANET16 STAR8) [1] 6: (TURN_TO SATELLITE4 STAR11 PHENOMENON7) [1] 6: (TAKE_IMAGE SATELLITE0 PLANET13 INSTRUMENT0 SPECTROGRAPH4) [1] 6: (TAKE_IMAGE SATELLITE1 PLANET14 INSTRUMENT3 THERMOGRAPH2) [1] 7: (TAKE_IMAGE SATELLITE2 STAR17 INSTRUMENT4 INFRARED0) [1] 7: (TAKE_IMAGE SATELLITE3 PLANET16 INSTRUMENT7 IMAGE3) [1] 7: (TAKE_IMAGE SATELLITE4 STAR11 INSTRUMENT8 INFRARED1) [1] 7: (TURN_TO SATELLITE0 PHENOMENON9 PLANET13) [1] 7: (TURN_TO SATELLITE1 STAR4 PLANET14) [1] Slide 40 Supra-Human Planning Industrial planning Lithographic printing production planning Complex set of discrete & geometric constraints Expert+ performance (40 job ganging) Domain-specific implementation of SAT techniques Slide 41 But, also Discovery of social relationships in consumer photo collections Recognition of activities of daily living using multi-scale, multi- perspective vision and RFID Applying SAT techniques to model-counting and probabilistic inference Slide 42 Conclusion An early goal of AI was to create programs that exhibited commonsense This goal proved elusive Missing efficient methods for logical & probabilistic reasoning Lacked grounding in real world Lacked compelling applications Today we have the tools, the sensors, and the motivation Slide 43 Acknowledgements Tongxin Bai, Jeff Bilmes, Gaetano Borriello, Tanzeem Choudhury, Kate Deibel, Oren Etzioni, Dieter Fox, Carla Gomes, Craig Harman, Harlan Hile, Eric Horvitz, Kurt Johnson, Lin Liao, Alan Liu, Joseph Modayil, Don Patterson, Parag, Sangho Park, Bill Pentney, Matthai Philipose, Yongshao Ruan, Ashish Sabharwal, Tian Sang, Bart Selman, Dan Weld, Danny Wyatt