Standard Test Methods For Response Robots ASTM ... Test Methods For Response Robots ASTM...

1

www.nist.gov/el/isd/ms/robottestmethods.cfm

Standard Test Methods For Response RobotsASTM International Standards Committee on Homeland Security Applications;

Operational Equipment; Robots (E54.08.01)

Overview of the Challenges and Opportunities forthe Testing and Use of Robotic Technologies

at Nuclear Facilities

Presenter:

Kamel S. Saidi, Ph.D.Intelligent Systems Division

National Institute of Standards and TechnologyU.S. Department of Commerce

Prepared for the:

U.S. NRC’s 28th Annual Regulatory Information Conference

Session T9 - Deployment of Robotic Technologies at Nuclear Facilities




2

Standard Test Methods for Evaluating, Purchasing, and Training with Response Robots

Test Director:

Adam JacoffIntelligent Systems Division

National Institute of Standards and TechnologyDepartment of Commerce

Sponsor:

Phil MattsonOffice of Standards

Science and Technology Directorate Department of Homeland Security




Definition of a Response Robot

“A remotely deployed device intended to perform operational tasks at operational tempos.” ASTM E2854

• A response robot should serve as an extension of the operator to improve remote situational awareness, provide means to project operator intent through the equipped capabilities, and reduce risk to the operator while improving effectiveness and efficiency of the mission.

3

2




Challenges for the Use of Robots…

• …at Nuclear Facilities– Radiation / HAZMAT / fire /

water / collapsed structures

– Confined spaces

– Indoor/outdoor, underwater, in air

– Communications

– No room for errors

– Relatively well-knownenvironments (except during accidents)

4




Challenges for the Use of Robots…

• …at Nuclear Facilities– Radiation / HAZMAT / fire /


– Confined spaces


– Communications


– Relatively well-known environments (except during accidents)

• …for Emergency Response– Explosives / HAZMAT / fire /


– Confined spaces


– Communications


– Unknown environments

5




Challenges for the Use of Robots

• The challenges are not technological, but technical.

• In most cases, the technologies needed to solve a particular problem robotically probably already exist.

• Robotic implementations are often not sufficiently technically developed or evaluated.

6

3




“Standard Test Methods”A consensus-based process for determining the performance of a product.

Apparatus: A repeatable, reproducible, and inexpensive representation of a

task you expect the robot to perform.

Procedure: A script for a user to follow.

Metric: A quantitative measure of performance.

Manipulator Dexterity TasksSensor Targets for Visual Acuity

Mobility Terrains and Obstacles

7




A Comprehensive Suite of 50 Test Methods for Ground, Aerial, and Aquatic Systems

Mobility

Dexterity

Endurance

Sensors

Radio Comms

Durability

Logistics

Safety

Autonomy

Proficiency

8




9

Apparatuses Scale to Intended EnvironmentsIncreasing Complexity with Same Procedure and Metric

Gravel

Sand

Crossing Ramps

SymmetricStepfields

Rubble Pile, Disaster City, College Station, TX

4




10

Apparatuses Scale to Intended EnvironmentsIncreasing Complexity with Same Procedure and Metric

Gravel

Sand

Crossing Ramps

SymmetricStepfields

Mini Arena Scale: 30cm (12 in) Lateral Clearance)




11

Aquatic

Examples of Aquatic Test MethodsBaseline Capabilities in a Water Tank Before Adding Variables




12

Aerial

Examples of Aerial Test MethodsBaseline Capabilities for Small UAS (<5kg)

5




Quantitative Comparison of CapabilitiesWorks for Robots and/or Operators on a Given Robot

0

10

20

30

40

50

60

70

80

ReconScout FirstLook Armad-Base SandFlea DR10-Base Armad-Manip DR10-SCK Pointman

480030

41

91176

25

1113

38

14171919

30

2115

64

191719

8121411

75

3830

3832303030

3838

25

383830

38

0

(met

ers

per

min

ute

)

Sand Gravel Flat Line Following Continuous Ramps Crossing Ramps Stepfields

ROBOT A ROBOT B ROBOT C ROBOT D ROBOT E ROBOT F ROBOT G ROBOT H

Average Rate of Advance on Terrain (at least 100m/10repetitions)

13




Standard Test Methods Help Different Stakeholders

Help Robot Developers to:

Better understand missions

Practice and refine robot designs, and make trade-off decisions

Highlight their “Best-In-Class” capabilities

Help Robot Users to:

Compare robots with objective data, not marketing

Specify purchases based on existing measured capabilities

Align expectations with deployment decisions

Help Program Managers to:

Describe objectives with a collection of tangible tasks

Challenge conventional approaches and inspire innovation

Measure baseline capabilities and document progress

14




Validation Exercises for Robot DevelopersVarious Robot Sizes and Capabilities To Ensure Tests Scale Effectively

Cobham Telemax

80kg (175lbs)

ICOR Caliber MK3

84kg (185lbs)

Remotec HD-SEL

111kg (245lb)

iRobot 710 Kobra

166.5kg (367lbs)ICOR Caliber T5

64kg (140lbs)

iRobot 110 FirstLook

2.4kg (5.2lbs)

Qinetiq Dragon Runner 10

4.5kg (10lbs)

ICOR Caliber Mini

27kg (65lbs)iRobot 310 SUGV

13.2kg (29lbs)

Remotec Titus

61kg (135lbs)

Remotec Wolverine

367kg (810lbs)

Remotec Mark 5-A1

358kg (790lbs)Remotec F6B

220kg (485lb)

WM Robotics Knight

249kg (550lbs)

15

6




Validation Exercises for Robot UsersMission Task Representations, Variable Settings and Complexity, Sequences

30 Test Methods: 10 Basic Skills, 15 IED/PBIED/VBIED, 5 Building Access

16




Validation Exercises for Operator Training(4) JPO C-IED Interoperability Exercises (aka Raven’s Challenges)

30 Test Methods: 10 Basic Skills, 15 IED/PBIED/VBIED, 5 Building Access

17




18

SponsorsAdopting and Developing New Test Methods

Robot PurchasesUsing Data From Existing Test Methods

7




Facilities in USA110 Requests from Bomb Squads

(of 466 nationwide) Existing Facilities (Validate/Train)

FY16 Plan

Washington

Oregon

California

Nevada

Idaho

Montana

Utah

ArizonaNew Mexico

Colorado

Wyoming

North Dakota

South Dakota

Nebraska

Kansas

Oklahoma

Texas

Minnesota

Iowa

Missouri

Wisconsin

IllinoisIndiana

Michigan

Ohio

Arkansas

Louisiana

Kentucky

Tennessee

MS AlabamaGeorgia

Florida

South

Carolina

North Carolina

Virginia

West

Virginia

Pennsylvania

New York

NJMD

VTNH MA

Maine

RICT

DE

DC

• NIST’s Robot Test FacilityGaithersburg, MD (2006)

• Southwest Research InstituteSan Antonio, TX (2010)

• University of MassachusettsLowell, MA (2013)

• Army Camp DawsonWV (2013)

• Air Force Tyndall AFBPanama City, FL (2014)

• San Diego Fire DepartmentSan Diego, CA (2014)

• Port of Seattle Police Dept.Seattle, WA (June 2015)

• NY State Dept. of Homeland Security Oriskany, NY (June 2015)

• Navy SPAWARSan Diego, CA (2016)

19




New Facilities

• US State Department Anti-Terrorism Assistance Kabul, Afghanistan (2015)

• Korean Atomic Energy Research InstituteSeoul, South Korea (2016)

• Japanese Atomic Energy AgencyFukushima, Japan (2016)

20




21

• DARPA Robotics Challenge Trials, Homestead, Florida – December 2013

• DARPA Robotics Challenge Finals, Pomona, California – June 2015.

• Validation of ground robot test methods, Japan Atomic Energy Agency (JAEA) and Tokyo Electric Power Company (TEPCO) Test Facility (under construction) in Naraha, Japan – Date TBD

• Validation of ground, aerial, and aquatic robot test methods, Japanese Ministry of Economy, Trade and Industry (METI) Robot Testing Field and International Collaboration Facility in the Hamadori area of Fukushima Prefecture, Japan – Date TBD

Extension of ASTM E54 Response RobotStandard Test Methods to Nuclear Applications

8




22

Questions?

[email protected]

[email protected]




Simple Rules for Evaluating Robots and Measuring Proficiency

Select all applicable tests (20+ typically apply) in scale of deployment environment.

Freeze robot configuration for entire suite of tests to capture trade-offs.

Operate remotely, out of sight and sound of robot as if downrange.

Use “expert” operators to capture best possible robot capabilities and 100% level of operator proficiency for comparisons.

Capture statistically significant repetitions (10-30 reps) to establish reliability.

Compare quantitative scores:

Robot capabilities across all tests.

Operator proficiency relative to 100% across all tests using the same robot.




Basic Skills: Maneuvering (5)

Drive forward then reverse | 10 minute time limits | 50 minutes overall

9




Basic Skills: Dexterity and Strength (5)

Drive as necessary | 10 minute time limits | 50 minutes overall




Mission Essential Tasks: Small Objects (5)





Mission Essential Tasks: Large Objects (5)


10




Mission Essential Tasks: Building Access (5)

Drive as necessary | 10 minute time limits | 50 minutes overall (covered with tarp for dark

environment)




Standard test methods are not intended to replace regular scenario training. They provide a rigorous method to prepare for and augment such training.

Similar to sports examples:

Circuit training —- 5 or 10 minute trials on each apparatus to train your entire skill set. Repeat often

Golf driving range — practice skills, gain muscle memory

Quick training tasks can be performed as a group rotating through several test methods simultaneously with robots remaining in each apparatus.

Techs can practice and train by themselves if they miss a training day.

Now you can measure your operator proficiency as a percentage of “expert” performance on the same robot!

“Standard Test Methods in a Box”A New Approach for Operator Training and Evaluation – Worldwide!

Standard Test Methods For Response Robots ASTM ... Test Methods For Response Robots ASTM...

Documents

Transcript of Standard Test Methods For Response Robots ASTM ... Test Methods For Response Robots ASTM...