Human Robot Teams: Concepts, Constraints, and Experiments Research Agenda Evaluation Technology

Human Robot Teams: Concepts, Constraints, and Experiments

Research Agenda

• Evaluation Technology

• Neglect Tolerance
• Behavioral Entropy
• Fan-Out

• Interface Design

• Mixed Reality Displays
• Principles
• HF Experiments

• Autonomy Design

• Team-Based Autonomy
• UAVs
• Perceptual Learning

The Presentation Agenda

• The types of questions

• Neglect tolerance: Is a team feasible?

• How do we compute neglect tolerances?

• Tradeoffs: workload and performance

• Is a team optimal?

• The problem with switch costs

• Some limits, ideas, and proposals

A Special Case: The Robotics Specialist

• One soldier

• Two UAVs

• One UGV

• Can one person manage all three assets?

• At what level of performance?

• At what level of engagement?

A More General Case: Span of Control

• How many “things” can be managed by a single human?

• How many robots?

• How do we measure Span of Control in HRI?

• Relationships between NT and IT

• How do we compare possible team configurations?

• Evaluate performance-workload tradeoffs
• Identify performance of feasible configurations

The Most General Case: Multiple Robots & Multiple Humans

• How many people are responsible for a single robot?

• How many robots can provide information to a single human?

The Presentation Agenda

• The types of questions

• Neglect tolerance: Is a team feasible?

• How do we compute neglect tolerances?

• Tradeoffs: workload and performance

• Is a team optimal?

• The problem with switch costs

• Some limits, ideas, and proposals

Neglect Tolerance: Neglect Time and Interaction Time

• How long can the robot “go” without needing human input?

• How long does it take for a human to give guidance to the robot?

Fan-Out (Olsen 2003,2004): How many homogeneous robots?

• How many interaction periods “fit” into one neglect period

• Two other robots can be handled while robot 1 is neglected

• Fan-out = 3

Can a human manage team T ? Fan-out and Feasibility

• Fan-out (homoeneous teams)

• Feasibility (heterogeneous teams)

• These are upper bounds

The Presentation Agenda

• The types of questions

• Neglect tolerance: Is a team feasible?

• How do we compute neglect tolerances?

• Tradeoffs: workload and performance

• Is a team optimal?

• The problem with switch costs

• Some limits, ideas, and proposals

Neglect Impact Curves

• A task is Neglected if attention is elsewhere

• Neglect impacts task performance: 2ndary tasks

Not Neglect Tolerant Enough

Too Neglect Tolerant

• Old Glory Insurance

Interface Efficiency Curves

• Recovery from “zero” point

• Imprecise switch costs

Efficient Interfaces

• PDA-based UAV control (versus command line)

Efficient Interfaces

• Phycon-based UAV control (versus command line)

Finding NT and IT from the curves

Example

Validation of Method: Complexity

• As complexity goes up, NT goes down and IT goes up

• Feasibility using NT/IT needs more work

The Presentation Agenda

• The types of questions

• Neglect tolerance: Is a team feasible?

• How do we compute neglect tolerances?

• Tradeoffs: workload and performance

• Is a team optimal?

• The problem with switch costs

• Some limits, ideas, and proposals

Existing Tradeoffs

Types of Autonomy

Using Tradeoffs to Select a Configuration

Tradeoffs Galore

• Higher workload means shorter missions

• Higher performance requires higher workload

• Higher workload implies smaller span of control

• Lower risk tolerance implies shorter neglect times

• Longer neglect times imply greater risk of unperceived failures

• Longer neglect times imply more complicated “other” tasks

• More complicated “other” tasks imply greater switch costs

The Presentation Agenda

• The types of questions

• Neglect tolerance: Is a team feasible?

• How do we compute neglect tolerances?

• Tradeoffs: workload and performance

• Is a team optimal?

• The problem with switch costs

• Some limits, ideas, and proposals

Predicting Performance of a Heterogeneous Team

• Each robot may have multiple autonomy modes and interaction methods

• Each interaction scheme yields NT, IT, and average performance values

Predicting Performance continued …

Using Predicted Performance

Accuracy of Predictions in a Three-Robot Team

The Presentation Agenda

• The types of questions

• Neglect tolerance: Is a team feasible?

• How do we compute neglect tolerances?

• Tradeoffs: workload and performance

• Is a team optimal?

• The problem with switch costs

• Some limits, ideas, and proposals

What are switch costs?

• The biggest unknown influence on span of control

• They come in several flavors:

• Time to regain situation awareness
• Time to prepare for switch
• Errors and Change Blindness

Before and After

Getting a Feel for the Experiment

The Experiment

• Primary task: Control a robot

• Vary type and duration of secondary task

• Measure speed and accuracy of change detection

• Measure speed and accuracy of change diagnosis

Preliminary Results

• 6 subjects, none naïve

• 207 correct change detections

• One-sided T-test, equal variances

Important Trends

• Differences not just from “time away”

• blank and tetris have same time
• UAV and tone have same time
• Averages nearly identical

• Differences not just from “counting”

• UAV and tone both count

• Differences not just from “motor channel”

• UAV and tone both select
• Tetris requires interaction

• Probably spatial reasoning and changing perspectives

Summary of Preliminary Results

• If it takes longer than 20 seconds to diagnose a change, the subject has probably failed

• Need to gather failure rate

• Averages show very strong trend

• We conclude:

• The test is sensitive enough to detect differences
• The type of secondary task affects recovery

The Presentation Agenda

• The types of questions

• Neglect tolerance: Is a team feasible?

• How do we compute neglect tolerances?

• Tradeoffs: workload and performance

• Is a team optimal?

• The problem with switch costs

• Some limits, ideas, and proposals

How Many Robots?

• Assumptions

• Goal: Gather battle-related information while minimizing risk
• Media: Mostly camera/video information

• Prediction

• Interpreting camera information difficult
• High robot autonomy won’t help enough

A Special Case: The Robotics Specialist

• Can one person manage multiple robot assets?

• At what level of performance?

• Goal: gather information

• Media: visual (camera/video)

• Belief: autonomy will help, but not enough

A Research Agenda

• Phase 1: Refine assessment technique

• Validate sensitivity
• Assess feasibility with switch costs

• Phase 2: Study interfaces

• Select plausible secondary tasks
• Compare information presentation techniques
• Compare PDAs, tablets, workstations
• Compare interaction while stationary with in-motion

• Phase 3: Study autonomy

• Vehicle control
• Team playbooks
• Interactive perception

Pushing the limits

• Mixed reality displays

• Let the operator control the camera, not the robot

• robot controls its motion to support camera

• Highlight changes

• Mitigate change blindness effects
• TiVO

• Support task switching

• Improve robot perception by teaching it

• Automate image understanding

• Effects of false alarms on the human?
• Costs of missed detections on the mission?

• 2 to 3, or 3 to 5 ratios: redundancy and responsibility swapping

Mixed Reality Displays

• Eliminate “The world through a soda straw”

• Integrate vision with active sensors

• Integrate display with autonomy

• Include sensor uncertainty

• Control pan-and tilt

• Study time delay effects

Real World Results

• Objective

• 51% Faster (p < .01)
• 93% Less Safeguarding (p < .01)
• 29% Lower Entropy (p < . 05)
• 10% Better on Memory Task (p < .05)

• Subjective

• 64% Less Workload / Effort (p < .001)
• 70% More Learnable (p < .0001)
• 46% More Confident (p < .05)

Several Thousand Words

Experiment Results

Ecological Display: Experiment Results

• Mixed-Reality better for

• Time to complete
• Number of collisions
• Subjective workload
• Entropy

• Even stronger results for real-world

• Twice as long
• Better performance on secondary task

Mixed Reality Displays (Pan and Tilt)

Control the Information Source, Not the Robot

• Phlashlight Concept

• What will UAV see?

Semantic Maps and Change Highlighting

• Video in context

• Icon-based maps w/ semantic labels

• “That was then, this is now comparison” --- change highlighting

• Information decay

Information in Context

Support Timely Shifts

• Prompt prospective memory

• Shift in a timely way

• Give time to prepare

Supporting Task Switching: Etc.

• History trails. Knowing recent past helps

• Tail on a map-based interface
• Virtual descent into video-based interface
• Change highlighting/morphing

• Plans: Knowing intention helps

• Planned path on map-based interface
• Predicted trajectory on video-based interface
• Quickened displays

• Task relationships: Knowing relationship between two tasks helps

• Relative spatial location on map-based interface
• Picture-in-picture on video-based interface
• Progress bar of task X on task Y’s display

Improve Perception and Scene Interpretation (Olsen)

• Use interaction and machine learning to make this robust

Future Concept (Proposed)

• Safe/Unsafe occupancy grids

• Evolutionary image classifier
• Evolutionary integration of vision and lasers
• Particle-based inverse perspective transform

• Path planning

• Uncertainty-based triggers for retraining

Conclusions

• We can evaluate team feasibility

• We can predict team performance

• We need to understand task switching better

• We need to support realistic task switching

• Via interfaces
• Via autonomy

Near-Term Future Work

• Complete validation of task switching experiment paradigm

• Compare “new and improved” interfaces against baseline

• Compare effects of type and size of interface

• Answer the questions for the special case