Intelligent sensor and learning challenges for context aware appliances >> Stéphane Canu

Intelligent sensor and learning challenges for context aware appliances

• >> Stéphane Canu

• [e-mail]

• asi.insa-rouen.fr/~scanu

• INSA Rouen, France - EU

• Laboratoire PSI

1984: La souris et leMacintoch

La technologie d'aujourd'hui

• Loi de Moore

• Communication "sans fil"

• L'ère des données

Wearable

IHM

Wearable

Reasearch on wearable

Wearable

context aware appliances

General Motors and CMU

Oops! Where is my car?

• Old fashion software design: process

• Match the sentence
• Send the query to the satellite
• Satellite send query to the car on its own frequency
• Car answers…
• Tell the computer what to do (where is the switch)

• Distributed software design: interaction

• Software agents talk together

• Future way: Programming by Example

• Show the computer what to do

• Today's solution: Louis my 3 years old son

Calm technology

• Ubiquitous computing

• One people - many computer

• Technology at our service

• Reactive to what user do
• Proactive - Prepare what to do next
• Situated – sharing context
• (Hans Gellersen, Sensing in Ubiquitous Computing)

• Adapted to our needs

• New functionalities and new behaviors
• New way of communicating
• Learn to adapt

What is the context?

• user

• activity (available/meeting)
• location,
• identity, profile

• environment monitoring

• time, day/night, temperature, weather,
• resources (networks, services…)

• appliance - proprioception

• usage - functionalities
• maintenance
• resources (energy…)

Sensing context from the environment presentation roadmap

Context from data

• Unbelievable capacity

• Moore’s law

• New sensors

• Artificial nose
• Bio sensor

• “Personal” data

• humor: affective computing

Biological sensors

Too much information kills information

• Critic of the "Data Era"

• Data smog

• Non measurable things

• Ethical consequences

• the Orwellian future

Intelligent sensors

• Requirements:

• Data
• Accuracy and confidence
• Self diagnostic
• Self calibration

• How to do it?

• Uncertainty management
• Learning ability
• Network + database
• Adaptation ability
• Fault detection mechanism

Data validation

• Mono sensor validation

• Static validation
• Mean, variance
• Dynamic validation
• Cusum (control charts)
• Trend analysis

• Multisensor validation

• Residual analysis
• Fusion: Joint probability estimation
• Prior knowledge: Balanced relations

• Hierarchical validation

• Multisensor perception

Software sensor

• Value + confidence interval + validity domain

• How to build it ?

• From a model: tracking = Kalman filter
• When no model is available: learn it!

Towards proprioceptors

• Learn

• How to learn?

• Gaussian mixture + EM
• Include prior: Bayesian networks
• Deal with uncertainty: Evidence framework

• Use to:

• Detect non nominal situations
• Replace missing data

What is data?

• Individuals or measurements

• Associated variables

• Data set (matrix)

• line = measurements
• column = variable

• Data: point clouds

• Data exploration: recognize patterns

Summarize data

• Non linear components analysis

• Feature space: kernel (PCA or ICA)
• Local linear
• Quantisation (SOM)
• Relevant distance

• Select features

• Local adapted representation
• Feature selection

• Select relevant situations

• Sparse learning
• Kernel learning

Kernel representation Distance maps

Example of kernel map

Looking for hiden shapes

• Data point = information + noise

• Principal curve

• Non linear PCA

• Independent curve

• Non linear ICA

Information retrieval

• What for

• User profiling
• User identification
• Battery discharge rate
• Sequence induction…

• Classification problem

• Decision theory
• Example based programming
• Learning machine

A brief historical perspective of machine learning

• Before machines

• Statistics: PCA, DA, regression, CART, kNN

• 70's - Learning is logic

• Grammatical inference in expert systems

• 80's - Learning is human

• Neural networks: backprop

• 90's - Learning is a problem: COLT

• Kernel machines: SVM
• Mixture of experts: adaboost

What is learning?

• Data

• Training set
• Test point looking for such that

• Learning is balancing

• Hypothesis set (Neural networks, Kernels)
• Fitting criterion (least square, absolute value)
• Compression criterion (penalization, Margin)
• Balancing mechanism (cross validation, generalization)

Linear discrimination separable case

Linear discrimination separable case

The classifier Margin

Maximize the margin Be sparse

What is learning?

Summarize Input adaptive scaling

• Enumerate all combination

• …and score

• Preprocessing

• Information theory
• Statistical test

• Wrapper

• Use a relevance index
• Learn and select together

Summarize patterns

Learning machines challenges

• Hypothesis set

• Multi scale data representation: wavelets
• Use context: mixture of experts

• Fitting criterion

• Sparse distance criterion
• Select relevant input (adaptive scaling)
• Relevant distance: adapt the kernel

• Compression criterion

• Information issues
• Global optimization

• Balancing mechanism

• Efficient direct algorithm (one shot learning)

Context assessment

• Deal with uncertainty

• plausibility / credibility
• unknown states / ability to evolve
• data fusion: evidence theory

• Take into account prior knowledge: transitions

• temporal representation
• uncertain transitions
• learn probabilities or possibilities

• Learn the model

• don't start from scratch
• create and delete contexts

• Adapt context determination to user

• from a global imprecise context to specific context

Context implementation

• Context = state

• List of variables
• Petri's nets

• State = stochastic

• Markov model
• Bayesian networks

• Identify = decision theory (data fusion)

• Information retrieval

• Learn context

• Knowledge discovery
• Create / delete
• Context hierarchy (time granularity)

New idea to deal with context

• Current context: working memory

• Prior knowledge: transition law

• Available information: evidence

• Data fusion

• Learn context

• Transition law
• Context retrieval from data

• Context is a language

• Speech recognition

• Markovian model
• Evidence
• Language + previous state
• Locator's adaptation

Context: Research chalenges

• Inputs

• Deal with uncertainty (and missing data)
• Representation
• Data fusion (multimedia fusion)

• Context

• Define a language
• Represent previous state
• Learn transition

• Feed Back to inputs

• Adapt transition to the user

• Loop the user: reinforcement
• Control mechanism (stability/plasticity dilemma)

Break through

• What is information?

• Computer science
• Coding
• Signal

• Mathematics

• Statistics & computer science
• Pattern recognition
• Functional analysis
• ??????

My long bet

Research challenges

• create context

• how to define prior contexts: user’s needs
• how to represent contexts: stochastic automaton
• learn from data: modify, create and destroy context

• decide context

• validate data software sensors
• select relevant inputs representation + distance
• select relevant patterns wavelets
• select relevant situations SVM and kernel
• make decision using data fusion Dempster-Shafer + EM

• loop with the user

• reinforcement learning
• user’s needs

Questions?

• Asia

• Scurry™, Wearable & Virtual Keyboard - Samsung,
• K. Doya for reinforcement

• America

• Context Aware Computing group - Media lab MIT
• CMU, Stanford
• Georgia tech: Future Computing Environments
• Smart Matter Integrated Systems (Xerox PARC)
• Montreal – learning lab

• Australia

• ANU for learning
• University of South Australia -  wearable computer lab

• Europe

• Telecooperation Office (TecO) at the University of Karlsruhe
• The disappearing computer, a EU-funded proactive initiative
• The Smart-Its project
• Equator project focuses on the integration of physical and digital interaction
• Perceptual Computing in general and Computer Vision in ETH Zurich
• IDIAP for machine learning and speech recognition
• PSI, France for learning

From macroscopic…

Emotion detection >> E-Motions

Learning accuracy

• Find (a,b) such that

• Model the model

• The sandwich estimator, (Tibshirani, 1996)
• Likelihood Based on the Hessian matrix
• Confidence machine (Gammerman RHC, 1999)
• Confidence: 73.11% - Credibility: 51.37%

• Sample the models

• Bootstrap (Heskes 1997)

• Learn the error

• Train using absolute error

Movie synthesis from text

From one expression to another

Non euclidian metrics

Hyperbolic Self-Organizing Map

Example on movies - HSOM

Example on movies

Curvlets

Curvlets

Select relevant situations

• Relevant representation

• "Invent" features
• Select features

• Relevant "distance"

• map
• Use kernel

• Summarize the examples

• Define a relevant global criterion to be minimized
• Support vector machines (SVM)

Learning architecture

• Agent - Data base - Communication

• Metadata

• Context language

• Adaptability: control mechanism

• Pre programming: anticipation

• Open – modular – distributed

• The Ektara Architecture (MIT for wearable)
• Nexus - A Platform for Context-Aware Systems
• The Context-Toolkit (Geargia Tech)

Future appliances?

• Deal with the context

• Recognize
• Adapt
• Create

• Inference, Learning, discovery,

• Represent
• Decide
• Deal with time

• From user interface to user interaction

• Reinforcement learning
• Human factors

• How to know what we need?