Adapted typographical annotations for language learning: user-profiled design solutions to problems of auditory perception and speech production
CNRS Project 2003-2005 : TCAN Traitement Cognitif Apprentissage et NTIC
The development of a prototype authoring system using visual cues for improving the perception of stress patterns in foreign languages, in particular in English
SWANS = Synchronised Web Authoring Notation System
Most francophone speakers do not consciously store information on stress patterns. During oral presentations , stress is placed randomly on the second or third syllable of the word “ deVELopment ” indicating not ‘deafness’ or a problem of production but negligence of memory storage.
Communication depends on “tolerance thresholds”. It can breakdown after only a few stress errors.
The challenges
To define the rules of English stress (Alain Deschamps 94, d’après Guierre) and an appropriate learning method (Théo van Leeuwen : Visual Design, Rudolph Arnheim : Visual Weight)
To develop protocols for evaluating learning (Sweller, Mayer, Paivio)
To develop an ‘Intelligent Environment for Human Learning’ where visual perception can help improve auditory perception
To develop a prototype authoring system to make the process of textual annotation semi-automatic
70% to 90% of our French students get the accent wrong
Reading and subvocalizing ( silent reading) can reinforce bad habits
A psychological barrier
In terms of pronunciation, one might argue, the Hundred Years’ War is far from ended. Many French students refuse to leave the comfortable foothills of their own relatively low pitch frequencies for a roller coaster ride in English high frequencies. These are French or Latin words on the screen. Why on earth should I employ such unnatural stress patterns?
Are the problems with English stress patterns the same for all nationalities?
No
In Czech, Latvian, Hungarian, Swiss German (Bernese dialect), Finnish, and Swahili, stress is always placed on the first syllable.
In French, Spanish, Portuguese, Turkish, and Polish stress is rarely placed on the first syllable.
Textual annotation must adapt to student mother-tongues
Language centres must offer texts which place annotations on words where English the stress pattern will probably cause problems of perception for novice learners according to their mother-tongue
A battering ram of a solution
It is suggested that adult L2 learners might counter L1 interference effects through managing to receive “exaggerated acoustic cues, multiples instances of many talkers, and massed listening experience” (McCandliss et al 2002; McClelland et al.1999)
Hypothesis of dual coding (Paivio)
Dual coding (sound + text + paralinguistic markers) should improve learning for novice learners while having no effect or a negative effect for those who already know the place of the accent.
Textual annotation techniques for learning pronunciation
How far can we go ?
When does the brain start to complain because reading becomes too slow or because codes seem too obscure ?
What constitutes cognitive overload ?
Robertson method circa 1850
The Robertson Method
Brazil 1994
Sounds Right 2002
Can typography change ?
Is our software capable of offering new more memorable techniques for learning pronunciation ?
Are we ready for a more flexible, animated and colourful alphabet ?
Does reading on a computer screen change learning ?
Simultaneous reading and listening
Cinema : 1931 subtitles to the ‘Jazz Singer’ in Danish
BBC television 1938, opera + subtitles
Karaoke :1970’s
Language Learning : Audio Partner Teleste 1995
How does memory behave after dual coding ?
Should synchronised reading be intensive or extensive ?
Does synchronised reading inhibit ‘L1 interference’ during silent reading?
How does memory behave when we change colours?
How does memory behave when text moves ?
Three « levels of representation for words in memory » (= triple storage):
conceptual level
semantic-syntaxical level (lemmes)
formal level (lexèmes)
Willem Levelt, Modèle de la production langagière
(résumé from de Bot, Paribakht & Wesche 1997, 312)
Levelt’s analysis is based on student reading of paper documents
Toulouse tests of dual coding at the word level and at the sentence level constitute an early and limited analysis of the potential of the new plasticity in computer based reading
Over 200 students participated
Sound recordings of student pronunciation were stored to examine stress perception
Tests with students in Toulouse
Problem : The adult brain is conditioned by typography
« Freeing ourselves from chirographic and typographic bias is probably more difficult than any of us can imagine » Ong 1982
Annotation techniques
Synchronisation with SMIL
The arrival of SMIL (Synchronised Multimedia Integration Language) in 1999 and the work of the W3 Consortium - ‘Timed-Text Initiative’ (2003) - constitute an important breakthrough. SMIL allows fine-tuned synchronisation of sound and text combined with a heightened degree of typographical plasticity for on screen textual annotation.
How do you synchronise sound and text ?
Problem : the absence of annotated and synchronised texts
Programming ‘by hand’ with ‘Magpie’ and ‘Dreamweaver’, it takes two hours to annotate and synchronise a short text with a film or a sound file and to transfer the code into a web page.
Early testing required 50 documents in 3 languages : English, Spanish & German
Solution 2005: SWANS
Synchronised Web Authoring Notation System
The programme generates SMIL code which is placed automatically in web pages to allow students to study stress by reading and listening
The generation of annotated and synchronised web pages has been cut from2 hours per one page document to 10 minutes.
The four stages of authoring with SWANS
SWANS : Synchronised Web Authoring Notation System
Annotation with SWANS
Can annotation become automatic?
Like spelling correctors accuracy levels are variable but ‘acceptable’
Method 1 rule-based expert system
According to Deschamps we can deduce the stress pattern of a word from its spelling in 90% of cases. Some rules, such as those concerning endings with <_ic, itis, osis> or the <ION> rule, are close to a 100% (Deschamps p.33, 1994) and are relatively easy to implement. Accordingly all such character chains occurring in a given text can be temporarily highlighted in SWANS to aid annotation. Unstressed suffixes can be highlighted in red (e.g. -able, -age, -ance, -ancy, ant, -cy, -ful, -hood, -ist, -ise, -ize, -less, ‘ly, ment, -ness, -or/-er, -ous) while stressed suffixes (-ade, -aire, -ee) are highlighted in blue if the user desires.
Can annotation be automatic ?
Method 2 voice recognition techniques
The integration of voice recognition techniques giving raw and optimised phonemic transcriptions adapted from the PERL algorithms of the Aix-Marsec project is a more complex challenge. The Aix-MARSEC project time-aligns speech data at the phoneme and syllable level. Aix- MARSEC tools consist of a set of reference files (grapheme-phoneme conversion dictionaries) and multiplatform Praat and Perl scripts.
a bank of pre-annotated scripts, from 2004-2005 3-minute video news bulletins, offers an alternative to rule-based or dictionary based systems (e.g. 71,000 entries of the Oxford Advanced Learner’s using the SAMPA alphabet) but, of course, requires manual adjustment for ambiguous cases (“convict” = the verb // or the noun / ?) .
Conclusions
Synchronised reading based on dual coding techniques opens promising new perspectives for working on pronunciation in an adaptive multilingual context.
Mother-tongue based annotation techniques offer an adaptive solution to individual learning needs which deserves further exploration in university language centres.
The ‘extensive reading hypothesis’
Extensive reading of synchronised text can help students to inhibit interference from the L1
Perception of stress improves in novice learners exposed to sound and annotated text.
Improved oral production cannot be demonstrated in the short term and may take time to appear. Explicit knowledge of stress patterns is a new springboard for turning savoir into savoir faire and savoir être.
Does SWANS announce the Swan-Song of the Gutenberg tradition for language learners studying pronunciation ?
Resistance to changing reading habits may ensure an erratic evolution of reform but we believe the way is open for new experiments with intensive and extensive reading on the computer screen with potentially important repercussions for perception and oral production.
Will teachers use SWANS ?
The eternal debate … should teachers become authors of on-line resources ?
Testing will take place in several European language centres affiliated to CERCLES in 2006
