Research into Same-Language Subtitling

Same Language Subtitling (SLS) is the idea of subtitling (or captioning) audio-visual content in the ‘same’ language as the audio. Word for word, what you hear is what you read, in perfect timing.

There is a rich body of evidence supporting the impact of Same Language Subtitling (SLS) or captioning for language learning (e.g., Robert Vanderplank’s work). There is strong and growing evidence that SLS also improves reading skills. While either of these impacts would provide a strong rationale for SLS, the evidence for both, not to mention media access, makes SLS compelling. 

The most sustained push for SLS on TV for reading literacy has been in India, through research, sustained TV pilots and policy (Kothari and Bandyopadhyay, 2020; forthcoming Stanford Social Innovation Review; available on request). However, the positive impact of subtitle use on reading literacy has also been affirmed in several English and non-English speaking countries. Taken together, these studies consistently prove that exposure to subtitles which match the audio directly, contribute to reading development and language acquisition.

How do subtitles aid reading?

Subtitles cause automatic reading behaviour among children and adults.

A key finding of eye-tracking research on subtitling, which studies the automatic reading behaviour of children and adults, is that viewers who have some decoding ability – even partial letter-to-sound correspondence – just cannot ignore the subtitles and will exhibit automatic reading responses.

The Belgian professor Géry van Outryve d’Ydewalle is considered to be the pioneer in proving that subtitles will be read inescapably and automatically. His list of publications is voluminous: http://lirias.kuleuven.be/cv?Username=U0015362. In one particular study confirming this, American subjects watched an American movie with English subtitles. Despite their lack of familiarity with subtitles, they spent considerable time in the subtitled area of the screen. As such, subtitle reading was found not to be due to habit formation from long-term experience. In the second part of the experiment, a movie in Dutch with Dutch subtitles was shown to Dutch-speaking subjects. They also looked extensively at the subtitles, suggesting that reading subtitles is preferred because of efficiency in following and understanding the movie. Evidence of such eye-tracking enables us to attribute possible learning outcomes to the subtitles.

The key to reading fluency is practice

An important learning from the neuro-cognitive sciences is that neurons that fire together, wire together. For proficient reading, they need to fire sufficiently and over a long enough period of time to achieve automaticity. As Frey and Fisher (2010) state, “When we experience something, neurons fire. Repeated firings lead to physical changes [in the brain] that, over time and with repetition, become more permanent.” They further point out that “The challenge, of course, with automaticity is to not allow repetition to turn into a rut.”

The key to reading practice is high-interest content

SLS rides content that is by definition interesting to the viewer. This is a critical point. MIT’s John Gabrieli, a leader in the field of cognitive neurosciences, explains how emotion and motivation “propel learning very powerfully.” SLS of audio-visual content that is interesting to children fires a steady stream of consistent grapheme-phoneme associations in the brain that already knows the language and letter-sound correspondence. In the case of popular songs, nursery rhymes and repeatedly watched cartoons there is the additional advantage of predictable text. The visual and auditory pathways involved in reading are strengthened gradually, subconsciously and incidentally, as a by-product of watching content that the child has an inherent motivation to watch.

Does SLS cause automatic reading engagement among good and struggling readers?

d’Ydewalle et al.’s (1991) eye-tracking research found that American subjects watching an English movie with SLS and Dutch subjects watching a Dutch movie with SLS, spent considerable time in the subtitle area. Reading SLS was inevitable and comparable for both groups, even though the Dutch subjects had much more experience with subtitles on TV. Reading SLS did not depend much on habit formation.

Several other studies have confirmed that reading along with SLS is inescapable, but the subjects have almost always been good readers. Hence, the critical question is, would struggling readers, especially those from economically disadvantaged backgrounds, also try and engage automatically with SLS?

PlanetRead (2018) completed an eye-tracking study of government school children in Grades 2-5 in rural Rajasthan, India, by showing them animated stories with and without SLS. It is an understatement that the subjects were struggling readers from low-income families. Most children (94%) engaged with SLS, exhibiting an increasing number of eye-fixations on the subtitles, with grade. The story at a ‘low’ level of reading difficulty (81 simple words/min) invited the most reading engagement, followed by the ‘medium’ one (82 words/min), but SLS was mostly ignored in the ‘high’ difficulty story (111 words/min).

The primary conclusion is that almost all viewers who are beginning, struggling or good readers, will automatically engage with SLS. SLS just cannot be ignored.

Does SLS exposure lead to improved reading skills?

The idea of leveraging Closed-Captioning (CC) on TV to enhance the reading skills of struggling readers, is as old as CC itself (Koskinen et al., 1986) but longitudinal studies are few. Linebarger et al. (2010) commented on one such by Koskinen et al. (1997): “In a longitudinal study of continuous caption use in the home, children who viewed with captions scored significantly higher on normative tests of word identification and passage comprehension when compared with non-caption viewers.”

In Linebarger et al.’s (2010) study with struggling readers in Grades 2-3 from economically disadvantaged urban locations in the US, SLS exposure was limited to just six 30-minute episodes from children’s TV. Still, “The majority of outcomes… indicated that children who viewed with captions outperformed their counterparts who viewed without captions,” and the improvement was most pronounced among children at risk for poor reading outcomes.

Similarly, in New Zealand, Parkhill & Johnson (2009) found that in their six-week ‘AVAILLL’ programme for children aged 5-13 years, which uses popular, subtitled movies and accompanying novels to engage students in reading literacy, the greatest gains occurred for ‘low-progress’ readers. A positive impact was also observed for average and higher-level readers.

A number of longitudinal studies have come out of the SLS project in India. Kothari and Bandyopadhyay (2014) evaluated the impact of SLS after sustaining it for 5 years on a weekly hour-long programme of Hindi film songs telecast nationally in prime time. Among school children who could not read a single letter in Hindi at the baseline (2002), 70% in the high-SLS viewing group became functional readers by the endline (2007) as compared to 34% in the low-SLS group. In the 15+ age group, 14% in the high-SLS and 5% in the low-SLS group went from non-decoding to functional-reading. Adults gained too but children benefited substantially more in what can be described as a schooling + SLS effect.

Given the SLS project’s goal of persuading broadcast policy in India to require SLS on all the film songs shown on TV in India, in every language, a maximum SLS exposure of an hour a week, albeit for 5 years, was still too little. That was addressed in Maharashtra state where, for 2 years, SLS had a strong broadcast presence on around 20 Marathi films per week (only the songs were subtitled) on two of the most popular Marathi channels (Kothari and Bandyopadhyay, 2015). The Gujarat (control) and Maharashtra samples were comparable at the baseline (2013). By the endline (2015), in Maharashtra, 68% in Grade 3 could read at Grade 1 level or better as compared to 43% in Gujarat.

The Annual Status of Education Report (ASER) independently found that over the same 2-year period, Maharashtra outperformed all states – 9% more children in Grade 5 were able to read a Grade 2 level text, as compared to no gain nationally. Among those who got regular SLS exposure, 30% more children in the early grades achieved basic reading ability. The impact of SLS on reading skills was considerably stronger for children in Grades 2-3. A separate study of SLS on TV in Gujarat state confirms its value on film songs for reading literacy (Kothari et al., 2004). What about SLS on children’s TV?

Universally, children love to watch cartoons. PlanetRead (2018a) found that struggling readers cannot but attend to SLS on animated stories. Does that contribute to reading skills? Linebarger et al. (2010) and Linebarger (2001) provide evidence that it does, while underscoring the importance of captions, “especially for children who might not have access to print.”

PlanetRead (2018b) conducted a year-long study in 10 primary schools in rural Delhi serving children in Grades 1-5 from low-income families. In 5 treatment schools, the teachers showed all the children in Grades 1-4, 30 minutes of animated stories in Hindi with SLS, three times a week. From a comparable starting point, the average reading score in the treatment schools was 70% higher than the control schools. The impact of the intervention on reading was most apparent in Grades 2-3, pointing again to the strong complementarity of SLS, during the early stages of reading acquisition.

The benefits of SLS or CC are not limited to reading literacy. For an overview of the range of benefits attributable to SLS – including reading, media access and language acquisition – see Gernsbacher (2015).

Does Same Language Subtitling improve reading skills in developed world markets as well as developing world ones?

Evidence of positive learning outcomes can be found in many studies covering various locations and socioeconomic circumstances:

United States

Linebarger, D., Piotrowski, J. T., & Greenwood, C. R. (2010). On-screen print: The role of captions as a supplemental literacy tool. Journal of Research in Reading, 33(2), 148–167.

“Children were randomly assigned to view videos with or without closed captions. Captions helped children recognise and read more words, identify the meaning of those words, generate inferences regarding programme content and transfer these skills to a normative code‐related skill task.”

Linebarger, D. L. (2001). Learning to read from television: The effects of using captions and narration. Journal of Educational Psychology, 93(2), 288–298.

“The present study indicated that beginning readers recognize more words when they view television that uses captions… the combination of captions and sound helped children identify the critical story elements in the video clips… In sum, television captions, by evoking efforts to read, appeared to help a child focus on central story elements.”

United Kingdom 

Eye-tracking research at the University at Nottingham confirms that subtitles will be read: BISSON, M., VAN HEUVEN, W., CONKLIN, K., & TUNNEY, R. (2014). Processing of native and foreign language subtitles in films: An eye-tracking study. Applied Psycholinguistics, 35(2), 399-418.

Bisson is now at De Monfort University, Leicester: https://www.dmu.ac.uk/about-dmu/academic-staff/health-and-life-sciences/marie-josee-bisson/marie-josee-bisson.aspx. The others are at the University of Nottingham.

New Zealand and Australia

Parkhill, Faye and Davey, Ronnie. ‘I used to read one page in two minutes and now I am reading ten’: Using popular film subtitles to enhance literacy outcomes [online]. Literacy Learning: The Middle Years, Vol. 22, No. 2, Jun 2014: 28-33.

“A series of New Zealand studies and one in Australia indicate that, by using subtitles of popular movies and associated literacy activities, both reading achievement and engagement are enhanced, particularly for diverse and low achieving students. Same language subtitling (SLS) appears to evoke unavoidable reading mileage where reading skills are practised subconsciously (Banks, 2012).”                                                                                                                                                                                                                   

Canada and France

Baltova, I., 1999. Multisensory Language Teaching in a Multidimensional Curriculum: The Use of Authentic Bimodal Video in Core French. Canadian Modern Language Review 56 (1), 32-48.

Nicolas Guichon, Sinead Mclornan. The effects of multimodality on L2 learners: Implications for CALL resource design. System, Elsevier, 2008, 36 (1), pp.85-93.

“The results indicate that comprehension improves when learners are exposed to a text in several modalities. In addition, they suggest that L2 subtitling [SLS] is more beneficial than L1 [translation subtitling] because it causes less lexical interference.”

Spain

Birulés-Muntané J., Soto-Faraco S (2016). Watching Subtitled Films Can Help Learning Foreign Languages. PLoS ONE 11(6)

“The results of the listening skills tests revealed that after watching the English subtitled version [of English content], participants improved these skills significantly more than after watching the Spanish subtitled or no-subtitles versions.”

Finland

One of the main factors behind the good PISA reading results in Finland is attributed by Pirjo Sinko, Finnish National Board of Education, to: “Foreign TV programmes not dubbed but have subtitles – improves children’s reading routine.”:
https://cf5-www.ifla.org/files/assets/school-libraries-resource-centers/conferences/2012/finnish-pisa-results-2012.pdf

See Book Development of literacy in kindergarten and primary school

“Alongside the overall finding that children’s reading skills contribute to the frequency of their out-of-school reading, a bidirectional prospective impact was also found between reading and reading habits: the higher the amount of book reading, and the more likely children were to read TV subtitles, the better word chain reading they showed later on.”

 

This summary has been authored by Prof. Brij Kothari, Centre for Educational Innovation at the Indian Institute of Management (Ahmedabad), with input from Marion Macgillivray-Harrison from ITV, UK.

 

References

d’Ydewalle, G., Praet, C., Verfaillie, K., & Rensbergen, J. V. (1991). Watching Subtitled Television: Automatic Reading Behavior. Communication Research, 18(5), 650–666.

d’Ydewalle, G., & Van Rensbergen, J. (1989). Developmental studies of text-picture interactions in the perception of animated cartoons with text. In H. Mandl & J. R. Levin (Eds.), Advances in psychology, 58. Knowledge acquisition from text and pictures (pp. 233-248). Oxford, England: North-Holland.

Gernsbacher, Morton Ann. “Video Captions Benefit Everyone” Policy insights from the behavioral and brain sciences vol. 2,1 (2015): 195-202. https://www.researchgate.net/publication/290395991_Video_Captions_Benefit_Everyone.

Koskinen, P.S., Bowen, C.T., Gambrell, L.B., Jensema, C.J. & Kane, K.W. (1997). Captioned television and literacy development: Effects of home viewing on learning disabled students. Paper presented at the Meeting of the American Educational Research Association, Chicago, IL.

Koskinen, P. S., Wilson, R. M., Gambrell, L. B., & Jensema, C. J. (1986). Using closed captioned television to enhance reading skills of learning disabled students. National Reading Conference Yearbook, 35, 61-65.

Kothari, B., & Bandyopadhyay, T. (2014). Same language subtitling of Bollywood songs on TV: Effects on literacy. Information Technologies & International Development, 10(4), 31–47.

Kothari, B. & Bandyopadhyay, T. (2015). An innovation to raise a nation’s reading skills: Scale up of Same Language Subtitling (SLS) on Zee in Maharashtra, https://www.planetread.org/images/pdf/research/Impact%20of%20SLS%20scale%20up%20in%20Maharashtra%20on%20Zee%20Talkies%202015.pdf. See also https://www.huffingtonpost.com/entry/what-caused-maharashtras-leap-in-reading_us_589d1277e4b0e172783a9a8f..

Kothari, B., Pandey, A., & Chudgar, A. (2004). Reading out of the “idiot box”: Same-language subtitling on television in India. Information Technologies & International Development, 2(1), 23–44.

Linebarger, D., Piotrowski, J. T., & Greenwood, C. R. (2010). On-screen print: The role of captions as a supplemental literacy tool. Journal of Research in Reading, 33(2), 148-167.

Parkhill, F., & Johnson, J. (2009). An unexpected breakthrough for rapid reading improvement: AVAILLL uses movies so students read it, see it and get it. set: Research Information for Teachers, 1, 28−34.

PlanetRead (2018a). AniBooks: Scalable and likeable, but readable? https://www.planetread.org/pdf/Eye%20Tracking%20Study%20of%20AniBooks%20Draft%20Report%20(June%202018).pdf.

PlanetRead (2018b). AniBooks for early-grade reading. https://www.planetread.org/pdf/AniBooks%20for%20EGR%20PlanetRead.pdf.