Television viewers in Japan have been watching their popular news presenter Yuki Goto describing sporting events in sign language, hands flitting from one pose to the next in a series of precise movements.
But it wasn’t exactly Goto on their screens. The 25-year-old reporter, who has a hearing impairment, was chosen to model for a computer-generated sign-language commentator unveiled by NHK during the recent Tokyo Games.
“I’d seen CG sign language before, and it didn’t look natural,” Goto told The Japan Times ahead of the Paralympics. “It wasn’t human, it was a little choppy.”
NHK’s effort to create sign-language content began in 2009 in response to growing demand from native users who struggled with Japanese-language subtitles. While Goto can converse normally when using her cochlear implant, it isn’t always ideal and regularly takes it off when at home.
In order to build a lexicon of signs, NHK staff first used motion-capture technology to record around 8,000 words for its database — ranging from “autumn colours” to “preschool” — with sign-language users providing feedback through an online portal.
According to Hiroyuki Kaneko, who manages the sign-language project at NHK, early efforts focusing on hand movements were quickly found to be insufficient.
“We realised that you can’t read sign language just from the hands, so then we started capturing facial expressions including eyebrows and mouth movements,” Kaneko said.
After eight years of development, NHK unveiled CG sign-language weather forecasts.
Soon after, the network decided to pursue the creation of sign-language sports content with Tokyo 2020 in mind. The text-to-speech commentary, known as “robot play-by-play,” debuted for four sports during the 2018 Winter Olympics in Pyeongchang, South Korea, as researchers continued to prepare sign-language commentary for Tokyo.
While initial plans called for an anime-style two-dimensional presenter, the one-year postponement of Tokyo 2020 allowed NHK to respond to viewer feedback asking for more identifiable, photorealistic commentators.
That’s when Goto got the call, with designers working until just before the Games to avoid the “uncanny valley” effect that often occurs when a CG human doesn’t quite hit the mark.
“When you do CG reproductions of a person, sometimes it resembles them and sometimes it doesn’t,” Goto said. “The production team said it looked like my sister, and when I saw it … it was so unusual. It was really natural, about the pace that sign-language speakers would use to talk.”
Kaneko and his team worked to develop signs that would be understood by a majority of listeners, although he admits that it remains an imperfect science.
“It took a lot of time to think about how to best express those phrases and ideas,” he said. “Before motion capture we did a lot of translation and editing. And then we’d do the capture and see how it looked, and we kept repeating that process.
“Even so, sign language is difficult … there’s not always a right answer. But we gradually made things easier to understand for viewers.”
Future iterations could see characters become more emotive: Viewers have suggested that the commentators’ moods should change depending on the state of the game, especially if Japan is doing well.
“They want the commentator to cheer for Japan,” Kaneko said. “So we’re looking for ways to get the commentator to be more enthusiastic when Japan scores, and to create sign-language commentary that excites the viewer.”
“You hear the words ‘adaptability,’ ‘diversity’ and ‘universal design’ a lot, but for those of us with hearing impairments the issue isn’t tangible barriers like elevator access, but intangible barriers like the struggle to get information,” Goto said.
“Information needs to be a part of universal design and diversity. There hasn’t been a lot of focus on changing that, but I feel like that’s starting to happen.
“People understand that ‘barrier-free’ isn’t just about wheelchair access. It’s changing slowly, but it’s changing.”