Bringing Us Together

As the world was shutting down early last March, Microsoft Research scientist Jaron Lanier found himself in a predicament that would become almost universal within days: He couldn’t get together in person with people he loves collaborating with.

In his case, Lanier, who is also a composer and accomplished player of multiple instruments, had to cancel a gig on Stephen Colbert’s late-night talk show, where he occasionally plays in the house band, Stay Human, led by Jon Batiste. COVID-19 wouldn’t allow for that kind of close, in-person interaction that the world once took for granted. That’s when Lanier, who coined the term “virtual reality” and is one of the field’s pioneers, dreamed up a bold new concept.

“Stephen was having a really hard time being a comedian without an audience, and I thought, ‘What if we could make a virtual audience so he could see people in real time reacting?’” Lanier says. He turned to his Microsoft brain trust of researchers, behavioral psychologists, and programmers, and soon enough, Together mode was born.

Drawing from the latest scientific understandings of cognition, social perception, and group communication, as well as influences as diverse as Tibetan mandalas and the Muppets, Together mode in Microsoft Teams freed online group meetings from the videoconferencing box for the first generation of truly all-digital workers. Using AI technology that assigns each attendee a virtual seat in an auditorium-like backdrop, Together mimics the way humans interact—from high fives to eye contact—when they’re sharing a physical space. Rather than feeling cut off, a sense of easy camaraderie emerges as the assembled group appears in what looks like a big virtual mirror.

“One of the first things we noticed, even with the earliest tests of Together mode, was the lightness people felt when they used it,” Lanier says. “People smiled. They laughed. They got a little playful, which actually seemed healthy given all that’s happening in the world. People told us they were able to speak without getting lost, and that they were having a shared experience in a shared space rather than carefully monitoring other people on-screen for potential threat or potential cooperation. It was group cohesion in a way I’d never seen in a telecommunication system with that many people before.”

Reading the Room

Human beings weren’t meant to stare at one another on screens all day. As reality shifted out of the physical workplace onto a virtual plane last year, researchers at the Human Factors Lab at Microsoft conducted experiments that proved what most of us sensed already: People find remote collaboration more mentally challenging than in-person meetings.

In particular, brain-wave activity triggered by stress and overwork is much higher when collaborating by screen than in real life. Something more troubling emerged as well. Test subjects who began working remotely reported more difficulties making a social

connection with their virtual peers than with those they met in a physical space. That suggests that remote work and education might be making it harder for people to return to normal once post-pandemic life resumes.

Another study concluded that video meetings elicit more anxiety generally than other forms of communication, such as email. The fatigue of sustained concentration, the lack of nonverbal cues, the challenges of “reading the room,” confusion around whose turn it is to talk—it’s all a burden on the brain. Add in the other pressures of life since 2020, from staying safe and juggling childcare to making sure your on-screen background is a 10/10, and we’re looking at one very stressed-out workforce.

“These collaboration tools are not just about transactional conversations,” says Jeff Teper, the head of Microsoft 365 and Teams engineering and the lead behind SharePoint and OneDrive. “With video and audio and chat, it’s about engagement, about connections, about feelings of belonging, and, for us, those psychological needs are front and center in a way we couldn’t have predicted a year ago.”

As Together mode developed, Teper and Lanier knew that collaboration software needed to connect to people before it could help them get work done. Teams usage quintupled in 2020, and total meeting minutes expanded by a factor of 20 in the last year. To address the psychological factors in play in remote communication, the Together team turned to Stanford University professor Jeremy Bailenson, founding director of the Virtual Human Interaction Lab, who looked back at decades of studies on how screens and cameras trick and tease the senses.

For example, in a video call, eye contact often does not track with what’s really going on; nonverbal data about someone’s trustworthiness or attention can get lost when the gaze wavers or drifts off to one side. The same is true for head tilts, posture, and other micro-gestures that don’t fully register in video chat. Even the size of someone’s

head on-screen can hijack better judgment. If someone’s face appears large in your visual sphere in real life, it’s typically a signal to fight, flee, or mate. When multiple faces (say, at an all-hands meeting on our desktop) appear before us, it can overwhelm the body’s nervous system.

These physiological reactions might have gone unnoticed had the entire world not shifted to meeting by screen at the outset of COVID-19. “There definitely was this sense that the current videoconferencing had been optimized for a world where videoconferencing was a little bit of a sidecar,” says Jared Spataro, corporate vice president for Modern Work, who oversees Teams. In March, Microsoft alerted its employees that office work was going remote, and by April, Microsoft’s own research showed that people had started using video in Teams twice as often as they had before the pandemic. “Once everybody was virtual and spread out, immediately there was this feeling of, Wow, the state of the art is not up to the task of helping us feel like we're together.”

Getting Creative

Along with needing immediate solutions for the cognitive and behavioral side effects of working at a distance, there were also organizational issues.

“Creativity requires shared working space, it requires space in your mind, it requires space to lay out things that you’re mulling over,” Spataro says. “All of that, until very recently, was done almost exclusively in a physical sense.”

You might think that those who mostly toil alone—such as busy engineers and programmers—may have welcomed the break from in-person meetings that working remotely provides. But you would be wrong. Longtime Microsoft design researcher and cognitive scientist Abigail Sellen recognized that work falters in the absence of what she dubs “reciprocity.”

“We have this stereotype of developers working by themselves, hunched over their computers, but they miss the collaboration, they miss the whiteboard, they miss confirming evidence and other factors at play,” she says. Nearly 60 percent of people surveyed by Microsoft feel less connected to their colleagues since working remotely more often. “When we’re in a room together, we have reciprocity. If I can see you, you can see me. If I can hear you, you can hear me, and we know how each other is perceived. As soon as we get mediated by technology, we have uncertainties. We lose our ability to perform in relation to one another, to engage with one another. Big-picture thinking as a team becomes really difficult in the situation we’re in right now.”

With all this in mind, the Teams team aimed to replicate the experience of physically gathering together, but somehow without leaving users feeling drained. Over a weekend in early March, Microsoft engineers David Zhao, Henrik Turbell, and Walid Boumerdassi built a prototype of Together mode, borrowing heavily on coding they’d done two years earlier for a Microsoft Hackathon project in conjunction with work by computer scientists Kori Inkpen and John Tang.

“Back then, they figured out a way to do videoconferencing without people in squares but couldn’t quite find an application for it,” says Lanier, who at first was still looking for a solution for Colbert’s audience problem. It never ended up being used by Colbert, but Microsoft embraced it. “As we tested and tweaked Together mode, we were seeing stunning improvements, mostly in wellbeing among users. People were more

focused in meetings, they demonstrated better retention and greater mutual awareness, lower levels of stress throughout the day and less video fatigue. It had this kind of magic to it.”

The breakthrough this time came by way of an illusion that cleared one of the classic hurdles in videoconferencing: People don’t make eye contact. The fix was a “left-right reversal” that made up to 49 users in Together mode appear to be looking through a big virtual mirror. People immediately began positioning themselves to react to one another rather than focusing on what researchers call “gaze misalignment.” And by putting everyone together loosely on a stage-like set, Together mode eased the fight-or-flight response that comes from having giant faces staring at you in a typical video call.

“When Jaron and I did a one-on-one in the earliest stages, he said, ‘Let’s turn sideways so we’re looking at each other as if we were in an actual room,’” Teper says. “He said, ‘Watch what happens to our eyes and our language and our bodies,’ and that’s when I said, ‘Oh, this could work! This could work!’”

Design-wise, cues came from inspirations high and low. Lanier says, “This is going to sound funny, but in religious iconography, both in the Eastern Orthodox and Tibetan traditions, you have images of saints arrayed together in a shared space—that’s Together mode.”

The 1980s cult comedy series Mystery Science Theater 3000 , with its silhouettes of people commenting on movies, was another influence. So was the iconic Muppet Show wall, albeit as a design to avoid. “As much as we love the Muppets,” Teper says, “the wall of characters reminded us that your neck gets tired when you’re straining to see each other. Nobody wants to sit in a meeting and do that.”

Within days, a working prototype emerged that liberated users from the grid of boxes that makes most video chat experiences so exhausting. Together mode participants found themselves in overlapping spaces, and users could even “touch” and fist-bump or high-five those around them. Instead of a Brady Bunch vibe, with people staring blankly from separated rectangles, Together mode felt more like the coffeehouse from Friends.

Staying Together

Lanier likes to point out that in a perfect world we wouldn’t need Together mode because everyone could be in the same room. “There’s no way to pretend the pandemic doesn’t suck,” he says. But even as some workers start heading back to offices, a hybrid style of work that includes remote work is likely to persist (71 percent of employees and managers surveyed by Microsoft reported a desire to continue working from home at least part-time, post pandemic). Together mode is an option that’s optimized for the millions of users who will need these kinds of tools that make this hybrid future possible.

A study conducted by Microsoft using physiological signal monitoring (EEG) last June compared brain activity of Teams users switching between the classic grid view and Together mode. Biomarkers indicated that the brain exerts less effort when participating in a meeting using Together mode, suggesting that it may help remedy the feeling of meeting fatigue many remote workers are experiencing.

Better Together

Brain activity suggests Together mode in Microsoft Teams may decrease meeting fatigue

From a study conducted by Microsoft in June 2020 comparing brain activity of people who systematically switched between the two meeting views in Microsoft Teams.

“One of the big learnings from the pandemic is that these tools are not here to optimize output, and our wellbeing and morale be damned,” Teper says. “It’s in a company’s interest to make work more fulfilling and effective for everybody, not to burn out its employees. We’re applying what we’ve learned to get better guidance and build better products and that will have lasting change.”

Part of that is about inventing tools to match the moment. As Jared Spataro puts it, “One of the best things I’ve learned...is don’t try and take the old and replicate it with a new medium. Instead, figure out what the spark was and see if you can bring that magic to the new environment. That happened with Together mode.”

Lanier sees the impact of Together mode extending beyond the office, school, and the world of sports. “Even when travel and live attendance return, we’ll see a niche for attending events remotely, whether that’s a theatrical production that gathers people from different coasts or countries, or a celebration or a meeting of scientists or artists or hobbyists. We can’t predict the future, but we know it’s going to be fascinating.”

Indeed, while the focus of Together mode’s development so far has been the workplace, Microsoft recently released new scenes for Together mode that speak to other forms of socializing: virtual backgrounds such as a coffee bar, pub, the streets of Paris, even under the sea.

“This won’t be just for work or for setting an agenda over a PowerPoint presentation. It’s for virtual happy hours and lunch breaks and more casual settings where people need time to connect. We don’t expect people to use the spaceship scene to do a serious monthly business review,” Teper says.

Lanier adds that they also want to find ways for Together mode to work when we return to a hybrid of in-person and remote work. If life goes back to something like normal, how can Together mode work when some people are in a conference room and others are at home? There’s also the personal application—parties, celebrations, family reunions. Indeed, the pandemic has seen people connect virtually on Teams for everything from weddings and kids’ soccer practice to happy hours and yoga classes. Colleges are even considering using the platform for student orientation—a defining experience for students centered on forming quick social bonds.

“This isn’t just online meetings,” Teper says. “There’s an aspect of this collaboration software that has very deep psychological and social implications for how people feel about themselves and what our shared goals are when we’re having an online experience.”

Teper says Together mode prompts a user reaction that’s outside the standard list of outcomes for an engineering product, particularly one dreamed up during a pandemic. “It’s one thing for a feature like Together mode to have utility,” Teper says. “The most gratifying aspect is to work on a project that brings people moments of human joy.”