DATE: 2020
PRODUCT: Â A social music experience for the Meta Portal TV launch.Â
ROLE: I promoted the creation of the AR Experiences Team, fully staffed with 4 designers, prototypers, PMs, an art team, and engineering. I led the conception, creative direction, and design of this brand-new AR experience to showcase the potential of Portal TV upon launch.Â
IMPACT: Due to the limitations of Spark AR, we developed Mic Drop using Unity for the first time on Portal. We proved that the processing power was sufficient for the call, smart camera, and an elaborated AR experience to run simultaneously with proper optimization. We launched in time for the Portal TV market release and got excellent reviews from the media and users. Initially, use got hindered by the limitation of working only between Portal TVs, which needed more critical mass.
An emotional journey
When designing with the team, I framed the AR music experience in emotional terms to help us think of what properties would make it more engaging and immersive. I introduced the concept of Alibi and Magic Circle for our immersive experiences:
The Interaction Alibi
The Interaction Alibi
“A rule, object, or change of state that allows a human to interact. This idea is central to designing for participation. An alibi might be a role, a rule, a narrative, a game, a mask, an instruction, or an introduction. An interaction alibi helps you understand what you’re expected to do, feel safe trying something new, and trust that the outcome will be worth your time.” source
It is the context built around an experience that gives permission and comfort for people to immerse and participate in an activity that requires behaviors different than the ones they would have in a regular setting.
The alibi helps create the Magic Circle (on the outside is the world, and on the inside is the game. And when players cross that threshold, the rules change, norms change, and people’s roles and behaviors change).
Here’s a mapping of the moments of the emotional journey and the UX elements that support/influence it:
Attention to detail
To create a feeling of immersion, attention to detail is essential. Low-quality or buggy AR can ruin the magic very fast.
And Mic Drop, despite being a relatively short UX flow, had many details to take care of. From the song catalog selection to the look of the singer’s character. From making you feel you are not in your living room to how to communicate your turn when people are in different locations. And sometimes there is more than one person on each side.Â
See below for examples of how we solved these challenges.Â
Examples of AR masks, accessories, and make-up for the singer characters. Effects had physics to respond to body and head movements. We created various music genre characters from a punk rocker or hippie to an ’80s rock star.
Early prototype using Microsoft Kinect for user research.
We ran research sessions regularly to learn about what was technically possible and desirable for people.
We ran research sessions regularly to learn about what was technically possible and desirable for people.
We did many iterations on the Jumbotron Opening Scene.
One of the latest versions.
We created a feeling of backstage dressing room before a performance.
The “Switch Performer” button was added due to situations with more than one person in front of the camera.
The “Switch Performer” button was added due to situations with more than one person in front of the camera.
Announcing your turn.
Show your chosen song and then count down to start.
We added several stage effects and camera cuts
for a live performance feeling.
for a live performance feeling.
Participants not currently performing
could send reactions from the remote control.
At the start, we provide an intro of what to expect.
At the end, you can share a highlight reel of your performance.
Well received by press media and Portal TV users
We launched in time for Portal TV market release and got excellent reviews from the media and users. Initially, use got hindered by the limitation of working only between Portal TVs, which needed more critical mass.
Combining music and AR in a magical way, Mic Drop received positive feedback
in press demos and social media for the launch of Portal TV.
It takes a village
At the beginning of the project, I motivated different XFN teams spread out on diverse floors to move to a shared area in the building to improve communication and collaboration.Â
Before, we relied on chat and scheduled meetings to get in sync. After, we dramatically reduced the number of meetings and the communication ambiguity.Â
We had my design team, product manager, engineering team, art team, and prototypers integrated and much more productive. This change, deviating from the company’s practices, made it possible for us to achieve higher quality and meet the tight deadline.Â
The Mic Drop team: designers, PM, engineers, prototypers, creative coders, art directors, sound designers...
