Social Intelligence

DATE: 2018-2019

PRODUCT: A mapping of behaviors, use scenarios, and a framework to ensure META's AR wearables and smart devices are welcomed as valued partners in human social interactions.

ROLE: Inspired by the issues with Audio AR wearables, I initiated and led the development of this framework, partnering with the Meta AI Research team, contractors, and hardware developers for feasibility assessment, prototyping, and definition of use cases.

IMPACT: By using the framework and feasibility assessment, the team was able to prioritize and plan the implementation of use cases based on social intelligence's current and future sophistication. Video scenarios helped leadership visualize the potential of different investments and make decisions around form factor, feature set, and roadmap. The scenarios still inform the development of the MR assistant.

A shift from thinking of devices as only impacting the direct user to impacting all the surrounding social dynamics.

Designing the relationship between AI and people

The new generation of devices we will be introducing in our homes and on our bodies need to dynamically know when to reach, what to share, and how to present content. Social Intelligence is the design of a relationship between AI and people with the purpose of connecting you with the people, communities, and the world around you in a deeper and more meaningful way.

Social Intelligence would work as the common foundation among products and new projects, unifying learnings and providing a cohesive and meaningful experience.

This relationship needs to be grounded on trust and clear communication. It is humans and computers working together to reach shared goals.

The three levels are not mutually exclusive. In the same way that sometimes you help your peers as a tool or work as a collaborator on their projects, these modes of relationships are fluid. But is it important to note that they are fluid once you establish a more advanced level, allowing free movement backward.

Problems & user scenarios

A person's mood, receptivity, cognitive capacity, and energy levels can vary greatly throughout the day, the week, and throughout their life, and these variations are greatly impacted by factors such as the presence of others, the current activity, and past events.

However, current technology often ignores these cues when deciding how to interact with users. Smart devices and AR wearables, in particular, can be disruptive if unaware of what is happening. To be accepted and integrated seamlessly into daily life, they must be designed to be unobtrusive and respectful of the user's current state and context.

To address this challenge, we conducted research based on user feedback and explored a range of scenarios to gain a deeper understanding of the need and utility of Social Intelligence in smart devices and AR wearables. We divided our findings into two sections: situations for smart devices placed in the home, and situations for wearables worn on-the-go or in public places.

Social Intelligence for smart devices at home

Examples of social situations at home that are currently challenging for Meta Portal:

Constant interruptions reduce the attention span of people to focus on what matters.

Despite this couple having a difficult conversation, they still show up as available for other people.

The same content that can be sweet and endearing can be embarrassing, depending on who is in the room.

We cherish so many rituals in our home, which makes a home a special place. We should make sure to preserve them from interruptions.

The same space has different uses during the day; learning about these uses helps make decisions about what is appropriate.

People should trust that their set limits are being respected.

Social Intelligence for wearables devices

We created several video scenarios for the brooch form factor of the Meta's AR Audio wearable. They mostly also apply to AR Glasses. The situations are grouped into Social Intelligence value propositions.

Understand which information is more meaningful to the recipient:

Surrounding snippets / AR lenses on the go

Birthday gift suggestion.

Be aware of when and how it is appropriate to reach someone:

Message while busy.

Message when not busy.

Infer what kind of suggestion is expected at a given moment:

Places to go recommendations.

Push recommendations based on the intersection of preferences.

Define the appropriate level of information to share:

Aura match and "hand wave" communication.

Date profile match for people in the proximity.

Identify which social hints would help the recipient better navigate the current situation:

The equivalent of a discrete tap to make you aware of something.

The equivalent of a friend's tips before you jump into a room with strangers.

Adapt messages based on how much complexity the recipient can handle at the current moment:

When engaged in a cognitively demanding activity, postpone or reduce the complexity of the message.

When in a more relaxed situation, assess the urgency of interruption.
If appropriate, deliver the full message.

Frameworks & Diagrams

Context understanding has been an elusive goal of UX and AI for a long time. For this exploration, we created frameworks and system maps to help us visualize the different steps and modules of the process so that we can tackle specific areas in parallel.

This diagram focuses on the role of Social Intelligence in facilitating meaningful social interactions between people. Social Intelligence is the core platform element that curates, formats, and frames the experience to be surfaced. Participation can be stimulated through subtle nudges or in explicit ways. The quality of the interaction can be as low-fidelity as a color code or text and as high-fidelity as an immersive experience.

Video showing the first iteration of the system map for Curation & Delivery. Each one of these cloud modules contains AI models and algorithms for understanding a person or situation and classifying what is meaningful.

Layered checkpoints for deciding whether and how to display messages on objects.

Deciding how to push content from surroundings.

Deciding how to reach and deliver an audio message.

Factors to take into account when evaluating emotional impact.

Building a mental model of the user's expectations.

“...in the literature, people talk about an orchestration needed — I like that metaphor of the orchestra — that the technology needs a conductor to help make the beautiful symphony of a social situation happen.”
Meta AI Researcher

Layered checkpoints for deciding whether and how to display messages on objects.

Deciding how to push content from surroundings.

Deciding how to reach and deliver an audio message.

Factors to take into account when evaluating emotional impact.

Building a mental model of the user's expectations.

You may also like