Spatial Software

April 9, 2020

I published Spatial Interfaces about seven months ago. This is a follow-up on that essay, starting with some updates on the over-arching thesis from the original piece, then moving on to a framework for thinking about spatial software. I close by discussing some examples of spatial software I've used in the past few months.

To recap the thesis from Spatial Interfaces: Software applications can utilize spatial interfaces to afford users powerful ways of thinking and interacting. Though often associated with gaming, spatial interfaces can be useful in any kind of software, even in less obvious domains like productivity tools or work applications. We will see spatial interfaces move into all verticals, starting with game-like interfaces for all kinds of social use-cases.

Thesis Updates

When I wrote the original piece last fall, I thought my thesis would play out gradually, over the next several years. The main update here is less of a thesis change and more of an acceleration in the timeline. While unpredictable and not part of my original thesis, this global pandemic and the social distancing that comes with it are pushing us towards spatial software more forcefully than any of the factors mentioned in the first essay.

Social distancing has placed the whole world in a new context. People everywhere are feeling a need for the presence of others, analogous to what we have in the real world. Social media isn't enough. In fact, none of our social apps are really enough. Because our current software is too plain, based on a purely utilitarian exchange of information.

Messaging apps are stacks of bubbles. Video calls are faces inside static rectangles. There are only so many degrees of freedom for users inside of these apps, which makes them simple to use. But this simplicity also strips away so much of the freedom we have during in-person interactions. You can type any message, but typing a message is all you can do. You can call any person, but talking directly into your camera is all you can do once you're connected. Without spatial interfaces, our current software doesn't give us much expressive control.

Additionally, when we use non-spatial social apps, we often understand that another's cognitive presence is there, but we can't feel the more human presence we're wired to need. When we're not social distancing, we fulfill this need elsewhere, outside of software. We meet for coffee. We go on a walk. We play a game, or show a friend something funny on our phone and watch them laugh. We have infinite options at our disposal for relating to others. Though the lack of these same options inside the software we use is sometimes inconvenient, we can usually get over it.

But things are different right now. We're constrained to rely almost exclusively on software for social interaction. What are usually minor inconveniences in our existing applications are now the main factor preventing us from fulfilling our social needs.

Because of this, we've already leaped into the first phase of the thesis above: spatial software gaining rapid adoption in social applications.

This is where the thesis has become more specific. In social applications, spatial software has at least three advantages. These three advantages often overlap and bleed into each other.

1. Afforded Intuition: Our natural understanding of space makes spatial software easier to use.
2. Expressiveness: In a spatial interface, more degrees of freedom mean users can interact more creatively with software and with each other.
3. Presence: In many spatial interfaces, we can sense other users through the presence of an avatar or figure.

Spatial Software Framework

Why does spatial software provide the benefits above? Mainly, spatial software takes advantage of how our brains understand space. It uses parallels to the real world as the organizing logic of its interface. After all, the real, outer world we live in day to day is characterized by the free movement of bodies and objects within a spatial environment.

Here is a picture of the world. It contains elements that can move about freely.

Optionally, we can separate elements into two categories: bodies and objects. A body is an element that represents a being. An object is an element that does not represent a being. Here is one more picture of the world.

Here is a picture of an interface. It puts constraints on the placement and movement of its contents.

Spatial software contains a world within its interface.

This is our definition of spatial software. It is characterized by the ability to move bodies and objects freely, in a parallel to the real world. This is opposed to traditional software, which uses some other logic to organize its interface.

Figma and Second Life are examples of spatial software. They contain worlds where the relationships between objects on a canvas, or bodies in an environment, respectively, are the organizing logic of the interface. WhatsApp is not spatial software. The organizing logic of its interface is recency of messages, not spatial relationships.

As mentioned above, the real, outer world we live in day to day is also characterized by the free movement of bodies and objects within a spatial environment. And all visual interfaces exist within this real world. We can zoom out on our previous diagram.

Now we have full context. When we zoom out, we see that the real world and a virtual world are characterized by the same core trait: free movement of bodies and objects in space. A virtual world can also mirror qualities of the real world in ways beyond that. It can have an environment full of mountains and trees, or physics that behave similarly to Earth, or objects that look the same as objects we use outside of software, to name a few examples. This adds color to the term "skeuomorphic spatial affordance" from my previous essay.

So, looking back at the previous examples, we now have clear guidelines for thinking about spatial software. Within this framework, there is a spectrum along which all spatial software lies, based on how closely its virtual world mirrors space as it exists in the real world.

On one end of that spectrum are more limited examples, like the app-switching interface in iOS. In the middle of the spectrum are applications like Figma, where the world is an open spatial canvas where objects can be moved freely. Finally, on the more literal end are MMO RPG's like Second Life or World of Warcraft, where users are given bodies and can move about an environment with landscapes and physics that mirror the real world.

Additionally, it's now clear that traditional messaging interfaces are not spatial software at all, since they do not present a world that allows for free movement of its contents.

References

1. Figma

Figma doesn't mirror the real world as closely as many game-like interfaces, but it has created an incredible virtual world. It allows for free movement and manipulation of all elements inside the world, which expands infinitely in every direction.

While it's positioned as design software, the openness of Figma's world means it can be used for almost anything. I use it for my job as a product designer, to design software applications, but I also use it to hang out and talk with friends. Others have used Figma for playing games of Pictionary or building virtual towns. I even used Figma with my friends at Blogger Peer Review to brainstorm concepts for this essay.

When you start to think about Figma as a world, and not just another app, you realize that this world can contain other applications within itself. For example, right now you can embed a Figma file inside a website. But what if you could deploy a website inside of a Figma file? You could place pages beside, above, or below each other. Visitors could see each others' cursors while they looked around the site. Or they could zoom out and see an entire website at once, then choose which page to visit based on which was the most visually interesting, instead of just choosing a link to click. Technically, a web browser could add this functionality by scraping an entire sitemap and rendering all the pages side by side at once. But if websites lived inside of Figma, this functionality would come for free, and it would give web designers more creative freedom.

Many applications could be reimagined to live inside Figma, and Figma's plugin system is already providing early evidence of that. I'll stop here for now, but I'll eventually write a full essay on the full potential of Figma's world.

2. Muze

Muze is a spatial chat app. It uses a virtual world very similar to Figma's. Users can place text, images, art, and gifs onto a vertically expanding canvas. Think of it like a more constrained, lightweight Figma on your phone.

It's striking that Muze and Figma are positioned for completely different tasks, while making use of nearly identical virtual worlds as their main interfaces. This is evidence of the unlimited versatility of spatial software. These two apps could really be made by the same company, or at least be running on the same engine.

3. Animal Crossing: New Horizons

Animal Crossing: New Horizons is probably the most popular game release from the past month. The timing of its release was perfect. It's a cute, nostalgic game that came out just as people everywhere went into self-quarantine. But it would be a mistake to attribute the popularity of this game to it being a well-known title that came out when people had a lot of free time on their hands - it's a lot more than that.

It's not just that people have a lot of free time right now; they're also socially isolated. And the new Animal Crossing isn't just any popular game. It's unique in two ways. First, it provides a customizable space to hang out. Second, the point of the game is mostly social, focused on creating your space and spending time in it. On top of this, it's easy to control and accessible to a mainstream audience. These factors combined mean the game is giving people a viable alternative to hanging out in real life. The openness of the virtual world is the game's primary strength. There's a reason people have used it for everything from weddings to a mini design conference to date nights to surprise birthday parties. You can't replicate these kinds of activities on Zoom or WhatsApp.

The downside to Animal Crossing is that there are still barriers to entry. You need a Switch to play, and even if you want one, those are sold out everywhere right now. I think there's potential for similar games to have much wider usage if built as free-to-play, multi-platform games like Fortnite.

Final notes on the accessibility of this game: Even more than games like Minecraft or Roblox, Animal Crossing has made 3D creation tools really easy to use. Through more constrained options and simpler controls, Animal Crossing makes landscaping a virtual environment easy for anyone. The game has struck a really nice balance between giving the player freedom but keeping some strong constraints.

This makes me think about design tools more generally. In the early days of the web, you could show off your creativity by creating your own static webpage. Maybe you'd design it using Photoshop or make it directly using CSS. Now that the web has evolved to a point where dynamic web applications are commonplace, we also have great design tools like Figma and Sketch.

It seems like we're now in the early days of the spatial web, and 3D design tools aren't yet tailored to the specific goals we might have moving forward. Lots of 3D tools exist, depending on what you're trying to make: Unity, Unreal, Blender, Maya, AutoDesk, etc. And there are also simpler options like the Roblox Studio, or just using Minecraft to design something. However, nothing has yet struck the right balance between ease-of-use and freedom for me.

What will design tools look like in a world where instead of a static webpage, we all have, for example, our own virtual island?

4. Online Town

This website launched last week, and while it's still very young, it's a really interesting application. Online Town is a game where players walk around a 2D space to chat and hang out.

The main innovation here is the way the game handles audio and video streams between players. Similar to real life, walking closer to someone means you can see and hear them. Walking far away means you can't. The same behavior exists in Mozilla Hubs, which is a similar app but in 3D.

This is a chance to point out the way a consumer social app can drive innovation that translates into business software.

I work on an entirely remote team that holds meetings on Zoom every week. Sometimes, our 11-person team needs to split into smaller group conversations, then reconvene afterwards. We use Zoom's "breakout rooms" feature to do this, but it's so clunky to configure that we frequently have to quit the call and start three new ones. Then we message on Slack when we're done and go back to the original call.

An application like Online Town works much better for this use-case. If you have people all talking in a video call, and you want to break out into a smaller group, the solution is really graceful. You just...walk away. The world gives you that kind of freedom. As the people in the smaller group walk away, their video drops away from the larger team, but they remain connected to each other, allowing them to have their own meeting.

This kind of behavior is much easier to understand and navigate, and could be used in business conferencing software as well.

5. Nototo

This one is actually not social, so it's a good example of spatial interfaces having explicitly useful qualities outside of social interaction.

Nototo is a spatial note-taking app. It lets you build an ever-expanding, topographical map containing your notes and writing. The app is designed this way to take advantage of another aspect of spatial interfaces: our brains remember spaces better than raw information. In this regard, Nototo is like a software manifestation of a memory palace.

Design-wise, Nototo is doing something really cool, taking a mechanic familiar to "no-code" apps and converting it to 3D. Whereas in many no-code apps, users can drag and drop different kinds of "blocks" around the app to control an interface, users in Nototo can drag 3D blocks onto their virtual map, customizing and expanding its territory.

6. Spatial Chatroom

One thing I've really loved about using the apps referenced above is how much playfulness and discovery comes built into spatial software. A perfect example comes from something I built recently.

Last week I built my own spatial chatroom app, where users get an avatar and a speech bubble for talking to each other. Within minutes of testing out the app with me, my friends discovered we could play tag for fun, or have little footraces around the map. Behaviors I didn't design into the app at all.

Because spatial software comes with more degrees of freedom, whether to move around or view things differently, there are creative ways of using the software, not just designing it.

What's Next

The global pandemic happening right now has brought with it an urgent need for software that can fulfill our social needs.

A whole wave of spatial social applications will be born out of this time, and these applications will serve as the breeding ground for insights that make their way out of social and into other major verticals. This will be an extremely generative time for insights and innovation in the way we build software. Designers and developers will begin using spatial interfaces in non-traditional ways, taking design patterns from gaming and applying them to non-competitive, explicitly social use-cases.

I hope that with the new framework introduced in this essay, it's easier to think of all kinds of ways we can build spatial software. Simply add a virtual world to an application.

Like a remake of AIM, where each user gets a house that can friends can stop by to visit. Or like a web browser with a topographical view, where users work together to build a spatial map of related websites. Or a video conferencing app where people can go on a walk together. In fact, you can put a world inside of almost any interface. Even a blog post.

Discuss this post

Contributors

Thank you to Toby Shorin, Kara Kittel, and Édouard Urcades for your thoughts and feedback, and for help developing this framework.

Note: My friends Toby, Kara, and Édouard are brilliant and all offer independent consulting in various capacities. If you ever get the chance, hire them to help you with design, writing, or exploratory research.