VR needs design and other observations.

1. VR needs design. As a programmer my perspective used to be that design would only emerge after the core technical issues had been solved. Somewhat surprisingly it turns out that most of those issues have been removed and that we’re now left with the bigger issue: what to make — not how to make it. That means you need to hire designers or learn to embrace your inner aesthetic. All the little burrs that tug on your experience, that mar the perfection of your interaction, that slow your navigation— these are the things we need to examine now.

Ive been asked to look at a few projects recently and I’m finding that companies that I’m consulting for appear to be asking for design — not code. It’s often phrased as a code problem but when I look closer the challenge is often not technical.

People want to know what experiences to deliver, in in what manner to deliver an experience, or which components to invent, and how to invent them in a modular way that they can get reuse across apps. There’s a lot of interest in defining a VR stack. What are the controls that we will use to interact with multiple applications? Will it be Leap Motion style gesture controls? Will it be gaze tracking? Voice? Simple gestures? How will applications notify us and call for our attention in return? How will apps cooperate together? What are the metaphors that we will use to introduce people to VR? We have to rebuild or at least re-examine the full stack. When asked “how” there is hand waving and the magic phrase “Unity3D” or “Unreal Engine”. The how has its unknowns but it is not an unknown unknown.

2. Nobody KNOWS what a good VR app is. Nobody has ever built one. VR apps have 0% market penetration. This is an industry led by hardware with zero successes in actual applications. We are speculating that there is value here. VR may be different because apps may need to work together. It may be more like being in a roomful of noisy avatars rather than a bunch of orderly buttons on a home page. There are no experts. There is no expertise. It’s a great opportunity for fresh thinking to come to the table.

3. Software sells hardware. When we released Dragon’s Lair at Readysoft it sold hardware. People were so excited by the software that they bought whatever hardware was required to deliver the experience. Hardware should follow, not lead. We’re in fact a bit spoiled by the low cost of hardware. People would routinely put down $1000~$1500 for the hardware to play that game. I see no reason why things will be any different today. The right VR experience will drive the rest. That experience may very well be just VR Facebook — but it will be something other than a video game in my mind.

4. A lost generation. Those of us who predate the microcomputer revolution may have some small jealousy at the power of the tools “the kids have these days” but mobile phones haven’t proven to be a terribly creative medium. It’s difficult to build and create with a small screen. VR gives us more screen real-estate, and if we find a way to build creative apps — apps that let people make rather than just consume — it may mean an artistic renaissance as millions of new computer users start to use the medium to explore their own creative passions.

5. The full flowering of what we think of as “mass market computing” may only be starting now. Everything we’ve seen in terms of the billions of dollars Apple has made selling applets for example may be insignificant compared to the market cap for interactive virtual applications.

For example imagine being able to stand over a timeline app of your past and future. Seeing not just 30 day calendar windows, but a single seamless whole — with long term goals visible on the horizon. That is a practical simple utility that would have tremendous value. It’s the kind of application that simply cannot be built without VR, and it is one that I’d probably use every day in VR.

Or, if you look at the emergence of card like information displays — what would it be like to be able to have a set of floating cards around you that each embodied specific information — such as say the weather in the town you were flying to, or a deck of cards showing an overview of a project you were working on. Card like displays may also be something that starts to reach full maturity only with VR. We see projects such as http://onuniverse.com/ and articles which hint at this as well: https://blog.intercom.io/why-cards-are-the-future-of-the-web/ .

6. VR value may not be in just games. I personally want and hunger for experiences that deliver me more powerful interfaces to my world. I’m also especially interested in being able to work in a way that other people can see and understand — not just hermetically in a trade language.

I would like to visually program with friends: to arrange and collaborate on a large software project as a series of virtual objects connected by data flow pipes. I’d love to be able to do collaborative architecture — a kind of multiplayer Kerbal Space Program. It was so much fun playing Minecraft with friends — I see similar opportunities for other kinds of experiences.

I’d like to simply arrange information around me in a large virtual space and organize it with more room than post-it notes or small screens allow. Barn raising may not be as fast or productive as working alone in all cases but it would be more scaleable and social and simply fun.

I can see a world where if the tools were easy enough you could order your phone or refrigerator or car or other personal digital appliance to present all of its code to you in a 3d floating model that you could then rewire. It might be as simple as playing a song when your house dips below a certain temperature, but there are many simple kinds of “programming” goals that many people would probably enjoy expressing or customizing if the interfaces were simple enough. VR might be a way to make things simple enough to understand and it could catalyze a revolution in our personal agency.

I can see home design using housing constraints where you can stretch out staircases to the second floor, or lay in windows and doors and have the wiring route around it and have the house generate a livability score against some kind of design pattern database.

Sure I’d love to play Skyrim in VR but I think that might just be an edge case. In fact what we think of as games will probably just become large shared collaborations that you can tweak and adjust in a god mode.

7. What is design in the context of VR? If the proposition is that design is king for the moment then what is design? Is design inventing? Is it a bridge between art and craft? Is it Steve Jobs yelling at somebody for having a pixel out of place?

I do feel that design pops out in surprising places, and often we’re asked to take on a design role even if we’re not formally trained in that role. For example this project to build better tarps for refugees ended up being assigned to a person who wasn’t technically qualified but rose to the challenge: http://www.wired.com/2016/01/tarpaulin/ . It’s similar in my mind for VR (or any discipline). Design in this context is a bridge between a person and some kind of personal agency or power.

The specific language I’m favoring now is “communication design”. This is slightly different from interaction design in my mind. It is more about building a bridge, solving the issue of how to communicate between people and machines or between people and people via machines.

Also design for me is more about defining the right process. It’s an attempt to get away from merely rote technique to a more intangible place that has to be searched for by iteration. It’s a process of refinement that helps us get to a goal without having to necessarily exhibit genius. It should be subject to measurable outcomes — design goals. Those goals should themselves be subject to re-evaluation after iteration.

Historically as a video game developer I used to write design tools that allowed designers to design fast. A designer would come to me and say “I want to build a video game with specific qualities, specific kinds of levels, specific kinds of critters in those levels”. I would take that set of loose goals and I’d write an authoring tool and then step out of the way. The artist or designer would then use my tool to more quickly be able to search a very large problem space looking for a sweet spot. The typical design challenge is that each of the variables that define a problem space acts as a single axis — and the intersection of all axes is an n-dimensional problems space that is extremely large. It is in fact too large for a computer to search, and the role of a designer is to use their intuition — assisted by computation — to find sweet spots or “islands” of stability where a meaningful user interaction could exist. For example a badly designed video game might have millions of orcs, or a currency crash, or no trees to harvest. Finding a sweet spot in the game that was stable, delivered an experience, was fun to play — was hard. Although I considered myself an artist, and a designer, it was my ability to build those bridges that was most valuable.

For me therefore, design is definitely about process. It’s about building tools that let us liberate our aesthetic sensibility — to bring our critical faculties to bear without undue frustration.

8. Taking storylines off the rails.

There is a popular idea that stories are a primary means of creating compassion or understanding. Stories however are just a technique. There are other ways to create understanding.

VR will probably not be about showing you a story — and having you watch it in 3D. That’s a rehash of the same old interfaces or tools that we’ve used up until now. There will probably be the kind of leap we saw between theater and movies, or between movies and television.

Modern computers let us simulate outcomes using brute force computation. When you combine that with putting somebody inside of an experience then it becomes less of telling them a story (or putting them on rails) and more of letting them try all the outcomes themselves. The learning experience is through trial and error, it is experiential. If there’s some kind of moral message or story that you want to convey you have to do it down at the level of the physics of the world, and let the outcomes emerge out of the participants interactions.

As my friend Talin used to say, a story is what is told after the experience is done.

SFO Hacker Dad Artist Canuck @mozilla formerly at @parcinc @meedan @makerlab

SFO Hacker Dad Artist Canuck @mozilla formerly at @parcinc @meedan @makerlab