Painting with Light
“Luminate” is a drawing program that lets you draw with a virtual paintbrush on the real world using a smartphone.
Unlike most drawing programs you draw by moving the phone itself. As the phone moves through 3d space it leaves behind a virtual brushstroke. The phone is the stylus and the world is the canvas.
When it is done I hope you’ll be able to print out your 3d drawings on a 3d printer or share them online. In fact the only way you can get your own copy of Luminate is by downloading the source code and building it yourself. I hope that you can help me finish it.
I’m making this project in part to explore a question: “what is the simplest possible creative experience one could have in Augmented Reality”? I am also interested in the young minds that will grow up without paper, that will never see a laptop or a computer display as we know it.
But that’s all the short story. For a more in-depth understanding of the program, we need to explore the concepts and concerns and how they inspire and motivate this work.
1. Drawing
One of the motivations for this project was early life and my mom, Anna Hook.
One could say my mom liked art, but this would be an understatement. I’m told she ran away from her childhood home when her own parents wouldn’t let her go to art school. I recall one time as a child looking very closely at the dining room chairs and table legs and realizing that everything had tiny flecks of multi-colored paint on them — from being constantly in the presence of artists making art. Another time my sister, when she was young, had to be rushed to the hospital because she drank some turpentine that had been left standing with white paint in it.
In fact almost all the memories of my mom, even late in her life are of her constantly painting. I remember my mom drawing two huge cranes on the living room wall of our house in Regina, Saskatchewan. I remember her inking black and white political cartoons. I remember her covering canvases stretched on wood with oil paintings of tigers and other endangered animals.
Our house was plastered with reference materials hung everywhere: piles of drawing books, art books, open oil paints, pastels strewn about. And as a child I would usually pore over drawings of Frazetta, Boris, or Mobius in her Heavy Metal magazines. Or look at drawings by R Crumb or “Freddy and The Freak Brothers” and other underground comics. National Geographic and nature magazines littered the tables.
Later, my sister Jennifer (who has an MFA in Sculpture) added her own folios, more books and pieces of art all over the house. There were moulded paraffin tigers and balsa wood sculptures and clay work from her classes.
Even when my mom was dying and could no longer speak she would sketch my sister with a shaky hand onto a sheet of paper held in front of her.
I imagine that whole world is coming back to us. As the age of “screens” draws to a close we’ll once again live in those kinds of noisy messy deeply human worlds.
2. Making things
More than just “art,” I was steeped in cultural criticism around the arts. Critical art, art inquiry, what constituted good or bad work was a constant topic. Art was both how we saw and how we communicated. The process of “making” was kind of how we lived. It was social. The world was our lego.
I suppose intrinsically I’ve come to love the creative act; especially the collaborative creative acts. In my mind all art was making, all making art and at some level it was all social.
“Making” seems to be where people put their own perception on display. One doesn’t have to be attached to praise or censure, but it’s part of a feedback loop that one engages in with the world. It’s a validation not just that the maker exists but that the world exists. Sometimes making can be an attempt to escape definition, to escape the known, to relieve the ennui of being trapped inside of the same four walls. Every work, both in being created and perceived; is a new lens for looking at the world.
I have a particular love for tools that help people be creative. Many people need to be able to express work but lack the technical ability. Providing tools that give people the power to participate is like being a meta-artist. In my experience it was always important to listen carefully, build the tools, and then get the hell out of the way. You never wanted to end up yourself being the medium the artist worked “through”.
As a result I’m often looking closely at creative tools and how well they facilitate creative process. Some tools surprise me with their power:
- Minecraft, aside from being a fun game, is also coincidentally just about the fastest way to model geometry I’ve seen. I often find myself reaching for it first rather than reaching for professional 3d modeling packages such as Alias or Blender3d.
- Paper is another lovely rapid prototyping tool. It’s hard to draw a bad drawing using Paper (and in some respects it was an inspiration for this project). It also has a wonderful addictive, dreamy and game-like quality.
- TiltBrush as well is a newer tool in a similar space — sitting somewhere between a formal 3d design tool and a childrens etch-a-sketch. It is also now starting to be a design inspiration for Luminate.
- Unity3D also impresses — similar to the kinds of things I would write when I used to make video games. Today the average teenager effectively has a kind of visual typewriter; they can create interactive narratives that would have taken entire teams and millions of dollars ten years ago.
If this project could be something like one of the above, something that helps others be creative together, was lightweight and fun I’d be more than happy.
3. Needs versus Wants
Building this drawing application is just a step along a path.
I see a deep need for a new kind of social collaboration, and one that is facilitated by being able to put everybody in the same virtual space. When we were kids we’d play together with Lego; it was so easy to collaborate. Yet today it seems so much more challenging.
Someday we’ll be able to organize our photos, projects, mail and our lives using a 3d metaphor of virtual objects floating around us. We’ll be able to mark the world with contextually appropriate information. We’ll be able to sketch out new buildings on the actual building site. This by implication is more collaborative. Video games will come out into the real world. Street signs will be contextually appropriate and animated. Even something as simple as navigation directions will become a glowing contrail showing you and your friends where to walk. “Work” and “making” will be visible rather than hermetic as they are now.
It’s almost like I “need” tools like Augmented Reality to be able to scale up to a new level of collaboration, to be able to do collective barn raising. These are tools I “need” more than “would like” or “want”.
3. Discriminants
There are things that are only possible in Augmented Reality. I like to refer to these things as the “discriminating characteristics” or “discriminants” of the domain. Augmented Reality seems to:
- offer a possibility of being “present” in a new or mixed reality world; to create a believable illusion of being somewhere different— fictitious, imaginary or superseding the real.
- allow for a much much larger space of interaction than conventional computer screens (your display screen real-estate is physically larger).
- offer a potential to implicate your whole body in how you navigate and interact with it (using hands, feet, head and voice is more intuitive when you’re in the real world).
- offers a possibility of contextually appropriate information better than any other medium so far except human memory (showing information about a person directly attached to a persons head for example).
- potentially be more collaborative; more so than Virtual Reality and more so than traditional computer interfaces which can be quite hermitic and private.
However the medium is so new we just don’t know which discriminants are valuable or not. It is unclear for example if contextual information is helpful. We believe it may be helpful to see big glowing arrows leading you to your destination but we just do not know for sure until we test against a large enough group of users.
When we’re formally studying a discriminant we try to dissect it into measurable aspects. For example one of the topics we’ve been researching is the idea of “presence”—when you believe you are “there” (in the place where the Augmented Reality experience is taking you). We speculate but don’t know if creating a sense of “presence” will translate into more people enjoying an experience. We think presence might be formed out of traits such as “high frame rate” or “high resolution display” but we don’t know if those are even important qualities that manifest the sense of presence. It may very well turn out that this idea is similar to the idea of “fun” or being “conscious”; something that is ineffable and comes out of a sense of real time interaction — not something we can conjure up on demand piecemeal.
4. The Big Book of UX for AR
Our R&D process at my office is to build prototypes and then test against users. We learn a lot through this. But even this can be challenging.
We are searching for a sweet spot where there is a valuable experience. There are many possible criteria we can explore in building our applications such as “quality of graphics”, “number of players”, “emotional nature of the interaction” just to name a few. The search space is incredibly large and it takes time to make a test, test against users and then iterate again.
And as researchers we are tempted to go with our passions, to trust our instincts, move further ahead faster, and find addictive compelling experiences (even if they are incredibly ugly + slow) and then polish that and then go back finally and get user feedback to validate or reject some experience that we actually crave.
But without testing or external validation we also collide with expert syndrome; so we test what we can while also practicing our intuition.
Basically what we need is “The Big Book Of UX for Augmented Reality”. This book has not yet been written. We have no idea what is in the book; most of the pages are blank. We’re completely guessing. It probably means throwing out a lot of lessons from traditional UX. It may mean a whole new generation of Experience Designers need to be grown. We keep grasping for this book but O’Reilly never seems to have it in stock. A good example for what this book might look like metaphorically speaking is the Google Cards UX Reference.
5. Augmented Reality has been a fail
Part of my curiosity is how to make something for AR that people actually want to use.
Nobody has yet to produce a successful Augmented Reality App. Every project has been a failure, business-wise, adoption-wise, revenue-wise. It has all been marketing propaganda. Many apps are used once if at all — as a throwaway novelty. So how do we get from where we are, where there is no adoption, no formalisms, no product, to where we need to be?
I’ve been working full time in the AR space now for 5 years. I’ve worked on this at Xerox Parc and at a number of smaller startups in San Francisco. The last project I worked on InsideMaps may actually end up being one of the early successes in the AR space as it matures (fingers crossed).
Now I’d like to see something, anything, that is mass market, that people actually use.
6. Simplest possible thing
Most of what this has been about is finding that simplest possible thing.
There are many kinds of complex experiences that we can imagine building but what are the experiences that are exactly the opposite of that — what is the minimum?
My friend Rich Gibson also poses this question and I like the discipline of the thought. I saw a possibility here in getting to the bottom of the medium, starting with something simple and using that as a way to work up.
If you had a giant paint brush and you could stand in a field and sketch out a building just by moving the brush around — this would be an intuitive interface. Or imagine sitting at a coffee shop with friends and using virtual Lego to collaboratively model a tree fort you’d like to build, or even just drawing an org chart of a team, or tracing out your friends face in 3d.
Drawing is such a simple basic interaction and yet we rarely get beyond 2d. Obviously many software programs allow full blown 3d modeling but they trapped behind a 2d screen. What is the 3d equivalent of doodling on paper?
If you look at products like 3Doodler clearly some people believe that drawing in 3D has merit. Lego and Makerbot also are clearly part of this heritage. People often model and create in 3d of course using clay and brick and sticks but we don’t have a virtual equivalent that is both contextual (in place) and fast.
Perhaps the core of the idea was simply the equivalent of a virtual sparkler; a stick with fire that you hold in your hand and wave around at night making big streaky lines in your persistence of vision. It’s like Pablo Picasso’s exercise of drawings with light, or like the light graffiti that you can find on the Internet.
7. Ongoing Design Tensions
Another motivation for releasing this is that there are design tensions that are beyond my ability to personally solve.
The core need is to simply be able to draw 3d shapes repeatably. The tool should be accurate enough to let you draw a simple thing like a box successfully over and over.
This means being able to estimate where the corners meet, where faces and shadows intersect. And this means there needs to be some kind of snap-to-edge or grid or regularity enforcing concept such as Minecraft has.
But I also want to permit the free flow of expressive painting. This is a currently unresolved tension. TiltBrush, notably, is tackling this by having a formal distinction between your viewing angle and a virtual plane that you are drawing on — but then you can’t free form draw in 3d… which is kind of the point of this project.
Another tension is color and swatch choices. It’s hard to figure out exactly what are the right powers to provide for drawing. It may make sense to offer a Minecraft brick mode as a way of solving a number of these issues.
If you look at this rough cut video from a few weeks ago — this has a long way to go — and perhaps the main challenge is finding the right community to help it grow:
So another motivation for releasing this is that I am hoping to get help so that it can be fully realized.
8. Other work
Aside from my own explorations there have been other people doing the same with lo-fi or lo-tech solutions. I’m leveraging a 3d SLAM solution that runs on a mobile device (Metaio for now, may try 13th Lab later or use Dekko which I helped write) but there are other people who do similar things in non-realtime without any technology. The “idea” exists, is powerful and shared. I feel like this work riffs on that earlier history. There are many examples:
I’ve also bookmarked similar work at http://delicious.com/anselm/3d+draw
I fully expect one of the first apps for NimbleVR or the Leap Dragonfly will be a finger painting in 3d in the real world project. I also see Meta or other pass through vision Augmented Reality rigs to provide similar interfaces. Personally I’m interested in moving beyond static drawings and into an animated space, dealing with occlusion and context appropriate lighting and reflections as well (but this is years down the road).
Space Sketcher is another new addition to the family (relying on 3d structured light hardware on the Tango) :
https://www.youtube.com/watch?v=FEv8BJmqiXw
I’ve also commented at length about similar topics in slides and talks at various conferences about drawing in 3d and AR in general for years ( http://ello.com/anselm, http://slideshare.com/anselm, http://blog.makerlab.com ). Oddly this is the first time I’ve bothered to actually release something to the public domain in all that time however. In a sense this is an attempt to get beyond the rhetoric into something real.
9. Help
Drawing in 3D on the real world in real time, in a persistent and durable way is a new medium. An new experience. It’s hard to figure out exactly what is the right way to do it.
I started this last summer and I’m still thinking about it and trying to figure out what it needs. Now I feel it is time to invite other people in to help.
Is it possible to take this work in new directions? To grow it beyond where I have been able to carry it? I decided to make it open source and I welcome help. I’d also like to start help making the magical book of UX design for Augmented Reality.
If there were other programmers or experience designers who wanted to step in and enhance this — there are so many beautiful things I can imagine this having — then please do so.
At the end of the day I need a simple good 3d drawing program that lets me easily draw and share ideas — and this is the best way I figure I can get there.
I know this will exist someday. I know somebody will eventually get it right and that drawing like this will just be an ordinary part of our everyday world. At some point there will be many similar programs, standards and bodies of work around this medium.
In any case I would like to have this 3d drawing program soon. I’d like it to be something that my mom could have used. I would like it to be something delightful and magical that blends together the real and the virtual seamlessly in a way that a 90 year old or a 6 year old would intuitively get.
You can find Luminate and the source code on github.
And please reach out to me on twitter!
Thank you for hearing me :-).
