Playing with gps spatialized web audio

I was at my friend Heather’s house and she mentioned that she’d like to do a spatialized audio game — so I threw together a tiny example of this here consisting of a few sound objects — four sound sources and yourself:

https://makerlab.com/soundgame

The source code is here:

https://gist.github.com/anselm/1a193d036788023dbb73be14960d6d0d

Visit this on your mobile phone (using say Safari or something that supports Web Audio). Make sure to tap the screen to allow audio to come through.

After you tap the screen (to start the audio) and as you move around in a physical space you move either closer or further away from virtual objects — and they should become louder or softer based on your distance from them in the real world.

The space is very small in this example — even GPS noise will move you. Also it ignores your orientation right now.

The basic idea is that you could be physically in a large open space (such as say Dolores Park) and be holding a mobile phone and wearing headphones that sonically place you in a different space (such the the interior of a spaceship). If you visit this page on your phone it loads up the experience and as you move around in the park you’d hear different parts of the space. You could hear some kind of sound cues if you were near a wall, or different objects in the virtual space — such as heavy engine machinery, or different kinds of floor gratings as you walk around.

To do this is pretty easy in with Web Audio. Here’s a breakdown of the approach:

First, for my own convenience I created a concept of “objects” in Javascript that embody both a location in space and an associated sound. I threw these into an array:

let artifacts = [
{ sound:”twitchininthekitchen.mp3", x:0, y:-20, z:0, color:”#ff00ff” },
{ sound:”owl.wav”, x:-0, y:20, z:0, color:”#000000" },
{ sound:”spaceship.wav”, x:20, y:0, z:0, color:”#ff0000" },
{ sound:”laugh.mp3", x:-20, y:0, z:0, color:”#ffff00" },
];

Then there’s a piece of code that ‘paints’ these objects — starting their audio if needed and also painting their current position to a visible canvas. This also does a lot of other things if the sound is not loaded yet — but the final goal is to just “make sure the sound happens”:

function paintObjects() {
for(let i = 0; i < artifacts.length; i++) {
let obj = artifacts[i];
ctx.fillStyle = obj.color;
ctx.fillRect(obj.x+512/2,obj.y+512/2,10,10);
playSound(obj);
}
}

Finally I setup a get location query to get your gps location from your phone. This kickstarts the engine:

function getLocation() {
navigator.geolocation.watchPosition(function(position) {
let lat = position.coords.latitude;
let lon = position.coords.longitude;
if(!pos) pos = { lat:lat, lon:lon };
listener.x = (lon-pos.lon)*100000;
listener.y = (lat-pos.lat)*100000;
repaintAll();
});
}

It’s worth noting how easy this is to do. Normally people probably expect to have to write these kinds of demos or games in Unity3D or some kind of native programming language — and to have to be expert developers — and to have to install and deploy these experiences through the app store rather than directly through the web. But it turns out that modern web-browsers expose everything you need, and it’s really just a tiny bit of javascript to test ideas like this.

There is one annoying thing which was that browser based location queries in javascript require you to setup https support. I think this was a poor decision on the part of whatever standards body defined browser security protocols. Systems should never be designed to inhibit playful exploration. A whole class of fun and easy location based applications is now more difficult for novice programmers to create. To circumvent this I decided to use cloudflare to proxy all my http traffic through to my ec2 instance for makerlab.com. Many other people would probably just host their apps on a third party cloud service that provides https.

At the end of the day this is just a small technical demo to show feasibility and could use a lot of improvement. There’s no story line or game play in this technical example, and the example doesn’t rotate the audio with respect to your phones orientation but it shows a possibility for a different kind of virtual reality — one based on imagination and pure audio.

SFO Hacker Dad Artist Canuck @mozilla formerly at @parcinc @meedan @makerlab

SFO Hacker Dad Artist Canuck @mozilla formerly at @parcinc @meedan @makerlab