About the Author
Mass Effect
Final Fantasy X
Batman:Arkham City
Borderlands Series
Weekly Column
Champions Online
World of Warcraft
DM of the Rings
Good Robot
Project Frontier

Exceedingly Clever Person

By Shamus
on Thursday Dec 27, 2007
Filed under:


Jennifer Snow was nice enough to send along a link to the following, which is a YouTube made by Jonny Chung Lee, a CMU student who has developed a way to track head movements (he uses Wii parts) and use the positioning of the user’s head to adjust the camera position in a 3d scene. The upshot is that moving your head around at the computer would let you see different parts of the scene, as if your monitor was a window into the scene. It sounds interesting, but you really have to see it in action:


I remember there was a push for VR goggles in the early / mid 90’s, but the whole effort suffered from a chicken / egg problem. Nobody wanted to buy a toy until it had widespread support, and developers didn’t want to support it because almost nobody owned them. The ~$300 price tag didn’t help. I remember Descent could be hacked to support the goggles. That always sounded exciting, but I wasn’t willing to pay that kind of money to see it in action. Goggles badly needed a “killer app”. It never materialized, and the idea was mostly abandoned.

(Goggles had other problems as well. Aparently strapping a couple of display screens to your eyeballs was often nausea-inducing, probably due to low refresh rates. Plus, computers were just barely fast enough to render 3d scenes to begin with, and rendering two (one for each eye) made things even choppier and less fun. Maybe the idea was doomed from the start, but maybe if it had come a little later and gotten support from a major A-list title (something like Quake II) it might have had a different fate. We’ll never know.)

I like Lee’s idea*, which offers a lot of “wow” factor for a very small investment. Could something like this catch on? The best chance of success would be for one of the major consoles to embrace the idea and make it part of their platform. I’d love to see it happen on the PC, but aside from Lee’s homebrew solution I think such a system would suffer the same fate as VR Goggles: Nobody would buy it because nobody supports it because nobody owns it.

Still, this is the sort of thing that really pushes gaming forward. New graphics are nice, but I’ve been content with graphics for a few years now. What I’m really interested in are new ways to interact with the game. The Wii does this with the nunchuck controller. Guitar Hero / Rock Band do this using their various instrument-based input devices. The PS3 sixaxis input was a good try, although it hasn’t caught on yet. (Not having tried it myself I can’t say if the idea is flawed or just their implementation.) Even DDR dance pads were an innovation in interface, even through the underlying technology was simple and obvious.

The idea demonstrated in the video is more compelling to me than any of the others I just mentioned. I hope I get to see it in action someday.

* Lee never claims that this is his idea, and it’s possible that he’s simply implemented what someone else suggested. In any case, he’s still a clever fellow.

Comments (42)

  1. Adam says:

    Fun, I use to do those VR shooter as the amusment park near my house. I ended up with vertigo for a few minutes afterwards, but no worse than a double corkscrew into a loop coaster. Anywho, this is an idea that is not far off from being put to market. The Wii is the first step in motion capture gaming that will make the controler go the way of the Joystick.

  2. Ian says:

    What impressed me more than anything else was how smooth and precise the movement seemed to be. That Wii be pretty nice. :)

    This demo kind of reminds me of an arcade shooter called Police 911. It uses infrared sensors to determine the player’s position and stance and doesn’t require any sort of head gear. This allows for many of the same motions as this demo, though it’s generally not as fluid or accurate due to it using external sensors and not relying on head tracking.

    Here’s a YouTube link if you want to see it in action: http://www.youtube.com/watch?v=8HIW8Fx-s5g. I used to have a Police 911-2 machine near me (at my DDR hang-out, heh) and it was a pretty fun diversion.

  3. Gwen says:

    The thing I found awsome in this vid is that the equipment required is so cheap. I mean, if you already have a Wii, you just need the glasses and something to put the Wiimote on. I just hope Nintendo will contact this guy to work with him.

  4. Marauder says:

    Makes me think of the TrackIR, it’s better known within flight sim circles, and supposedly great for combat sims. It tracks IR reflectors on your head and is mostly used to allow the player to to move their head to pan the view. I’ve not actually played with one myself so and how easily something like this sort of “virtual window” simulation could be done with it.


  5. Ranneko says:

    The difference headtracking makes is simply astounding, it is one conceptually simple change but the effect is incredibly impressive.

  6. blizzardwolf says:

    Excuse me while I pick my jaw up off the floor.

    Seriously, I haven’t seen VR look that nice and fluid since… well ever. And the equipment to make it work is already there, really all you need is an extra Wii-mote. I can see Wii-owners, as soon as they see this, trying it right away.

    Think about it: Now you can ACTUALLY peek around corners, and get the drop on that poor sap who doesn’t even know you’re there.

  7. K says:

    I am literally shocked at how good this looks. I really hope some game developper implements this, packages cheap glasses with IR leds (how the hell am I going to wear two pairs of glasses?) to the game box, requests that two controllers are to be used and makes a game. I would instantly buy it, even if it was mediocre. Red Steel was great fun, even if it wasn’t actually a good title. This would *rock*.

  8. João says:

    Go to his website and see the other projects he has based on the Wii remote. I love the one where you use a wall-projected image as a tablet PC :-D

  9. Mark says:

    I think his idea could work in its current form. A developer just has to paackage the game with the googles (or other head mounted IR source) and a stand to put a wiimote in. most wii owners have at least 2 wiimotes allready, and Jonny has allreaady pointed out that the technique wont work with more than one playefr.

    sorry aboutthe spelling, lying in bed and my arm has fallen asleep

  10. ngthagg says:

    It’s times like this I love being a geek.

    I love how the stadium demo is done using a flat picture. It shows how little effort is needed to create an immersive effect.

    There are some many games where this would add some significant depth. I was going to describe my favourites, but listing them would only exclude example equally as valuable.

  11. Deoxy says:

    Wow – it was far more impressive than I expected!

    I still goggles (with built-in screen) would be MUCH better, but this is a HUGE step in the right direction.

  12. Seracka says:

    Oh my HECK…

    That was so very cool!!

    Though it would make scary games terrifying and might make some not so scary games really intense.

  13. Renacier says:

    Johnny Lee for Cyber President.

    This could be the basis for some very cool puzzle games. Tetris with multiple layers, anyone?

  14. yd says:

    The market is definitely more supportive of gimmick hardware. As you mentioned, Guitar Hero isn’t suffering for lack of sales. Like Mark said, all it would take is a good game idea and $2 worth of extra equipment (stand and LED glasses). If the game was good enough, and this demo shows the potential, then the hardware expense wouldn’t be an issue.

    I wonder how well the Wii itself could react to the head-tracking. The calculations are fairly simple, but is re-rendering a complex scene that fluidly going to be doable? Keep in mind he was using a PC for the rendering, and the scene was very simple.

    This is probably best suited for a Mario level graphics game to start. The video demonstrates how simple the graphics can be and still fool your mind into seeing 3d.

    I personally find it fascinating how much our brain fills in just based on a slightly changing perspective based on movement. This is the stuff you never realize – but it’s happening constantly. Too cool

  15. Pitros says:

    i have to agree with Marauder here, it looks very close to a TIR, or other TIR analog, such as FreeTrack (a DIY headtracking device ) http://www.free-track.net/english/

    here are a couple videos using freetrack :

    in a flight sim

    in Armed Assault

    and in a racing sim

  16. food4worms says:

    Here’s to all those first-time Doom players I caught trying to peer around corners by moving their heads!

  17. Telas says:

    I first saw this on Gizmodo…


    It’s cool at first, but make sure you watch the last bit of this video. With a simple Wiimote and a projector, this genius replicates a significant chunk of MS Surface.

  18. Vyolynce says:

    Even DDR dance pads were an innovation in interface, even through the underlying technology was simple and obvious.

    …and used by Nintnedo on the original 8-bit NES over a decade ago. Of course, the NES Power Pad only supported like two games (World Class Track Meet and Dance Aerobics), but still.

  19. Jeremiah says:

    Heck, I’d almost be willing to pay a few bucks just for the basic demo he shows. Move your head around, shoot targets. That’d be pretty fun, and offer a diversion for a little while. Wouldn’t even take that much to do a little more with it. Moving targets, obstacles to shoot around, etc.

  20. Davesnot says:

    That’s sweet… and simple.. and not bulky.. but you still wouldn’t be able to “look around corners” like someone suggested.. it’s like a window.. you can press your nose against it to see more.. but if you turned your head 90-degrees.. well.. you would be looking at whatever artwork you have hanging on your wall.. awesome stuff, though.

    I love when people tear stuff open, hit radio shack and build what they really want… just awesome.

  21. MSchmahl says:

    Instead of targets, there should be fishies. That would be a great conversation piece.

  22. THOR says:

    So, looking at his other videos, I’m seeing the Wiimote-whiteboard demo, right? Then he mounts the Wiimote to the projector and points it at his tabletop and mucks about with not just one light pen, but two simultaneously.

    I totally just saw a virtual, light pen-based, interactive gaming board for my D&D sessions. Nice.

  23. Vegedus says:

    Wow, that is incredibly neat. Also, I can’t see two player wouldn’t be a problem if you had used splitscreen.

    Anyway, I just love this weird Wii videos. Even though the technology itself isn’t really new, the tube has blossomed with all sorts of videos with people using their wii for all sorts of inventive purposes. Now that I’ve finally got one myself, I should probably try some of it out.

  24. DocTwisted says:

    The DDR pad wasn’t even new… the 8-bit NES had a “Power Pad” that was used for a set of olympic-style games.

  25. Cadamar says:

    Very cool!
    The next couple of years should be very exciting for gaming. The Wii was an experiment and a huge risk for Nintendo, but with it’s enormous success it should open the door to a lot more inovation in Human-machine interfaces. So “cheers” to Nintendo for having some balls. Things should start getting interesting here really soon.
    Personally, I can’t wait to get my hands on a mind-reading controller.

  26. Martin says:

    Set that up with a web-cam that can do eye-tracking and you don’t even need a special sensor.

  27. Alden says:

    I suspect that the 2D vision of the camera he used to record this is making it look more impressive than it is, but still, it does look very clever. :)

  28. My dad did his dissertation on presence in Virtual Reality at Virginia Tech, i.e. on what makes you feel most “present” in the artificial world, which is why I was intrigued by this and sent you the link. I don’t really know enough about technology to have anything useful to say about it, myself.

    I got to have a small part in his project (I took the tests and wandered around in his VR world). This was back in the day when Pentiums just came out, and they used a separate machine to control each eye. Even so, it was awkward and patently artificial.

    Feeling dizzy or ill is a result of simulator sickness, most likely, where your brain isn’t receiving the motion input your eyes are saying you ought to be. This is a problem with flight instructors that teach students inside flight simulators . . . the simulator moves, but not much, and the instructor often gets ill because he knows what he *should* be feeling, and he doesn’t. The student doesn’t get ill because he doesn’t have conditioned responses to confuse his actual perceptions.

  29. Cadamar said: “Personally, I can't wait to get my hands on a mind-reading controller.”

    You can get your hands on one *now* . . . if you have a huge pile of cash. My dad is working on developing this sort of thing for Boeing. The primary use is for pilots controlling large numbers of drones from a remote location. When numerous things are going on at the same time, the system detects exceptional stress in the decision-making centers of your brain and begins automatically compensating, stepping up its AI problem-handling and presenting you only with a very few, immediate problems that need to be solved. When the crisis diminishes, it lets the human pilot take over again. It’s quite fascinating. If you go to http://www.boeing.com/news/frontiers/archive/2006/december/cover.pdf and scroll ALL the way to the VERY end of the document, you can see a photo of my dad with his equipment and a little blurb about their project. I think there’s a video floating around somewhere, too, but I forget where to locate it.

  30. roxysteve says:

    Now all you need to do is combine this “headset Wii” thing with my patent-pending Ronco Hedzup (TM) (which suspends your widescreen 19″ flatscreen monitor a foot or so in front of your face using a sort of bent wire coathanger thing attached to your skull with either bolts, nails or double-sided sticky tape) and you are in business.


  31. Cineris says:

    Amazing! I’m not quite sure how you’d get this to work in a live game (how do you handle turning and moving with this thing?), but I’m looking forward to seeing some attempts.

  32. SimeSublime says:

    Well, the obvious place to start would be an on-the-rails shooter such as Time Crisis. Let the game run you around a track, whilst you look around corners and boxes to shoot at enemies with another Wii remote. To be honest, it’s hard to think of that not selling well.

  33. Crusader Corim says:

    I think that’s a great idea, SimeSublime. It makes me wish that Umbrella Chronicles could be played that way. Curse you for reminding us what we’re still short of.

  34. dragonbane says:

    This reminds me of The Cave down at the University of Illinois. I was down there about 12 years ago and they had some very cool games and demos – including those fish that MSchmahl wanted. It was a bit more extensive though, a 3-sided 10×10 room, and you with goggles and a sort of joystick sensor.

  35. Gobo says:

    Wow. Hadn’t seen that one yet. Just the other two demos on his site (which are also quite cool, though in a more geeky kind of way).

    Now I really have to consider getting a Wiimote and try this out.

    (hmm.. also, I appear to have a gravatar… wonder when I made that… :)

  36. Lord_Lothar says:

    Wow! That’s really cool! While I do agree that it might take a while to catch on, I also think that unlike VR goggles, this is something that could be fairly easy to implement from a development side. After all, you would only need the mount for one Wii remote, and then the sensor bar headpiece, which Lee was able to create fairly inexpensively. Nintendo has already shown a willingness to create new and innovative peripherals before, and this would fit right into their way of doing business. In fact, if they’re smart, they’ll hire this guy into their R&D department as soon as possible.

  37. Oleyo says:

    While he was explaining it, I had no idea the effect would be so dramatic. VERY cool.

  38. kamagurka says:

    Fuck wow, that is brilliant!

  39. Phlux says:

    I think I have everything needed to use this except the IR goggles. I could rip my sensor bar off my tv, but it’s on there with strong double-sided tape, so that would make a mess.

    I’m sure a couple of remote controls with the buttons mashed down will do the trick, though. I saw a video about the Wii sensor bar a long time ago and was amazed at what a simple device it was. It’s literally just an infrared LED on each end, and the wire is the power source. It doesn’t even communicate with the Wii…that’s all done with the remote.

    I’m gonna try it out this weekend.

  40. Kayle says:

    This is VR stuff that’s been around in research labs, made possible for consumers due to Nintendo’s good engineering, manufacturing, and massive sales volume. Very cool, I’ve been on my mind all day.

    The goggles and glasses that started to become available for consumers in the mid-90s were LCD stereoscopic shutter glasses, mostly rather low quality. Stereoscopic displays basically show a slightly different image for each eye, to give a 3D impression. Head tracking wasn’t available at consumer prices back then, and had low accuracy and latency issues even with high end gear then. Head tracking gives a 3D effect when you are moving your head, stereoscopic displays give a 3D effect when you are standing still, together they are synergistic.

    I did a bit of googling and it looks like the quality of the low end consumer stereoscopic gear is better now, but there’s a new problem, they don’t work as well with LCD displays compared to CRTs. Nvidia apparently has driver support for stereoscopic displays.

    Just possibly, the time has come for the combination.

  41. Rats says:

    I realise these comments have had nothing new for 2 months, but a friend was going through accademic papers for a wireless tracking project recently and came accross this paper on a “head mounted three dimensional display”, published in 1968.

    Just for anyone who is still interested.

Leave a Reply

Comments are moderated and may not be posted immediately. Required fields are marked *


Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun.

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>