Archive for February, 2010

Stereoscopic 3D TVs and Project Natal

One thing we haven’t heard much from Microsoft about is stereoscopic 3D (aka S3D) support for Xbox 360 and specifically for Project Natal. A couple of games that I can think of support stereoscopic 3D on the Xbox 360: Avatar, and G-Force. And there are lots of new stereoscopic 3D TVs hitting the market now.

Stereoscopic 3D means each eye sees a separate image because it is seeing the scene from a slightly different angle. Your brain automatically looks at the difference between the two eyes and calculates depth from it. That’s why people have two eyes.

Anything that’s at the same screen position for both eyes will look like it’s at the same distance as the screen is. Which is what normal TV looks like. But anything that’s drawn further to the left on your right eye, and further to the right on your left eye, looks like it’s popping out in front of the screen. On the other hand, anything that’s drawn further to the left on the left eye, and further to the right on the right eye looks like it’s behind the screen. Your eyes are very sensitive, and even one pixel difference is enough for you to see the difference in depth.

By now you should have seen that effect at a stereoscopic 3D movie. If not, go and watch Avatar 3D. It is really, really cool. You feel like you can reach out and grab things that are in front of the screen. And just as good is feeling like there is a whole huge world stretching far out into the distance behind the screen.

atokirina': Avatar 3D seeds

Of course in Avatar when you reach out and touch the floating seeds that seem just in front of you, you can’t actually grab them and hold them and move them around in your hand. You’re probably thinking: “Well, duh! Of course not!”. But with Project Natal you can actually do that!

You see, project Natal already knows exactly where your hand is in 3D space. And it knows exactly where your eyes are in 3D space. So if you tell Project Natal exactly where your TV is in 3D space, by telling it what size TV you have and where you put the Natal sensor bar, then it can very easily calculate the line from your eye to your hand to the point on the screen that you are grabbing. So it knows which part of the scene you are touching.

More importantly, it knows exactly where to draw something so it looks like you are holding it in your hand! Imagine that 3D in front of the screen effect in Avatar but with a sword, gun, baseball bat, or even a weighted companion cube, that is drawn at the exact location of your hand and follows your hand precisely whenever you move or rotate your hand, and compensates for moving your head. It would look exactly like you were holding the virtual object in your real hand right in front of your eyes. You wouldn’t have to mime anything, because you can really see and hold the virtual object in your hand.

So, what’s the catch? Well, it only works when your hand is in front of the TV. When the 3D object looks like it should cross in front of the edge of the TV, the edge of the TV actually goes in front of it instead, and half the object disappears and it spoils the illusion. So you need a big enough TV, and you need to be close enough to the TV.

And there’s the catch that the hilt of a sword or the handle of a gun that is supposed to be drawn in front of your hand can’t actually be drawn in front of your hand because it’s drawn on the screen and your hand is in the way. The rest of the gun or sword would look right, but not the part that should be covering your hand but instead your hand seems to be covering.

The other catch is that the image on the TV is a bit out of focus and blurry when you look at your hand. Even if the image is in stereoscopic 3D and looks like it is right next to your hand, either the image or your hand will be out of focus because in reality they are at completely different depths. That would be OK, except that it is actually hard for your eyes to focus on one depth while they are converging (aiming) at a different depth. That makes it hurt to look at 3D that is too far in front of the screen. So you need to be reasonably close to the screen, and have your hand a reasonable distance away from your eyes. Or you just need to not look directly at the object in your hand and focus more on the rest of the scene.

Another catch is that there is lag. You would move your hand, and 100 ms later the thing you are holding will move. The same with moving your head.

I still think it would be awesome though. Especially with a big screen.

Ricochet's avatar (from Scientific American)

Think about the Project Natal game Ricochet. You don’t actually hit balls with your body. Your avatar on the screen, in it’s own virtual world, copies your movements and hits the virtual balls. Meanwhile you are outside in the real world and balls never come out of the screen towards you. But with Stereoscopic 3D, the real world and the virtual world can share the same space out in front of your TV in your living room with you. So the balls would come all the way out to your real hand, and you can hit them with your real hand, or catch them with your real hands, and even hold them and move them around in your hands. The same with fighting games, or sports games, or shooting games. Wouldn’t it be cool to have to physically duck projectiles that are really coming out of the screen at you.

Most project Natal games that we have seen have an avatar interacting with things on your behalf, instead of you interacting with them. Except for Milo and Kate, Burnout Paradise (the racing game), the quiz game, and the first part of the fighting game when you talk to the opponent. Wouldn’t Milo and Kate be better though if the thrown goggles really did seem to come out of the screen? And if you really could catch the goggles? And if you could see the goggles in your hands? And if you could really put them on? Obviously you wouldn’t be able to feel them, but you could see the goggles in your hands and responding to how you move them.

To some extent the things that I’ve said above can also be done without stereoscopic 3D, and only using the Johnny Lee head-tracking 3D effect, but it wouldn’t be quite as realistic (unless you only have one eye).

With stereoscopic 3D support, Natal would need a new slogan:
“No avatar required. You are the avatar.”

Project Natal Innovation Journey Parody

If you watched the video “Innovation Journey” from CES 2010 that I posted a while ago, then you might enjoy this parody of it: “Transformation Journey into Sameness” by ScrewAttack. If you haven’t watched the original CES video, then view Project Natal at CES 2010 first.

I think Project Natal will be great, so don’t be offended or think I am criticising Natal by posting these parodies.

For more spoof videos about Project Natal, just click on the “parody” tag below.

Augmented Reality Façade

This is a follow up to Façade would be great on Project Natal.

For those of you who don’t read the comments, Ryan Burke informed me about an Augmented Reality version of Façade that they made at Georgia Tech in 2006. They recreated Grace and Trip’s apartment in real life, and you walk around it with an eMagin Z800 Virtual Reality visor and camera on your head and can see Grace and Trip superimposed on the real world. You can thus walk around naturally and talk to them naturally. It does Voice Recognition the cheating way, by having a human type what you say into a computer. You can even pick up the objects around the apartment and have Grace and Trip react to it. Apparently it is very immersive.

So let’s hope Michael Mateas and Andrew Stern get together with Microsoft and make Façade for Xbox 360’s Project Natal.

Façade would be great on Project Natal

The Milo and Kate game looks like really impressive technologically. But the setting of playing with a child and helping him with his homework isn’t the most interesting application for it.

Trip and Grace in their apartment in Facade

But back in 2005, a freeware PC game called Façade was released. They had similar ideas, but didn’t have such good technology.

Façade is a First Person Drama game. The aim of the game is to save the marriage of your friends Trip and Grace, while you are invited to their apartment for drinks. They will try to spend the whole night arguing with each other, and will normally get angry and want a divorce before the night is out.

You can walk around the 3D apartment with the arrow keys and look around. And you can interact with or pick up objects with the mouse. You can also hug, kiss or comfort either Trip or Grace by clicking in the appropriate place on their body.

But most importantly, you can talk to them by typing whatever you feel like saying. They will talk back to you with recorded voice samples like Milo does. Dialog is in real time, and you can interrupt other people if you want. At the start of the game you can choose your name from a list and then Grace and Trip will talk to you by name, the way Milo does. Unfortunately, “Claire” isn’t one of the names you can choose, and neither was my name: Carl. But strangely I could choose the Vietnamese name Khá, which is what half my friends call me anyway.

I found typing in real time to be too hard, especially since they don’t respond to the fact that you started typing, like a real person would when you start speaking. They wait until you press Enter. I also found they didn’t seem to respond as much to what I said as they should, and I didn’t seem as in control of what happened as I should. But it is a good effort, and it’s possible to have lots of different outcomes.

But this game would be great if Microsoft or Lionshead got together with the developers of this game and made a Project Natal version with proper speech recognition, along with tone of voice and facial expression recognition, and added more dialog possibilities. It could also do with better graphics.

You can download this game from http://www.interactivestory.net/ and experience a very different genre of gaming.

Project Natal on Smallville

Smallville a couple of nights ago (for US viewers) featured Lois Lane’s cousin, Chloe, playing the Project Natal game Ricochet on Lois Lane’s Xbox 360.

It is Season 9, Episode 12: “Warrior”. Chloe doesn’t know it, but the man is actually a child called Alec Abrams who discovered an enchanted comic book and used it to temporarily become a grown-up superhero, and then saved Chloe. Apparently Chloe was impressed, so when she invited him up to Lois Lane’s room above the Talon, Chloe was apparently hoping for some adult entertainment, but was disappointed he just wanted to play Project Natal instead.

The implication is that Project Natal is slightly childish (although Lois Lane obviously owns it). Although the episode does end with the idea that Chloe needs to have more fun, so the episode isn’t completely harsh on Natal.

Watch the video at: Project Natal makes a Smallville cameo, does not guarantee ability to fly

This reminds me, I have to start watching Smallville again. I actually thought Smallville had been cancelled when they stopped showing it in Australia. But apparently it is still going with 9 seasons.

VisionPlay brings motion tracking to PC

VisionPlay are trying to cash in on the Project Natal craze with their own full-body tracking system for the PC. It is designed to work just with any webcam, but still track the position and orientation of your body parts. They are working on a Tennis game called VisionTennis.

Unlike Natal, they don’t have a 3D camera or an array microphone to work with, so they are doing things the hard, and error-prone way. Which means this probably won’t work anywhere near as well as Natal. But if what they say in the video is true, it will still be very impressive. Personally, I think they should have used simple props in a tennis game rather than hand gestures that look like handball though.

See http://trendyent.com/projects/visionplay.

Halo: Reach won’t support Project Natal

Sad news. Despite the rumours, Bungie just officially stated that there will be no Project Natal support for Halo: Reach.

Halo: Reach Screenshot Bug #15163

Bungie say:

“Myth: Reach is being built for Natal! (See that crosshair to the left as proof!)

Truth: Halo: Reach is NOT a Natal title and is being developed expressly with the traditional Xbox 360 controller in mind.”

See Bungie’s Halo: Reach Mythbusters.

Live dead for Xbox v1 to make way for Natal birth

Microsoft is permanently shutting down Xbox Live for both the original Xbox, and for original Xbox games, such as Halo 2, played on an Xbox 360. And Natal is taking the blame.

The shut-off will happen on April 15.

Microsoft says: “To reach our aspiration, we need to make changes to the service that are incompatible with our original Xbox v1 games… and I couldn’t be more excited about what we have in store with “Project Natal” and LIVE.”

You can read Microsoft’s press release by Marc Whitten.

Project Natal prototype image

Brier Dudley took a photo of this Project Natal prototype at a third party developer’s office:

Photo by Brier Dudley of the prototype Project Natal

If you followed my instructions with the last video I posted about, you might recognise this ugly duckling as much the same as the Project Natal you briefly saw the back of in the Innovation Journey video.

The thing that strikes me as odd about this hardware, is that it’s not level. Surely something like this needs to either be horizontal or else have accelerometers in it so it can measure it’s tilt. For example, in the car racing game we have seen, people steer by holding up their two hands and tilting them. But if the device itself is tilted, it will look like you are steering one way when you are actually holding your hands level.

The i Think Different blog thought this device reminded him of WALL•E. I think it’s cute.

This is not what the final device will look like when it is manufactured by Pegatron.

The stickers say “Caution: Potentially hazardous laser energy. Do not open enclosure.”, “Class 1 Laser” and “MS Equipment E690422″.

In case you were wondering “Class 1 Laser” means a laser that is perfectly safe and can’t hurt your eyes. Higher classes are dangerous. I’m assuming this is an infra-red laser for measuring depth. But even class 1 lasers are dangerous if you open the case and mess around with the insides.

Project Natal: Made In Taiwan

According to this report in Taipei’s DigiTimes online newspaper, and passed on via this Seattle Times blog post, Project Natal will be manufactured by a company called Pegatron in Taipei, Taiwan.

You can view the manufacturer Pegatron’s website here: http://www.pegatroncorp.com/, although they don’t say much and don’t mention Project Natal.

Of course the real magic is in the software, which is manufactured in Redmond, Washington, USA (and also around the world) by Microsoft. There is a huge amount of software required to turn the raw inputs into usable information such as where each of your joints are, who you are, what you are saying, and how you are saying it.