Posts Tagged ‘Milo’

Stereoscopic 3D TVs and Project Natal

One thing we haven’t heard much from Microsoft about is stereoscopic 3D (aka S3D) support for Xbox 360 and specifically for Project Natal. A couple of games that I can think of support stereoscopic 3D on the Xbox 360: Avatar, and G-Force. And there are lots of new stereoscopic 3D TVs hitting the market now.

Stereoscopic 3D means each eye sees a separate image because it is seeing the scene from a slightly different angle. Your brain automatically looks at the difference between the two eyes and calculates depth from it. That’s why people have two eyes.

Anything that’s at the same screen position for both eyes will look like it’s at the same distance as the screen is. Which is what normal TV looks like. But anything that’s drawn further to the left on your right eye, and further to the right on your left eye, looks like it’s popping out in front of the screen. On the other hand, anything that’s drawn further to the left on the left eye, and further to the right on the right eye looks like it’s behind the screen. Your eyes are very sensitive, and even one pixel difference is enough for you to see the difference in depth.

By now you should have seen that effect at a stereoscopic 3D movie. If not, go and watch Avatar 3D. It is really, really cool. You feel like you can reach out and grab things that are in front of the screen. And just as good is feeling like there is a whole huge world stretching far out into the distance behind the screen.

atokirina': Avatar 3D seeds

Of course in Avatar when you reach out and touch the floating seeds that seem just in front of you, you can’t actually grab them and hold them and move them around in your hand. You’re probably thinking: “Well, duh! Of course not!”. But with Project Natal you can actually do that!

You see, project Natal already knows exactly where your hand is in 3D space. And it knows exactly where your eyes are in 3D space. So if you tell Project Natal exactly where your TV is in 3D space, by telling it what size TV you have and where you put the Natal sensor bar, then it can very easily calculate the line from your eye to your hand to the point on the screen that you are grabbing. So it knows which part of the scene you are touching.

More importantly, it knows exactly where to draw something so it looks like you are holding it in your hand! Imagine that 3D in front of the screen effect in Avatar but with a sword, gun, baseball bat, or even a weighted companion cube, that is drawn at the exact location of your hand and follows your hand precisely whenever you move or rotate your hand, and compensates for moving your head. It would look exactly like you were holding the virtual object in your real hand right in front of your eyes. You wouldn’t have to mime anything, because you can really see and hold the virtual object in your hand.

So, what’s the catch? Well, it only works when your hand is in front of the TV. When the 3D object looks like it should cross in front of the edge of the TV, the edge of the TV actually goes in front of it instead, and half the object disappears and it spoils the illusion. So you need a big enough TV, and you need to be close enough to the TV.

And there’s the catch that the hilt of a sword or the handle of a gun that is supposed to be drawn in front of your hand can’t actually be drawn in front of your hand because it’s drawn on the screen and your hand is in the way. The rest of the gun or sword would look right, but not the part that should be covering your hand but instead your hand seems to be covering.

The other catch is that the image on the TV is a bit out of focus and blurry when you look at your hand. Even if the image is in stereoscopic 3D and looks like it is right next to your hand, either the image or your hand will be out of focus because in reality they are at completely different depths. That would be OK, except that it is actually hard for your eyes to focus on one depth while they are converging (aiming) at a different depth. That makes it hurt to look at 3D that is too far in front of the screen. So you need to be reasonably close to the screen, and have your hand a reasonable distance away from your eyes. Or you just need to not look directly at the object in your hand and focus more on the rest of the scene.

Another catch is that there is lag. You would move your hand, and 100 ms later the thing you are holding will move. The same with moving your head.

I still think it would be awesome though. Especially with a big screen.

Ricochet's avatar (from Scientific American)

Think about the Project Natal game Ricochet. You don’t actually hit balls with your body. Your avatar on the screen, in it’s own virtual world, copies your movements and hits the virtual balls. Meanwhile you are outside in the real world and balls never come out of the screen towards you. But with Stereoscopic 3D, the real world and the virtual world can share the same space out in front of your TV in your living room with you. So the balls would come all the way out to your real hand, and you can hit them with your real hand, or catch them with your real hands, and even hold them and move them around in your hands. The same with fighting games, or sports games, or shooting games. Wouldn’t it be cool to have to physically duck projectiles that are really coming out of the screen at you.

Most project Natal games that we have seen have an avatar interacting with things on your behalf, instead of you interacting with them. Except for Milo and Kate, Burnout Paradise (the racing game), the quiz game, and the first part of the fighting game when you talk to the opponent. Wouldn’t Milo and Kate be better though if the thrown goggles really did seem to come out of the screen? And if you really could catch the goggles? And if you could see the goggles in your hands? And if you could really put them on? Obviously you wouldn’t be able to feel them, but you could see the goggles in your hands and responding to how you move them.

To some extent the things that I’ve said above can also be done without stereoscopic 3D, and only using the Johnny Lee head-tracking 3D effect, but it wouldn’t be quite as realistic (unless you only have one eye).

With stereoscopic 3D support, Natal would need a new slogan:
“No avatar required. You are the avatar.”

Project Natal Innovation Journey Parody

If you watched the video “Innovation Journey” from CES 2010 that I posted a while ago, then you might enjoy this parody of it: “Transformation Journey into Sameness” by ScrewAttack. If you haven’t watched the original CES video, then view Project Natal at CES 2010 first.

I think Project Natal will be great, so don’t be offended or think I am criticising Natal by posting these parodies.

For more spoof videos about Project Natal, just click on the “parody” tag below.

Augmented Reality Façade

This is a follow up to Façade would be great on Project Natal.

For those of you who don’t read the comments, Ryan Burke informed me about an Augmented Reality version of Façade that they made at Georgia Tech in 2006. They recreated Grace and Trip’s apartment in real life, and you walk around it with an eMagin Z800 Virtual Reality visor and camera on your head and can see Grace and Trip superimposed on the real world. You can thus walk around naturally and talk to them naturally. It does Voice Recognition the cheating way, by having a human type what you say into a computer. You can even pick up the objects around the apartment and have Grace and Trip react to it. Apparently it is very immersive.

So let’s hope Michael Mateas and Andrew Stern get together with Microsoft and make Façade for Xbox 360’s Project Natal.

Façade would be great on Project Natal

The Milo and Kate game looks like really impressive technologically. But the setting of playing with a child and helping him with his homework isn’t the most interesting application for it.

Trip and Grace in their apartment in Facade

But back in 2005, a freeware PC game called Façade was released. They had similar ideas, but didn’t have such good technology.

Façade is a First Person Drama game. The aim of the game is to save the marriage of your friends Trip and Grace, while you are invited to their apartment for drinks. They will try to spend the whole night arguing with each other, and will normally get angry and want a divorce before the night is out.

You can walk around the 3D apartment with the arrow keys and look around. And you can interact with or pick up objects with the mouse. You can also hug, kiss or comfort either Trip or Grace by clicking in the appropriate place on their body.

But most importantly, you can talk to them by typing whatever you feel like saying. They will talk back to you with recorded voice samples like Milo does. Dialog is in real time, and you can interrupt other people if you want. At the start of the game you can choose your name from a list and then Grace and Trip will talk to you by name, the way Milo does. Unfortunately, “Claire” isn’t one of the names you can choose, and neither was my name: Carl. But strangely I could choose the Vietnamese name Khá, which is what half my friends call me anyway.

I found typing in real time to be too hard, especially since they don’t respond to the fact that you started typing, like a real person would when you start speaking. They wait until you press Enter. I also found they didn’t seem to respond as much to what I said as they should, and I didn’t seem as in control of what happened as I should. But it is a good effort, and it’s possible to have lots of different outcomes.

But this game would be great if Microsoft or Lionshead got together with the developers of this game and made a Project Natal version with proper speech recognition, along with tone of voice and facial expression recognition, and added more dialog possibilities. It could also do with better graphics.

You can download this game from http://www.interactivestory.net/ and experience a very different genre of gaming.

Project Natal at CES 2010

Project Natal’s most recent appearance was at CES 2010 earlier this month.

At CES, Microsoft gave a really boring introduction, then showed a new video called “Innovation Journey”.

This is my favourite video, since it actually explains how Project Natal works, shows what Project Natal’s depth sensor is actually seeing, and talks a bit to some of the people behind it. And of course it tells us the release date!

Note in particular the scenes where it shows on the screen a 3D view of what the depth sensor is seeing, with the different object types colour-coded. Notice how there is a black hole in the background where the camera can’t see what’s there because it’s blocked by you standing in front of it. But aside from that, it has a very good 3D model of the scene. You can see how easy it would be to recognise different kinds of objects from this sort of information, even without the colour data that you also get. Much more powerful than just a camera like eye-toy.

We also get to see a (blurry) close-up of the back of the project natal hardware.

At CES 2010, Microsoft also talked about Xbox Live, and their own version of Nintendo’s Virtual Console. The Xbox is getting a virtual arcade called Games Room, where your avatar can walk into a 3D arcade and see and hear all their purchased arcade games in their 3D cabinets in attract mode before you play them. You can also invite friends and play against them online. They have 30 different arcade games that you can buy.

And they talked about how people will be able to rent and watch videos online quickly, using Xbox Live.

More fighting and zombie games for Project Natal

Capcom’s Head of R&D, and Managing Corporate Officer, Keiji Inafune was interviewed at TGS 2009 too.

Capcom are the creators of Street Fighter, Resident Evil, Dead Rising, Devil May Cry, Bionic Commando, and Lost Planet, among many other games.

I’m sure you can imagine the potential of some of these games on Project Natal, and let’s hope Capcom can too. He doesn’t say much useful, other than that Natal will allow them to have more complex and detailed game actions by not being limited by the number of buttons on the controller. I’m not sure how he’s going to make videos that are interactive like he says at the end. I’m imagining a cross between say the cutscenes in Resident Evil 5, and a simplified Project Milo, would make for interesting and immersive story telling between chapters in any type of game.

Japanese talking head videos are getting a bit boring, but here’s the video link: http://www.youtube.com/watch?v=tU9TWrikZBI

And here’s the transcript:

“I believe many creators feel suppressed or even bored with the current trend of game development. But at the same time it’s a challenge for us to produce something beyond what we are so used to doing. I am certain that Natal will bring on that challenge. This doesn’t mean that it’s being replaced, but rather, it’s being enhanced. And by it being enhanced, it allows us to do more.

Take for instance, the number of buttons on an average controller. We were working on a very limited interface. But if we could create games without limits, it allows us to make it more interesting, for example, the physical experience. Its not something that reacts only to your fingertips. You can use your fingers too, but it doesn’t limit you to them. Being able to use a variety of things is what makes it so ground-breaking and innovative.

In that sense, by mixing in videos for instance, and a variety of other interactive elements may be the key to creating new gaming experience.”

Johnny Lee

Johnny Chung Lee is perhaps the most famous and coolest among the Wii Remote (or Wiimote) hackers. – Man, that makes me jealous. I could so nearly have been him. But compared to him, I’m only the Radek Zelenka of Wiimote hacking.

My favourite Wiimote project of his is this one:

You don’t actually have to have the Sensor Bar on your head and the Wiimote on your TV. Doing it that way you are actually throwing away valuable information about head rotation (only needed for stereoscopic 3D with this kind of VR) and giving yourself a smaller range that you can move around in (since the Wiimote camera has a lower FOV than the sensor bar does, and if the Wiimote was on your head it would turn to face the screen when you do). The reason why Johnny Lee does it the backwards way, is because the Wiimote is heavier, more awkward and hard to mount on your head.

This isn’t a new invention, it’s actually based on the original super-sized version by Carolina Cruz (no, not the Colombian model):

The virtual objects that appear to be attached to the woman’s wand controller are really drawn on the walls and floor several feet away. So are the laser beams you see coming out of her wand. And the illusion works perfectly when we move the camera, because the camera and the wand are very precisely tracked in 3D space and the images drawn on the wall are drawn exactly how they would look from that angle if they were really attached to the wand. There are 3 walls and a floor all at right angles, but you can hardly see the seams because it compensates for them being at right angles by drawing each individual screen with its own head-tracking algorithm. This is exactly the same as Johnny Lee is doing, but with several huge screens. This kind of Virtual Reality is called a CAVE, and costs several million dollars. It’s a lot more fun than a head-mounted display.

I actually implemented this same head tracking Virtual Reality on the computer long before Johnny Lee famously showed it. But I used an (even cheaper) Essential Reality P5 Glove on my head since Wii Remotes didn’t exist yet, and I used a monitor and a TV at right angles, and I added red/cyan 3D glasses. I made it so that it could be used in existing Direct3D games, and I played it with Hitman 2 (because it comes free with the glove) and Lord Of The Rings: Return Of The King. It was very cool. But I never finished it enough to release it, and never made a video. Maybe I will one day.

But why am I talking about Johnny Lee, Wii Remotes, and Virtual Reality gloves when this isn’t my GlovePIE blog?

Because Johnny Lee was recruited onto the Xbox: Project Natal team! And this kind of head-tracked virtual reality is coming to Project Natal. The Milo and Kate game is known to include this feature, and hopefully all the other games do to.

Johnny Lee’s blog post is here: http://procrastineering.blogspot.com/2009/06/project-natal.html

How Project Natal Works

Project Natal has two input devices:

First is a 3D camera, which can measure both the colour and the distance of every pixel.

Second is a microphone array, which is several directional microphones arranged in a pattern. They can separate out different sound sources and determine where each sound came from, and can thus filter out any noise.

The 3D camera is actually made out of two cameras, one that senses distance using infra-red and one normal one that senses colour. But it helps to think of it as a single camera that can also measure depth.

The 3D camera makes Project Natal much more powerful than a 2D camera like the Eye-toy. It makes it dead easy to filter out background objects, and find only the objects that you are looking for. And it doesn’t need to rely on colours to recognise things, it can just look at their 3D shape.

It could operate in complete darkness and still see the 3D shape of everything, but knowing the colours would make it easier to recognise hands and faces and other things.

The 3D camera would also allow it to easily scan in any 3D object and convert it into a virtual object in the game. It has only been demonstrated with the 2D image on a skateboard, but there is no reason why Natal couldn’t scan in any object you showed it, if you showed it from a few different angles.

One downside of the 3D camera is that it relies on line-of-sight and can’t see behind things. So it can’t track your hand when it goes behind your back, or when someone stands in front of you. But a Wii Remote, or magnetic 6DOF tracker could still track it.

Another downside is that it could get confused about the distance of transparent or reflective objects, since the IR light that goes back might have first come from somewhere further away.

The microphone array is important for speech recognition, since it can filter out noise, and could separately recognise several speakers talking at once. Especially since the TV will also be making noise. It could also be used to have virtual characters still look at you when you go out of sight of the camera and still talk to it. And in a multiplayer game it helps it know which player is speaking by knowing which sound came from which body that it can see.

It doesn’t just recognise words though. In games like Milo and Kate it can recognise the emotion of the speaker, and can tell things like whether they are telling a joke. It could also be used for rhythm games by using anything that makes a sound, or by clapping. It’s not known whether the monster game used the microphone or facial expression tracking to control breathing fire. It would have to be a good quality microphone, because normally speech recognition requires a microphone a few cm from your mouth.

But the most important part of how Project Natal works is the software. Microsoft, with some help from Peter Molyneux, went around to all their different research projects that they’d been working on for other projects, and collected their software technology and put it all together.

First, there’s the speech recognition. Microsoft has been working on their own speech recognition engine and API for a long time. You can use the same speech recognition engine as Natal for free in gaming right now if you want. Download GlovePIE, and train speech recognition in the Speech control panel. If you only have Windows XP, you will need to first either install the speech recognition from Microsoft Office (best way) or download and install the SAPI 5 SDK with speech recognition.

Then there’s facial recognition. You might have seen this in other Microsoft products, such as Windows Live Photo Gallery, which you can download here for free, and start your computer automatically recognising all the faces in your photos.

Peter Molyneux also mentions handwriting recognition, although we haven’t seen it used yet. But I’m guessing we will in some games.

And there was no doubt a lot of other code.

Then there’s a huge amount of new software which Microsoft had to write. It has to find the shapes of people, and from there convert the surface data into 48 skeletal points for each player. It can do that for 4 people at once, 30 times per second. It can even identify individual fingers if they are close enough.

The need for all that software is why Project Natal could only have been made fully by Microsoft, not Nintendo, Sony, or Sega.

More Milo and Kate Parodies

Here are some more well-made Milo and Kate parody videos. Some of these spoofs are very humorous:

Milo and Kate Parody

Here is a humorous parody video by IGN Originals of Peter Molyneux’s game Milo and Kate for Project Natal on the Xbox 360: