I think Project Natal will be great, so don’t be offended or think I am criticising Natal by posting these parodies.

For more spoof videos about Project Natal, just click on the “parody” tag below.

First is a 3D camera, which can measure both the colour and the distance of every pixel.

Second is a microphone array, which is several directional microphones arranged in a pattern. They can separate out different sound sources and determine where each sound came from, and can thus filter out any noise.

The 3D camera is actually made out of two cameras, one that senses distance using infra-red and one normal one that senses colour. But it helps to think of it as a single camera that can also measure depth.

The 3D camera makes Project Natal much more powerful than a 2D camera like the Eye-toy. It makes it dead easy to filter out background objects, and find only the objects that you are looking for. And it doesn’t need to rely on colours to recognise things, it can just look at their 3D shape.

It could operate in complete darkness and still see the 3D shape of everything, but knowing the colours would make it easier to recognise hands and faces and other things.

The 3D camera would also allow it to easily scan in any 3D object and convert it into a virtual object in the game. It has only been demonstrated with the 2D image on a skateboard, but there is no reason why Natal couldn’t scan in any object you showed it, if you showed it from a few different angles.

One downside of the 3D camera is that it relies on line-of-sight and can’t see behind things. So it can’t track your hand when it goes behind your back, or when someone stands in front of you. But a Wii Remote, or magnetic 6DOF tracker could still track it.

Another downside is that it could get confused about the distance of transparent or reflective objects, since the IR light that goes back might have first come from somewhere further away.

The microphone array is important for speech recognition, since it can filter out noise, and could separately recognise several speakers talking at once. Especially since the TV will also be making noise. It could also be used to have virtual characters still look at you when you go out of sight of the camera and still talk to it. And in a multiplayer game it helps it know which player is speaking by knowing which sound came from which body that it can see.

It doesn’t just recognise words though. In games like Milo and Kate it can recognise the emotion of the speaker, and can tell things like whether they are telling a joke. It could also be used for rhythm games by using anything that makes a sound, or by clapping. It’s not known whether the monster game used the microphone or facial expression tracking to control breathing fire. It would have to be a good quality microphone, because normally speech recognition requires a microphone a few cm from your mouth.

But the most important part of how Project Natal works is the software. Microsoft, with some help from Peter Molyneux, went around to all their different research projects that they’d been working on for other projects, and collected their software technology and put it all together.

First, there’s the speech recognition. Microsoft has been working on their own speech recognition engine and API for a long time. You can use the same speech recognition engine as Natal for free in gaming right now if you want. Download GlovePIE, and train speech recognition in the Speech control panel. If you only have Windows XP, you will need to first either install the speech recognition from Microsoft Office (best way) or download and install the SAPI 5 SDK with speech recognition.

Then there’s facial recognition. You might have seen this in other Microsoft products, such as Windows Live Photo Gallery, which you can download here for free, and start your computer automatically recognising all the faces in your photos.

Peter Molyneux also mentions handwriting recognition, although we haven’t seen it used yet. But I’m guessing we will in some games.

And there was no doubt a lot of other code.

Then there’s a huge amount of new software which Microsoft had to write. It has to find the shapes of people, and from there convert the surface data into 48 skeletal points for each player. It can do that for 4 people at once, 30 times per second. It can even identify individual fingers if they are close enough.

The need for all that software is why Project Natal could only have been made fully by Microsoft, not Nintendo, Sony, or Sega.

First they showed a video that they had prepared earlier, featuring a happy family playing on their XBox 360, using only their bodies, voices, and a real skateboard:

First it demonstrated recognising when a user entered the room, and which user it was. This is a confirmed feature that Project Natal will be able to do.

And it demonstrates a virtual character maintaining eye contact with the user as they walk around in front of the screen. That is also quite easy for Natal to do.

And it demonstrates the virtual character addressing the user by name in a sentence full of emotion. Which means either Microsoft has made a massive leap in speech synthesis technology; or more likely, that game makers could get their voice actors to record many of the most common names in various contexts, so users can feel personally involved. I’d like to see this more in games, even if some users with uncommon names miss out.

Then it demonstrates the virtual character responding to spoken dialog like “Let’s do it.” with an appropriate response and doing what it’s told.

Then it demonstrates how Natal could be used for 1-to-1 fighting games, where real punches, kicks, and blocks are used, and the avatar follows all the users moves perfectly. I can’t wait to try that kind of game.

Next came car racing with tracked hands steering in the air, and changing gears. I can’t see how they are controlling acceleration and braking, but Natal could do that by tracking your feet. It allowed multiplayer and didn’t get confused by others’ hands. And it allowed actual pit-work with 1-to-1 motions. That’s pretty impressive. But I suspect gamers would rather have a split-screen multiplayer racing option.

As we’d already seen, there was tracking of all your limbs in a rampage-like game. But this time they went for a mirror view rather than looking at the back of a transparent avatar. The impressive part is the roaring and breathing fire. I don’t know if that was done with the microphone or the 3D camera, but it looked cool.

Then they showed soccer with full body tracking. I can see that being popular, since sports games have always been successful with innovative but realistic controls. I’ve never been able to get into sports games much, but it was fun on the Wii.

I really like the part when the XBox 360 scans the appearance of the skateboard, and uses it as an in-game object. Users like being able to customise their game characters, but consoles have never before had an easy enough way to draw or send designs to the program. I doubt Project Natal can really project a flash of visible light onto it, but the video could be showing us the camera view as it’s displayed on the TV screen.

And the hand gestures to control the menu are good, but could get annoying if you have to go through a long list one item at a time. In real life when you spin something it has momentum, and spinning it harder makes it go faster. Unfortunately there is no haptic feedback, so there is no easy way to measure how hard someone is pushing, only how fast they are pushing.

Skateboarding games could work well, without the need of a Balance board like on the Nintendo Wii.

Then it goes back to facial recognition like at the start of the clip, to log into instant messaging, and voice commands to call someone. Who surprisingly answers the call instantly without complaining about being interrupted in the middle of something.

The video phone system looks good. But I can’t help noticing the video of your friend is in 2D. Project Natal has a 3D camera. So it wouldn’t be hard to have the friend extending out in front of the frame, especially when she does the hand gestures. It would look better on a stereoscopic 3D TV, which I really hope XBox 360 will be supporting one day, with or without Project Natal.

Since Project Natal can make a 3D scan of your whole body and use it as an avatar, the dressing up your avatar while online shopping could be useful. You could virtually try it on, and move around in it. But that would require significant effort on the part of clothes shops or manufacturers. Perhaps it would just be used for downloading clothes for your avatars.

The quiz game looks interesting, if it can do the hand tracking accurately enough to tell who pressed their “buzzer first”. But the video gets one thing wrong. The host is trying to draw out suspense with “Millard Filmore is… ” before saying “correct”, and the players are worried. But the fact that the game show host recognised the name and was able to say it back shows that the game makers programmed it in, which they wouldn’t have done for a random wrong name.

It also shows online play against another family.

And last of all, it shows a media player controlled by gestures and voice commands.