That video begs the question… why does it show an orange blob (from the 3D camera) with no texture, when Kinect can also see the colour of each pixel like a normal camera? I don’t know the answer, but I suddenly had a disturbing thought… Perhaps the USB data rate is limiting the pixel data that Kinect sends back. If you’ve ever wondered why all webcams are low resolution (but lie and say they’re a much higher resolution), it’s because USB 2 can’t even handle 640×480 at 60Hz in 24-bit colour. So I’m wondering if Kinect really can send all the depth and colour data at the same time. We haven’t seen as many Augmented Reality applications for Kinect as one might expect. On the other hand, perhaps UbiSoft are just lazy. Milo and Kate did use colour and depth data at the same time when Claire was splashing in the pond, although that was an older version of the hardware.

Interview with the producer, Claudine Cezac:

For comparison, here’s the Wii Trailer from last year’s E3 2009:

First it shows Kinectimals which is the Kinect (Project Natal) equivalent of Sony’s PlayStation Move game EyePet. But it’s been super-sized. You get a life-size tiger, lion, or other big cat to play with. And it doesn’t appear to have augmented reality.

Then it shows the latest version of Ricochet, the game where your avatar deflects balls back to destroy blocks. Microsoft have been showing this game almost exclusively for the past year.

Then it shows Kinect Adventures, which is the river rafting game for two players.

Then it shows the media player with voice controls which was in the video last year. They were watching Harry Potter. I wonder if there will be Harry Potter games for Kinect, since they would be an obvious choice for gesture and voice control. Lego Harry Potter (Parts 1 and 2) and Harry Potter and the Deathly Hallows (Parts 1 and 2) are still to be released, so keep your fingers crossed. There are two franchises players have been dreaming of Kinect controls for since long before Project Natal: Star Wars and Harry Potter.

Then it shows a new dancing game, called “Dance Central”. Unlike the standard Dance Dance Revolution/Guitar Hero rhythm and timing games, this game doesn’t have scrolling notes that you have to hit in time with the music. Instead it has a wheel on the side of the screen that tells you what dance move you have to do and what the next ones will be, and you have to spend a few seconds doing that move correctly. And of course this game requires no extra hardware, due to Kinect.

Then it shows two Kinect Sports games hurdles and table tennis. Sony’s PlayStation Move also does table tennis, as does Nintendo’s Wii Motion Plus. Microsoft’s version seems the worst since you are using your open hand instead of a table tennis bat. If I was Microsoft, I would let players hold any random prop they have lying around the house and use it as a table tennis bat. A spatula, a carrot, your TV remote control, even a real table tennis bat, or a Wii Remote. I don’t see why Kinect can’t use random household objects as props.

Then it shows Your Shape: Fitness Evolved. You might remember this from the Nintendo Wii, it came with a Wii webcam peripheral. Now it doesn’t need one. This was an obvious choice for Kinect, and I’m surprised it never occurred to me that we would see this game for Kinect.

Then it shows Kinect Joy Ride. This is actually an old game that was going to be released last year for free on Xbox Live Arcade with traditional controls, but it was delayed and made into a Kinect launch title. It is a cartoony car racing game with obstacles that includes split-screen multiplayer.

Lots more games are being announced though, so stay tuned for a low more info and videos.

It is Sony’s version of Project Natal. The differences are: PlayStation Move uses a 2D camera (PlayStation Eye) instead of a 3D camera like Natal, PlayStation Move doesn’t come with software for tracking your whole body (just your hands), and PlayStation Move uses a pair of wand controllers like the Wii remote and Nunchuk.

Sony demonstrated augmented reality with their Motion Controller a lot more than Microsoft did. But augmented reality is actually where Natal has the advantage once everyone has stereoscopic 3D TVs. The PlayStation 3 will be getting stereoscopic 3D support later this year, but without a 3D camera Sony’s augmented reality is going to be strictly 2D, unlike Project Natal’s potential for stereoscopic 3D augmented reality.

Sony haven’t demonstrated voice recognition for PlayStation Move, although it has an array microphone the same as Natal. Sony previously demonstrated speech recognition in their SingStar game though, and claimed it was in their SDK.

Sony have demonstrated a lot of features that would be very useful in existing games and would be easy to add to existing games. Which is something Microsoft hasn’t been good at yet. For example, they’ve shown using it as a pointer, using it in RTS games, FPS games, sports games, etc. Any existing game could be ported from PC or Wii to PlayStation Move in an obvious way with very little effort. And any PlayStation game could be ported to PlayStation Move with very little effort, except for a few missing buttons. But Project Natal hasn’t shown how it will be used as a pointer, or how it will be used in RTS and FPS games, although I’m imagining lots of very cool ways Natal could do that. So I’m expecting PlayStation Move will have more game support, but those games will be less revolutionary than Natal games.

Anyway, see my PlayStation Move blog.

Stereoscopic 3D means each eye sees a separate image because it is seeing the scene from a slightly different angle. Your brain automatically looks at the difference between the two eyes and calculates depth from it. That’s why people have two eyes.

Anything that’s at the same screen position for both eyes will look like it’s at the same distance as the screen is. Which is what normal TV looks like. But anything that’s drawn further to the left on your right eye, and further to the right on your left eye, looks like it’s popping out in front of the screen. On the other hand, anything that’s drawn further to the left on the left eye, and further to the right on the right eye looks like it’s behind the screen. Your eyes are very sensitive, and even one pixel difference is enough for you to see the difference in depth.

By now you should have seen that effect at a stereoscopic 3D movie. If not, go and watch Avatar 3D. It is really, really cool. You feel like you can reach out and grab things that are in front of the screen. And just as good is feeling like there is a whole huge world stretching far out into the distance behind the screen.

atokirina': Avatar 3D seeds

Of course in Avatar when you reach out and touch the floating seeds that seem just in front of you, you can’t actually grab them and hold them and move them around in your hand. You’re probably thinking: “Well, duh! Of course not!”. But with Project Natal you can actually do that!

You see, project Natal already knows exactly where your hand is in 3D space. And it knows exactly where your eyes are in 3D space. So if you tell Project Natal exactly where your TV is in 3D space, by telling it what size TV you have and where you put the Natal sensor bar, then it can very easily calculate the line from your eye to your hand to the point on the screen that you are grabbing. So it knows which part of the scene you are touching.

More importantly, it knows exactly where to draw something so it looks like you are holding it in your hand! Imagine that 3D in front of the screen effect in Avatar but with a sword, gun, baseball bat, or even a weighted companion cube, that is drawn at the exact location of your hand and follows your hand precisely whenever you move or rotate your hand, and compensates for moving your head. It would look exactly like you were holding the virtual object in your real hand right in front of your eyes. You wouldn’t have to mime anything, because you can really see and hold the virtual object in your hand.

So, what’s the catch? Well, it only works when your hand is in front of the TV. When the 3D object looks like it should cross in front of the edge of the TV, the edge of the TV actually goes in front of it instead, and half the object disappears and it spoils the illusion. So you need a big enough TV, and you need to be close enough to the TV.

And there’s the catch that the hilt of a sword or the handle of a gun that is supposed to be drawn in front of your hand can’t actually be drawn in front of your hand because it’s drawn on the screen and your hand is in the way. The rest of the gun or sword would look right, but not the part that should be covering your hand but instead your hand seems to be covering.

The other catch is that the image on the TV is a bit out of focus and blurry when you look at your hand. Even if the image is in stereoscopic 3D and looks like it is right next to your hand, either the image or your hand will be out of focus because in reality they are at completely different depths. That would be OK, except that it is actually hard for your eyes to focus on one depth while they are converging (aiming) at a different depth. That makes it hurt to look at 3D that is too far in front of the screen. So you need to be reasonably close to the screen, and have your hand a reasonable distance away from your eyes. Or you just need to not look directly at the object in your hand and focus more on the rest of the scene.

Another catch is that there is lag. You would move your hand, and 100 ms later the thing you are holding will move. The same with moving your head.

I still think it would be awesome though. Especially with a big screen.

Ricochet's avatar (from Scientific American)

Think about the Project Natal game Ricochet. You don’t actually hit balls with your body. Your avatar on the screen, in it’s own virtual world, copies your movements and hits the virtual balls. Meanwhile you are outside in the real world and balls never come out of the screen towards you. But with Stereoscopic 3D, the real world and the virtual world can share the same space out in front of your TV in your living room with you. So the balls would come all the way out to your real hand, and you can hit them with your real hand, or catch them with your real hands, and even hold them and move them around in your hands. The same with fighting games, or sports games, or shooting games. Wouldn’t it be cool to have to physically duck projectiles that are really coming out of the screen at you.

Most project Natal games that we have seen have an avatar interacting with things on your behalf, instead of you interacting with them. Except for Milo and Kate, Burnout Paradise (the racing game), the quiz game, and the first part of the fighting game when you talk to the opponent. Wouldn’t Milo and Kate be better though if the thrown goggles really did seem to come out of the screen? And if you really could catch the goggles? And if you could see the goggles in your hands? And if you could really put them on? Obviously you wouldn’t be able to feel them, but you could see the goggles in your hands and responding to how you move them.

To some extent the things that I’ve said above can also be done without stereoscopic 3D, and only using the Johnny Lee head-tracking 3D effect, but it wouldn’t be quite as realistic (unless you only have one eye).

With stereoscopic 3D support, Natal would need a new slogan:
“No avatar required. You are the avatar.”

For those of you who don’t read the comments, Ryan Burke informed me about an Augmented Reality version of Façade that they made at Georgia Tech in 2006. They recreated Grace and Trip’s apartment in real life, and you walk around it with an eMagin Z800 Virtual Reality visor and camera on your head and can see Grace and Trip superimposed on the real world. You can thus walk around naturally and talk to them naturally. It does Voice Recognition the cheating way, by having a human type what you say into a computer. You can even pick up the objects around the apartment and have Grace and Trip react to it. Apparently it is very immersive.

So let’s hope Michael Mateas and Andrew Stern get together with Microsoft and make Façade for Xbox 360’s Project Natal.

At CES, Microsoft gave a really boring introduction, then showed a new video called “Innovation Journey”.

This is my favourite video, since it actually explains how Project Natal works, shows what Project Natal’s depth sensor is actually seeing, and talks a bit to some of the people behind it. And of course it tells us the release date!

Note in particular the scenes where it shows on the screen a 3D view of what the depth sensor is seeing, with the different object types colour-coded. Notice how there is a black hole in the background where the camera can’t see what’s there because it’s blocked by you standing in front of it. But aside from that, it has a very good 3D model of the scene. You can see how easy it would be to recognise different kinds of objects from this sort of information, even without the colour data that you also get. Much more powerful than just a camera like eye-toy.

We also get to see a (blurry) close-up of the back of the project natal hardware.

At CES 2010, Microsoft also talked about Xbox Live, and their own version of Nintendo’s Virtual Console. The Xbox is getting a virtual arcade called Games Room, where your avatar can walk into a 3D arcade and see and hear all their purchased arcade games in their 3D cabinets in attract mode before you play them. You can also invite friends and play against them online. They have 30 different arcade games that you can buy.

And they talked about how people will be able to rent and watch videos online quickly, using Xbox Live.

My favourite Wiimote project of his is this one:

You don’t actually have to have the Sensor Bar on your head and the Wiimote on your TV. Doing it that way you are actually throwing away valuable information about head rotation (only needed for stereoscopic 3D with this kind of VR) and giving yourself a smaller range that you can move around in (since the Wiimote camera has a lower FOV than the sensor bar does, and if the Wiimote was on your head it would turn to face the screen when you do). The reason why Johnny Lee does it the backwards way, is because the Wiimote is heavier, more awkward and hard to mount on your head.

This isn’t a new invention, it’s actually based on the original super-sized version by Carolina Cruz (no, not the Colombian model):

The virtual objects that appear to be attached to the woman’s wand controller are really drawn on the walls and floor several feet away. So are the laser beams you see coming out of her wand. And the illusion works perfectly when we move the camera, because the camera and the wand are very precisely tracked in 3D space and the images drawn on the wall are drawn exactly how they would look from that angle if they were really attached to the wand. There are 3 walls and a floor all at right angles, but you can hardly see the seams because it compensates for them being at right angles by drawing each individual screen with its own head-tracking algorithm. This is exactly the same as Johnny Lee is doing, but with several huge screens. This kind of Virtual Reality is called a CAVE, and costs several million dollars. It’s a lot more fun than a head-mounted display.

I actually implemented this same head tracking Virtual Reality on the computer long before Johnny Lee famously showed it. But I used an (even cheaper) Essential Reality P5 Glove on my head since Wii Remotes didn’t exist yet, and I used a monitor and a TV at right angles, and I added red/cyan 3D glasses. I made it so that it could be used in existing Direct3D games, and I played it with Hitman 2 (because it comes free with the glove) and Lord Of The Rings: Return Of The King. It was very cool. But I never finished it enough to release it, and never made a video. Maybe I will one day.

But why am I talking about Johnny Lee, Wii Remotes, and Virtual Reality gloves when this isn’t my GlovePIE blog?

Because Johnny Lee was recruited onto the Xbox: Project Natal team! And this kind of head-tracked virtual reality is coming to Project Natal. The Milo and Kate game is known to include this feature, and hopefully all the other games do to.

Johnny Lee’s blog post is here: http://procrastineering.blogspot.com/2009/06/project-natal.html

It features facial recognition, tone-of-voice recognition, some simple speech recognition, recognition of the colour of your clothes, head-tracked perspective, gesture recognition, augmented reality with your real reflection showing in the rippling lake along with virtual objects, and recognition of when you are holding up a sheet of paper to the camera. While it can see and recognise what’s on the piece of paper, and insert the paper into the virtual world, it has been confirmed that this XBox 360 game never actually does that properly, because they were worried about people drawing rude things.

One thing you might notice in the video… You can’t see Claire’s goggles that she put on, even though you can see her reflection in the water. I think the augmented reality would be better if they added in the goggles, which is perfectly possible to do with Project Natal.

“Milo and Kate” is a real game, with real objectives.

Here is an interview Peter Molyneux gave at E3 after the video was shown:

In other interviews, Peter Molyneux revealed more information. This game will automatically download updated dialogue from the internet: “For example, Milo will download dialogue very regularly behind the scenes which updates his database with current events. So, if Britain’s Got Talent is on, he’ll comment on certain acts, so that makes him feel so relevant and real that you start believing it, even if you’re a sceptic.”

He explained how it creates the illusion you’re talking to Milo: “We’re building up a profile of the emotion in your voice so we know when your voice is strained or excited or calm or passive. That coupled with Milo’s vocabulary of words he understands and what he’s doing on screen creates this illusion…”

“Milo can recognise the emotions on your face and the emotions in your voice. He can recognise certain words you say. You can have conversations with him, you can read stories to him. We’re trying to bring all these things together. Some of them are tricks – I’ll be absolutely honest with you – to make you believe Milo’s real.

He can recognise what you’re wearing. If he notices you’ve got dark bags under your eyes he will say, ‘You look tired today.’”

But there’s a catch: “The interesting thing is you can only talk to him when the Talk icon appears at the bottom of the screen. That’s when he’s listening to you; the rest of the time, he’s not. He’s listening to you because there’s a context in which you can talk to him.”

And he described the gameplay: “You can buy stuff for Milo’s world, like a bicycle or a trampoline. He’ll come back from school one day and say, ‘Oh, Alex’ – Alex is this character at school who always does a bit better than Milo – ‘Alex has got a new bike. When can we get a new bike?’

To get that bike you need to earn money by doing activities. There are three activities you can do, and the amount of time you spend on each activity sculpts your Milo in different ways – so everybody’s Milo will be completely unique to them.

If you do lots of work, your Milo will be very studious. His hair will have a side parting. He’ll be quite worried about his appearance and he won’t like to get dirty. Whereas if you do more of the play stuff with Milo, he’ll be more of a kid who goes out and scratches his knees.

Your character doesn’t have to be a boy, it can be a girl. At the start you can choose whether to be play as Milly or Milo.”

The game is planned to be ready for the release of Project Natal, and at E3 they had already implemented a large percentage of the game content.

Lionsgate Studios is the company that made the game “Black & White” which was practically the only game to support the Essential Reality P5 Virtual Reality Glove.