Archive for January, 2010

Project Natal at CES 2010

Project Natal’s most recent appearance was at CES 2010 earlier this month.

At CES, Microsoft gave a really boring introduction, then showed a new video called “Innovation Journey”.

This is my favourite video, since it actually explains how Project Natal works, shows what Project Natal’s depth sensor is actually seeing, and talks a bit to some of the people behind it. And of course it tells us the release date!

Note in particular the scenes where it shows on the screen a 3D view of what the depth sensor is seeing, with the different object types colour-coded. Notice how there is a black hole in the background where the camera can’t see what’s there because it’s blocked by you standing in front of it. But aside from that, it has a very good 3D model of the scene. You can see how easy it would be to recognise different kinds of objects from this sort of information, even without the colour data that you also get. Much more powerful than just a camera like eye-toy.

We also get to see a (blurry) close-up of the back of the project natal hardware.

At CES 2010, Microsoft also talked about Xbox Live, and their own version of Nintendo’s Virtual Console. The Xbox is getting a virtual arcade called Games Room, where your avatar can walk into a 3D arcade and see and hear all their purchased arcade games in their 3D cabinets in attract mode before you play them. You can also invite friends and play against them online. They have 30 different arcade games that you can buy.

And they talked about how people will be able to rent and watch videos online quickly, using Xbox Live.

Other Japanese developers supporting Natal

Namco Bandai weren’t in the panel at TGS 2009, but that doesn’t mean they weren’t at TGS to support Project Natal.

We already saw one Namco Bandai game for the Project Natal demonstrated live for many people to try: Beautiful Katamari.

But Namco Bandai are better known for their fighting games: Tekken, SoulBlade/SoulEdge/SoulCalibur, Naruto: Ultimate Ninja, Dragon Ball Z, etc.
They’ve also produced a few baseball games over the years, and many, many other kinds of games.

video link

Yozo Sakagami
General Manager, Planning Unit 2
Planning Division, Contents Production Group
NAMCO BANDAI Games Inc.

“You don’t need anything but your hands. Like in Science Fiction films, where people control holographic interfaces by their hands. Being able to do this in reality is very futuristic and exciting. We have worked with motion capturing in fighting games for quite some time. Realistic movement has been one of our strongest interests.

“We were all excited, like kids with a new toy. Everyone on the team was coming up with new ideas. We talked about things like how a ball was thrown in a baseball game. We were all quite excited. In fact, it’d been a while since we had so much fun. We hadn’t encountered anything as innovative.

“On the other hand, the experience itself that people enjoy, really hasn’t changed. So it may not be about creating something that is completely new and different, but rather to deliver an experience that everyone could imagine and relate to.

“Motions and gestures are very real in form, and they are also very personal features. It’s not only about the looks, but how a person moves is also very unique. I think being able to integrate this into a game could be a big step forward.”

Tecmo are famous for their Dead Or Alive fighting games. And also for Ninja Gaiden, Fatal Frame/Project Zero, Monster Rancher, Gallop Racer, and others.
Since this interview, Tecmo has merged with Koei, so hopefully that brings Koei’s games on-board too.

video

Keisuke Kikuchi
Executive Producer
Tecmo

We’ve put a lot of thought, but there’s others who do more. That’s why we keep putting more thoughts into it. We want to do better than them. Our team has started planning for games to play on Natal and we came up with 72 ideas all together. When we saw these ideas, we started seeing a trend. My first impression was to become something in a game, but as we went on, we started seeing these ideas in categories.

For example, let’s say there is a small wall. In real life, you do a little jump over this wall. But if your character jumps to the 2nd floor of a building, it makes you feel as if you really were jumping that high.

This is more about the physical experience, but the experience as a whole. I am thinking about a content where you interact physically, and enjoy a more intimate experience within the game.

From Software were also interviewed. They are the developers of a couple of my favourite games: Tenchu and Armored Core.

I can just imagine playing Tenchu and accidentally making a sound in the real world and having the guards you are trying to sneak past hear you and try to catch you. I did find some of the gestures in the Wii version of Tenchu disturbing though, such as having to mime breaking someone’s neck.

video

Masanori Takeuchi
Managing Director / Game Producer
FromSoftware, Inc.

“I think Natal is beyond the concept of any existing game. With Natal, I thought that we’d be able to do something new on Xbox 360 again. I honestly thought it was going to be a lot of fun.

I think Natal is more about enhancing the experience. If evolution would leave behind our current fans, I want to be careful not to do so. I hope to expand the current gaming experience, rather than to evolve gaming into something irrelevant to current gamers.

Things like movement and facial expressions could not be reflected into the game through a handheld controller. But with Natal, this becomes possible.

I believe Natal will become very interesting, when combined with games that work with the players’ emotions.”

Kojima are the developers of Metal Gear Solid. But they also made Boktai. Boktai was a revolutionary gameboy advance game that uses a solar sensor as an input device, along with a clock. The game character had a solar gun that he used to shoot vampires, but the gun only charged when the player was outdoors in the sunlight. The stronger the sunlight, the faster your gun charged. Sunlight also affected other things in the game, such as slowly rotting fruit and nuts in your inventory. Sometimes in the game darkness is preferable. Time also affected things in game. That gives you an idea of where they are coming from.

They also made the game PoliceNauts for some ancient consoles, which was unusual in that it supported various console peripherals such as Mice and Lightguns.

video

Kenichiro Imaizumi
Senior Producer
Kojima Productions

During our previous production, we tried to build our own controller. We ran a lot of experiments. We tried running electric shockwaves, install a pulse reader, we also tried to build a controller that’s receptive to the strength of the grip. We tried a lot, even a transforming controller. Kojima came up with these ideas, and we worked directly with the manufacturer.

But all the way along, we thought we had to have a controller. And now Natal tells us that we don’t need a controller. We can use our movements, our voice, or the recognition of colours. That’s how we interact with games. It was quite revolutionary to us, we really never thought of it that way. We had imagined it before, but we never thought it would come so quickly. So it took us by surprise.

What we will create on Natal would be something that has no limitations. Ideas from outside the box. I have a strong feeling that Natal will allow us to create things that we really want to create.

More fighting and zombie games for Project Natal

Capcom’s Head of R&D, and Managing Corporate Officer, Keiji Inafune was interviewed at TGS 2009 too.

Capcom are the creators of Street Fighter, Resident Evil, Dead Rising, Devil May Cry, Bionic Commando, and Lost Planet, among many other games.

I’m sure you can imagine the potential of some of these games on Project Natal, and let’s hope Capcom can too. He doesn’t say much useful, other than that Natal will allow them to have more complex and detailed game actions by not being limited by the number of buttons on the controller. I’m not sure how he’s going to make videos that are interactive like he says at the end. I’m imagining a cross between say the cutscenes in Resident Evil 5, and a simplified Project Milo, would make for interesting and immersive story telling between chapters in any type of game.

Japanese talking head videos are getting a bit boring, but here’s the video link: http://www.youtube.com/watch?v=tU9TWrikZBI

And here’s the transcript:

“I believe many creators feel suppressed or even bored with the current trend of game development. But at the same time it’s a challenge for us to produce something beyond what we are so used to doing. I am certain that Natal will bring on that challenge. This doesn’t mean that it’s being replaced, but rather, it’s being enhanced. And by it being enhanced, it allows us to do more.

Take for instance, the number of buttons on an average controller. We were working on a very limited interface. But if we could create games without limits, it allows us to make it more interesting, for example, the physical experience. Its not something that reacts only to your fingertips. You can use your fingers too, but it doesn’t limit you to them. Being able to use a variety of things is what makes it so ground-breaking and innovative.

In that sense, by mixing in videos for instance, and a variety of other interactive elements may be the key to creating new gaming experience.”

Sega interested in Project Natal voice controls

Sega’s R&D Creative officer, Toshihiro Nagoshi, was also interviewed at TGS 2009 (Tokyo Game Show).

He was interested in both speech recognition and facial expression recognition, or perhaps tone of voice recognition. So I hope that means we’re going to have a lot of Sega games we can talk to, or games where we can order people around. Or maybe we will get games that sense when you are frustrated and start swearing at the game, so the game will then make it easier or give you hints.

“Like when gaming changed from 2D to 3D the genres of gaming had also been renewed, if this new system leads to renewing the current format, I think it’s a very positive change.

“But it wouldn’t be interesting if it were just a rereun of what we already have, so I hope that it becomes something new that would move gamers.

“If it’s not the game that tells you what to do, but instead if it’s your emotions that decide how the game proceeds, I think that may be the ultimate form.

“The most innovative side to it is that its controller free, and that it focuses on voice recognition. When you play games, you’re basically saying something, whether it be pleasant or disturbing. Voices do not affect game play today [Carl: in some games they do], but I think it may be the last input source left. With or without the controller, you’re still using your hands. So I guess it’s our voice that we’ve never really used before, and I see a lot of potential in that field.”

DDR and Guitar Hero Natal Now With Stagediving!

On September 25, Naoki Maeda gave an interview at TGS 2009. He is the Producer at Bemani Production of Konami Digital Entertainment Co.

Bemani is of course the company that brought you Dance Dance Revolution, Dancing Stage, BeatMania, Dance Maniax, Pop’n Music, Pop’n Stage, GuitarFreaks, DrumMania, Rock Revolution, Keyboardmania, ParaParaParadise, Ubeat, Bemani Pocket, and Pop’n Rhythm.

They invented the whole genre of rhythm and dancing games, and had games like Guitar Hero and DJ Hero long before Guitar Hero itself.

These games are such an obvious usage for Project Natal, so it’s not surprising Bemani is very keen on adding Project Natal support. You wouldn’t need to buy a dance mat anymore, which makes games much more affordable and gives them a much wider audience and profit margin. I don’t know if we’re going to have air-guitar games. But there’s no reason it can’t use normal guitar peripherals while using Natal for things like starpower, avatar control, avatar creation, etc.

But Naoki Maeda wants to go a lot further than that… (transcript after the video).

“I feel we could try a lot on Natal. My specialty is games that work with music.

My instincts tell me that Natal would be ideal for dancing. Any form of dancing could be emulated on Natal. [starts dancing] … I really can’t dance, obviously.

The fundamentals of music are about performing. Well, dancing too.

For example, when you’re on stage, the audience reacts to your performance. This is where Natal may prove to have a lot of potential. You’re up on the stage, and you see the audience. You might want to call out to them. And then the crowd also calls back to you. You throw something at them, and somebody in the crowd catches it. The responses from the audience are what makes it so exciting for the performer. You’re on stage, performing to your audience, and they react to your performance.

I’d like to create a more realistic music entertainment with Natal for my fans.”

Japanese Game Developers Back Project Natal

With any luck the Xbox 360 should be getting lots of new games supporting Project Natal. Which is good, because most existing Xbox 360 games probably aren’t going to have Project Natal support added.

At TGS 2009 (that’s the Tokyo Game Show), Microsoft featured a panel with Japanese game developers. And they announced that most of the big game development companies are keen on supporting Project Natal.

The following game companies have been confirmed as currently developing for Project Natal: Activision Blizzard, Bethesda Softworks, CAPCOM, Disney Interactive, Electronic Arts, Konami, MTV Games, Namco Bandai, Sega, Square Enix, THQ Inc., and Ubisoft.

Those companies make up 70% of the third party games sold for recent consoles. So don’t fear that we won’t get any games for Natal.

As mentioned in the video above, representatives from Konami, Capcom and Sega (left to right) gave a special panel presentation at TGS. Here’s a humorous snippet:

Johnny Lee

Johnny Chung Lee is perhaps the most famous and coolest among the Wii Remote (or Wiimote) hackers. – Man, that makes me jealous. I could so nearly have been him. But compared to him, I’m only the Radek Zelenka of Wiimote hacking.

My favourite Wiimote project of his is this one:

You don’t actually have to have the Sensor Bar on your head and the Wiimote on your TV. Doing it that way you are actually throwing away valuable information about head rotation (only needed for stereoscopic 3D with this kind of VR) and giving yourself a smaller range that you can move around in (since the Wiimote camera has a lower FOV than the sensor bar does, and if the Wiimote was on your head it would turn to face the screen when you do). The reason why Johnny Lee does it the backwards way, is because the Wiimote is heavier, more awkward and hard to mount on your head.

This isn’t a new invention, it’s actually based on the original super-sized version by Carolina Cruz (no, not the Colombian model):

The virtual objects that appear to be attached to the woman’s wand controller are really drawn on the walls and floor several feet away. So are the laser beams you see coming out of her wand. And the illusion works perfectly when we move the camera, because the camera and the wand are very precisely tracked in 3D space and the images drawn on the wall are drawn exactly how they would look from that angle if they were really attached to the wand. There are 3 walls and a floor all at right angles, but you can hardly see the seams because it compensates for them being at right angles by drawing each individual screen with its own head-tracking algorithm. This is exactly the same as Johnny Lee is doing, but with several huge screens. This kind of Virtual Reality is called a CAVE, and costs several million dollars. It’s a lot more fun than a head-mounted display.

I actually implemented this same head tracking Virtual Reality on the computer long before Johnny Lee famously showed it. But I used an (even cheaper) Essential Reality P5 Glove on my head since Wii Remotes didn’t exist yet, and I used a monitor and a TV at right angles, and I added red/cyan 3D glasses. I made it so that it could be used in existing Direct3D games, and I played it with Hitman 2 (because it comes free with the glove) and Lord Of The Rings: Return Of The King. It was very cool. But I never finished it enough to release it, and never made a video. Maybe I will one day.

But why am I talking about Johnny Lee, Wii Remotes, and Virtual Reality gloves when this isn’t my GlovePIE blog?

Because Johnny Lee was recruited onto the Xbox: Project Natal team! And this kind of head-tracked virtual reality is coming to Project Natal. The Milo and Kate game is known to include this feature, and hopefully all the other games do to.

Johnny Lee’s blog post is here: http://procrastineering.blogspot.com/2009/06/project-natal.html

Science Behind Project Natal

Scientific American just published an article on the Science of Project Natal. Unfortunately, the article is a bit light on actual science, as scientific thinking seems to be dying a slow death in the West:

http://www.scientificamerican.com/article.cfm?id=microsoft-project-natal

It talks about how Microsoft had to train their software to recognise where people’s joints are from video frames.

Project Natal at Tokyo Game Show 2009 (TGS)

Back in September 2009, the Xbox 360’s Project Natal made another public appearance, this time at the Tokyo Game Show.

Here we got to see new Project Natal controls for Beautiful Katamari:

And new Project Natal controls for Space Invaders Extreme (see the end of this portuguese video), plus more Ricochet and Burnout Paradise (car racing):

How Project Natal Works

Project Natal has two input devices:

First is a 3D camera, which can measure both the colour and the distance of every pixel.

Second is a microphone array, which is several directional microphones arranged in a pattern. They can separate out different sound sources and determine where each sound came from, and can thus filter out any noise.

The 3D camera is actually made out of two cameras, one that senses distance using infra-red and one normal one that senses colour. But it helps to think of it as a single camera that can also measure depth.

The 3D camera makes Project Natal much more powerful than a 2D camera like the Eye-toy. It makes it dead easy to filter out background objects, and find only the objects that you are looking for. And it doesn’t need to rely on colours to recognise things, it can just look at their 3D shape.

It could operate in complete darkness and still see the 3D shape of everything, but knowing the colours would make it easier to recognise hands and faces and other things.

The 3D camera would also allow it to easily scan in any 3D object and convert it into a virtual object in the game. It has only been demonstrated with the 2D image on a skateboard, but there is no reason why Natal couldn’t scan in any object you showed it, if you showed it from a few different angles.

One downside of the 3D camera is that it relies on line-of-sight and can’t see behind things. So it can’t track your hand when it goes behind your back, or when someone stands in front of you. But a Wii Remote, or magnetic 6DOF tracker could still track it.

Another downside is that it could get confused about the distance of transparent or reflective objects, since the IR light that goes back might have first come from somewhere further away.

The microphone array is important for speech recognition, since it can filter out noise, and could separately recognise several speakers talking at once. Especially since the TV will also be making noise. It could also be used to have virtual characters still look at you when you go out of sight of the camera and still talk to it. And in a multiplayer game it helps it know which player is speaking by knowing which sound came from which body that it can see.

It doesn’t just recognise words though. In games like Milo and Kate it can recognise the emotion of the speaker, and can tell things like whether they are telling a joke. It could also be used for rhythm games by using anything that makes a sound, or by clapping. It’s not known whether the monster game used the microphone or facial expression tracking to control breathing fire. It would have to be a good quality microphone, because normally speech recognition requires a microphone a few cm from your mouth.

But the most important part of how Project Natal works is the software. Microsoft, with some help from Peter Molyneux, went around to all their different research projects that they’d been working on for other projects, and collected their software technology and put it all together.

First, there’s the speech recognition. Microsoft has been working on their own speech recognition engine and API for a long time. You can use the same speech recognition engine as Natal for free in gaming right now if you want. Download GlovePIE, and train speech recognition in the Speech control panel. If you only have Windows XP, you will need to first either install the speech recognition from Microsoft Office (best way) or download and install the SAPI 5 SDK with speech recognition.

Then there’s facial recognition. You might have seen this in other Microsoft products, such as Windows Live Photo Gallery, which you can download here for free, and start your computer automatically recognising all the faces in your photos.

Peter Molyneux also mentions handwriting recognition, although we haven’t seen it used yet. But I’m guessing we will in some games.

And there was no doubt a lot of other code.

Then there’s a huge amount of new software which Microsoft had to write. It has to find the shapes of people, and from there convert the surface data into 48 skeletal points for each player. It can do that for 4 people at once, 30 times per second. It can even identify individual fingers if they are close enough.

The need for all that software is why Project Natal could only have been made fully by Microsoft, not Nintendo, Sony, or Sega.