Stereoscopic 3D TVs and Project Natal
One thing we haven’t heard much from Microsoft about is stereoscopic 3D (aka S3D) support for Xbox 360 and specifically for Project Natal. A couple of games that I can think of support stereoscopic 3D on the Xbox 360: Avatar, and G-Force. And there are lots of new stereoscopic 3D TVs hitting the market now.
Stereoscopic 3D means each eye sees a separate image because it is seeing the scene from a slightly different angle. Your brain automatically looks at the difference between the two eyes and calculates depth from it. That’s why people have two eyes.
Anything that’s at the same screen position for both eyes will look like it’s at the same distance as the screen is. Which is what normal TV looks like. But anything that’s drawn further to the left on your right eye, and further to the right on your left eye, looks like it’s popping out in front of the screen. On the other hand, anything that’s drawn further to the left on the left eye, and further to the right on the right eye looks like it’s behind the screen. Your eyes are very sensitive, and even one pixel difference is enough for you to see the difference in depth.
By now you should have seen that effect at a stereoscopic 3D movie. If not, go and watch Avatar 3D. It is really, really cool. You feel like you can reach out and grab things that are in front of the screen. And just as good is feeling like there is a whole huge world stretching far out into the distance behind the screen.
atokirina': Avatar 3D seeds
Of course in Avatar when you reach out and touch the floating seeds that seem just in front of you, you can’t actually grab them and hold them and move them around in your hand. You’re probably thinking: “Well, duh! Of course not!”. But with Project Natal you can actually do that!
You see, project Natal already knows exactly where your hand is in 3D space. And it knows exactly where your eyes are in 3D space. So if you tell Project Natal exactly where your TV is in 3D space, by telling it what size TV you have and where you put the Natal sensor bar, then it can very easily calculate the line from your eye to your hand to the point on the screen that you are grabbing. So it knows which part of the scene you are touching.
More importantly, it knows exactly where to draw something so it looks like you are holding it in your hand! Imagine that 3D in front of the screen effect in Avatar but with a sword, gun, baseball bat, or even a weighted companion cube, that is drawn at the exact location of your hand and follows your hand precisely whenever you move or rotate your hand, and compensates for moving your head. It would look exactly like you were holding the virtual object in your real hand right in front of your eyes. You wouldn’t have to mime anything, because you can really see and hold the virtual object in your hand.
So, what’s the catch? Well, it only works when your hand is in front of the TV. When the 3D object looks like it should cross in front of the edge of the TV, the edge of the TV actually goes in front of it instead, and half the object disappears and it spoils the illusion. So you need a big enough TV, and you need to be close enough to the TV.
And there’s the catch that the hilt of a sword or the handle of a gun that is supposed to be drawn in front of your hand can’t actually be drawn in front of your hand because it’s drawn on the screen and your hand is in the way. The rest of the gun or sword would look right, but not the part that should be covering your hand but instead your hand seems to be covering.
The other catch is that the image on the TV is a bit out of focus and blurry when you look at your hand. Even if the image is in stereoscopic 3D and looks like it is right next to your hand, either the image or your hand will be out of focus because in reality they are at completely different depths. That would be OK, except that it is actually hard for your eyes to focus on one depth while they are converging (aiming) at a different depth. That makes it hurt to look at 3D that is too far in front of the screen. So you need to be reasonably close to the screen, and have your hand a reasonable distance away from your eyes. Or you just need to not look directly at the object in your hand and focus more on the rest of the scene.
Another catch is that there is lag. You would move your hand, and 100 ms later the thing you are holding will move. The same with moving your head.
I still think it would be awesome though. Especially with a big screen.
Ricochet's avatar (from Scientific American)
Think about the Project Natal game Ricochet. You don’t actually hit balls with your body. Your avatar on the screen, in it’s own virtual world, copies your movements and hits the virtual balls. Meanwhile you are outside in the real world and balls never come out of the screen towards you. But with Stereoscopic 3D, the real world and the virtual world can share the same space out in front of your TV in your living room with you. So the balls would come all the way out to your real hand, and you can hit them with your real hand, or catch them with your real hands, and even hold them and move them around in your hands. The same with fighting games, or sports games, or shooting games. Wouldn’t it be cool to have to physically duck projectiles that are really coming out of the screen at you.
Most project Natal games that we have seen have an avatar interacting with things on your behalf, instead of you interacting with them. Except for Milo and Kate, Burnout Paradise (the racing game), the quiz game, and the first part of the fighting game when you talk to the opponent. Wouldn’t Milo and Kate be better though if the thrown goggles really did seem to come out of the screen? And if you really could catch the goggles? And if you could see the goggles in your hands? And if you could really put them on? Obviously you wouldn’t be able to feel them, but you could see the goggles in your hands and responding to how you move them.
To some extent the things that I’ve said above can also be done without stereoscopic 3D, and only using the Johnny Lee head-tracking 3D effect, but it wouldn’t be quite as realistic (unless you only have one eye).
With stereoscopic 3D support, Natal would need a new slogan:
“No avatar required. You are the avatar.”
Categories: Project Natal Tags: augmented reality, baseball, fighting, hardware, Milo, quiz, racing, Ricochet
Other Japanese developers supporting Natal
Namco Bandai weren’t in the panel at TGS 2009, but that doesn’t mean they weren’t at TGS to support Project Natal.
We already saw one Namco Bandai game for the Project Natal demonstrated live for many people to try: Beautiful Katamari.
But Namco Bandai are better known for their fighting games: Tekken, SoulBlade/SoulEdge/SoulCalibur, Naruto: Ultimate Ninja, Dragon Ball Z, etc.
They’ve also produced a few baseball games over the years, and many, many other kinds of games.
Yozo Sakagami
General Manager, Planning Unit 2
Planning Division, Contents Production Group
NAMCO BANDAI Games Inc.“You don’t need anything but your hands. Like in Science Fiction films, where people control holographic interfaces by their hands. Being able to do this in reality is very futuristic and exciting. We have worked with motion capturing in fighting games for quite some time. Realistic movement has been one of our strongest interests.
“We were all excited, like kids with a new toy. Everyone on the team was coming up with new ideas. We talked about things like how a ball was thrown in a baseball game. We were all quite excited. In fact, it’d been a while since we had so much fun. We hadn’t encountered anything as innovative.
“On the other hand, the experience itself that people enjoy, really hasn’t changed. So it may not be about creating something that is completely new and different, but rather to deliver an experience that everyone could imagine and relate to.
“Motions and gestures are very real in form, and they are also very personal features. It’s not only about the looks, but how a person moves is also very unique. I think being able to integrate this into a game could be a big step forward.”
Tecmo are famous for their Dead Or Alive fighting games. And also for Ninja Gaiden, Fatal Frame/Project Zero, Monster Rancher, Gallop Racer, and others.
Since this interview, Tecmo has merged with Koei, so hopefully that brings Koei’s games on-board too.
Keisuke Kikuchi
Executive Producer
TecmoWe’ve put a lot of thought, but there’s others who do more. That’s why we keep putting more thoughts into it. We want to do better than them. Our team has started planning for games to play on Natal and we came up with 72 ideas all together. When we saw these ideas, we started seeing a trend. My first impression was to become something in a game, but as we went on, we started seeing these ideas in categories.
For example, let’s say there is a small wall. In real life, you do a little jump over this wall. But if your character jumps to the 2nd floor of a building, it makes you feel as if you really were jumping that high.
This is more about the physical experience, but the experience as a whole. I am thinking about a content where you interact physically, and enjoy a more intimate experience within the game.
From Software were also interviewed. They are the developers of a couple of my favourite games: Tenchu and Armored Core.
I can just imagine playing Tenchu and accidentally making a sound in the real world and having the guards you are trying to sneak past hear you and try to catch you. I did find some of the gestures in the Wii version of Tenchu disturbing though, such as having to mime breaking someone’s neck.
Masanori Takeuchi
Managing Director / Game Producer
FromSoftware, Inc.“I think Natal is beyond the concept of any existing game. With Natal, I thought that we’d be able to do something new on Xbox 360 again. I honestly thought it was going to be a lot of fun.
I think Natal is more about enhancing the experience. If evolution would leave behind our current fans, I want to be careful not to do so. I hope to expand the current gaming experience, rather than to evolve gaming into something irrelevant to current gamers.
Things like movement and facial expressions could not be reflected into the game through a handheld controller. But with Natal, this becomes possible.
I believe Natal will become very interesting, when combined with games that work with the players’ emotions.”
Kojima are the developers of Metal Gear Solid. But they also made Boktai. Boktai was a revolutionary gameboy advance game that uses a solar sensor as an input device, along with a clock. The game character had a solar gun that he used to shoot vampires, but the gun only charged when the player was outdoors in the sunlight. The stronger the sunlight, the faster your gun charged. Sunlight also affected other things in the game, such as slowly rotting fruit and nuts in your inventory. Sometimes in the game darkness is preferable. Time also affected things in game. That gives you an idea of where they are coming from.
They also made the game PoliceNauts for some ancient consoles, which was unusual in that it supported various console peripherals such as Mice and Lightguns.
Kenichiro Imaizumi
Senior Producer
Kojima ProductionsDuring our previous production, we tried to build our own controller. We ran a lot of experiments. We tried running electric shockwaves, install a pulse reader, we also tried to build a controller that’s receptive to the strength of the grip. We tried a lot, even a transforming controller. Kojima came up with these ideas, and we worked directly with the manufacturer.
But all the way along, we thought we had to have a controller. And now Natal tells us that we don’t need a controller. We can use our movements, our voice, or the recognition of colours. That’s how we interact with games. It was quite revolutionary to us, we really never thought of it that way. We had imagined it before, but we never thought it would come so quickly. So it took us by surprise.
What we will create on Natal would be something that has no limitations. Ideas from outside the box. I have a strong feeling that Natal will allow us to create things that we really want to create.