As many designers are wont to do, I've been goofing around with game ideas in my head lately. No doubt being in a period of console platform transition tends to spur this kind of thought, and no console is more inspiring during this particular transition than Nintendo's "Wii".

What the Wii brings to the table is an idea - gestural control input - which is very simple and not at all new. However the fact that it is enabled by default on a device which will be sold by the millions, now that is new, and that gets a person thinking.

What is Gestural Control?
It's hardly coincidence that one of the first demonstrations Nintendo chose to introduce the Wii to the public was a simulation of conducting an orchestra. Orchestral conducting is a completely gestural system. When I was in college I studied music and, among other things, learned the gestures for conducting. Despite how it looks, the conductor actually uses some very specific, pre-defined gestures. So for instance in 4/4 time (by far the most common time signature in music), the gestures are, in sequence: down, left, right, up. For a three-beat time signature (such as 3/4), the gestures are down, right, up.

What makes orchestra conducting interesting is that within these prescribed set of gestures is infinite variability. Western classical music, though written with a "time signature", is rarely designed to be played with a steady "beat" or "groove" (that was an addition of African influence, but that's a whole other story). Rather, the orchestra follows the conductor's interpretation of each division of time, speeding and slowing to match the "feeling" or "meaning" of the piece. To do this, the conductor utilizes analog control. In fact, when you learn conducting, you're taught not to draw straight lines, but rather to make an asymmetrical, curved diamond with the four (or three) beats. All of this is to make the gesture more dramatic and varied, and to provide the infinite variability the conductor needs within each beat.

So, using gestural control is about variability and expressiveness. And once you use it, it's very easy. Conducting is not difficult.

Gestural Control and Computers
Interestingly, computers have had excellent gestural control input devices for more than twenty years, yet have very rarely used them as such. The mouse, with its combination of analog and precision, is a nearly perfect gestural input device. Yet we use it mostly just for "pointing and clicking". When I installed the Firefox plugin "mouse gestures" (highly recommended!), it took some getting used to, but opened up a whole new world: I no longer needed to either point or click in order to do common tasks like closing windows, going forward or backward in a browse sequence, etc. This freeing from the spatiality of the GUI of Firefox is really eye-opening.

A different type of gestural control was popularized by the movie Minority Report, in which the lead character, needing to browse large volumes of information quickly, dons some weird gloves (it's a sci-fi movie - there have to be weird gloves), and moves stuff around a large virtual screen. In fact this kind of interface has been demonstrated (without the goofy gloves) in several different touchscreen-driven forms here in the real world. It looks really cool. One has to imagine that in the future, driven by technologies such as those behind the Tablet PC, nearly all computers will incorporate some sort of touchscreen-driven gestural input systems.

The Limits of Gestures
This is a good time to discuss the limits of gestural controls. If you watch some of the demonstrations of the existing touchscreen systems, they're wonderful for organizing a desktop, for viewing photos and the like. But when it's time to enter data, they always pop up a "virtual" QWERTY keyboard. When I saw this, it reminded me of some other computers that relied on gestural input: the Newton and Palm Pilot.

Apple made the Newton back in the early '90s, and it was revolutionary, a handheld computerized organizer designed to be used by anybody. And a big part of the "fun and easy to use" ethos of the Newton was handwriting recognition. Because of its special software, you could just write on the Newton, and it would accept this input. The problem was, the handwriting recognition software sucked. Turns out, handwriting recognition is one of the Hard Problems of computer science. And the little battery-powered handheld wasn't even remotely up to it. Cataloging "Newton Translations", unintentional jokes created by Newton handwriting recognition errors, was one of the earlier hobbies of the World Wide Web.

Poor Newton. Great idea, bad software.

Right?

Well that's what the guys at PalmPilot thought. They realized that the Newton had just been too ambitious, attempting to recognize "natural" handwriting. If the user could be trained to use a simplified type of handwriting, still gestural but not idiosyncratic, the device could easily recognize the gestures as text.

They were right. And the Palm Pilot made a very crucial good design decision, which was to require letters to be entered in sequence in the same space - this enforced that the user would use independent gestures for each letter, with a start and end. I had a couple different Palm Pilots, and they really did work.

But they were a pain in the ass. Was an "I" a stroke up, or a stroke down? Which stroke first for a "T"? How do I enter capital letters? You eventually learned it, but it took time. Then Palm created a "better" input system, which changed about 20% of the gestures. I had to learn the whole damn thing over again.

The problem was, they were missing the point. Entering text is a shitty use of gestural input. Look at the devices that replaced the Palm - the Blackberry, the Sidekick, even Palm's own Treo - pretty much all of them use keyboards for input. As great as gestural input is for analog or expressive actions, it's just crap for anything that requires speed and/or accuracy. Such as, for instance, entering data.

Gestural Input and Games
Oh yes, games. Gestural input is far from new in games - we've been playing with this for ages. Ironically, Sony was the first to bring real gestural input to consoles with EyeToy, which they first demonstrated at Siggraph in 1998. The EyeToy story is a classic example of why the games business should never be driven by technology. EyeToy was a technology R&D project to test the capabilities of software to recognize and use video input. It was a dramatic success. But what came of it (the brilliantly fun EyeToy: Play) was not great because of its video recognition. It was great because it was an example of gestural input. If you had seen my daughter, only two at the time, play the 'Soapy Window' game endlessly, you'd have clearly seen the potential of gestural input. Or at least, you should have.

EyeToy however is limited by its technology: video input is fussy, and imprecise. So Sony has relegated it to a niche device - a classic case of viewing a product as technology instead of entertainment. Had Sony viewed the success of EyeToy: Play on its merits as an interactive system, rather than a technology demonstration, I have a suspicion that Nintendo would not be the sole console manufacturer offering gestural input this fall.

Gestural input hit the bigtime though with the Nintendo DS. When the DS was launched, it seemed like its key feature was the dual screens (hell, the thing was even called "DS"). Wrong. Two screens actually pretty much sucks. But one of those screens has a touchscreen input, and a stylus is stuck in the back of the device. Aha. With stylus in hand, great new products like Brain Age, the fascinating Electroplankton, and the before-its-time Kirby's Canvas Curse suddenly become possible. And there are many more to come.

Meanwhile Okami has convinced me that gestural control on a traditional analog controller is mostly a lost cause. And though I admire Sony for putting the six-axis momentum sensor in the PS3 controller, it's really just a giant and wonky third analog stick. Its lack of spatiality makes it useless as a gestural device.

That Wii Thing
Honestly, I'm concerned about the Wii's controller as a gestural system. The video I've seen shows the input to be pretty jumpy and imprecise. I really don't think you could do Brain Age on the Wii-mote. It will be wonderful for many things. For others, it will be highly imperfect. Remember: great for expressive and analog; poor for quick and precise. Nintendo were wise to include the "nun-chuck" traditional controller adapter in the box.

Still, it does get the mind turning. I remember how fun EyeToy: Play was, and though it was too primitive to maintain interest, there was something there. And as I turn over these little game ideas in my head, the quick, accurate gestures possible on the DS are really attractive (indeed for me, the DS remains the more attractive of the two, at least for now). I'm not going to tell you my ideas just now, but suffice to say, the potential in gestural control for games is fantastic.