@elonmusk: We figured out how to design rocket parts just w hand movements through the air (seriously). Now need a high frame rate holograph generator.
@elonmusk: Will post video next week of designing a rocket part with hand gestures & then immediately printing it in titanium
As there are no further details, and the video is now slightly delayed (per Twitter as of September 2nd: @elonmusk: Video was done last week, but needs more work. Aiming to publish link in 3 to 4 days.), it’s time to speculate! I was hoping to have seen the video by now, but oh well. Deadline is deadline.
First of all: what’s he talking about? My best guess is a free-hand, direct-manipulation, 6-DOF user interface for a 3D computer-aided design (CAD) program. In other words, something roughly like this (just take away the hand-held devices and substitute NURBS surfaces and rocket parts for atoms and molecules, but leave the interaction method and everything else the same):
If you ask me, that would be a very effective tool to design rocket parts, or really anything three-dimensional. And, according to Twitter, Elon Musk agrees (emphasis mine):
@elonmusk: And, uhh no (zillionth person who asked), I am not going to make an IM suit, however design by hand-manipulated hologram is actually useful.
This is great. I’ve been saying for a long time that, if nothing else, VR is a useful and effective method for interacting with 3D data, but if Elon Musk says it, people will take notice. Side note: why did a zillion people ask him about an IM (Iron Man) suit? It appears Elon Musk has taken on the persona of a real-life Tony Stark (director Jon Favreau confirmed that Elon Musk was indeed the inspiration for Stark’s portrayal in the movies), and now with talk of holograms and holographic design programs, the boundaries between real person and movie character are starting to get somewhat blurry. The functional similarities between a design program as alluded to in the pair of Twitter posts and the movie version of same are surely not coincidental:
As I’ve written before, the remarkable thing about this sequence (and other similar ones from the Iron Man movies) is that — ignoring the fact that real holograms don’t work as depicted here — this interface makes sense, and would actually work as shown (unlike the equally famous Minority Report interface, zing). Instead of holograms it would use VR or AR as a display medium, but that’s merely a technical detail.
Now, then, what will the video actually show once it hits the Internets? In principle, any proper VR system could be used. But, given the posts’ emphasis on hand gestures, and that augmented reality (AR) is generally considered “cooler” than virtual reality(VR), I’m guessing we’ll see an integrated system of a pair of AR goggles and a 3D camera for hand tracking and gesture recognition. By sheer not-coincidence, such a device currently exists: the meta. I saw a prototype myself at the 2013 Augmented World Expo, demonstrated by Steve Mann. The meta consists of a SoftKinetic time-of-flight 3D camera directly attached to a pair of AR glasses. This combination enables 3D interfaces, where users can manipulate the virtual 3D objects shown by the AR glasses using their bare hands.
I am very curious to see how well this will work in practice. The meta’s tiny field-of-view (23° per eye) will probably not be an issue, but optical hand/finger tracking might be. Look carefully at this video:
You can see how the red dots indicating the finger tips are somewhat jittery, and sometimes align not with the tip, but with the first joint or other parts of the finger, due to occlusion. That’s a fundamental problem in optical hand tracking. For the kinds of precise interaction required to effectively design in 3D, it’s definitely an issue.
A second problem is how to use hand tracking to trigger events. To go beyond simple sculpting, the user needs to be able to make something happen (create a vertex, drag a face, split an edge, …) at a well-defined 3D location. How do you do that? Complex hand gestures, such as sign language phonemes, are out because they require moving the hands away from the intended action point to perform the gesture. Simple gestures like finger pinches don’t, and are more intuitive to boot, but finger pinches are hard to detect in depth images. It’s a subtle problem: how do you tell from depth images, suffering from occlusion, whether two fingers are actually pinched, or just lightly brushing against each other? If the user and the system do not exactly agree when a pinch occurred, it will lead to missed or spurious events, and user frustration. I’m using pinches as an example here, but other simple gestures have the same problem. This is where old-fashioned push buttons and their direct physical feedback really shine.
To recap: the question is not whether direct-manipulation 6-DOF user interfaces work (do they ever!), but whether optical hand tracking is the ideal way to implement them. The problems are jitter, tracking jumps due to occlusion and joint mislabeling, and flaky event detection. Hand-held 6-DOF devices with buttons are not nearly as futuristic and cool, but they are reliable, predictable, and lead to little user frustration. See this video, starting at 6:35, where my Hydra’s left handle malfunctions and generates spurious events. It’s extremely annoying and derails my workflow. This happened because my Hydra was broken; in optical hand tracking, it’s the normal state of affairs unless one takes extra precautions. This is why I’m very curious to see the promised video. And I’m hoping the “video was done last week, but needs more work” quote doesn’t mean they need more time to edit around the bad bits. I really want to see this work. KeckCAVES‘ users are clamoring for free-hand interactions, but so far nothing worked reliably enough.
Because if it does work, it’s going to be good for VR as a whole. If someone like Elon Musk comes out and says that VR/AR/3D user interfaces are useful, it will have a huge impact. After that, I’m just hoping that people don’t forget that this kind of thing is not entirely new, but has already existed — and been used successfully — for a long time. We, at least, have been building things with our hands in air since 1998.