Gaze-directed Text Entry in VR Using Quikwrite

Text entry in virtual environments is one of those old problems that never seem to get solved. The core issue, of course, is that users in VR either don’t have keyboards (because they are in a CAVE, say), or can’t effectively use the keyboard they do have (because they are wearing an HMD that obstructs their vision). To the latter point: I consider myself a decent touch typist (my main keyboard doesn’t even have key labels), but the moment I put on an HMD, that goes out the window. There’s an interesting research question right there — do typists need to see their keyboards in their peripheral vision to use them, even when they never look at them directly? — but that’s a topic for another post.

Until speech recognition becomes powerful and reliable enough to use as an exclusive method (and even then, imagining having to dictate “for(int i=0;i<numEntries&&entries[i].key!=searchKey;++i)” already gives me a headache), and until brain/computer interfaces are developed and we plug our computers directly into our heads, we’re stuck with other approaches.

Unsurprisingly, the go-to method for developers who don’t want to write a research paper on text entry, but just need text entry in their VR applications right now, and don’t have good middleware to back them up, is a virtual 3D QWERTY keyboard controlled by a 2D or 3D input device (see Figure 1). It’s familiar, straightforward to implement, and it can even be used to enter text.

Figure 1: Guilty as charged — a virtual keyboard in the Vrui toolkit, implemented as a GLMotif pop-up window with rows and columns of buttons.

The problem is that a straight-up virtual keyboard is, in my opinion, pretty much the worst possible way of going about it. While the old legend about QWERTY being specifically designed to slow down typists is mostly that, a legend, the truth is that keyboards rely on ten fingers and tactile feedback. In VR, where the current reality is no tactile feedback and at most two 3D devices tracking the user’s hands instead of fingers, users cannot touch-type, but instead have to virtually pound the virtual keyboard with their fists. In other words, virtual keyboards are an implementation of hunt&peck typing: find virtual key with input device by visually scanning the UI (“muscle memory” doesn’t work), press button on device to press virtual key, repeat. Granted, that can actually be quite fast, but it is very repetitive and annoying. Having to type extended text by pressing the same real button for each character in turn is a one-way ticket to carpal tunnel town. On top of that, virtual keys have to be fairly large — there is a direct relationship between the size of a button and how quickly it can be selected — and as a result, virtual keyboards take up a lot of display real estate. That large size leads to the need for large movements across the keyboard when typing, and then Gorilla Arm says hi, too.

So what else is out there? A lot, actually, and I’m not even aware of most of it because text entry isn’t my main focus. It just so happens that I think Vrui has a pretty neat way of entering text, and because I got involved in a discussion about text entry, I made a video about it:

The method shown in this video is called Quikwriting or Quikwrite (not QuikWrite as in the video title, my mistake), and it was developed by Ken Perlin (yes, that Oscar-winning Ken Perlin) and patented by New York University. The three main benefits of Quikwrite over virtual keyboards are: (1) entering text of any length is a single stroke, without the need to press/release a button or otherwise signal an event for each character, (2) the Quikwrite UI is very compact and naturally see-through, and (3) the UI can be used without having to look at it at all times.

The main reason why Quikwrite is fast and can be used almost without looking at the UI is that characters are not selected by locating a small rectangle in a sea of similar rectangles, but by moving an input device in one of only eight directions. Specifically, a character is selected by moving the input device from the UI’s center into one of eight major regions arranged around the center along the edges and corners of a square, then potentially from that major region into an adjacent minor region, and then back to the center, which fluidly links to the sequence for the next character.

How fast is it? I haven’t done any formal studies, but it’s faster than using a virtual keyboard. For me, it’s not much faster at this point, but then I’ve used keyboards for 30 years, and Quikwrite for maybe 30 minutes total while developing and debugging it in Vrui  (yeah, I don’t use VR applications to write novels). It obviously takes some training to commit the position of characters and special symbols to muscle memory, but after extended use I expect it to be a lot faster. Speed isn’t even the main reason why I like Quikwrite: for me, it’s not having to play staccato on an input device button. The UI being compact is a nice benefit, too.

But back to the video: the fluid single-stroke nature of Quikwrite enables using it even when there is no button at all on the input device driving it. Such as when using gaze direction for hands-free text entry. With a virtual keyboard or similar methods, there would have to be some way to select the key you’re currently looking at, such as holding gaze while a timer runs out. That’s a problem because the timer can neither be too short, or there will be spurious characters, nor too long, or typing will be slow (and annoying). In Quikwrite, users can type as slowly or quickly as they want (or can), and pause at any time, without having to perform any special actions.

While we’re on the topic: Quikwrite lends itself well to implementation via thumb stick, too. Always start and end characters from the center? Check. Only need to move in one of eight directions? Check. Once in a major region, the minor region is along the same square edge? Check. I really need to implement that (update: I did). And what about Valve’s way of entering text using a game controller? It’s a similar idea, but I find it a bit more clunky. The character positions on the “flower” take just as long to learn, but they’re alphabetic, not optimized by frequency analysis, and there’s still that button mashing going on. In the end it’s going to be a matter of taste, as always in VR.

Finally: what about Swype (pretty much the only other single-stroke method I’m aware of)? It supposedly works well on smart phones, but I’m not so sure about it in VR. It requires a large UI and constant visual focus just like a virtual keyboard, and I am guessing it only works as well as it does due to having a physical surface against which to slide one’s finger. It also needs to be backed by a dictionary to decide which characters have actually been entered, so that’s a bit of a drag.

10 thoughts on “Gaze-directed Text Entry in VR Using Quikwrite

  1. Haha, I must be stupid >_O I have such a hard time seeing you moving in one of eight directions. I see the sections, but it looks like you aim for the letters directly and then you turn around and can go any way back to the center square to register it. Did I get that right?

    So with a controller I would go with the stick to one of the major regions and then rotate it to find a minor region? In my head it sounds like quite smooth text entry, I imagine pulling the stick one way, rotate a little to get the correct letter in the group, then just let it snap back. Might turn into muscle-memory fighting game typing 😛 haha.

    In any case, I found this very interesting, especially as there are no button presses and the fact you don’t have to rest on a letter… just calculating vectors and directions seems slick. Looking forward to eventually trying this in a VR app :3

    • Best via example: to enter an “a” (top-left corner), you first go up-left into the major region, and then right back to the center because the minor region for “a” (up-left) matches the major region. Not surprisingly, that’s true by design for the most frequently used letters.

      To enter an “m” (left in up-left region), you go up-left into the major, then move down to left into the minor, then back to the center.

      To enter a “k” (right in up-left region), you go up-left, then right to up-right, then back to center.

      It doesn’t show up in the video that well because you don’t have to go all the way to the edge. Just dipping into the regions is enough, and after some use you develop the shortest paths.

      It would work the same with a thumbstick: to make a “k”, push top-left, then slide along the top boundary to go top-right, then move or snap back to center.

      • Ah, you actually describe it quite well in the video. I guess I didn’t listen too well 😡 I think what confused me were the regions with five letters. I saw you pick an ‘m’ and it looked like aiming at the minor region directly and then out the side. Blah, I guess just using it would make sense of everything 😛

  2. Does the Rift got enough space to let you add a something like piece of glass at 45 degrees between each eye and the screen, and then cut a hole on the sides (or on top) and add cameras for eye tracking? (I guess perhaps you might need to also add some IR LEDs to make sure the eyes are always well lit; IR so that the user doesn’t see any light interfering with the screen. And i guess a non-glossy IR filter covering the camera would be good to cut on reflections.)

    I thought of asking this back on that post about the Rift’s projection model, but it’s relevant here too, since i was about to ask about using eyetracking to write with Quickwrite. You think Stephen hawking could use Quikwrite to speed up his talking? (He still got full control of his eyeballs, right?)

  3. Btw, last i checked, Youtube allows you to edit pretty much all the text associated with a video even after it’s posted, including the title.

  4. Pingback: Quikwriting with a Thumbstick | Doc-Ok.org

    • Cool! I found Quikwrite only relatively late. I was looking to adapt Dasher to VR by arranging the characters in a 2D probability-weighted quadtree, but that didn’t really work out, and then stumbled upon Quikwrite through an unrelated Google search. I think that was 2008 or 2009.

  5. Pingback: An Eye-tracked Oculus Rift | Doc-Ok.org

Please leave a reply!