2D Desktop Embedding via VNC

There have been several discussions on the Oculus subreddit recently about how to integrate 2D desktops or 2D applications with 3D VR environments; for example, how to check your Facebook status while playing a game in the Oculus Rift without having to take off the headset.

This is just one aspect of the larger issue of integrating 2D and 3D applications, and it reminded me that it was about time to revive the old VR VNC client that Ed Puckett, an external contractor, had developed for the CAVE a long time ago. There have been several important changes in Vrui since the VNC client was written, especially in how Vrui handles text input, which means that a completely rewritten client could use the new Vrui APIs instead of having to implement everything ad-hoc.

Here is a video showing the new VNC client in action, embedded into LiDAR Viewer and displayed in a desktop VR environment using an Oculus Rift HMD, mouse and keyboard, and a Razer Hydra 6-DOF input device:

So how does this work? On the desktop side, it’s a regular VNC server, such as can be started using any number of “remote desktop” packages, like the one integrated into the Gnome desktop environment. On the VR side, it’s a bit more custom. I used the libvnc library as backend and wrapped it into a Vrui Vislet (a dynamic plug-in that can run inside an arbitrary Vrui application), using some GLMotif widgets and custom Vrui tools for the user interface. Fortunately GLMotif already has widget classes to represent images and scrolled images, so this was fairly easy.

This is a very early version of the client, so it’s still quite rough around the edges, and the code updating the displayed image based on messages from the VNC server is embarrassingly unoptimal. The tricky issue here is that the VNC client can’t just update parts of the displayed image whenever it feels like it; in a VR application, data visible to the user must never be touched out of turn, and the application’s main loop must never block to wait for updates to happen. This requires passing images between several buffers and threads, which, incidentally, has been an ongoing topic in recent Vrui developers’ meetings. But I have an idea of how to make this a lot more efficient.

But those are details, relatively speaking. The biggest problem is that this version of the VNC client will not work in the CAVE. Our CAVE is run by a cluster of six Linux boxes: one head node, four render nodes (one for each CAVE screen), and one audio node. Of these six, only the head node is visible to the Internet at large. The problem is that, when connecting to a local or remote desktop, the entire cluster has to act as a single machine to present a seamless 3D environment to users. It would not work to open separate VNC connections from each cluster node to the desktop server — even if the slave nodes were on the Internet, this would lead to inconsistencies in the display. Instead, all nodes have to receive the exact same data from the VNC server, at the exact same time, in perfect lockstep.

Vrui has an entire sub-architecture dealing with this. Internally, data is transmitted between cluster nodes using a custom broadcast-based stream protocol, which fits the special communication pattern of a VR cluster very well and leads to high data throughput. To simplify development, there are several high-level abstractions layering virtual files or TCP pipes on top of that broadcast protocol. Alas, libvnc is an old-school C library, and does not use any of those abstractions. What I’ll need to do is replace all socket calls in libvnc with the equivalent virtual socket calls from Vrui’s supporting library — or implement the entire VNC protocol from scratch. Not a happy thought.

But that’s what it takes to create a truly system-independent VR development toolkit. 🙂

Update: After I published this article and the video above, I realized I could have gone deeper in my demonstration. Check out this follow-up post (and video) to see what that would have looked like. And to see how one can interactively manipulate 3D protein structures in VR.

24 thoughts on “2D Desktop Embedding via VNC

  1. Haha, I see what you did there.

    Using VNC is quite clever, I hadn’t thought of that. I guess as it’s done locally the latency will be quite good. And as long as it’s normal desktop interactions it should be fine even with some latency, perhaps as long as the mouse cursor is rendered on top of the stream… I know I’ve had to replace monitors because of input lag making my cursor being delayed, it had me go crazy! 😛

    On a related topic, I have been waiting for Steam to get the VR screen to work with games. They could basically do the same thing you do with VNC with the already working in home streaming that is currently in beta. Not sure what kind of solution they will finally do, but it’s exciting in any case. Even if the DK1 has such low resolution, getting a virtual huge screen is awesome. And from Steam a curved screen makes so much sense when it is large, lessens perspective distortion, super nice!

    Oh, and I’ve totally watched too many of your videos. Now every time I read “display” in my head it’s said with your accent ._.

    • The latency is really very good, hardly noticeable. While watching the video just now, I realized that exactly when I’m saying “the video runs very smoothly,” it’s all choppy. That’s an effect of how I made the video: I first recorded my interactions in the software (like recording a MIDI file), and then I played those back on a second run while dumping movie frames to disk (like converting a MIDI into a WAV). Because I can’t control firefox of oocalc like that, I had to run them “live” in the second pass as well, and they really didn’t like the other part of the program dumping gigabytes of image data at the same time.

      Bottom line: it’s way choppier and higher latency in the video than it was when I recorded it. You gotta take my word for it. 😉

  2. Thanks for sharing! Any plans to make the VNC client software available to try out? Noted that it’s a bit rough and ready and will probably be a pain to support though!

        • That entirely depends on the Vrui application(s) you want to run, or the data you want to load into them. The only basic requirement would be some kind of 3D graphics acceleration. We were successfully running 3D Visualizer, on serious datasets, on regular (non-pro) Macbooks six years ago, and my laptop is a 2008 Macbook Pro, and runs all applications and most data sets fine. Running LiDAR Viewer with half a billion points (like in this video) is maybe a bit too much for baseline computers. My main development rig is a 2.66 GHz Intel Core i7 with 4GB of RAM (forgot the RAM specs), a 2 TB two-disk RAID, and, as of recently, a Geforce 680.

          • That doesn’t seem like a lot of RAM. I noticed LidarViewer has an option to set a cache size, does it significantly benefit from a lot of RAM?
            I suppose fast disk access is most important though.
            I tried a 10+ billion dataset this week and I noticed my imac started performing a bit choppy (The whole thing ran from just one regular disk though).

          • Reply to Ylannl: I hear that a lot 😉 More RAM in the computer acts mostly as cache for the disk; the really important number is GPU RAM size. If you noticed choppiness with large data sets, you could try experimenting with the memory cache size. In very dense data, when you have more memory cache than GPU cache, the renderer will stream points that don’t fit into GPU RAM from main memory, and that can cause delays. I’ve experimented with leaving memory cache at slightly smaller than GPU cache, which makes it run smooth unless something goes wrong.

  3. Have you experimented with making a virtual keyboard with fat keys you punch with the 6dof input device? (or perhaps instead of punching, give the virtual representation sort of a beak, a hook in the front with the tip pointing down, so the user can use the rotation of the device more when typing, to reduce the amount of movement needed to reach all keys. Or perhaps just make it like those software keyboards many OSes got, and use the laser pointer mode + a button to type; though i feel, the beak pecking mode would feel more natural. Perhaps make a mix of the two, a drum-set keyboard, replacing the laser pointer with a limited range solid pointer/pokey thing and bang on the keys like you’re playing the drums?)

    • The original Vrui VNC client had its own virtual keyboard, because back then Vrui didn’t have a text channel. It didn’t really work. To make it non-frustrating, you have to make each key fairly big, and then the entire keyboard becomes so big that you have to move your arm(s) a lot to type. It’s quite tiring. That keyboard was a regular GLMotif dialog, so it required pointing and clicking. Your “beak” idea is very similar to something I’ve had on the to-do list for a while, in its more general incarnation of simulating button presses by touching a virtual 3D GUI instead of pressing a real button. I have a hunch that it will work quite well.

      Now that Vrui has a text channel, it’s up for experimentation. Right now I have a built-in QuikWrite tool ( https://mrl.nyu.edu/projects/quikwriting/ ), which works really well after you spend some time practicing. I haven’t used it much in a long time, but while developing it, I became really good at it, much better than with a virtual 3D keyboard. With QuikWrite, you can spell out entire paragraphs by just making fluid gestures with your hand. It’s all in the wrist.

      There are a bunch of other methods that I haven’t tried. I’m hoping that some third party will develop some additional text input tools for Vrui some time soon. 🙂

      • That QuickWrite thing quite an interesting approach!

        What is the logic for the placement of the characters though? Is it something like based on character frequency in the English language? I tried the java demo, it feels like Space and Backspace would’ve felt more natural as Right-Right and Left-Left respectively.

        Btw, do you usually type with the stick or by rotating the 6dof device?

        • The character layout was driven by frequency distribution, so that the most common characters have the shortest paths to/from the center, and that common character sequences have smooth gestures when you chain their respective paths.

          I hadn’t actually thought about using the thumbstick for QuikWrite; it’s worth testing. I’m using the same 6-DOF pointing method as for GUI interaction. When you attach a QuikWrite tool to a button, pressing that button will pop up the wheel directly in front of the device, so that you’re already pointing at the center. Character selection is then based on ray intersection, so it only needs relatively small rotations of the wrist to do.

        • Oops, i was using the older version of the applet, the more recent one got the space and backspace the way i described.

    • Compositing window managers should be a good starting point. Once you have window contents in a compositing buffer, it should be trivial to map them onto a textured quad. I went the VNC route because that was tried and true, works with remote computers (scientists like remoting their laptops into the CAVE), and is OS-independent. RDP would probably work as well or better, but I have no experience with it.

  4. Have you considered projecting the windows over cylindrical/spherical surfaces? just thought it would be more comfortable that if you moved the screen to the side with the mouse it would still be viewable if you looked to your side.

    Also another idea is to use the cylindrical/spherical surface as an anchor for the plane window.

    http://imgur.com/0uRFtqq < a drawing of that idea, considering that:
    * the light blue square is the initial position of the window.
    * the light green square is the final position of the window when moved with the mouse to the left side.
    * the red circle is the viewer position.
    * the black line is the "projection surface".

    • Yes. One of the reasons it took me so long to support the Rift in Vrui was not the basics — tracking, projection setup, lens distortion — those only took a few hours or days, but creating a proper user interface without hard-coding anything. Vrui is environment-agnostic, meaning that it works unchanged on anything from a regular 2D desktop to a CAVE or HMD. But window layout and interaction is an environment-specific task, so there needs to be an abstraction layer that handles it, in other words, a VR window manager. I have been working towards that, but it isn’t quite there yet. But once there is a proper window manager, things like spherical or cylindrical layouts would be manager policies, selected by whoever sets up some environment.

      Right now there is the planar pseudo-manager I’m showing in the video, and I also have one where the interaction plane is fixed to the Rift, which is an approximation of a spherical layout, in which the mouse cursor moves with a combination of mouse and head movement. I didn’t use it for the video. The default manager in environments with 6-DOF devices is one where there is no layout at all; users can place GUI widgets at arbitrary positions and orientations. I didn’t turn it on for this video because I wanted to start with the mouse.

      I’m not in favor of actually projecting GUI elements onto a sphere or cylinder. The widgets here are 3D, so they get visibly distorted under that kind of projection. A VNC client would be a special case where projection would work well in an HMD environment, but it wouldn’t work so well in less constrained cases. This is something where generality will probably win out.

  5. Pingback: More on Desktop Embedding via VNC | Doc-Ok.org

  6. Pingback: 2D Desktop Embedding via VNC | Doc-Ok.org | Res...

  7. Pingback: New Adventures in Blogging | Doc-Ok.org

  8. Pingback: Will Virtual Reality Replace The Virtual Desktop? »

Please leave a reply!