There have been several discussions on the Oculus subreddit recently about how to integrate 2D desktops or 2D applications with 3D VR environments; for example, how to check your Facebook status while playing a game in the Oculus Rift without having to take off the headset.
This is just one aspect of the larger issue of integrating 2D and 3D applications, and it reminded me that it was about time to revive the old VR VNC client that Ed Puckett, an external contractor, had developed for the CAVE a long time ago. There have been several important changes in Vrui since the VNC client was written, especially in how Vrui handles text input, which means that a completely rewritten client could use the new Vrui APIs instead of having to implement everything ad-hoc.
Here is a video showing the new VNC client in action, embedded into LiDAR Viewer and displayed in a desktop VR environment using an Oculus Rift HMD, mouse and keyboard, and a Razer Hydra 6-DOF input device:
So how does this work? On the desktop side, it’s a regular VNC server, such as can be started using any number of “remote desktop” packages, like the one integrated into the Gnome desktop environment. On the VR side, it’s a bit more custom. I used the libvnc library as backend and wrapped it into a Vrui Vislet (a dynamic plug-in that can run inside an arbitrary Vrui application), using some GLMotif widgets and custom Vrui tools for the user interface. Fortunately GLMotif already has widget classes to represent images and scrolled images, so this was fairly easy.
This is a very early version of the client, so it’s still quite rough around the edges, and the code updating the displayed image based on messages from the VNC server is embarrassingly unoptimal. The tricky issue here is that the VNC client can’t just update parts of the displayed image whenever it feels like it; in a VR application, data visible to the user must never be touched out of turn, and the application’s main loop must never block to wait for updates to happen. This requires passing images between several buffers and threads, which, incidentally, has been an ongoing topic in recent Vrui developers’ meetings. But I have an idea of how to make this a lot more efficient.
But those are details, relatively speaking. The biggest problem is that this version of the VNC client will not work in the CAVE. Our CAVE is run by a cluster of six Linux boxes: one head node, four render nodes (one for each CAVE screen), and one audio node. Of these six, only the head node is visible to the Internet at large. The problem is that, when connecting to a local or remote desktop, the entire cluster has to act as a single machine to present a seamless 3D environment to users. It would not work to open separate VNC connections from each cluster node to the desktop server — even if the slave nodes were on the Internet, this would lead to inconsistencies in the display. Instead, all nodes have to receive the exact same data from the VNC server, at the exact same time, in perfect lockstep.
Vrui has an entire sub-architecture dealing with this. Internally, data is transmitted between cluster nodes using a custom broadcast-based stream protocol, which fits the special communication pattern of a VR cluster very well and leads to high data throughput. To simplify development, there are several high-level abstractions layering virtual files or TCP pipes on top of that broadcast protocol. Alas, libvnc is an old-school C library, and does not use any of those abstractions. What I’ll need to do is replace all socket calls in libvnc with the equivalent virtual socket calls from Vrui’s supporting library — or implement the entire VNC protocol from scratch. Not a happy thought.
But that’s what it takes to create a truly system-independent VR development toolkit. 🙂
Update: After I published this article and the video above, I realized I could have gone deeper in my demonstration. Check out this follow-up post (and video) to see what that would have looked like. And to see how one can interactively manipulate 3D protein structures in VR.