I am a research computer scientist at the University of California, Davis. My research areas are scientific visualization, particularly in immersive ("virtual reality") environments, human/computer interaction in immersive environments, and 3D computer graphics.
My primary work is software development, from architecture over design to implementation and coding. I am the primary software developer for the UC Davis W.M. Keck Center for Active Visualization in the Earth Sciences (KeckCAVES). Some of my released packages are Vrui (a VR development toolkit), CollaborationInfrastructure (a tele-collaboration plug-in for Vrui), Kinect (a driver suite to capture 3D video from Microsoft Kinect cameras), LiDAR Viewer (a visualization package for very large 3D point clouds), 3D Visualizer (a system for interactive visual analysis of 3D volumetric data), Nanotech Construction Kit (an interactive molecular design program), and SARndbox (an augmented reality sandbox).
I also dabble in VR hardware, in the sense that I take existing custom or commodity hardware components (3D TVs, head-mounted displays, projectors, tracking systems, Wiimotes, Kinect cameras, ...) and build fully integrated immersive environments out of them. This includes a fair share of driver development to support hardware that either doesn't have drivers, or whose vendor-supplied drivers are not up to par.
It’s been more than two years since the last time I posted set-up instructions for Vrui and HTC Vive, and a lot has changed in the meantime. While Vrui-5.0 and its major changes are still not out of the kitchen, the current release of Vrui, Vrui-4.6-005, is stable and works very well with the Vive. The recent demise of our CAVE, and our move towards VR headsets until we figure out how to fix it, have caused a lot of progress in Vrui’s set-up and user experience. The rest of this article contains detailed installation and set-up instructions, starting from where my previous step-by-step guide, “An Illustrated Guide to Connecting an HTC Vive VR Headset to Linux Mint 19 (“Tara”),” left off.
If you use a Linux distribution that is not Ubuntu-based, such as my own favorite, Fedora, or another desktop environment such as Gnome Shell or Cinnamon, you will have to make some adjustments throughout the rest of this guide.
This guide also assumes that you have already set up your Vive virtual reality system, including its tracking base stations, and that your Vive headset is connected to your PC via HDMI and USB (I will publish a detailed illustrated guide on that part soon-ish). Continue reading →
Running Vrui-based applications in glorious VR on an HTC Vive head-mounted display requires some initial set-up before Vrui itself can be installed and configured. This step-by-step guide will build upon an already-installed Linux operating system with high-performance graphics card drivers, specifically upon the current (as of 12/17/2018) version 19, code-named “Tara,” of Linux Mint, one of the most popular and user-friendly Linux distributions. This guide picks up right where the previous one in this series, “An Illustrated Guide to Installing Linux Mint 19 (“Tara”),” left off.
If you did not follow that guide, this one assumes that you have a “VR ready” or “gaming” PC with a powerful Nvidia GeForce graphics card, an installation of the 64-bit version of Linux Mint 19 (“Tara”) with the MATE desktop environment, and the recommended proprietary Nvidia graphics card driver. And an HTC Vive VR headset, of course.
Graphics Card Driver Set-up
Using a Vive headset with Vrui requires a change to the Nvidia graphics card driver’s configuration. Nvidia’s driver scans connected display devices for known VR headsets, and hides detected headsets from the desktop environment. This does make sense, as headsets are not standard monitors, and it would be awkward if windows or dialogs were to show up on a headset’s display. That said, here’s one relatively large quibble: headset filtering should happen earlier during the boot sequence, not just when the graphics card driver is loaded. As it is, headsets are still enumerated during boot, meaning that boot screens, BIOS menus, boot menus, etc. often show up on the headset, causing real problems. Anyway, carrying on.
Unfortunately for Vrui, there is currently no way to activate a hidden headset from inside an OpenGL-based VR application. For the time being, this means headset filtering in the driver needs to be disabled. To do so, open a terminal window (click on the terminal icon in the panel along the bottom screen edge, or right-click anywhere on the desktop and select “Open in Terminal” from the pop-up menu), enter exactly the following command into it (also see Figure 1) and press the Enter key (the $ sign indicates the terminal’s input prompt; don’t type it):
The first step towards installing any Vrui-based software, including the Augmented Reality Sandbox, is installing some version of the Linux operating system on a new computer, which might sound like a daunting proposition to those who have never done that kind of thing before, but is actually very straightforward. This guide will be using the current (as of 12/17/2018) version 19, code-named “Tara,” of Linux Mint, one of the most popular and user-friendly Linux distributions.
As this guide is geared towards installing and running Vrui-based 3D graphics applications, it assumes that the computer onto which Linux is to be installed is some type of “gaming” or “VR ready” PC, containing an Nvidia GeForce graphics card. The exact model of graphics card, as well as the exact model of CPU, amount of main memory, and hard drive size are not really important (that said, to run 3D graphics applications effectively, the PC should have a recent CPU, at least 4GB of main memory, and at least 60GB of hard drive space). While AMD/ATI graphics cards are otherwise perfectly serviceable, they have traditionally had inferior Linux driver support, and therefore Vrui and Vrui applications have not been tested on them in quite a while. In other words, we do not recommend them for these purposes.
Before installing Linux, one needs to download an installation image for one’s chosen Linux distribution and flavor, and copy it to an installation medium, like a CD/DVD or USB stick. For this guide, we will be using the 64-bit version of Linux Mint 19 (“Tara”), with the MATE desktop environment. The page in the preceding link offers a 1.9GB disk (“iso”) image via a wide selection of download sites all over the world. Click the link for the site that is located most closely to you, and wait for the download to finish.
If the installation medium created in the previous step is a USB drive, plug it into a USB port on the new computer before turning it on for the first time. If it is a CD/DVD, turn the computer on first, and then insert the medium as quickly as possible.
The next step is to tell the computer to boot from the installation medium instead of from its internal hard drive. There is usually a key that needs to be pressed shortly after powering on the computer; typically either the “Delete” key to enter the computer’s BIOS, or “F8” to enter a boot menu. If this does not work on the first try, and the computer “hangs” or boots into whatever operating system was previously installed on it, don’t wait until it finishes booting — just turn it right back off (or press the reset button if it has one) and try again. We are going to erase any previous operating system anyway, so there’s no harm. However, do wait for about 20-30 seconds between turning the computer off and on again to avoid any danger of electrical damage. If neither the “Delete” nor “F8” keys work, look in the computer’s manual or online for the correct key sequence. Amazingly, finding the correct key to boot from the installation medium is by far the most difficult step of installing Linux Mint.
Once a BIOS screen or boot menu show up, select to boot from the installation medium. From there, it will take a few seconds to boot into a “live” Linux Mint environment, see Figure 1.
Figure 1: Linux Mint’s “live” installation environment.
Computer reviews aren’t my thing, but for this one I had to make an exception. My 3.5 year old laptop, the HP Spectre x360 I had scored as swag at the 2015 Microsoft Build conference, suddenly died a few months ago. I had taken a liking to that thing, so when I had to leave for a conference in early November, and realized I should probably bring a laptop with me, I decided to replace it with the current version of the same model. Fortunately they had one in stock at my neighborhood Best Buy (alas, only the silver one and not the pretty black and gold one), so I was able to pick it right up.
I then had the bad idea to search online for Linux support on the x360 after already having bought it, and was dismayed by what I found. A lot of people reported poor performance, too-hot-to-handle operating temperatures, and very poor battery life. Not having much of a choice at that point, I decided to go ahead anyways and install Fedora 28 on it, the then-current release of my go-to Linux distribution. Long story short: installation was a breeze, everything worked out-of-the-box, performance is great, the laptop runs barely warm, and battery life is awesome (so awesome, in fact, that I initially thought the readings were wrong). In order to provide a counter-narrative to those other reports, this is my experience of installing and running Linux on a 2018 HP Spectre x360. Continue reading →
I wrote an article earlier this year in which I looked closely at the physical display resolution of VR headsets, measured in pixels/degree, and how that resolution changes across the field of view of a headset due to non-linear effects from tangent-space rendering and lens distortion. Unfortunately, back then I only did the analysis for the HTC Vive. In the meantime I got access to an Oculus Rift, and was able to extract the necessary raw data from it — after spending some significant effort, more on that later.
With these new results, it is time for a follow-up post where I re-create the HTC Vive graphs from the previous article with a new method that makes them easier to read, and where I compare display properties between the two main PC-based headsets. Without further ado, here are the goods.
The first two figures, 1 and 2, show display resolution in pixels/°, on one horizontal and one vertical cross-section through the lens center of my Vive’s right display.
Figure 1: Display resolution in pixels/° along a horizontal line through the right display’s lens center of an HTC Vive.
Figure 2: Display resolution in pixels/° along a vertical line through the right display’s lens center of an HTC Vive.
Back in the olden days, in the summer of 1996 to be precise, I was a computer science Master’s student at the University of Karlsruhe, Germany, about to take the oral exam in my specialization area, 3D computer graphics, 3D user interfaces, and geometric modeling. For reasons that are no longer entirely clear to me, I decided then that it would be a good idea to prepare for that exam by developing a 3D rendering engine, a 3D game engine, and a game, all from scratch. What resulted from that effort — which didn’t help my performance in that exam at all, by the way — was “Starglider Pro:”
In the mid to late 80s, one of my favorite games on my beloved Atari ST was the original Starglider, developed by Jez San for Rainbird Software. I finally replaced that ST with a series of PCs in 1993, first running DOS, and later OS/2 Warp, and therefore needed something to scratch that Starglider itch. Continue reading →
The short answer is, of course, that it depends on your model of headset. But if you happen to have an HTC Vive, then have a look at the graphs in Figures 1 and 2 (the other headsets behave in the same way, but the actual numbers differ). Those figures show display resolution, in pixels/°, along two lines (horizontal and vertical, respectively) going through the center of the right lens of my own Vive. The red, green, and blue curves show resolution for the red, green, and blue primary colors, respectively, determined this time not by my own measurements, but by analyzing the display calibration data that is measured for each individual headset at the factory and then stored in its firmware.
Figure 1: Resolution in pixels/° along a horizontal line through my Vive’s right lens center, for each of its 1080 horizontal pixels, for the three primary colors (red, green, and blue).
Figure 2: Resolution in pixels/° along a vertical line through my Vive’s right lens center, for each of its 1200 vertical pixels, for the three primary colors (red, green, and blue).
At this point you might be wondering why those graphs look so strange, but for that you’ll have to read the long answer. Before going into that, I want to throw out a single number: at the exact center of my Vive’s right lens (at pixel 492, 602), the resolution for the green color channel is 11.42 pixels/°, in both the horizontal and vertical directions. If you wanted to quote a single resolution number for a headset, that’s the one I would go with, because it’s what you get when you look at something directly ahead and far away. However, as Figures 1 and 2 clearly show, no single number can tell the whole story.
And now for the long answer. Buckle in, Trigonometry and Calculus ahead. Continue reading →
How do all the technical components of VR headsets, e.g., screens, lenses, tracking, etc., actually come together to create realistic-looking virtual environments? Specifically, why do virtual environment in VR look more “real” compared to when viewed via other media, for example panoramic video?
The reason I’m bringing this up again is that the question keeps getting asked, and that it’s really kinda hard to answer. Most attempts to answer it fall back on technical aspects, such as stereoscopy, head tracking, etc., but I find that this approach somewhat misses the point by focusing on individual components, or at least gets mired in technical details that don’t make much sense to those who have to ask the question in the first place.
I prefer to approach the question from the opposite end: not through what VR hardware produces, but instead through how the viewer perceives 3D objects and/or environments, and how either the real world on the one hand, or virtual reality displays on the other, create the appropriate visual input to support that perception.
The downside with that approach is that it doesn’t lend itself to short answers. In fact, last summer, I gave a 25 minute talk about this exact topic at the 2016 VRLA Summer Expo. It may not be news, but I haven’t linked this video from here before, and it’s probably still timely:
There is an on-going, but already highly successful, Kickstarter campaign for a new VR head-mounted display with a wide (200°) field of view (FoV): Pimax 8k. As I have not personally tried this headset — only its little brother, Pimax 4k, at the 2017 SVVR Expo — I cannot discuss and evaluate all the campaign’s promises. Instead, I want to focus on one particular issue that’s causing a bit of confusion and controversy at the moment.
Early reviewers of Pimax 8k prototypes noticed geometric distortion, such as virtual objects not appearing in the correct places and shifting under head movement, and the campaign responded by claiming that these distortions “could be fixed by improved software or algorithms” (paraphrased). The ensuing speculation about the causes of, and potential fixes for, this distortion has mostly been based on wrong assumptions and misunderstandings of how geometric projection for wide-FoV VR headsets is supposed to work. Adding fuel to the fire, the campaign released a frame showing “what is actually rendered to the screen” (see Figure 1), causing further confusion. The problem is that the frame looks obviously distorted, but that this obvious distortion is not what the reviewers were complaining about. On the contrary, this is what a frame rendered to a high-FoV VR headset should look like. At least, if one ignores lenses and lens distortion, which is what I will continue to do for now.
Figure 1: Frame as rendered to one of the Pimax 8k’s screens, according to the Kickstarter campaign. (Probably not 100% true, as this appears to be a frame submitted to SteamVR’s compositor, which subsequently applies lens distortion correction.)
Why does everything in my VR headset look so pixelated? It’s supposed to be using a 2160×1200 screen, but my 1080p desktop monitor looks so much sharper!
This is yet another fundamental question about VR that pops up over and over again, and like the others I have addressed previously, it leads to interesting deeper observations. So, why do current-generation head-mounted displays appear so low-resolution?
Here’s the short answer: In VR headsets, the screen is blown up to cover a much larger area of the user’s field of vision than in desktop settings. What counts is not the total number of pixels, and especially not the display’s resolution in pixels per inch, but the resolution of the projected virtual image in pixels per degree, as measured from the viewer’s eyes. A 20″ desktop screen, when viewed from a typical distance of 30″, covers 37° of the viewer’s field of vision, diagonally. The screen (or screens) inside a modern VR headset cover a much larger area. For example, I measured the per-eye field of view of the HTC Vive as around 110°x113° under ideal conditions, or around 130° diagonally (it’s complicated), or three and a half times as much as that of the 20″ desktop monitor. Because a smaller number of pixels (1080×1200 per eye) is spread out over a much larger area, each pixel appears much bigger to the viewer.