A Clarification About “Black Smear”

Here’s another frequently-asked question about VR headsets, or at least those that use LED-based displays:

Why does my headset show dark grey when it’s supposed to show black? Shouldn’t LED displays be able to show perfect blacks?

I addressed this in detail a long time ago, but the question keeps popping up, and it is often answered like the following: “LED display pixels have a memory effect when they are turned off completely, which causes ‘black smear.’ This can be avoided by never turning them off completely.”

Unfortunately, that answer is mostly wrong. LED display pixels do have a memory effect (for reasons too deep to get into right now), but it is not due to being turned off completely. The obvious counter argument is that, in the low-persistence displays used in all LED-based headsets, all display pixels are completely turned off for around 90% of the time anyway, no matter how brightly they are turned on during their short duty cycle. That’s what “low persistence” means. So having them completely turned off during their 1ms or so duty cycles as well won’t suddenly cause a memory effect.

The real answer is mathematics. In a slightly simplified model, the memory effect of LED displays has the following structure: if some pixel is set to brightness b1 in one frame, and set to brightness b2 in the next frame, it will only “move” by a certain fraction of the difference, i.e., its resulting effective brightness in the next frame will not be b2 = b1 + (b2 – b1), but b2‘ = b1 + (b2 – b1)⋅s, where s, the “smear factor,” is a number between zero and one (it’s usually around 0.9 or so).

For example, if b1 was 0.1 (let’s measure brightness from 0 = completely off to 1 = fully lit), b2 is 0.7, and s = 0.8, then the pixel’s effective brightness in frame 2 is b2‘ = 0.1 + (0.7 – 0.1)⋅0.8 = 0.58, so too dark by 17%. This manifests as a darkening of bright objects that move into previously dark areas (“black smear”). The opposite holds, too: if the pixel’s original brightness was b1 = 0.7, and its new intended brightness is b2 = 0.1, its effective new brightness is b2‘ = 0.7 + (0.1 – 0.7)⋅0.8 = 0.22, so too bright by 120%(!). This manifests as bright trails following bright objects moving over dark backgrounds (“white smear”).

The solution to black and white smear is to “overdrive” pixels from one frame to the next. If a pixel’s old brightness is b1, and its intended new brightness is b2, instead of setting the pixel to b2, it is set to an “overdrive brightness” bo calculated by solving the smear formula for value b2, where b2‘ is now the intended brightness: bo = (b2 – b1)/s + b1.

Let’s work through the two examples I used above: First, from dark to bright: b1 = 0.1, b2 = 0.7, and s = 0.8. That yields bo = (0.7 – 0.1)/0.8 + 0.1 = 0.85. Plugging bo = 0.85 into the smear formula as b2 yields b2‘ = 0.1 + (0.85 – 0.1)⋅0.8 = 0.7, as intended. Second, going from bright to dark: b1 = 0.7, b2 = 0.1, and s = 0.8 yields bo = (0.1 – 0.7)/0.8 + 0.7 = -0.05. Oops. In order to force a pixel that had brightness 0.7 on one frame to brightness 0.1 on the next frame, we would need to set the pixel’s brightness to a negative value. But that can’t be done, because pixel brightness values are limited to the interval [0, 1]. Ay, there’s the rub.

This is a fundamental issue, but there’s a workaround. If the range of intended pixel brightness values is limited from the full range of [0, 1] to the range [bmin, bmax], such that going from bmin to bmax will yield an overdrive brightness bo = 1, and going from bmax to bmin will yield an overdrive brightness bo = 0, then black and white smear can be fully corrected. The price for this workaround is paid on both ends of the range: the high brightness values (bmax, 1] can’t be used, meaning the display is a tad darker than physically possible (a negligible issue with bright LEDs), and the low brightness values [0, bmin) can’t be used, which is a bigger problem because it significantly reduces contrast ratio, which is a big selling point of LED displays in the first place, and means that surfaces intended to be completely black, such as night skies, will show up as dark grey.

Let’s close by working out bmin and bmax, which only depend on the smear factor s and can be derived from the two directions of the overdrive formula: 1 = (bmax – bmin)/s + bmin and 0 = (bmin – bmax)/s + bmax. Solving yields bmin = (1 – s)/(2 – s) and bmax = 1/(2 – s). Checking these results by calculating the overdrive values to go from bmin to bmax, which should be 1, and from bmax to bmin, which should be 0, is left as an exercise to the reader.

In a realistic example, using a smear factor of 0.9, the usable brightness range works out to [0.09, 0.91], meaning the darkest the display can be is 9% grey.

Quantitative Comparison of VR Headset Fields of View

Although I’ve taken many through-the-lens pictures of several common VR headsets with a calibrated wide-angle camera, until recently I was still struggling how to compare the resulting fields of view (FoV) quantitatively, how to put them in context, and how to visualize them appropriately. When trying to answer questions like “which VR headset has a bigger FoV?” or “by how much is headset A’s FoV bigger than headset B’s?” or “how does headset C’s FoV compare to the field of vision of the naked human eye?”, the basic question is: how does one even measure field of view, in a way that is fair and allows comparison across a wide range of possible sizes. Does one report a single angle, and how does one measure it? Across the “diagonal” of the field of view? What if the field of view is not a rectangle? Does one report a pair of angles, say horizontal⨉vertical? Again, what if the field of view is not a rectangle?

Then, if FoV is measured either as a single angle or a pair of angles, how does one compare different FoVs fairly? If one headset has 100° FoV, and another has 110°, does the latter show 10% more of a virtual 3D environment? What if one has 100°⨉100° and another has 110°⨉110°, does the latter show 21% more?

To find a reasonable answer, let’s go back to the basics: what does FoV actually measure? The general idea is that FoV measures how much of a virtual 3D environment a user can see at any given instant, meaning, without moving their head. A larger FoV value should mean that a user can see more, and, ideally, an FoV value that is twice as large should mean that a user can see twice as much.

Now, what does it mean that “something can be seen?” We can see something if light from that something reaches our eye, enters the eye through the cornea, pupil, and lens, and finally hits the retina. In principle, light travels towards our eyes from all possible directions, but only some of those directions end up on the retina due to various obstructions (we can’t see behind our heads, for example). So a reasonable measure of field of view (for one eye) would be the total number of different 3D directions from which light reaches that eye’s retina. The problem is that there is an infinite number of different directions from which light can arrive, so simple counting does not work.

Continue reading

How to Track Glowing Balls in 3D

Note: I started writing this article in June 2017, because people kept asking me about details of the PS Move tracking algorithm I implemented for the video in Figure 1. But I never finished it because I couldn’t find the time to do all the measurements needed for a thorough error analysis, and also because the derivation of the linear system at the core of the algorithm really needed some explanatory diagrams, and those take a lot of work. So the article stayed on the shelf. I’m finally publishing it today, without error analysis or diagrams, because people are still asking me about details of the algorithm, more than four years after I published the video. 🙂

This one is long overdue. Back in 2015, on September 30th to be precise, I uploaded a video showing preliminary results from a surprisingly robust optical 3D tracking algorithm I had cooked up specifically to track PS Move controllers using a standard webcam (see Figure 1).

Figure 1: A video showing my PS Move tracking algorithm, and my surprised face.

During discussion of that video, I promised to write up the algorithm I used, and to release source code. But as it sometimes happens, I didn’t do either. I was just reminded of that by an email I received from one of the PS Move API developers. So, almost two years late, here is a description of the algorithm. Given that PSVR is now being sold in stores, and that PS Move controllers are more wide-spread than ever, and given that the algorithm is interesting in its own right, it might still be useful. Continue reading

Visual Simulation of the Resolution of VR Headsets

While I was writing up my article on the PlayStation VR headset’s optical properties yesterday, and specifically when I made the example images for the sub-section about sub-pixel layout comparing RGB Stripe and PenTile RGBG displays, it occurred to me that I could use those images to create a rough and simple simulator to visually evaluate the differences between VR headsets that have different resolutions and sub-pixel layouts.

The basic idea is straightforward: Take a test image that has some pixel count, for example WxH=640×360 as the initial low-resolution full-RGB picture seen in Figure 1. If you then blow up that image to fill a monitor that has the same aspect ratio (16:9) and some diagonal size D, the resolution of that image in terms of pixels per degree depends both on D and the viewer’s distance from the monitor Y: the larger the ratio Y/D, the higher is the image’s resolution as seen by the viewer. In detail, the formula for distance Y to achieve a desired resolution R in pixels/° is:

Y = (D / sqrt(W*W + H*H))/(2*tan(1/(2*R)))

where W and H are in pixel units, R is in pixels/°, and D is in some arbitrary length unit (inch, meter, parsec,…). Y will end up being in the same unit as D.

If a viewer then positions one of their eyes at a distance of Y from the center of the monitor and closes the other one, the resolution of the image on the monitor will be R. In other words, if R is the known resolution of some VR headset, the image on the monitor will look the same resolution as that VR headset.

There is one caveat: when looking at a flat monitor, the resolution of the displayed image increases away from the monitor’s center, while in a VR headset, the resolution generally decreases away from the center direction (see this article for reference). Meaning, for a correct evaluation, the viewer has to focus on the area in the center of the monitor. Unfortunately there is no easy way to simulate resolution drop-off using a flat monitor, at least not while also simulating sub-pixel layout.

Simulation Procedure

Here’s what you need to do: Continue reading

Field of View and Resolution of the PlayStation VR Headset

It has been a very long time since I did the original optical measurement of then-current VR headsets. I have owned a PlayStation VR headset (PSVR from now on) for almost a year now, and I finally got around to measuring its optical properties in the same way. I also developed a new camera calibration algorithm (that’s a topic for another post), meaning I am even more confident in my measurements now than I was then.

One approach to measuring the optical properties of a VR headset, which includes measuring its field of view, its resolution in pixels/°, and its lens distortion correction profile, is to take a series of pictures of the headset’s screen(s) through its lenses using a calibrated wide-angle camera. In this context, a calibrated camera is one where each image pixel’s horizontal and vertical angles away from the optical axis are precisely known.

If one then displays a test pattern that lets one identify a particular pixel on the screen, one can measure the viewer-relative angular position of that pixel in the camera image, which is all the information needed to generate the projection matrices and lens distortion correction formulas that are essential to high-quality VR rendering.

Without further ado, here is a series of 7 images taken with the camera lens at increasing distances from the headset’s right lens (Figures 1-7, and yes, I forgot to clean my PSVR’s lens). The camera was carefully positioned and aligned such that it was sliding back along the lens’s optical axis, and looking straight ahead. The first image was captured with an eye relief value of 0mm, meaning that the camera lens was touching the headset’s lens. The rest of the images were captured with increasing eye relief values, or lens-lens distances, of 5mm, 10mm, 15mm, 20mm, 25mm, and 30mm: Continue reading

KeckCAVES On Mars, pt. Oh-I-lost-count

Last weekend, we had yet another professional film crew visiting us to shoot video about our involvement in NASA’s still on-going Mars Science Laboratory (MSL, aka Curiosity rover) mission. This time, they were here to film parts of an upcoming 90-minute special about Mars exploration for the National Geographic TV channel. Like last time, the “star” of the show was Dawn Sumner, faculty in the UC Davis Department of Earth and Planetary Sciences, one of the founding members of KeckCAVES, and member of the MSL science team.

Unlike last time, we did not film in the KeckCAVES facility itself (due to the demise of our CAVE), but in the UC Davis ModLab. ModLab is part of an entirely different unit — UC Davis’s Digital Humanities initiative — but we are working closely with them on VR development, they have a nice VR environment consisting of two HTC Vive headsets and a large 4.2m x 2.4m screen with a ceiling-mounted ultra-short throw projector (see Figure 1), their VR hardware is running our VR software, and they were kind enough to let us use their space.

Figure 1: Preparation for filming in UC Davis’s ModLab, showing its 4.2m x 2.4m front-projected screen and ceiling-mounted ultra-short throw projector, and two Lighthouse base stations.

The fundamental idea here was to use several 3D models, created or reconstructed from real data sent back either by satellites orbiting Mars or by the Curiosity rover itself, as backdrops to let Dawn talk about the goals and results of the MSL mission, and her personal involvement in it. Figure 1 shows a backdrop in the real sense of the word, i.e., a 2D picture (a photo taken by Curiosity’s mast camera) with someone standing in front of it, but that was not the idea here (we didn’t end up using that photo). Instead, Dawn was talking while wearing a VR headset and interacting with the 3D models she was talking about, with a secondary view of the virtual world, from the point of view of the film camera, shown on the big screen behind her. More on that later. Continue reading

New Adventures in Hi-Fi

I’ve been spending all of my time over the last few weeks completely rewriting Vrui‘s collaboration infrastructure (VCI from now on), from scratch. VCI is, in a nutshell, the built-in remote collaboration / tele-presence component of my VR toolkit. Or, in other words, a networked multi-player framework. The old VCI was the technology underlying videos such as this one:

Figure 1: Collaborative exploration of a 3D CAT scan of a microbial community, between a CAVE and a 3D TV with head-tracked glasses and a tracked controller.

Continue reading

Hilarity Ensues

A funny thing has been happening for the last few days. There is a link farming or SEO or whatever site out there that is currently doing a deep clone of this blog (I’m obviously not going to provide a link here), but unlike all the other times that’s happened, these guys aren’t just straight-up copying, they’re running the text through what looks like a game of Google Translate Telephone (or a simple synonym replacer, but I’m going to pretend it’s the former). I’m guessing the reason is to obfuscate the source and throw off plagiarism detectors, but the result is unintentionally(?) hilarious.

For example, my post title “The Display Resolution of Head-mounted Displays” becomes “The Show Decision of Head-mounted Shows,” and “How Does VR Create the Illusion of Reality” becomes “How Does VR Create the Phantasm of Actuality.”

For a fuller experience, here are two paragraphs from the first article I mentioned above:

“The quick reply is after all that this relies on your mannequin of the headset. However should you occur to have an HTC Vive, view the graphs in footage 1 and a couple of (the opposite headsets behave in the identical method, however the precise numbers differ). These figures present the display decision, in pixels / °, alongside two traces (horizontal and vertical, respectively) that undergo the middle of the suitable lens of my very own Vive. The purple, inexperienced and blue curves present the decision for the purple, inexperienced and blue main colours respectively, this time not on the idea of my very own measurements, however by analyzing the show calibration knowledge measured for every particular person headset on the manufacturing unit after which saved within the firmware.

At this level it’s possible you’ll surprise why these graphs look so unusual, however for that you’ll have to learn the lengthy reply. Earlier than I’m going into that, I need to throw away a single quantity: proper in the course of the suitable lens of my Vive (on pixel 492, 602) the decision for the inexperienced shade channel is 11.42 pixels / °, each horizontal and vertical instructions. When you needed to cite a single decision quantity for headphones, that will be the one I used to be going to, as a result of it’s what you get once you have a look at one thing instantly in entrance of you and much away. As figures 1 and a couple of clearly present, no quantity can inform the entire story.”

Now, having posted this article, I am waiting with bated breath for what kind of hash their cloning bot will make out of it.

Welcome New VR Users!

Apparently, there were good sales numbers for VR equipment prior to the holiday season, and therefore a host of new VR users are coming in just about now. This meta-post collects a bunch of stuff I’ve written (or presented) in the past that might be of interest to some of those new users. These questions/answers are not hardware-specific, meaning they apply to any current-generation VR system (Oculus Rift, HTC Vive, all the Windows Mixed Reality headsets, PlayStation VR, …), and go beyond basic tech questions such as “how do I plug this in, install drivers, …).

There is one other issue for which I do not have a full article, but it’s quite important for new users: VR sickness (aka motion sickness, simulator sickness, …). Today’s VR headsets, at least the ones doing full head tracking (that means Rift/Vive et al., and not Gear VR, Oculus Go, Google Cardboard, …) should not cause VR sickness per se. These days, it is primarily caused by artificial locomotion in games or applications, as I explain in the second presentation I linked above.

The important message is: do not attempt to fight through VR sickness! If you try to stomach it out, it will only get worse. Stop using VR the moment you feel the first symptoms, take a long break, and then try again if you want to continue with the application/game that made you sick. If you try to power through repeatedly, your body might learn to associate sickness with VR, and that might cause you to get sick even when merely thinking about VR, or smelling the headset, or similar triggers. Just don’t do it.

That’s about it; now go ahead and enjoy your shiny new VR systems!

Want to Know More?

Here are a couple of other, more hardware-specific, topics:

A Milestone

Good news, everyone! This here blog, Doc-Ok.org, hit 1,000,000 (that’s one million) total page views early in the morning of Christmas Day, 12/25/2018, just about six years and four months after I started it.

I have no idea whether that’s something to be embarrassed or proud of, given the narrow range of topics I’m covering and the niche audience I’m targeting. Either way, it’s a nice, round number.