Here’s another frequently-asked question about VR headsets, or at least those that use LED-based displays:
Why does my headset show dark grey when it’s supposed to show black? Shouldn’t LED displays be able to show perfect blacks?
I addressed this in detail a long time ago, but the question keeps popping up, and it is often answered like the following: “LED display pixels have a memory effect when they are turned off completely, which causes ‘black smear.’ This can be avoided by never turning them off completely.”
Unfortunately, that answer is mostly wrong. LED display pixels do have a memory effect (for reasons too deep to get into right now), but it is not due to being turned off completely. The obvious counter argument is that, in the low-persistence displays used in all LED-based headsets, all display pixels are completely turned off for around 90% of the time anyway, no matter how brightly they are turned on during their short duty cycle. That’s what “low persistence” means. So having them completely turned off during their 1ms or so duty cycles as well won’t suddenly cause a memory effect.
The real answer is mathematics. In a slightly simplified model, the memory effect of LED displays has the following structure: if some pixel is set to brightness b1 in one frame, and set to brightness b2 in the next frame, it will only “move” by a certain fraction of the difference, i.e., its resulting effective brightness in the next frame will not be b2 = b1 + (b2 – b1), but b2‘ = b1 + (b2 – b1)⋅s, where s, the “smear factor,” is a number between zero and one (it’s usually around 0.9 or so).
For example, if b1 was 0.1 (let’s measure brightness from 0 = completely off to 1 = fully lit), b2 is 0.7, and s = 0.8, then the pixel’s effective brightness in frame 2 is b2‘ = 0.1 + (0.7 – 0.1)⋅0.8 = 0.58, so too dark by 17%. This manifests as a darkening of bright objects that move into previously dark areas (“black smear”). The opposite holds, too: if the pixel’s original brightness was b1 = 0.7, and its new intended brightness is b2 = 0.1, its effective new brightness is b2‘ = 0.7 + (0.1 – 0.7)⋅0.8 = 0.22, so too bright by 120%(!). This manifests as bright trails following bright objects moving over dark backgrounds (“white smear”).
The solution to black and white smear is to “overdrive” pixels from one frame to the next. If a pixel’s old brightness is b1, and its intended new brightness is b2, instead of setting the pixel to b2, it is set to an “overdrive brightness” bo calculated by solving the smear formula for value b2, where b2‘ is now the intended brightness: bo = (b2 – b1)/s + b1.
Let’s work through the two examples I used above: First, from dark to bright: b1 = 0.1, b2 = 0.7, and s = 0.8. That yields bo = (0.7 – 0.1)/0.8 + 0.1 = 0.85. Plugging bo = 0.85 into the smear formula as b2 yields b2‘ = 0.1 + (0.85 – 0.1)⋅0.8 = 0.7, as intended. Second, going from bright to dark: b1 = 0.7, b2 = 0.1, and s = 0.8 yields bo = (0.1 – 0.7)/0.8 + 0.7 = -0.05. Oops. In order to force a pixel that had brightness 0.7 on one frame to brightness 0.1 on the next frame, we would need to set the pixel’s brightness to a negative value. But that can’t be done, because pixel brightness values are limited to the interval [0, 1]. Ay, there’s the rub.
This is a fundamental issue, but there’s a workaround. If the range of intended pixel brightness values is limited from the full range of [0, 1] to the range [bmin, bmax], such that going from bmin to bmax will yield an overdrive brightness bo = 1, and going from bmax to bmin will yield an overdrive brightness bo = 0, then black and white smear can be fully corrected. The price for this workaround is paid on both ends of the range: the high brightness values (bmax, 1] can’t be used, meaning the display is a tad darker than physically possible (a negligible issue with bright LEDs), and the low brightness values [0, bmin) can’t be used, which is a bigger problem because it significantly reduces contrast ratio, which is a big selling point of LED displays in the first place, and means that surfaces intended to be completely black, such as night skies, will show up as dark grey.
Let’s close by working out bmin and bmax, which only depend on the smear factor s and can be derived from the two directions of the overdrive formula: 1 = (bmax – bmin)/s + bmin and 0 = (bmin – bmax)/s + bmax. Solving yields bmin = (1 – s)/(2 – s) and bmax = 1/(2 – s). Checking these results by calculating the overdrive values to go from bmin to bmax, which should be 1, and from bmax to bmin, which should be 0, is left as an exercise to the reader.
In a realistic example, using a smear factor of 0.9, the usable brightness range works out to [0.09, 0.91], meaning the darkest the display can be is 9% grey.
Why is black smear only an issue with OLED but not LCD panels? Is it only due to the poor contrast ratio of LCD panels?
The real explanation requires a deep dive into electronics, but basically the problem lies in how individual pixel’s brightnesses are set by the display controller at the circuit level. The way it’s done for (AM)OLED displays is different than for LCDs.
What about LCoS? Since AR isn’t big yet would be nice to know whether we should expect this artifact in a future AR headset.
One can both drive LCD cells with negative values without damage (because they can switch bidirectionally), and the supermajority of LCD controllers already have pixel overdrive built in; meaning that yo do not need to futz with modifying the pixel values you send to the panel, and instead send the ‘true’ values to the panel and the panel controller will apply the appropriate offset itself before energising that LCD cell.
This is the source of an early ‘gotcha’ with DP Adaptive Sync & G-sync: when driving at variable update intervals, the pixel overdrive offset needed changes with both the time since last refresh AND the previous pixel value. The stonkingly expensive FPGA-based G-sync controller had sufficient memory to keep a record of past frames in order to calculate this in-panel, needing no modification of the input image. DP Adaptive Sync’s early implementations were utilising existing panel controllers’ available sync range (rather than controllers designed from the outset to vary refresh intervals dynamically) and which had no provision for this. This resulted in early DP Adaptive Sync displays exhibiting image smearing under variable-refresh-rate operation. This was ‘resolved’ for the first (and some second) gen monitors by having the AMD driver modify the pixel values sent to the panels, which corrected the majority of the smear.
Oh, and that because LCD panel switching time is decoupled from backlight energisation, you can start the pixel switching immediately after a backlight pulse, and then wait for as many of your pixels to complete transition as possible before pulsing the backlight again for the next display refresh, though at the cost of adding that wait time to the overall motion-photons latency. A direct tradeoff between colour accuracy and panel response time.
So you’re saying that memory effect happens at any pixel brightness, not just zero brightness, but the reason that zero brightness is a problem is because there’s no room for overdrive to correct the memory effect?
That’s an excellent way of putting it.
How does changing display brightness affect our ability to view a scene inside a VR headset.
Valve have introduced variable brightness with updates to the Index software, it’s possible to set from 0 to 140.
The obvious solution to the greyish blacks is to reduce brightness but this reduces contrast.
Some people have mentioned it’s better to set brightness to maximum due to contrast being more important than display brightness due to the way the eye works.
Can you explain more?
Changing brightness doesn’t actually affect contrast. Contrast is the ratio between the brightest and darkest colors that can be displayed. Increasing overall brightness increases both extremes by the same factor, which cancels out in the ratio. In addition, reducing overall brightness won’t make the darkest greys appear black, because our eyes will adjust to the reduced brightness (by widening the pupils, for example). We would perceive an overall brightness increase or decrease, but not a reduction in the brightness of the darkest greys that are being displayed.
Brightness actually helps with perceived contrast, but only if there are very bright and very dark sections in a frame. If the frame is mostly dim or mostly bright it won’t help. A good way to test this is with a video projector in a room with some ambient light: dark pixels appear darker than the part of the projection screen that isn’t projected onto at all if there are bright pixel regions surrounding the dark section. To test make a large black and white checkerboard pattern image, like 16×9 and view it with a projector, you will notice the black boxes seeming darker than the screen where it is not illuminated by the projector. I don’t know what this effect is called but it’s real.
That’s an effect from contrast enhancement / edge detection in the retina (technical term “lateral inhibition“), it is also independent of the actual brightness level of the overall image.
You mention ambient light, and that does factor into contrast perception, as it “locks” the brightness levels that are perceived as “normal.” But in VR headsets ambient light is mostly absent.
Just from subjective experience I would have to disagree that what I describe is independent of brightness. It may be at the retina level (before brain processing), but definitely my brain experiences it far more with mroe brightness.
But thanks for explaining what the effect is.
I mentioned the experiment with ambient light only because it’s the easiest way to experience this. It’s also experienced in a pitch black room where ambient light illuminating the screen may be about as much as the diffusely reflected light from your skin around your eyes through the lens back to the display panel in a VR headset.
“… may be about as much as the diffusely reflected light from your skin around your eyes through the lens back to the display panel in a VR headset.”
Note that the intensity of reflected display light in a headset is also proportional to the overall brightness of the headset, meaning it won’t affect perceived contrast like external ambient light of constant brightness would.
“Note that the intensity of reflected display light in a headset is also proportional to the overall brightness of the headset, meaning it won’t affect perceived contrast like external ambient light of constant brightness would.”
With video projectors in a pitch black room like in my example all of the ambient light is generated by the diffuse reflections of the projector beams reflected from the screen. But on the other hand LCD or OLED is much darker than most projection screens, so we are probably going on a tangent here.
Any ideas what causes a similar smear with some LCD monitors?
Even a 240Hz monitor I have leaves a smear the color of the moving pixel(s) behind it. Most noticeable with white pixels.
Is it due to liquid crystal switching speeds or circuit level cause?
Thanks.
I haven’t researched it in detail, but liquid crystal switching times are on the order of, or higher than, upper-tier monitor refresh rates. I’d say that smearing on LCDs is dominated by LC switching times. I have a range of monitors spanning more than 15 years of production at my various workstations, and can guess their ages just by how much smear there is.
LCDs driving circuits should also be quite different from active-matrix (AM) LED circuits, in which smear is caused by the level-holding capacitors. If LCoS is driven by the same type of AM circuitry, I would expect about the same level of smearing there.
So I’m still confused : I kinda get the memory effect and overdrive work. What I don’t understand is that since pixels are completely turned off between each frame on a low persistence display, I would expect b1 to ALWAYS be 0, since each frame starts from a fresh black screen.
Does that mean that some “memory” from the previous brightness gets carried over when the pixel is turned back on ?
I didn’t get into that in the post in detail because there are multiple implementations, but the gist of it is that there is memory, but not in the LEDs themselves, instead in their control logic. Showing an image on an AMOLED screen is divided into two phases: in the first phase, the brightness each pixel is supposed to show in the second phase is uploaded into a per-pixel circuit (hence “active matrix”); in the second phase, the pixel is actually turned on at its pre-selected brightness by supplying power to all LEDs at the same time.
It is the programming step that has memory. In the simplest circuit implementation, the per-pixel brightness is held as a charge in a small capacitor; during the display phase, the voltage level of the capacitor controls the amount of current flowing through the LED through a field-effect transistor. Setting a capacitor to a desired voltage is not immediate, and that’s where the memory effect comes from.