About okreylos

I am a research computer scientist at the University of California, Davis. My research areas are scientific visualization, particularly in immersive ("virtual reality") environments, human/computer interaction in immersive environments, and 3D computer graphics. My primary work is software development, from architecture over design to implementation and coding. I am the primary software developer for the UC Davis W.M. Keck Center for Active Visualization in the Earth Sciences (KeckCAVES). Some of my released packages are Vrui (a VR development toolkit), CollaborationInfrastructure (a tele-collaboration plug-in for Vrui), Kinect (a driver suite to capture 3D video from Microsoft Kinect cameras), LiDAR Viewer (a visualization package for very large 3D point clouds), 3D Visualizer (a system for interactive visual analysis of 3D volumetric data), Nanotech Construction Kit (an interactive molecular design program), and SARndbox (an augmented reality sandbox). I also dabble in VR hardware, in the sense that I take existing custom or commodity hardware components (3D TVs, head-mounted displays, projectors, tracking systems, Wiimotes, Kinect cameras, ...) and build fully integrated immersive environments out of them. This includes a fair share of driver development to support hardware that either doesn't have drivers, or whose vendor-supplied drivers are not up to par.

Are Math Textbooks Written by People who Hate Math?

Now that I’m basically home-schooling my daughter due to The Lockdown, I’m realizing how ridiculous math textbooks and workbooks are. Who writes these things / creates these problem sets? Today’s homework assignment had these nuggets in it:

“Kelly subtracted 2.3 from 20 and got 17.7. Explain why this answer is reasonable.”

The obvious answer is “because it is correct.” But that would get the student zero points. The expected (I assume) answer is about number sense / estimation, e.g., “If I subtract 2 from 20 I get 18, but I have to subtract a little bit more, and 17.7 is a little bit less than 18, so 17.7 is a reasonable answer.” Now my issue with this problem is that the actual arithmetic is so simple that it is arguably easier to do just do it than it is to go the estimation route. The problem sets the students up for failure, and undercuts the point of the unit: that estimation is a valuable tool. A better problem would have used numbers with more digits to hint that the students were supposed to estimate the result instead of calculating it, and to show that estimation saves time and effort.

“At a local swim meet, the second-place swimmer of the 100-m freestyle had a time of 9.33 sec. …”

This one made me laugh out loud, and I’m not even a sports fan who follows swimming. But even I know that swimming is a lot slower than running, and upon checking, I found that the world record for the 100m freestyle is 46.91 seconds. Who was competing in this “local swim meet?” Aquaman? My issue here is that the problem creator failed to understand the reason for using this type of word problem: reinforcing the important notion that math is important in the real world. But by choosing these laughable numbers, the creator not only undercut that notion, but created exactly the opposite impression in the students: that math has no relationship to the real world.

And from today’s section of the textbook, this table:

LocationRainfall amount in a
typical year (in inches)
Macon, GA 45
Boise, ID 12.19
Caribou, ME 37.44
Springfield, MO 44.97

Followed by this question: “What is the typical yearly rainfall for all four cities?” The book expects 139.6 inches as the answer, but that answer makes no sense. Rainfall amounts measured in inches can not be added up between multiple locations, because they are ratios, specifically volume of rain per area. How is that supposed to work? Stacking the four cities on top of each other? As in the previous example, this problem undercuts the goal of showing that math has a relationship to the real world. These students, being in fifth grade, wouldn’t necessarily realize the issue with this problem, but it really makes me think whether the person creating this example has advanced beyond fifth grade. Or, even worse, if that person is actively trying to create the impression that math is just some numbers game that happens in a vacuum. If so, good job.

My daughter was actually stumped by this last one, having no idea what the book meant by “typical yearly rainfall for all four cities,” and I had to explain to her that the question makes no sense, and reassure her that math is important, even if the math textbook goes out of its way to teach the students that math is frustrating, incomprehensible, and has no point. Again, good job, textbook writers.

In violation of Betteridge’s Law, I will answer the question posed in this post’s headline with a resounding “YES!

Idle Hands etc. etc.

A friendly redditor sent me this link to a popular post on /r/funny yesterday (see Figure 1 for the picture). I might have mentioned before how it was that exact scene in the original Star Wars movie that got me into 3D computer graphics and later VR, so it got me thinking how that particular shot would have looked like if the miniature ILM used to film the trench run scene had not been flat, but exhibited the proper scaled curvature of the Death Star.

Figure 1: Death Star and trench from attack scene in A New Hope, showing the flat miniature that was used to shoot the scene. Source.

Two hours and 153 lines of code later, here are a couple of images which are hopefully true to scale. I used 160km as the Death Star’s diameter, based on its Wookiepedia entry (Wikipedia states 120km, but I’m siding with the bigger nerds here), and I assumed the meridian trench’s width and depth to be 50m, based on the size of an X-Wing fighter and shot compositions from the movie.

Side note: I don’t know how common this misconception is, but the trench featured in the trench run scenes is not the equatorial trench prominently visible in Figure 1. That one holds massive hangars (as seen in the scene where the Millennium Falcon is tractor-beamed into the Death Star) and is vastly larger than the actual trench, with is a meridian (north-south facing) trench on the Death Star’s northern hemisphere, as clearly visible on-screen during the pre-attack briefing (but then, who ever pays attention in briefings).

The images in Figures 2-6 are 3840×2160 pixels. Right-click and select “View Image” to see them at full size.

Figure 2: Meridian trench on spherical Death Star, approx. 12.5m above trench floor.Horizon distance: 1.4km.
Figure 3: Meridian trench on spherical Death Star, approx. 25m above trench floor.Horizon distance: 2km.
Figure 4: Meridian trench on spherical Death Star, approx. 37.5m above trench floor.Horizon distance: 2.5km.
Figure 5: Meridian trench on spherical Death Star, precisely at Death Star’s surface.Horizon distance: 2.8km.
Figure 6: Meridian trench on spherical Death Star, approx. 100m above Death Star’s surface.Horizon distance: 4.9km.

As can be seen from Figures 2-6, the difference between the flat miniature used in the movie, and the spherical model I used, is relatively minor, but noticeable — ignoring the glaring lack of greebles in my model, obviously. I noticed the lack of curvature for the first time while re-watching A New Hope when the prequels came out, but can’t say I ever cared. Still, this was a good opportunity for some recreational coding.

Is TCP really that slow?

I’m still working on Vrui’s second-generation collaboration / tele-presence infrastructure (which is coming along nicely, thankyouverymuch), and I also recently started working with another group of researchers who are trying to achieve similar goals, but have some issues with their own home-grown network system, which is based on Open Sound Control (OSC). I did some background research on OSC this morning, and ran into several instances of an old pet peeve of mine: the relative performance of UDP vs TCP. Actually, I was trying to find out whether OSC communicates over UDP or TCP, and whether there is a way to choose between those at run-time, but most sources that turned up were about performance (it turns out OSC simply doesn’t do TCP).

Here are some quotes from one article I found: “I was initially hoping to use UDP because latency is important…” “I haven’t been able to fully test using TCP yet, but I’m hopeful that the trade-off in latency won’t be too bad.”

Here are quotes from another article: “UDP has it’s [sic] uses. It’s relatively fast (compared with TCP/IP).” “TCP/IP would be a poor substitute [for UDP], with it’s [sic] latency and error-checking and resend-on-fail…” “[UDP] can be broadcast across an entire network easily.” “Repeat that for multiple players sharing a game, and you’ve got a pretty slow, unresponsive game. Compared to TCP/IP then UDP is fast.” “For UDP’s strengths as a high-volume, high-speed transport layer…” “Sending data via TCP/IP has an ‘overhead’ but at least you know your data has reached its destination.” “… if the response time [over TCP] was as much as a few hundred milliseconds, the end result would be no different!”

Continue reading

A Question About VR Headset Resolution

I received a question via reddit a few moments ago, and I think the answer might be of general interest, so I decided to answer it here:

“Would you happen to know the effective or perceived resolution of the [Valve Index headset] when viewing a 50″ virtual screen from say.. 5 feet away? Do you think its equivalent to a 50″ 1080p tv from 5 ft away yet? I was also wondering why when I look at close up objects on the index that I can see basically no screen door effect, but when looking into the distance at the sky then suddenly the sde becomes very noticeable.”

Okay, so that’s actually two questions. Let’s start with the first one, and do the math.

The first thing we have to figure out is the resolution of a 50″ 1080p TV from 5 feet away. That’s pretty straightforward: a 1080p TV has 1920 pixels horizontally and 1080 pixels vertically. Meaning, it has √(19202 + 10802) = 2202.9 pixels along the diagonal, and – assuming the pixels are square – a pixel size of 50″/2202.9 = 0.0227″. Next we have to figure out the angle subtended by one of those pixels, when seen from 5 feet away. That’s α = tan-1(0.0227″/(5⋅12″)) = 0.0217°. Inverting that number yields the TV’s resolution as 46.14 pixels/°.

Figuring out a VR headset’s resolution is more complex, and I still haven’t measured a Valve Index, but I estimate its resolution in the forward direction somewhere around 15 pixels/°. That means the resolution of the hypothetical 50″ TV, viewed from 5 feet away, is approximately three times as high as the resolution of a Valve Index. The interested reader can simulate the perceived resolution of a VR headset of known resolution by following the steps in this article.

The second question is about screen-door effect (SDE). As shown in Figure 1, SDE is a high-frequency grid superimposed over a low-frequency (low-resolution) pixel grid, which makes it so noticeable and annoying. But why does it become less noticeable or even disappear when viewing virtual objects that are close to the viewer? That’s vergence-accommodation conflict rearing its typically ugly, but in this case beneficial, head. When viewing a close-by virtual object, the viewer’s eyes accommodate to focus on a close distance, but the virtual image shown by the VR headset is still at its fixed distance, somewhere around 1.5‒2m away depending on headset model. Meaning, the image will be somewhat blurred, and SDE, being a high-frequency signal, will be affected much more than the lower-frequency actual image signal.

Figure 1: Low resolution vs. screen-door effect (SDE). (Right-click and “view image” to see in full resolution.)

A Clarification About “Black Smear”

Here’s another frequently-asked question about VR headsets, or at least those that use LED-based displays:

Why does my headset show dark grey when it’s supposed to show black? Shouldn’t LED displays be able to show perfect blacks?

I addressed this in detail a long time ago, but the question keeps popping up, and it is often answered like the following: “LED display pixels have a memory effect when they are turned off completely, which causes ‘black smear.’ This can be avoided by never turning them off completely.”

Unfortunately, that answer is mostly wrong. LED display pixels do have a memory effect (for reasons too deep to get into right now), but it is not due to being turned off completely. The obvious counter argument is that, in the low-persistence displays used in all LED-based headsets, all display pixels are completely turned off for around 90% of the time anyway, no matter how brightly they are turned on during their short duty cycle. That’s what “low persistence” means. So having them completely turned off during their 1ms or so duty cycles as well won’t suddenly cause a memory effect.

The real answer is mathematics. In a slightly simplified model, the memory effect of LED displays has the following structure: if some pixel is set to brightness b1 in one frame, and set to brightness b2 in the next frame, it will only “move” by a certain fraction of the difference, i.e., its resulting effective brightness in the next frame will not be b2 = b1 + (b2 – b1), but b2‘ = b1 + (b2 – b1)⋅s, where s, the “smear factor,” is a number between zero and one (it’s usually around 0.9 or so).

For example, if b1 was 0.1 (let’s measure brightness from 0 = completely off to 1 = fully lit), b2 is 0.7, and s = 0.8, then the pixel’s effective brightness in frame 2 is b2‘ = 0.1 + (0.7 – 0.1)⋅0.8 = 0.58, so too dark by 17%. This manifests as a darkening of bright objects that move into previously dark areas (“black smear”). The opposite holds, too: if the pixel’s original brightness was b1 = 0.7, and its new intended brightness is b2 = 0.1, its effective new brightness is b2‘ = 0.7 + (0.1 – 0.7)⋅0.8 = 0.22, so too bright by 120%(!). This manifests as bright trails following bright objects moving over dark backgrounds (“white smear”).

The solution to black and white smear is to “overdrive” pixels from one frame to the next. If a pixel’s old brightness is b1, and its intended new brightness is b2, instead of setting the pixel to b2, it is set to an “overdrive brightness” bo calculated by solving the smear formula for value b2, where b2‘ is now the intended brightness: bo = (b2 – b1)/s + b1.

Let’s work through the two examples I used above: First, from dark to bright: b1 = 0.1, b2 = 0.7, and s = 0.8. That yields bo = (0.7 – 0.1)/0.8 + 0.1 = 0.85. Plugging bo = 0.85 into the smear formula as b2 yields b2‘ = 0.1 + (0.85 – 0.1)⋅0.8 = 0.7, as intended. Second, going from bright to dark: b1 = 0.7, b2 = 0.1, and s = 0.8 yields bo = (0.1 – 0.7)/0.8 + 0.7 = -0.05. Oops. In order to force a pixel that had brightness 0.7 on one frame to brightness 0.1 on the next frame, we would need to set the pixel’s brightness to a negative value. But that can’t be done, because pixel brightness values are limited to the interval [0, 1]. Ay, there’s the rub.

This is a fundamental issue, but there’s a workaround. If the range of intended pixel brightness values is limited from the full range of [0, 1] to the range [bmin, bmax], such that going from bmin to bmax will yield an overdrive brightness bo = 1, and going from bmax to bmin will yield an overdrive brightness bo = 0, then black and white smear can be fully corrected. The price for this workaround is paid on both ends of the range: the high brightness values (bmax, 1] can’t be used, meaning the display is a tad darker than physically possible (a negligible issue with bright LEDs), and the low brightness values [0, bmin) can’t be used, which is a bigger problem because it significantly reduces contrast ratio, which is a big selling point of LED displays in the first place, and means that surfaces intended to be completely black, such as night skies, will show up as dark grey.

Let’s close by working out bmin and bmax, which only depend on the smear factor s and can be derived from the two directions of the overdrive formula: 1 = (bmax – bmin)/s + bmin and 0 = (bmin – bmax)/s + bmax. Solving yields bmin = (1 – s)/(2 – s) and bmax = 1/(2 – s). Checking these results by calculating the overdrive values to go from bmin to bmax, which should be 1, and from bmax to bmin, which should be 0, is left as an exercise to the reader.

In a realistic example, using a smear factor of 0.9, the usable brightness range works out to [0.09, 0.91], meaning the darkest the display can be is 9% grey.

Quantitative Comparison of VR Headset Fields of View

Although I’ve taken many through-the-lens pictures of several common VR headsets with a calibrated wide-angle camera, until recently I was still struggling how to compare the resulting fields of view (FoV) quantitatively, how to put them in context, and how to visualize them appropriately. When trying to answer questions like “which VR headset has a bigger FoV?” or “by how much is headset A’s FoV bigger than headset B’s?” or “how does headset C’s FoV compare to the field of vision of the naked human eye?”, the basic question is: how does one even measure field of view, in a way that is fair and allows comparison across a wide range of possible sizes. Does one report a single angle, and how does one measure it? Across the “diagonal” of the field of view? What if the field of view is not a rectangle? Does one report a pair of angles, say horizontal⨉vertical? Again, what if the field of view is not a rectangle?

Then, if FoV is measured either as a single angle or a pair of angles, how does one compare different FoVs fairly? If one headset has 100° FoV, and another has 110°, does the latter show 10% more of a virtual 3D environment? What if one has 100°⨉100° and another has 110°⨉110°, does the latter show 21% more?

To find a reasonable answer, let’s go back to the basics: what does FoV actually measure? The general idea is that FoV measures how much of a virtual 3D environment a user can see at any given instant, meaning, without moving their head. A larger FoV value should mean that a user can see more, and, ideally, an FoV value that is twice as large should mean that a user can see twice as much.

Now, what does it mean that “something can be seen?” We can see something if light from that something reaches our eye, enters the eye through the cornea, pupil, and lens, and finally hits the retina. In principle, light travels towards our eyes from all possible directions, but only some of those directions end up on the retina due to various obstructions (we can’t see behind our heads, for example). So a reasonable measure of field of view (for one eye) would be the total number of different 3D directions from which light reaches that eye’s retina. The problem is that there is an infinite number of different directions from which light can arrive, so simple counting does not work.

Continue reading

How to Track Glowing Balls in 3D

Note: I started writing this article in June 2017, because people kept asking me about details of the PS Move tracking algorithm I implemented for the video in Figure 1. But I never finished it because I couldn’t find the time to do all the measurements needed for a thorough error analysis, and also because the derivation of the linear system at the core of the algorithm really needed some explanatory diagrams, and those take a lot of work. So the article stayed on the shelf. I’m finally publishing it today, without error analysis or diagrams, because people are still asking me about details of the algorithm, more than four years after I published the video. 🙂

This one is long overdue. Back in 2015, on September 30th to be precise, I uploaded a video showing preliminary results from a surprisingly robust optical 3D tracking algorithm I had cooked up specifically to track PS Move controllers using a standard webcam (see Figure 1).

Figure 1: A video showing my PS Move tracking algorithm, and my surprised face.

During discussion of that video, I promised to write up the algorithm I used, and to release source code. But as it sometimes happens, I didn’t do either. I was just reminded of that by an email I received from one of the PS Move API developers. So, almost two years late, here is a description of the algorithm. Given that PSVR is now being sold in stores, and that PS Move controllers are more wide-spread than ever, and given that the algorithm is interesting in its own right, it might still be useful. Continue reading

Visual Simulation of the Resolution of VR Headsets

While I was writing up my article on the PlayStation VR headset’s optical properties yesterday, and specifically when I made the example images for the sub-section about sub-pixel layout comparing RGB Stripe and PenTile RGBG displays, it occurred to me that I could use those images to create a rough and simple simulator to visually evaluate the differences between VR headsets that have different resolutions and sub-pixel layouts.

The basic idea is straightforward: Take a test image that has some pixel count, for example WxH=640×360 as the initial low-resolution full-RGB picture seen in Figure 1. If you then blow up that image to fill a monitor that has the same aspect ratio (16:9) and some diagonal size D, the resolution of that image in terms of pixels per degree depends both on D and the viewer’s distance from the monitor Y: the larger the ratio Y/D, the higher is the image’s resolution as seen by the viewer. In detail, the formula for distance Y to achieve a desired resolution R in pixels/° is:

Y = (D / sqrt(W*W + H*H))/(2*tan(1/(2*R)))

where W and H are in pixel units, R is in pixels/°, and D is in some arbitrary length unit (inch, meter, parsec,…). Y will end up being in the same unit as D.

If a viewer then positions one of their eyes at a distance of Y from the center of the monitor and closes the other one, the resolution of the image on the monitor will be R. In other words, if R is the known resolution of some VR headset, the image on the monitor will look the same resolution as that VR headset.

There is one caveat: when looking at a flat monitor, the resolution of the displayed image increases away from the monitor’s center, while in a VR headset, the resolution generally decreases away from the center direction (see this article for reference). Meaning, for a correct evaluation, the viewer has to focus on the area in the center of the monitor. Unfortunately there is no easy way to simulate resolution drop-off using a flat monitor, at least not while also simulating sub-pixel layout.

Simulation Procedure

Here’s what you need to do: Continue reading

Field of View and Resolution of the PlayStation VR Headset

It has been a very long time since I did the original optical measurement of then-current VR headsets. I have owned a PlayStation VR headset (PSVR from now on) for almost a year now, and I finally got around to measuring its optical properties in the same way. I also developed a new camera calibration algorithm (that’s a topic for another post), meaning I am even more confident in my measurements now than I was then.

One approach to measuring the optical properties of a VR headset, which includes measuring its field of view, its resolution in pixels/°, and its lens distortion correction profile, is to take a series of pictures of the headset’s screen(s) through its lenses using a calibrated wide-angle camera. In this context, a calibrated camera is one where each image pixel’s horizontal and vertical angles away from the optical axis are precisely known.

If one then displays a test pattern that lets one identify a particular pixel on the screen, one can measure the viewer-relative angular position of that pixel in the camera image, which is all the information needed to generate the projection matrices and lens distortion correction formulas that are essential to high-quality VR rendering.

Without further ado, here is a series of 7 images taken with the camera lens at increasing distances from the headset’s right lens (Figures 1-7, and yes, I forgot to clean my PSVR’s lens). The camera was carefully positioned and aligned such that it was sliding back along the lens’s optical axis, and looking straight ahead. The first image was captured with an eye relief value of 0mm, meaning that the camera lens was touching the headset’s lens. The rest of the images were captured with increasing eye relief values, or lens-lens distances, of 5mm, 10mm, 15mm, 20mm, 25mm, and 30mm: Continue reading

KeckCAVES On Mars, pt. Oh-I-lost-count

Last weekend, we had yet another professional film crew visiting us to shoot video about our involvement in NASA’s still on-going Mars Science Laboratory (MSL, aka Curiosity rover) mission. This time, they were here to film parts of an upcoming 90-minute special about Mars exploration for the National Geographic TV channel. Like last time, the “star” of the show was Dawn Sumner, faculty in the UC Davis Department of Earth and Planetary Sciences, one of the founding members of KeckCAVES, and member of the MSL science team.

Unlike last time, we did not film in the KeckCAVES facility itself (due to the demise of our CAVE), but in the UC Davis ModLab. ModLab is part of an entirely different unit — UC Davis’s Digital Humanities initiative — but we are working closely with them on VR development, they have a nice VR environment consisting of two HTC Vive headsets and a large 4.2m x 2.4m screen with a ceiling-mounted ultra-short throw projector (see Figure 1), their VR hardware is running our VR software, and they were kind enough to let us use their space.

Figure 1: Preparation for filming in UC Davis’s ModLab, showing its 4.2m x 2.4m front-projected screen and ceiling-mounted ultra-short throw projector, and two Lighthouse base stations.

The fundamental idea here was to use several 3D models, created or reconstructed from real data sent back either by satellites orbiting Mars or by the Curiosity rover itself, as backdrops to let Dawn talk about the goals and results of the MSL mission, and her personal involvement in it. Figure 1 shows a backdrop in the real sense of the word, i.e., a 2D picture (a photo taken by Curiosity’s mast camera) with someone standing in front of it, but that was not the idea here (we didn’t end up using that photo). Instead, Dawn was talking while wearing a VR headset and interacting with the 3D models she was talking about, with a secondary view of the virtual world, from the point of view of the film camera, shown on the big screen behind her. More on that later. Continue reading