To my surprise and delight, I recently found out that Valve has been releasing Linux versions of most of their SteamVR/OpenVR run-time/SDK for a while now (OpenVR just hit version 1.0.0, go get it while it’s fresh). This is great news: it will allow me to port Vrui and all Vrui applications to the Vive headset and its tracked controllers in one fell swoop.
But before diving into developing a Lighthouse tracking driver plug-in for Vrui’s input device abstraction layer, I decided to cobble together a small testing utility to get a feel for OpenVR’s internal driver interface, and for the Lighthouse tracking system’s overall tracking quality.
Figure 1: The Lighthouse 6-DOF tracking system, disassembled (source).
The neat thing about OpenVR’s driver architecture is that it uses a well-defined — and published! — internal interface to talk to different tracking hardware, and that I can use that interface to read tracking data directly from Valve’s Lighthouse driver code without having to mess with client/server architectures. As an additional benefit, this also gives me access to all tracking data, not just the parts that Valve’s own vrserver sees fit for application consumption.
In a nutshell, my testing utility directly loads Valve’s driver_lighthouse.so dynamic library, which contains the sensor fusion and tracking code, and then calls a single exported function to retrieve a driver object. That driver object in turn offers several callbacks, such as one receiving button events, and, most importantly, one receiving tracking data updates as the tracking code calculates them from raw device sensor measurements. And that’s great.
So, let’s have a close look at Lighthouse tracking, specifically its implementation in the Vive Pre development kit, which is the one I happen to have.
Update rate and Latency
Not surprisingly, there is a lot of wild speculation and misinformation about Lighthouse’s update rate floating around. One known fact is the operating principle of the laser base stations that are the foundation for Lighthouse’s drift-free positional tracking. A quick refresher: each Lighthouse base station contains two lasers (see Figure 2). One laser projects a horizontal line of light (in the base station’s coordinate system) that sweeps through the tracking volume in front of the base station from bottom to top (when looking at the base station’s front); the other laser projects a vertical line of light that sweeps through the tracking volume from left to right (also when looking at the base station’s front).
Figure 2: Slow-motion video showing a Lighthouse base station in action (source).
Both lasers rotate around their respective axes at 3,600 rpm, or sixty revolutions per second. As there can (currently) only be one laser sweeping the tracking volume at any time, the two lasers inside one base station, and the four lasers of two linked base stations (A and B), are interleaved: First base station A’s vertical laser sweeps left to right; then, half a revolution or 8.333ms later, A’s horizontal laser sweeps bottom to top; then, another 8.333ms later, A’s lasers turn off and B’s vertical laser sweeps left to right; then after another 8.333ms B’s horizontal laser sweeps bottom to top; and finally B’s lasers turn off, A’s turn on, and the entire dance repeats from the beginning. To allow two (or potentially more) base stations to synchronize with each other, and to give tracked devices a way to measure their relative angles to each base station, each base station also contains an LED array that flashes a wide-angle synchronization pulse at the beginning of each 8.333ms laser sweep period.
This means that any tracked object (headset or controller) inside the tracking volume is hit by a laser every 8.333ms, or at a rate of 120Hz (potential occlusion notwithstanding). This might lead one to assume that Lighthouse’s tracking update rate is 120Hz, and that the worst-case tracking latency is 8.333ms (if an application happened to query a tracking pose just before one laser sweep hits a tracked object, the application would receive data that’s stale by a tad less than 8.333ms).
But this assumption does not bear out. When polled in a tight loop, SteamVR’s vrserver tracking server spits out a new pose (position and orientation) for each controller at a rate of 250Hz, and a new pose for the headset at a rate of 225Hz (that’s a strange number, and Alan Yates himself could not confirm it, but it’s what I measured). Digging even deeper, by talking directly to the sensor fusion code as I explained above, the update rate is even higher: a new pose is calculated for the headset at a rate of 1,006Hz (again, weird but measured), and for each controller at a rate of 366Hz (also measured). As an aside, unlike with Oculus’ Rift DK2, poses are not batched in sets of two, but delivered independently, more or less 0.994ms and 2.732ms apart, respectively.
My explanation is that these are the rates at which raw sensor samples, i.e., linear accelerations and angular velocities from each tracked object’s built-in inertial measurement units, and timing deltas for laser hits on each photodiode, arrive at the host, either via USB (for the headset) or wirelessly (for the controllers). The sensor fusion code inside driver_lighthouse.so then integrates these raw samples into a best guess for the current position and orientation of each device, and hands them off to the driver host via the aforementioned callback function. Apparently, vrserver subsequently bundles those measurements and sends them to VR applications over its client/server interface at a reduced rate, probably not to overwhelm the interface or applications with too much — and mostly redundant — data.
This means that at OpenVR’s internal interface, worst-case latency for head tracking data is about 1ms, and worst-case latency for controller tracking data is about 2.7ms, assuming that wire(-less) transmission and pose calculation add negligible latency. At vrserver’s client/server interface, on the other hand, worst-case latency is 4.444ms and 4ms, respectively (assuming my 225Hz and 250Hz measurements are correct).
Residual Noise or Tracking Jitter
The next important spec of a positional tracking system is residual noise, or the amount by which the reported position of a tracked device jitters when the device is actually fixed in space. I measured noise by placing the headset (see Figure 3) and one controller on a chair in the middle of my tracked space, which has two Lighthouse base stations about 2.4m above the floor, and about 4m apart.
Figure 3: Measuring residual headset tracking noise.
As can be seen in Figure 3, residual noise with two base stations is isotropic, and has a range of about 0.3mm. With only a single base station, the noise distribution turns highly anisotropic, with 0.3mm laterally to the remaining base station, and 2.1mm in the distance direction. The noise magnitude and distribution for controllers is very similar. This anisotropy is to be expected, as Lighthouse-based pose estimation boils down to the same perspective-n-point reconstruction problem that’s encountered in camera-based tracking and has lateral-to-camera error that grows linearly with distance, and distance-to-camera error that grows quadratically with distance.
Inertial Dead Reckoning and Drift Correction
As can already be deduced from the high tracking update rates described above, Lighthouse does not simply update a tracked object’s position and orientation when it is hit by a laser sweep. Instead, the current tracking estimate is primarily advanced by integrating linear acceleration and angular velocity measurements from each device’s built-in inertial measurement unit (IMU) via dead reckoning, and is updated at the rate at which samples arrive from said IMUs. The Lighthouse base stations merely control the build-up of positional and orientational drift that is inherent in integrating noisy and biased measurements, as explained in this video. Figure 4 shows the precise effect of drift build-up and drift correction on headset tracking data, and Figure 5 shows the same for controller tracking data.
Figure 4: Inertial dead reckoning, drift build-up, and drift correction in headset tracking.
Figure 5: Inertial dead reckoning, drift build-up, and drift correction in controller tracking.
Partial Interleaved Drift Correction
Due to its design of using two lasers per base station that sweep the tracking volume alternatingly left-to-right and bottom-to-top, Lighthouse adds an extra wrinkle to drift correction (see Figure 5). Unlike camera-based tracking systems such as Oculus’ Constellation, which measure the camera-relative X and Y position of all tracked markers at the same point in time, Lighthouse only measures a tracked object’s photodiodes’ X positions during a left-to-right sweep, and their Y positions during a bottom-to-top sweep, 8.333ms offset from each other.
This has two major effects: For one, it adds complexity to the sensor fusion algorithm. Instead of constraining the pose estimate of a tracked object to the full pose computed by a single camera image taken at a single point (or very brief interval) of time, the sensor fusion code has to constrain the tracking solution in independent steps. Digging even deeper, it turns out that there isn’t even a partial (in X or Y) Lighthouse-derived pose at any time, as a device’s photodiodes are hit at different times as the laser sweeps through space. The device’s current (estimated) motion has to be taken into account to calculate a full pose estimate over at least one laser sweep period. Fortunately, widely-used sensor fusion algorithms such as the Kalman filter are in principle flexible enough to support such partial state updates.
A second effect is that the total amount of information provided by the Lighthouse system to the sensor fusion code is only half of what a camera-based system would provide at the same frame rate. Specifically, this means that, even though Lighthouse sweeps the tracking volume in intervals of 8.333ms or a rate of 120Hz, it only provides the same total amount of information as a camera-based system with capture frame rate of 60Hz, as the camera delivers X and Y positions of all tracked markers for each frame. Meaning, a dead-reckoning tracking system with Lighthouse drift correction running at 120Hz is not automatically twice as “good” as a dead-reckoning tracking system with camera-based drift correction running at 60Hz. To compare two such systems, one has to look in detail at actual tracking performance data (which I hope to do in a future post).
Precision and Accuracy
The final two parameters to investigate are precision, or how close multiple subsequent measurements of the same point in space are to each other, and accuracy, or how close a measurement of a point in space is to the actual position of that point in space. Both are important data points for 6-DOF tracking systems.
To measure them, I placed a 36″ long ruler onto the floor in the center of my tracked space, and measured the 3D position of each 1″ mark using a small probe tip I attached to one of the tracked controllers (the probe tip’s position in the controller’s local coordinate frame, which is essential for repeatable point measurements, was derived from a simple calibration procedure). I then compared the resulting set of 3D points to the “ideal” set of 3D points, generated by computing each mark’s theoretical position in some arbitrary coordinate system, by running a non-linear point set alignment algorithm (see Figure 6). The RMS distance between the two point sets, after alignment, was 1.9mm, an estimate of Lighthouse’s tracking accuracy.
Figure 6: Accuracy of Lighthouse 3D position data, measured by aligning a set of collected points against with their theoretical counterparts. Maximum residual distance is 2.4mm, RMS is 1.9mm. Measured object is a 36″ ruler, here viewed along its length. Green points: ideal points; purple points: measured points.
Figure 6 shows some non-linear distortion in the measured point set, but overall accuracy is very good. To judge precision, I compared the measured point set against a second measurement of the same points, yielding a slightly smaller RMS distance of 1.5mm.
As a practical result, it is therefore possible to use a Lighthouse controller with an attached and calibrated probe tip as a large-area 3D digitizer, with an expected accuracy of about 2mm.
Fantastic write up. Thank you for doing this!
nice work
another interesting point to analyse is the relation between jitter amount and distance from nearest basestation.
That should be approximately linear.
Glad you were able to take the lead on just loading the driver_lighthouse and run with it, and get such nice results. Looks like they haven’t rebuilt the Linux drivers yet with an incompatible set of headers yet: back on Window drivers, this is what we get to enjoy (haven’t checked to see if the new release synced things up yet) https://github.com/ValveSoftware/openvr/issues/122#issuecomment-221140946
I did run into exactly that after everything suddenly stopped working for no apparent reason due to a SteamVR auto-update, but was able to get the osvrsvvr beta that fixed it (somewhat). The 0.9.21 SDK update Joe Ludwig posted yesterday has the new driver version string already, but I haven’t tried it yet.
hello author, you write “and finally B’s lasers turn off, A’s turn on, and the entire dance repeats from the beginning”. I do not think there would be the case of lasers’ turing on/off.
Watch the video in Figure 2 again. The sequence is flash – X laser – flash – Y laser – flash – no X laser – flash – no Y laser. The other base station does it exactly the other way around.
Excellent write up and thank you very much for posting!
We have done our own tests, measuring the timing of the sweeps and flashes (hooking up an oscilloscope to a single light sensor). Here is a summary of what we found, only looking at the optical path.
With a single lighthouse station connected (the second unit is not powered on!), everything still works well and the sequence is:
Fa Ha Fa Va – repeat. Fa is a flash by unit a, Ha and Va are horizontal and vertical sweeps of unit a. If only one unit is connected the flashes are still 8.33 ms apart, so the headset and controllers are hit with a horizontal and vertical sweep (giving them an absolute 3D position lock) 60 times/sec. The two sweeping lasers are 180 deg out of phase in this case of only having one operational lighthouse unit.
With two units connected (a and b) the rotating lasers appear to spin at half the speed! As you point out in your write-up, there can be only one sweep of the volume at any given time. The sequence now goes like this:
FaFb Ha FaFb Va FaFb Hb FaFb Vb – repeat. The interesting part here is that both units provide a sync flash within about 1 ms of each other. I believe this is used by the headset and the controllers to realize there are 2 lighthouses in the volume (again, very elegant and hats off to the engineers behind this tech!). Here lasers Ha and Va are 90 deg out of phase, while lasers Ha and Hb are 180 deg out of phase.
The IMU data fusion is critical to this system as a whole working as well as it does. Keep in mind though that with two lighthouse units, your effective 3D position fix is ‘only’ 60 times/sec (since it needs both sweeps for a 3D position fix) and actually drops to 30 times/sec if one of the lighthouses is blocked from view.
Are you sure about that? Based on my understanding, the slow-motion video in Figure 2, and the time difference between drift correction jumps in the tracking data when I had a controller occluded from one base station, the lasers still rotate at 3600rpm, but are turned off on every other revolution.
Meaning the sequence for a single base station would be flash – h sweep (on) – flash – v sweep (on) – flash – h sweep (off) – flash – v sweep (off), and the h and v lasers of a single base station are still offset by 180°.
I stand corrected. Looking at the video it does not make sense that the laser rotation changes speed. So this video would be shot from a unit that is synced with another unit (two base stations active).
But, with only one unit powered up we saw a flash and a sweep every 8.33 ms period. With a single unit powered on the optical sensors registered:
F – H – F – V – F – H – F – V, the unit does not do a sweep with the lasers off.
If you power down one of the lighthouse units and collect the data again (very nicely done by the way), you should still see the drift correction jumps every 8.33 ms.
So basically the base stations change between two modes, depending on wehter or not they receive a sync signal from a second lighthouse?
Interesting, I would love to have that inspected further. Especially if they do this on the fly, each time they don’t get a sync signal?
The lighthouse units change mode for sure, but not on the fly. If a second unit is powered on, it will take seconds for them to a) realize there is another one that came on-line and then b) to synchronize their outputs (phase locking).
Very nice!
It should be interesting to see how it will compare with other trackers.
My hunch is that it will be more or less on par with Constellation. I already know it’s significantly better than anything else I’ve seen so far.
I assume you mean in consumer tracking… it’s nice but no ART…
Good point. I did some work with A.R.T. back in the day, and while its positional tracking quality was top-notch, it did have noticeable issues with orientational jitter (same thing holds for OptiTrack).
Using a virtual laser pointer was definitely hit-and-miss. I don’t know if newer A.R.T. trackables have built-in IMUs to reduce that jitter, but whenever I use a virtual laser pointer with a Vive wand, I’m amazed by how stable it is.
Yes, orientational jitter can lead to a bad experience in VR, especially when wearing an HMD or using a virtual laser pointer like described. With the rise of HMDs we have seen over the past years ART has started to address this issue by implementing improved filtering algorithms in the DTrack2 software. As a next major step a substantial solution for the rotational jitter problem is already in development. We expect to be able to launch it by end of the year 2016. Feel free to get in touch with ART at http://www.ar-tracking.com/home/
This is fantastic, thanks a lot. What do you think of the feasibility of integrating GearVR with a lighthouse system?
That should be pretty easy.
Nice! 🙂
I am interested in the performance in outdoor conditions. Do you think sunlight will significantly degrade the performance?
The sweeping lasers are modulated with a high-frequency signal to allow filtering out constant IR sources such as the Sun, but with enough ambient IR light to push the photodiodes into saturation, none of that will help.
I recently saw a video of someone using a Vive on top of their roof, in Sydney sunshine. So I guess it works at least somewhat.
Thanks for the reply. That makes sense.
I found Alan Yates’ comments:
https://www.reddit.com/r/Vive/comments/467bad/alan_yates_retweets_picture_of_someone_using_the/?
” …. I can’t recommend you operate the Vive in direct sunlight, but it will probably work at some loss of range.”
Interesting. I remember reading that Alan Yates has said it should work fine in sunlight, presumably because of the modulation. (Not sure why you’d use the hmd in sunlight, except maybe as a part of an AR thing, but if you’re outside you might also have a tricky time getting a nice rock solid mount for the bases, or at least the one labeled B, which is apparently the one that determines the coordinate system…. yes, I picked the wrong letter/channel to assign to the one i had to stick on a tripod instead of a wall mount, but I haven’t wanted to redo the room setup…)
I will concede that’s one place where those of us who work on active market IR computer vision based trackers are simply outmatched: if nothing else, that much IR ambient illumination makes extracting and filtering down to “legitimate” blobs more time consuming. But hey, that’s why lots of different tracking methods are nice to have around: different strengths and weaknesses.
“The sweeping lasers are modulated with a high-frequency signal to allow filtering out constant IR sources such as the Sun”
My understanding is that the BPW34 sensors filter out any MHz modulation of laser or flash, due to their rise/fall time. For the same reason, this generation of Lighthouse sensors/trackers (controllers/HMDs) cannot handle FDM.
nairol did a simulation (and measurements with an oscillsocope):
https://imgur.com/a/SVsX4
https://github.com/nairol/LighthouseRedox/tree/master/simulation/sensor
Alan Yates has repeatedly referred to a 2MHz modulation, and stated that carrier frequency would only increase in the future. But I have not seen any explanation as to what that modulation actually does for the Vive trackers. I also haven’t seen any measurement of the flash and/or laser signal with a scope that actually shows the modulation. Did you measure the sweep laser modulation, and was it at 2MHz?
I didn’t investigate laser modulation myself (I don’t have the equipment for that). I took that from a series of comments by Alan Yates.
Both the LEDs and the lasers modulate at 1.84 MHz. I have measured them with an oscilloscope, using my build of the circuit posted by Alan Yates elsewhere.
Thanks for confirming.
Any chance you want to post the code for your testing utility?
I second that request. I’m doing things client/server right now, and would like to get a better understanding of how you were able to take advantage of the driver architecture to directly access the tracking. I’m sure if I bang my head against the openvr codebase long enough I’ll figure it out, but I’d appreciate the jumpstart.
I recently found this very nice article. The measurements you showed here resulted in the idea to attach one of the controllers to an object I wanted to track and use the system as some kind of high precision in-room GPS.
Getting controller poses via the the client/server model was quite straight forward. However, I still haven’t managed to get the information via driver_lighthouse.so. Can you provide some more information/hints how your program interfaces with the driver_lighthouse.so? One really annyoing thing is that the headset have to be connected for the client/server to work. I’m hoping when accessing via driver_lighthouse.so this requirement can be dropped.
I’m sure it will eventually be released as a part of the next (eventual) VRUI release (Right, Oliver ;-D?) – but substantially similar code in ability to load the driver and make it do stuff is in OSVR-Vive https://github.com/OSVR/OSVR-Vive which you can either use as-is to run the Vive in an OSVR environment, or you can use the (Apache 2.0 licensed) pieces to do other unsupported things with the lighthouse driver. (If doing that breaks, though, you get to keep both pieces – don’t ruin OSVR-Vive and VRUI on Vive for everyone by complaining so much to Valve that they make it impossible for us to do this direct loading of the lighthouse driver…)
I’m wondering if it is possible to set up multiple base stations to enlarge the tracking area, for example make it a 10x10m area by using 8-10 base stations.
From my understanding if still using TDM the revolution speed should speed up or the data frequency would drop even more. Using FDM could possibly solve the problem however it requires more expensive parts.
I’m not sure it’s realistic to up the revolution speed, and without doing that, as you mentioned the data frequency would drop significantly with TDM. FDM is a good alternative, but (probably) would require changes to the base stations. Alternatively the combination of stock basestations with a secondary more coarse location sensor could resolve redundant readouts from the stock lighthouse sensors.
For instance if you had 3 base stations in a line ~5m apart configured as C-B-C, with no overlap of areas covered by the two base stations running the same channel (this is tricky) there could be 2 physical locations that would get the same “coordinates” from a channel C sensor. This would essentially be a spatial alias. There are several ways that this could be anti-aliased. If channel B is not occluded then the coordinates from B could be used to differentiate, but if that is not possible then a secondary system with lower resolution such as radio signal strength (RSS) could be used.
Let’s say for the sake of argument that this is a location on a plane 4′ off the ground on the axis that the sensors are mounted. Imagine that axis running left to right like an x axis. Let’s say that at x=1 and x=-1 on channel C a lighthouse sensor get’s the same coordinate information. If the tracked object had RSS sensing capability in addition to lighthouse sensors, and two radio beacons were installed at the same approximate location as the two basestations on channel C, then the RSS sensor could differentiate whether the object is at x=1 or x=-1.
Obviously, there are several hurdles to overcome in both software and hardware to achieve this. You’d have to have a system to manage the synchronization (by wire) of the base stations, implement an RSS system, create new drivers, and very carefully locate the basestations. However, in my opinion these are more practical than re-engineering the basestations and sensors themselves to increase revolutions speeds for TDM or implement FDM.
There are references in the driver to an “async” mode in addition to the “tdm” mode that normally is used. I have no idea how you’d go about using it, if it requires special firmware, etc – I assume it’s not “production ready” or otherwise not suitable for mass consumption or there’d be more about it, but TDM isn’t the only way that Lighthouse can function apparently.
I’m really curious about how anything except TDM would work hardware-wise. The base stations’ lasers are all the same color, and the photo diodes are color-blind. They could theoretically distinguish between base stations by the high-frequency signal that’s modulated onto the lasers, but from what I gather, the receptor units are hard-wired to latch onto a specific signal frequency via a resonance circuit, and therefore wouldn’t be able to tag laser pulses with base station IDs. Maybe there could be multiple circuits per receptor, akin to a filter bank, in future receptor hardware versions, or maybe there already are. I don’t know the hardware at that level.
Alan Yates has hinted that lighthouse has several modes and we’ve only primarily seen one. I thought I’ve seen setups from Valve with more than 3, and you can configure which mode the system runs in, iirc the two I’ve seen in the current code are tdm and “async”. I have no idea if publicly available base stations are usable in anything but the stock tdm mode. (I’ve only got the basic two base stations, and they cover my space fine so I haven’t needed to fiddle with that bit of the system 🙂 )
Great work! Thank you
How do you think is it possible, at this moment, to make a third-party device with use of Lighthouse tracking?
Both Oculus and Valve promised to open their tracking API for third-party developers, and none of them made it so far.
As of now, you’d have to reverse-engineer the internal protocol between tracked devices and the Lighthouse tracking driver on the host PC. That includes wireless communication, sensor data transfer, and in-device calibration data such as 3D photosensor locations in device space. If you had all that, the Lighthouse driver would possibly pick up and track your device just like an original Vive controller.
From a practical point of view, you’ll have to wait until Valve publish the necessary internal hardware and software protocols.
AFAIK, Valve are working on it (confirmed by Alan VK2ZAY), but when and how it will be released isn’t known yet. Meaningful SDK would likely need both the sw and some hw “kit”, because building the circuitry that is in one of the Vive wands is just not economical for one-offs, even though it is certainly possible if someone really wants to do it.
Excellent work, thank you for sharing. The need for a high performing IMU becomes very clear from your test. The dead reckoning error, i.e. the kinks that appear when the lighthouse positioning corrects for the gyro drift, may be reduced with a more stable gyroscope. BMI055 comes to my mind here as many VR developers report excellent results for similar applications. It may reduce the kinks especially since the lighthouse does not correct for all axes at a time as show. It is worth a try.
Pingback: Analysis of Valve's 'Lighthouse' Tracking System Reveals Accuracy - Road to VR
Can the Vive controllers be used by themselves without the headset?
I need to test that once I am set up again, but my guess is no, at least with the OpenVR version I tried. For one, the controllers talk to the computer through radios integrated into the headset. This could potentially be circumvented by plugging the controllers directly into USB ports, but I don’t know right now whether they are set up to send tracking data over USB. The PS Move controllers, for example, which also can be plugged in and programmed via USB, only send tracking data over wireless.
Second, Valve’s OpenVR Lighthouse driver might refuse to start up if no headset is detected.
Not sure, but it does seem like the sensors are receiving a fair bit of data about each base station over the laser pulse (easiest to verify with a Pre, where there is no communication channel available from the base stations except optical) – they get a serial number, firmware version, and gravity vector, IIRC. I think the tracked sensors only pick up the laser, and not the sync pulse (as evidenced by the fact you can run with just one or with a wired rather than optical sync), so that’s a fair bit of info. Whether they get that all from one “hit” or have to combine multiple hits, I don’t know, but I know they get that data somehow and there’s no other way for that data to get in.
The sensors must see both the sync pulse and the laser for precise timing to measure their angles relative to the base. I’ve assumed that any data from the base stations is communicated by the sync pulse, as it hits all sensors at the same time, and isn’t further complicated by very brief “hit” times. Whether the lasers’ modulated-on signals transmit data, or are simply there to distinguish the lasers from ambient IR light, is the big question. I’m at “no data on laser” for the moment.
True about serial no. and firmware version, but I don’t think the base stations send a gravity vector. Each base station’s orientation can be deduced from the IMU on the currently tracked devices (the orientation relative to base station is calculated from laser hit times, and orientation relative to ground by the IMU), and given that the reported orientation varies slightly as trackables lock onto the base stations, that points towards the base stations not having built-in IMUs.
My thinking is that if the base stations had IMUs, their orientation would be fixed immediately when turned on, and not update and converge as tracked objects connect and stabilize.
I thought I had read an offhand comment by vk2zay on reddit or somewhere about the base stations reporting gravity vectors but how that was really just for speedier startups and not strictly necessary. Apparently the sync pulse is some 340-bits “ootx” modulation (I haven’t been able to figure out what that means either), so that would probably give enough bits to put all that data in…
Oh, and fwiw, a casual run of “strings” on the macOS lighthouse driver reveals a bunch of mentions of Ceres (the Google-originated non-linear optimization toolkit), which I imagine they’re using to do their tracking math – perhaps not a Kalman filter after all.
Re: gravity vector – I missed that comment, but having an IMU in the base station makes sense. It’s just that the lighthouse driver doesn’t seem to behave as if it’s getting data from one. I’ll leave that as a “maybe” until we learn more.
Re: Ceres – in this recent podcast interview Alan explicitly discusses Kalman filters and their use in Lighthouse sensor fusion. He also talks a lot about calibration to work around some issues from making Lighthouse hardware cheap, and it’s possible that they are using Ceres for dynamic in-situ calibration through non-linear optimization.
Pingback: Accuracy of the touch controllers – VRomit's World
Oliver, have you attempted to calculate the motion-to-photon latency for Vive using the functions provided in the OpenVR API? If so, what was your exact method?
No, I have not looked into that yet.
Hi Okreylos/Oliver,
Very interesting post, nice read! I was wondering if you have a more detailed idea on how the 3D pose is constructed from the angles. What I can think of is:
1. Find the pinhole of the basestation from the angles
2. Impose a plane in front of the basestation and find ray intersections
3. Solve as PnP problem then by finding 2D to 3D correspondences
But I guess finding the pinhole would be most error prone in this approach. Have you got another idea of how it could be done? Just curious really…
Thanks,
Error323
The standard perspective-n-point algorithm is posed in terms of a camera projection matrix and points in a camera’s image plane. To apply that algorithm unchanged to one laser sweep (horizontal or vertical) of a Lighthouse base station, you need to assume that the sweeping laser rotates around a central axis. You pick some point on that axis and call it the origin. Then you pick a vector parallel to the rotation axis and call it y, and a vector orthogonal to y and the general forward-looking direction of the base station and call it x. The vectors x and y define a virtual image plane. Then you create a virtual camera with arbitrary intrinsic parameters, say center of projection = (0.0, 0.0), and focal length = 1.0. Taken all together, this defines a virtual camera projection matrix in base station-aligned space. There is no error introduced yet, as all values are arbitrarily chosen.
When you get a laser hit and an angle phi, you create a virtual image point by setting x=tan(phi). You don’t have a y, because the laser is a line along the rotation axis.
In the PnP algorithms central step, where you create the least-squares linear system, you normally add two linear equations for each image point. In this case, you can only add one equation, for the current laser sweep’s x axis. That will leave you with an underdetermined system. But half a cycle later, you add more linear equations using the other laser’s virtual projection matrix. The wrinke here is that the two lasers don’t have the same center of projection, but you know the distance vector between them based on how you built the base station. Given that, you now calculate the linear equations for the second laser sweep, and add them to the linear system from the first step. Taken all together, this system should now be fully determined, and result in the pose of the model relative to the first laser’s coordinate system.
The real algorithm is trickier because it has to take motion of the model during the laser sweep into account, which is why I believe PnP is only used during initial tracking establishment, and afterwards laser hits and their linear equations are directly integrated into Kalman filter state as they occur.
Thank you for your reply! I think I’m almost there!
Given your description and pen and paper I’m still missing a crucial step. How can one determine the central axis of a base station – given that you only have a local coordinate system of the sensors and the angles w.r.t. that base station.
There must be some step which transforms this local sensor coordinate system to world before the PnP algorithm can be applied. And I think it’s connected to the determination of the central axis of the base station. Somehow a triangulation of all the angles takes place which determines the basestation position? I’m just rambling here…
Oh I understand now! You do need some kind of reference for the ground plane. But this can be obtained from the headset IMUs.
I’m amazed at how awesome this works. Really cool.
That is correct. The system uses the headset’s IMUs to determine the direction of gravity. Tracking coordinates in the primary base station’s local coordinate system are post-rotated so that the positive Y axis lines up with the gravity acceleration vector (i.e., y points straight up). Those are the coordinates delivered to OpenVR applications at the API. If you look at the tracking server’s log output, you can see how it establishes a first guess of “up” during startup, and then slowly corrects that estimate over time as more tracking data comes in.
Here’s such log output from my tracking driver:
Log message: LHR-8F706079 H: ----- BOOTSTRAPPED base 00BF444F (best) distance 4.09m velocity 0.09m/s base pitch ~46.3 deg roll ~-5.5 deg -----
Log message: LHR-8F706079 H: ----- CALIBRATED base 00BF444F at pitch 47.35 deg roll -7.80 deg -----
The base pitch and roll come from the IMU’s gravity vector. This system cannot estimate yaw, so it keeps the z axis pointing straight at the primary base station.
Pingback: Vive la Vrui! | Doc-Ok.org
Oliver, I have floated this question to you before, but having acquired some more data, I’d like to ask again. After reviewing Vlachos 2015 & 2016 GDC talk, I now assume that fPredictedSecondsFromNow [= fFrameDuration – fSecondsSinceLastVsync + fVsyncToPhotons] which is computed inside IVRSystem::GetDeviceToAbsoluteTrackingPosemore, is equal to the motion-to-photon latency for the Vive, minus the Lighthouse tracking latency (~1ms), minus the display latency (~2ms). In one of our simple test scenes, we are getting fPredictedSecondsFromNow = ~14ms, so total MTP latency ~17-18. I am curious if the logic I have described stands to reason based on your knowledge of the sensor fusion code and having worked extensively with the OpenVR API, or if you have attempted to calculate the MTP latency in a controlled setting since my last inquiry. Best wishes.
I don’t know the external OpenVR API well; my code interfaces only with the internal driver API.
Anyway, I measured frame-to-photon latency of the Vive using a latency tester. I get consistent results of 21-22ms, which makes sense: The frame starts immediately after vsync, swaps buffers at the next vsync (11ms later), then the frame buffer content is scanned out to the display over the next approx. 10ms, at the end of which it’s displayed all at once.
I noticed that there’s code in OpenVR that delays frame processing to a certain offset after vsync, basically assuming that an application’s render loop only takes a few ms, and active-waiting for those few ms before the next vsync. That would explain the 14ms you are seeing, as I recall the default frame delay being 8ms. I’ve used the same method in Vrui, where I could get frame-to-photon latency down to just above 11ms, at the risk of missing a vsync entirely once in a while.
With time warp, rotational latency goes down to about 11ms quite reliably, as expected.
Really cool! Exciting! Can you offer the detail method and the source code of calibrate the probe tip?
The “RoomSetup” utility in the Vrui package contains the probe tip calibration code.
Thank you very much for your reply！
But I can’t find The calibration code in “RoomSetup”,Only hardcode Position for initialize like this:
void RoomSetup::controllerTypeValueChangedCallback(GLMotif::DropdownBox::ValueChangedCallbackData* cbData)
{
/* Update the probe tip position: */
switch(cbData->newSelectedItem)
{
case 0: // Raw from device driver
probeTip=Point::origin;
break;
case 1: // Custom controller
probeTip=customProbeTip;
break;
case 2: // Vive DK1 controller
probeTip=Point(0.0,-0.015,-0.041);
break;
case 3: // Vive and Vive Pre controller
probeTip=Point(0.0,-0.075,-0.039);
break;
}
/* Update the probe tip text fields: */
for(int i=0;isetValue(probeTip[i]);
}
Did I miss something?I really appreciate your help
My mistake; the calibration routine was supposed to make it into Vrui-4.2, but it got pushed down the to-do list. Here is a Vrui utility to measure 3D positions using a tracked input device: MeasurePoints.cpp. It contains the calibration routine and runs it when the user presses “c”.
To use it, first run the Vive tracking driver as usual (“/usr/local/bin/RunViveTracker.sh”). Then build MeasurePoints (see below) and run it like so:
./bin/MeasurePoints localhost 8555 -t <trackerIndex> -b <buttonIndex>To use the trigger button on the first controller, use “-t 1” and “-b 4”.
MeasurePoints is supposed to be a Vrui utility, so to build it, you need to drop it into the Vrui source tree. Best place for now is the Vrui package root directory. Once you copied it in there, cd into the directory, and run
make PACKAGES=MYVRUI MeasurePointswhich will create ./bin/MeasurePoints.
Thank you very much.
Did I need to rotate the vive controller around a point to collect calibration data?
Yes, you need to collect at least three poses, ideally more, where the same point on the controller touches the same point in world space from different directions.
Would you be willing to share the software you wrote (shown in the video) to observe the drift and accuracy of the lighthouse?
I don’t have that exact program anymore, as it was merely a (very thin) graphical wrapper around a stand-alone OpenVR tracking driver host. The OpenVR host transformed into a VRDeviceDaemon tracking module, and I haven’t updated the stand-alone version to new OpenVR APIs.
But the actual tracking visualization is basically a fifty-line Vrui application. Make a ring buffer, put tracking positions into it, draw ring buffer contents as polyline. There is a TrackingTest utility in Vrui that has most of the infrastructure built in, specifically getting asynchronous data from a VRDeviceDaemon. Add a ring buffer to that, and you’re good.
Pingback: Positional Tracking, MoCap, HeadTracking | Ronny Gey
Doc OK,
Do you know of any way to parse the tracking code that can be generated from the SteamVR Developer menu, e.g. Record Tracking Data? The output is in some unknown binary format (.pb).
No, sorry, I’ve never looked at that.
Very nice article. Do you know where the tracked position of the controller is located on the controller itself?
Wow! Thanks for the great post! I’ve noticed that the tracking is significantly more jittery when line-of-sight is occluded for either basestation (or in a single lighthouse configuration). The jitter is mostly along the axis between the basestation and the tracked object. It is much more noticeable for the controllers and gets worse the farther they are from the basestation. Since my play space is fairly large (16ft between the basestations, synced via the cable A+B configuration) the jitter is quite noticable in the center of my play space. I’d like to find out if what I’m experiencing is an atypical amount of jitter, or if it is within the norm. So I have 2 questions for you… 1) could you share a plot of the jitter from a single basestation as the distance from the basestation increases? 2) could you share a link to some jitter measuring tools? …Thanks again for your excellent post!
Thanks for your work on this topic, particularly the accuracy tests. I’ve been trying to build a measurement system with the LH and am currently stuck on on the details. Alan Yates posted on my video that the angle+cotan=”image” strategy is only approximately true.
I saw above that you mentioned the center of projection for each rotor is not the same. I’ve thought about this a great deal and it still seems like the rotor’s offset from the actual center shouldn’t matter since they are projecting a plane, as long as they’re in that plane why does it matter where they’re projecting from?
My video with the comment: https://www.youtube.com/watch?v=f7E-s6CnsYM
Any thoughts on this topic are welcome as I’ve experimented a lot with applying the factory calibration values in different ways and believe that my reliance on the pinhole model is the actual problem. Thank you.
Thanks for the post. I’ve learned a lot. Do you think an external calibration can improve the accuracy of the tracking system?. My system needs a 0.5mm accuracy, and the Lighthouse tracking is close to it.
That depends. I’d recommend installing a Lighthouse system under the same conditions as your desired production system, and mapping out its accuracy and distortion profile. That should give you an idea whether external (non-linear) calibration might get you where you need to be.
I’m not very optimistic about achieving sub-millimeter accuracy, though. At typical tracking distances, even residual jitter is already getting close to the 0.5mm threshold. You might be able to reduce influence of jitter by strong filtering, if you primarily need to measure non-moving objects.
Thanks for your post. I’ve done some experiments myself and I am wondering about the accuracy of tracking an object in real space using lighthouse.
I mounted the HMD and a Vive controller onto a table on wheels. The distance between the HMD and the controller (in real space) is therefore fixed. Using GetDeviceToAbsoluteTrackingPose() and .mDeviceToAbsoluteTracking I get the transform matrices and I calculate the distance between the HMD and the controller. I expected this calculated distance to remain constant when I move the table around the play area, because this distance is fixed in real space. Unfortunately, it varies a couple of centimeters.
The real distance is about 550 mm. The distance I get from the lighthouse/SteamVR tracking varies between 521 and 585 mm. It depends on the distance from the table to a lighthouse and also on orientation to a lighthouse.
I did a total of 60 measurements at 5 locations in the play area (4 corners and center). At every location I did 4 measurements with the table (with HMD and controller) pointed north, east, south, and west. At every location and orientation I did the measurements three times. See this Google sheet for the results:
https://docs.google.com/spreadsheets/d/1OdZuhH8InHfKJ5RWPmeRvJiBPXFaqrfDvJA1KFgWZBo/edit?usp=sharing
The measured distances at the same location and orientation are reasonably constant. But I get great differences when the table is at another location or is oriented differently. Maximum distance measured: 585 mm
Minimum distance measured: 521 mm
Am I doing something wrong? Or is controller tracking not that accurate? I was expecting submillimeter precision. Or, after reading your post, at least millimeter precision. But I get centimeter precision.
All info is much appreciated!
Good experiment! I’ll have to try that myself.
Before I do, one potential reason for this difference could be interference in the room, say from some smaller reflectors and therefore small regions of bad tracking. If one trackable is in a bad region, and the other one is not (or in a different bad region), their position measurements might be relative to different coordinate systems, and therefore uncorrelated.
I did some more tests using sheets (cloth) around the play area to eliminate interference from reflections.
This time I mounted two controllers to the table (and one HMD).
The results are roughly the same when looking at the HMD to controller distances. But the distance between the two controllers does stay relatively constant when moving the table around the play area. So, the cause seems to be in the HMD. Maybe my HMD did not get a good calibration in the factory? I read that after production the HMD is calibrated; probably meaning that the actual IR sensor positions and orientations are measured and stored in the HMD. Somebody from Valve wrote that you couldn’t take an HMD apart and re-assemble it perfectly because of this post production calibration.
It would be great if you (or somebody else) could do the same experiment as I did.
Pingback: Lighthouse Tracking: Calibration | Oculus Rift Blog
Could you do this with the Vive Pro and 2.0 base stations? I want to know if there’s a jitter difference. Thank you.
Gladly, but I don’t have a Vive Pro (yet).
Donations are welcome. 😉
The Lighthouse approach explained. https://www.youtube.com/watch?v=cJT0lLkqDTQ
Might be interesting to everybody trying to understand the principle. The lecture is a couple of years old, but still quite interesting.
Also there is a publication mentioned, that describes the principle. The minnesota scanner https://experts.umn.edu/en/publications/the-minnesota-scanner-a-prototype-sensor-for-three-dimensional-tr
Thanks for your very interesting article btw.!
Pingback: HTC Relative Lighthouse Pose Estimation without Open- / SteamVR | Oculus Rift Blog
steamVR position offset after vrserver restart
i setup a room with two lighthouse(1.0) and a HTC vive tracker, the tracker is connect to Linux host, which is battery powered.
it works fine most of time, but I found if move tracker to different position then restart the SteamVR, absolute position has offset about 7-10mm when Tracker move to a same docker position.
How can I resolve the problem?
Thank you very much!
I don’t think there is very much you can do. The two base stations automatically calibrate their positions relative to each other through the data received from a trackable that is seen by both stations when the tracking service starts, and then report tracking data in a coordinate system relative to the “master” base station. Random noise or spurious reflections during that initialization phase can cause small-ish coordinate system offsets. During longer use, the system will continuously re-calibrate itself, and I noticed that initial coordinate mismatches get slowly fixed over time.
There might be a way to disable auto-calibration and stick with previous calibration data through driver configuration options, but I haven’t figured out how yet. A practical approach might be to measure things like boundaries or important positions after the service has been running, and devices have been moved around, for several minutes, instead of immediately after it’s been started. That way, tracking data from later sessions will converge back to the measured session.