Build your own Augmented Reality Sandbox

Posted on by

Update: There is now an AR Sandbox support forum with detailed complete installation instructions starting from a blank/new PC, and a video showing a walk-through of same instructions.

Earlier this year, I branched out into augmented reality (AR) to build an AR Sandbox:

Photo of AR Sandbox, with a central “volcano” and several surrounding lakes. The topographic color map and contour lines are updated in real time as the real sand surface is manipulated, and virtual water flows over the real sand surface realistically.

I am involved in an NSF-funded project on informal science education for lake ecosystems, and while my primary part in that project is creating visualization software to drive 3D displays for larger audiences, creating a hands-on exhibit combining a real sandbox with a 3D camera, a digital projector, and a powerful computer seemed like a good idea at the time. I didn’t invent this from whole cloth; the project got started when I saw a video of such a system done by a group of Czech students on YouTube. I only improved on that design by adding better filters, topographic contour lines, and a physically correct water flow simulation.

The idea is to have these AR sandboxes as more or less unsupervised hands-on exhibits in science museums, and allow visitors to informally learn about geographical, geological, and hydrological principles by playing with sand. The above-mentioned NSF project has three participating sites: the UC Davis Tahoe Environmental Research Center, the Lawrence Hall of Science, and the ECHO Lake Aquarium and Science Center. The plan is to take the current prototype sandbox, turn it into a more robust, museum-worthy exhibit (with help from the exhibit designers at the San Francisco Exploratorium), and install one sandbox each at the three sites.

But since I published the video shown above on YouTube, where it went viral and gathered around 1.5 million views, there has been a lot of interest from other museums, colleges, high schools, and private enthusiasts to build their own versions of the AR sandbox using our software. Fortunately, the software itself is freely available and runs under Linux and Mac OS X, and all the hardware components are available off-the-shelf. One only needs a Kinect 3D camera, a data projector, a recent-model PC with a good graphics card (Nvidia GeForce 480 et al. to run the water simulation, or pretty much anything with water turned off) — and an actual sandbox, of course.

In order to assist do-it-yourself efforts, I’ve recently created a series of videos illustrating the core steps necessary to add the AR component to an already existing sandbox. There are three main steps: two to calibrate the Kinect 3D camera with respect to the sandbox, and one to calibrate the data projector with respect to the Kinect 3D camera (and, by extension, the sandbox). These videos elaborate on steps described in words in the AR Sandbox software’s README file, but sometimes videos are worth more than words. In order, these calibration steps are:

Step 1 is optional and will get a video as time permits, and steps 3, 6, and 8 are better explained in words.

Important update: when running the SARndbox application, don’t forget to add the -fpv (“fix projector view”) command line argument. Without it, the SARndbox won’t use the projector calibration matrix that you so carefully calibrated in step 7. It’s in the README file, but apparently nobody ever reads that. ;)

The only component that’s completely left up to each implementer is the sandbox itself. Since it’s literally just a box of sand with a camera and projector hanging above, and since its exact layout depends a lot on its intended environment, I am not providing any diagrams or blueprints at this point, except a few photos of our prototype system.

Basically, if you already own a fairly recent PC, a Kinect, and a data projector, knock yourself out! It should be possible to jury-rig a working system in a matter of hours (add 30 minutes if you need to install Linux first). It’s fun for the whole family!

62 thoughts on “Build your own Augmented Reality Sandbox”

  1. Pingback: Build your own Augmented Reality Sandbox | Doc-Ok.org

  2. Pingback: … and they did! | Doc-Ok.org

    • There is nothing special to enable water in the sandbox software; it’s on by default. You’ll need to set up a water simulation bounding box (part of the BoxLayout.txt file), as described in the documentation. To test the water simulation — rain can be a bit finicky — create a water tool. Press any key (say ’1′), and move the mouse while holding that key to the “Manage Water” menu item, then release the key. When the dialog box pops up, press another key, say ’2′. Then, when you press (and hold) ’1′, the sandbox will be flooded with water; if you press (and hold) ’2′, the sandbox will be drained.

      The biggest practical issue is graphics driver support. You need a powerful discrete graphics card, something like an Nvidia GeForce 480, and the vendor-supplied binary drivers, to run the shaders necessary for water simulation. There are good tutorials on installing the drivers for a variety of Linux distributions online.

      Reply ↓

    • The GPU water simulation code is based on GLSL shaders, not on CUDA or other GPGPU packages. I don’t know anything about nvidia-smi, but that’s probably why nothing shows up. If the water simulation were running on your CPU, you’d get maybe one new frame every 10 seconds or so.

      Another way to check is to use the nvidia-settings utility, click on the tab for your GPU (“GPU 0 – (GeForce GTX 680″ in my case), and look at the”GPU Utilization” value. When the water simulation code is running, it should be anywhere between 20% and 100%.

      Reply ↓

        • Oh, it’s you! I missed this comment originally and just saw it yesterday. I have a picture of your AR Sandbox on my external installations page, with a link to the article in your newspaper. Do you have a more appropriate / permanent web page to which I can link?

          Reply ↓

  5. Hello, I have this message:

    [root@localhost SARndbox-1.5]# ./bin/SARndbox -c 0
    0.155481 x 0.15066
    Caught exception Sandbox: Not all required extensions are supported by local OpenGL while initializing rendering windows

    can you help me?
    (it is with CentOS-Linux)

    Thank you

    Reply ↓

  6. Pingback: AR Sandbox Support Forum | Doc-Ok.org

  7. Hi,
    This is really amazing, I want to build a sandbox in a large format 4x3m for a light festival. Is there any limitations to the size of the kinects field of view?
    best regards
    Edvin

    Reply ↓

    • 4x3m is much too large to be useful. The fundamental problem is the Kinect’s limited field-of-view and resolution. To cover a 4x3m sandbox, the Kinect would have to be mounted 4m above the sandbox, and 4m is way outside its useable distance range. A 2mx1.5m sandbox would already be very low-resolution. 1×0.75m is about the ideal size.

      At the moment the only feasible approach for large-area sandboxes would be multiple Kinects, but that would be a major installation and maintenance problem, and the software doesn’t support it, either.

      Reply ↓

      • hi, im working on building this and i would appreciate some measurements, the height of the sandbox (depth as well) and the height of the metal pole where the projector/kinect are hung.

        Reply ↓

        • We found the ideal AR Sandbox size is 40″x30″, which places the Kinect (and the projector) approximately 40″ above the intended average sand surface.

          Reply ↓

  8. Hi! I’ve compiled all the programs and having the system almost working. Here I’m making my test on the wall and having some issues. Can you answer me some questions?

    1- When I run the sandbox (there’s no object on the wall), I don’t obtain the green textured image but a running simulation instead with the water and all environment. What you think would be doing this?

    2- About the BoxLayout file, can you explain me about the coordinates system and how modifications in it reflect on sandbox simulation?

    3- How can be a object from the wall? When i put my hand close the wall, it is recognized in the simulation but if a put something thicker (about 8cm) it start the rain simulation

    I’ll put here some images from calibration and the simulation running.

    http://imageshack.com/a/img537/2954/EyOQA6.png

    http://imageshack.com/a/img538/2831/wztoGI.png (Here in Step 5 My reconstruction always be Over or behind the axis, i didn’t manage to put it between the two coordinates axis like in the video )

    http://imageshack.com/a/img912/8937/tcojHD.png
    http://imageshack.com/a/img633/8228/JV5CRG.png

    Congratulations for your great work! I can’t wait to start it here

    Reply ↓

    • 1: Without setting up a BoxLayout.txt file, the Sandbox software does not know what the zero-level plane relative to the camera is. The result will be some more or less random topography model.

      2: The first line sets up the geometric plane equation for the zero-level plane relative to the camera, in camera units (can be mm or cm, depending on camera model, in the current software version). The first three numbers define the plane’s orientation as a normal vector. For example, if the camera points straight down, then the vector should be (0, 0, 1) meaning that the terrain’s “up” direction points along the camera’s Z axis. If the camera points down at a 45 degree angle, the vector should be (0, 0.7071, 0.7071), etc. The fourth number is the distance from the zero-level plane to the camera coordinate system’s origin, which is the lens center of the (virtual) depth camera, in camera units. For example, if the camera points straight down, i.e., the normal vector is (0, 0, 1), and the camera is 120cm above the desired zero level, then the fourth number would be -120.0 (negative because the camera looks along the negative Z axis).

      During calibration, you determine the plane equation as described, resulting in a distance value in the appropriate unit, i.e., centimeters or millimeters. You can then adjust the distance value to move the zero-level plane higher or lower, as convenient for your specific setup.

      3: The sandbox has a range of valid topographic elevation values, by default the range of the elevation color map. The software ignores surfaces outside that range. Similarly, there is a valid range for “rain objects.” Both ranges are relative to the zero-level plane defined in BoxLayout.txt, using the camera’s native measurement unit. The BoxLayout.txt file and height color map shipped with the SARndbox software are in centimeters. If your camera uses millimeters (check the distance value in your plane equation), you will have to multiply all elevation values in the color map file by ten.

      Looking at your pictures, your camera works in millimeters. Do the adjustments I mention above, and also run “SARndbox -h” to see a full list of command line options. You need to pass some non-default values to make the software work well, such as a different contour line interval.

      Reply ↓

      • Wonderful. My other question is if you think this is a feasible task for someone that has never done anything like this before. I don’t have experience with computer systems or programming. Is there enough direction in your videos and discussions that I could manage to build one of these? Or would I need someone that has experience with programming?

        Reply ↓

        • Hard to say for me, but it should be fine. There’s no programming knowledge required to set up an AR Sandbox, but you’ll have to be willing to work inside a terminal window, and ideally to install Ubuntu Linux.

          We have detailed installation instructions on the AR Sandbox Forum; you can check those out and see if they look doable.

          Reply ↓

  13. Just wondering about costs, approx is fine :) …I want to create a budget for myself and set a goal to save towards. I am blown away by this project and would really love to make one!!

    Reply ↓

    • Short-throw projector around USD 600, first-generation Kinect around USD 100. If you already have a computer you can use, the rest is a bunch of plywood, two-by-fours, duct tape, baling wire, and a few buckets of sand. If you want to run the water simulation in a responsive way, you’ll have to invest around USD 300 into a good gaming graphics card, such as an Nvidia GeForce GTX 770.

      Reply ↓

      • Got it. We’ll give it a try. Wondering how I can tailor the application to compliment the rainforest environment and our eco approach. Any ideas?

        Reply ↓

        • That’s a good question. We haven’t thought about it in that context, and it’s probably not directly applicable as it is, but one potential addition we’ve discussed was a simplified simulation of vegetation growth in response to water availability. It’s a pretty big piece of work, though.

          Reply ↓

          • That would be useful for a site like this. It is for an eco reserve retreat center. Sitting on 250 acres of primary rainforest. here is our website: http://www.samasati.com . Perhaps a simulation that can help us to better manage our water resources would be helpful. We have been catching water on site for almost 20 years now.

  15. hei, first up: thanks for your wonderful project!
    I`m in the middle of setting up my own sandbox for a school project, but unfortunatelly the server is still down.
    any news on when its going online again?

    Reply ↓

  16. The site that was hosting your code seems to have taken it offline. The URL is returning a 404 now. Is there a mirror somewhere else? Tks!

    Reply ↓

  18. Pingback: From-scratch AR Sandbox Software Installation | Knowledge

  20. Hi. I managed to build a sandbox by your description – thank you for the tips! I tried to register for the sandbox forum but did not receive an email with my password – help please (via e-mail). Greetings from Poland.

    Reply ↓

  21. Hi, my engineering class and I have recently built an AR Sandbox and we are really enjoying it! I was just wondering if it was possible to change the color of contour lines to make them seem more visible?

    Reply ↓

    • You can set the color by editing line 96 of the SurfaceAddContourLines.fs shader file, which is located in the share/SARndbox-<version>/Shaders/ subdirectory of the SARndbox source directory:

      baseColor=baseColor=vec4(0.0,0.0,0.0,1.0);

      The first three numbers are the red, green, and blue color components, respectively, normalized to [0.0, 1.0]. After editing the file, you need to restart the SARndbox application to see the change.

      Reply ↓

  22. Hi, first off thank you for the support you have been giving people, it has been a really fun project to work on!

    So my sandbox is built and calibrated, the red x was perfect in high/low positions across the sandbox when running calib. however I think the plane equation I have is wrong or the file is in the wrong location. If you were looking from where the kinect is, the bottom of the sandbox has the water level much much higher than at the top. please see photos below.

    http://imgur.com/EAsu7iX,aOjwAxO,InR4Itd,PBIDwQy#0

    first pic is boxlayout.txt next 3 are sandbox, sorry for rotated images. but you can see what I mean about water super high on bottom and really low towards top. From your above post I adjusted the first 3 numbers and 4th in the plane equation hoping to correct it, ie the (0,1,1) method described above, but of the 50 different adjustments I made nothing changed at all when relaunching the sandbox software, I am wondering if I have the boxlayout.txt in the wrong location hence the changes not reflecting in the sandbox? its current path is

    ~/src/kinect-2.8-001/kinect/boxlayout.txt

    everything is installed under the src folder..

    once I get the plane level with the sandbox how do you raise and lower the water level (sea level)? like what file is that and what line do you adjust sea level.

    and lastly I am making this to simulate lava flows so I am wanting to use a color scheme of grays and change the water to orange/yellow for lava. I saw your post above about changing contour line colors which I tested and it works. Can you elaboration on what file you have to edit and line to adjust colors for terrain elevations? I would be really appreciate it.

    Thank you so much for all the work you have done on this.

    Josh

    Reply ↓

    • ok, I re-read the above posts and searched for boxlayout.txt and found it in another location, I copied the good data into that file and now the plane is no longer tilted. yay!

      now I have been in heightcolormap file for about an hour messing with elevations and colors and have made my volcano file and an island/mountain file. yay

      So I guess my only question is where is the setting to change the color of the water for the lava, like what file and line?? I’m pretty sure i read through this and did not fine that info, once again thank you for your time.

      Reply ↓

  23. Hi
    I am new here and before i go ahead and possibly make one of these, I want to know
    1). Is it possible to do some programming on the project, I am thinking of an idea and want to know what language I can use to build on the system or what language it is written in.
    2). Are there any plans to create one for the X Box ONE
    Thanks
    Greg

    Reply ↓

      • okay thanks for the reply, I know Java and Python, guess I will have to start learning C++ on my off days from work and while building the Sandbox. Will be back with more questions then no doubt.
        Thanks
        Greg

        Reply ↓

  24. Hi forum,

    I find this very interesting and would like to experiment it to design some of my tutorials.

    However, to be more specific and to avoid any hiccups, I would be grateful if I can be recommended with some specifications for the below.

    1. 3D Camera (model, approx cost)
    2. Projector (model, approx cost)
    3. Sand (is it a special type of sand?, if yes where can I buy it? cost?)

    Thanking you

    Sandeep

    Reply ↓

  25. I’m working to put together specs for a PC to run the software. Would there be any issues with utilizing a Quadro card as opposed to a GeForce card?

    Reply ↓

    • No, the Quadro will work fine, but you most probably won’t get any benefits from it, either. Bottom line, you’ll be paying ten times as much for the same result. Unless you already own a Quadro, of course.

      Reply ↓

      • What aspects of the video card is the software utilizing? Is it the amount of VRAM on the card, the bus size (128-bit, 192-bit, 256-bit, etc..), core speed, memory speed? I read a comparative blog you wrote up between the variance between the Quadro and GeForce cards. I work for the City and the department I support is wanting to set up an exhibit. They’ve given me a budget to work with that I’m trying to keep within. The hard part is they want it to run from a laptop, so that everything is self-contained within the exhibit without the need for an external monitor needing to be setup each time calibration is needing to be done. Obviously this makes it slightly more expensive since laptops are more costly than desktop setups. My vendors we utilize unfortunately can’t quote me pricing for consumer grade laptops, only business grade and the closest to the specs mentioned for the sandbox would be engineering grade laptop with a Quadro or Firepro card in it.

        So I’m just trying to understand a little bit about the specs and the utilization of the hardware by it, so I can properly assess which hardware to purchase.

        Thanks,

        Reply ↓

        • It’s hard to break it down by specs; only a benchmark can really tell how each individual system performs. As a rough guide, the most important criteria are number of CUDA cores and core clock. The code has to do a certain number of calculations per second, and the number it can do is roughly clock speed times number of cores (assuming that the cores are identical between systems, which they happen to be between Quadro and GeForce). VRAM is a binary threshold: you need a certain amount to represent the simulation state in memory. If you have less VRAM, it won’t work at all; if you have more VRAM than you need, it won’t improve performance at all. The AR Sandbox uses less than 2GB of VRAM with standard settings (probably much less than 2GB, but I don’t have exact numbers at hand).

          Would you be able to buy from Dell, specifically a gaming laptop from their Alienware line (this one here seems pretty high up in specs, for $1500), or from MSI or Acer? You could get one with a high-end latest-generation mobile GeForce. I think the most expensive ones run up to $2000, probably still less than a business laptop.

          I understand that you work under certain constraints, but in general I advise against using a laptop to run the sandbox. Not only do you pay much more for the same performance, but you might also run into reliability issues such as overheating. There are a couple of users on the AR Sandbox Forum who use laptops. You could ask there for their experiences.

          Reply ↓

          • Thanks for the info. I think this definitely helps me better understand what I need.

            Dell is actually our primary computer vendor. They can’t quote be a business price as their Alienware computers are consumer grade and wanted to quote me pricing for the Precision line. The issue I was running into especially now knowing about more about the software, is how the cores are calculated between the two cards. Comparably the 970M GeForce 1280 CUDA Cores and 3GB VRAM, whereas lets say the Quadro k2100M has only 576 Parallel CUDA Cores. Is the parallel nature on the Quadro cards the equivalent of say (576 * 2) or are you literally looking at less than half the amount of CUDA cores on that card? Cause to get close to the same amount of CUDA cores on a Quadro you are looking more along the lines of the K4100M and roughly about $2200 laptop instead of $1500-1700.

            Do you know how that works between “Parallel cores” on the Quadro cards and the regular cores on the GeForce cards?

          • They can’t quote be a business price as their Alienware computers are consumer grade and wanted to quote me pricing for the Precision line.

            Jeez.

            are you literally looking at less than half the amount of CUDA cores on that card

            That’s right, it’s just a phrasing difference. All GPU cores are parallel. A Quadro 2100M should be less than half the performance of a GeForce 970M, as I think it also has a lower core clock. In general, Quadros and GeForces are the same GPU architecture, just aimed at different markets. The lowest-level Quadros will be comparable to lowest-level GeForces, and the same is true at the highest level. A GeForce 970M is pretty high up in the mobile range, and you’d have to go to a Quadro K4100M or even K5100M to get the same computing power.