Sandbox Software Unstable – Constantly Freezing

Home Forums AR Sandbox Forum Sandbox Software Unstable – Constantly Freezing

Viewing 6 posts - 1 through 6 (of 6 total)
  • Author
    Posts
  • #2757
    cwj
    Participant

    Hello,

    I am working with a AR Sandbox installation that was set up a few years ago. It has been working great, but about a month ago it began experiencing issues with freezing. The map would still be displayed, but the topography would no longer update when changes were made to the sand. It can be resolved by stopping and restarting the program, but I am looking for a permanent solution to this issue.

    The most notable recurring issues from the program logs are:
    1. Repeated messages of “Ran out of time by x.xx” with the X’s being varying numbers
    2. Sometimes it reports:
    terminate thrown after an instance of 'std.runtime_error'
    what (): USB:: Device: :writecontrol: Device has been disconnected
    Aborted (core dumped)

    I noted that the “ran out of time” errors occurred when making it rain, and sometimes made it freeze if too much water was added. I modified the evaporation rate and that made it seem more stable in the short-term. However, it very consistently freezes after running for a while, usually within a few hours.

    Is the Kinect camera failing or is there some sort of performance bottleneck on the computer? It is running 360 Kinect, Ubuntu 18.04 LTS, 8GB memory, Intel i7-4770, and a GTX 970. The 970 is slightly underpowered compared to the officially recommended GTX 1060, but I’m not sure if this is the only explanation for the issues.

    Initially, it was running SARndbox 2.6 but it has been updated to SARndbox 2.8 and re-calibrated, although the issues remain the same. I confirmed that the graphics drivers are installed properly as well.

    Any thoughts or ideas on what could be going wrong here? Any help is appreciated.

    #2760
    cwj
    Participant

    Just as a brief update, the ‘ran out of time by’ error originally was repeated and were small numbers like 4.286, but now it only appears once before the program freezes and the number is significantly larger, something like 42000.

    I have not seen the USB error in a while, but I am wondering if the program is timing out because the Kinect is failing? The rest of the computer seems to be running fine when the program is frozen. It has been running v2.6 of SARndbox for years now, so it would be strange to be attributed to a configuration issue.

    I also found the new installation instructions on this forum and used the PullPackage tool to install the newer version of SARndbox (upgraded from 2.8 to 2.12) but unfortunately this did not resolve the issue at all.

    #2797
    ramonvsbr
    Participant

    I’ve been experiencing the same problem, after about 5 minutes the screen freezes. I would at least like to know if there is a way to program an automatic restart.

    #2798
    cwj
    Participant

    @ramonvsbr

    I did manage to put together a basic script that will auto-start and restart the program every 30 minutes, which seems to be effective for mitigating the issue. If it is freezing every 5 minutes for you, you may want to change the timeout (in seconds) to something shorter. Adding the “-evr” flag to change the evaporation rate also helped lessen the freezing. Also check that the directory matches where your program is stored. This is definitely a band-aid fix, though, and I would love to see a proper solution implemented.

    I recommend scheduling a cronjob for an automatic restart each day, since brute-forcing it this way could cause memory issues if you do it for too long. You’ll need to set the account to auto-login on startup if not already, and then add this script as a startup program (through the system settings menu).

    #!/bin/bash
    while true; do
        # Kill running instance
        kill -9 $(pgrep -f "SARndbox -uhm -fpv -evr -0.03")
    
        # Wait for previous instance to terminate
        sleep 7
    
        # Change directory
        cd /opt/SARndbox-2.12
    
        #Run program with 30 minute timeout
        timeout 1800 SARndbox -uhm -fpv -evr -0.03
    
    done 
    #2801
    okreylos
    Keymaster

    The “ran out of time by …” messages mean that the graphics card was not able to keep up with the requirements of the water simulation. This is not fatal by itself, it just means that there is a momentary slow-down in the visible speed of water propagation in the sandbox. This issue should not cause stability problems, lock-ups, or crashes.

    The USB::Device::writecontrol: Device has been disconnected error is serious, it means that there is a connection problem with the Kinect camera. Maybe the camera itself is bad, or the cable or power supply are not up to spec, or the PCs USB ports have issues.

    Either way, this is a hardware problem.

    First I would try plugging the Kinect into different USB ports on the host PC. If that does not improve stability, I would try replacing the camera’s power / USB adapter cable. Those often seem to have issues. If that doesn’t help, either, I would try to find a different Kinect camera.

    For comparison, the AR Sandbox in UC Davis DataLab has been running continuously (no application restarts) for about three months now, with no issues.

    #2808
    cwj
    Participant

    Thank you @okreylos for the insight and confirmation that this is a hardware issue. I definitely came to that conclusion after all the troubleshooting, so that will be the next step for my situation.

    I don’t have plans to make any changes (i.e. replacing Kinect) to our sandbox deployment right now, but if I ever do I will come back here to provide any updates.

Viewing 6 posts - 1 through 6 (of 6 total)
  • You must be logged in to reply to this topic.