ExtractionBrowserTroubleshooting

1/11/2012 The demo seems to fail on 2 to 3 day intervals. When trying to login as knowall to fix, all attempts to execute commands in bash yield the error "fork retry: resource temporarily unavailable". Googling is unhelpful due to how generic the error is. Solution for now: You must remotely kill the demo process from another machine. From a different account, run: ssh knowall@reliable "ps aux | grep knowall" to find the demo process. Then run "kill <demoPID>" to kill the demo's Jetty process. Once the demo has been terminated, you should be able to login as knowall and run commands. Follow the normal restart instructions from here, but don't use a daemon screen. (Which terminates when it's given child process does) (NOTE: Killing the Jetty process also kills the SCREEN instance, so any output it contained is lost) (NOTE: maybe not running --headless has been the problem somehow?)

1/17/2012 The error discussed above happened again, and was even more of a pain to deal with this time. The procedure I suggested above did not work, instead it continued to yield the same "fork: retry: resource temporarily unavailable", along with other weird errors when trying to run commands remotely, like "exec request failed on channel 0". Turns out that "kill <pid>" wasn't enough. In the future, use "kill -s 9 <pid>" to more forcibly kill the process -- note this error -- "Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated" The output redirect I set up was successful at capturing some debugging info. It revealed that, as the demo ran longer, the memcached client I was using began throwing OutOfMemoryErrors when trying to create a new thread (the cause of the error could be anything, not just Memcached). Looks like it may be a more normal-ish memory leak after all.