Add more robust ssh ping to be run on reboot for updates.

remote_access.sh

Index: remote_access.sh
diff --git a/remote_access.sh b/remote_access.sh
index 794deff8e755ed0a98a35513016e86c3239cdbad..bf6e05836a9ea279f436907b9de543a91d3ad5ec 100644
--- a/remote_access.sh
+++ b/remote_access.sh
@@ -34,8 +34,39 @@ function remote_sh() {
   return ${PIPESTATUS[0]}
 }
 
+# Robust ping that will monitor ssh and not hang even if ssh hangs.
+# This method attempts to run true on the remote device.  We check to see
+# if ssh is still running after a timeout (is hanging).  If so, we retry
+# 3 times before giving up.  We wait up to 5 seconds for true to return each
+# time.
+function ping_ssh() {
+  local timeout=5
+  local timeout_inc=.5
+  local retries=3
+  while [ ${retries} -gt 0 ]; do
+    nohup -- remote_sh "true" 2> /dev/null &
+    local ssh_pid=$!
+    local current_timeout=0

[petkov, 2011/01/12 18:38:52]

this site shows an interesting way to implement a timeout that doesn't require
waiting for the full timeout yet doesn't do a loop:
http://redflo.de/tiki-index.php?page=Bash+script+with+timeout+function

+    while ps -p ${ssh_pid} > /dev/null && \

[petkov, 2011/01/12 18:38:52]

no need for trailing \ I think

+        [ ${current_timeout} -gt ${timeout} ]; do

[petkov, 2011/01/12 18:38:52]

current_timeout is 0, timeout is 5 -- this evaluates to false, no? so the loop
will never iterate?

+      sleep ${timeout}
+      echo "blah"

[petkov, 2011/01/12 18:38:52]

blah? :)

+      current_timeout=$((current_timeout+timeout_inc))

[petkov, 2011/01/12 18:38:52]

add spaces around +

+    done
+    # If pid has been killed, kill will return 0.
+    if ! kill ${ssh_pid} &> /dev/null; then
+      break
+    else

[petkov, 2011/01/12 18:38:52]

s/else/fi/, or better:

kill ... || break

are you sure kill SIGTERM won't hang? should you just kill -9 instead?

+      retries=$((retries-1))
+    fi
+  done
+}
+
 function remote_sh_allow_changed_host_key() {
   rm -f $TMP_KNOWN_HOSTS
+  # Often on first boot we can have a race condition where the initial ssh
+  # can hang.  Pinging first mitigates this.
+  ping_ssh
   remote_sh "$@"
 }