Have service_manager start a cloudtail attached to each service it starts.

infra/libs/service_utils/_daemon_nix.py

Index: infra/libs/service_utils/_daemon_nix.py
diff --git a/infra/libs/service_utils/_daemon_nix.py b/infra/libs/service_utils/_daemon_nix.py
index f23d070656043ee877cfe86b5622f7657b71a37f..72212a97393aaa92ee86dc0bfd8429f49df194e3 100644
--- a/infra/libs/service_utils/_daemon_nix.py
+++ b/infra/libs/service_utils/_daemon_nix.py
@@ -103,21 +103,10 @@ def _fork_then_exit_parent():
     os._exit(0)
 
 
-def become_daemon(keep_fds=None):
-  """Makes this process a daemon process.
-
-  Starts a new process group, closes all open file handles, opens /dev/null on
-  stdin, stdout and stderr, and changes the current working directory to /.
-  """
-
+def close_all_fds(keep_fds):
   if keep_fds is None:
     keep_fds = set()
 
-  _fork_then_exit_parent()
-  os.setsid()
-  _fork_then_exit_parent()
-
-  # Close all open files.
   for fd in reversed(range(2048)):

[Vadim Sh., 2015/09/23 17:40:21]

(I don't remember this code)

Why 2048? There's no guarantee it's all of them...

Reading subprocess.py I found following interesting snippets:

try:
    MAXFD = os.sysconf("SC_OPEN_MAX")
except:
    MAXFD = 256

and

if hasattr(os, 'closerange'):  # Introduced in 2.6
    @staticmethod
    def _closerange(fd_low, fd_high):
        os.closerange(fd_low, fd_high)
else:
    @staticmethod
    def _closerange(fd_low, fd_high):
        for fd in xrange(fd_low, fd_high):
            while True:
                try:
                    os.close(fd)
                except (OSError, IOError), e:
                    if e.errno == errno.EINTR:
                        continue
                    break

I didn't know close_fds implementation is so... unsophisticated.

[dsansome, 2015/09/24 04:34:13]

I just picked a high number :(
It looks like Python 3 is a bit smarter and looks in /proc/self/fd for a list of
things to close:
https://github.com/python/cpython/blob/829b49cbd2e4b1d573470da79ca844b730120f...

     if fd in keep_fds:
       continue
@@ -131,6 +120,29 @@ def become_daemon(keep_fds=None):
   # Open /dev/null on stdin, stdout and stderr.
   null = os.open(os.devnull, os.O_RDWR)
   for i in range(3):
-    os.dup2(null, i)
+    if i not in keep_fds:
+      os.dup2(null, i)
+
+
+def become_daemon(keep_fds=None):
+  """Makes this process a daemon process.
+
+  Starts a new process group, closes all open file handles, opens /dev/null on
+  stdin, stdout and stderr, and changes the current working directory to /.
+
+  Args:
+    keep_fds: One of:
+      None: close all FDs
+      True: keep all FDs (don't close any)
+      a set: close all FDs except the ones in this set.
+  """
+
+  _fork_then_exit_parent()
+  os.setsid()
+  _fork_then_exit_parent()
+
+  if keep_fds != True:
+    # Close all open files.
+    close_all_fds(keep_fds)
 
   os.chdir('/')