Introduce the constrained network server.

media/tools/constrained_network_server/cns.py

+#!/usr/bin/python
+
+# Copyright (c) 2011 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Constrained Network Server. Serves files with supplied network constraints.
+
+The CNS exposes a web based API allowing network constraints to be imposed on
+file serving.
+"""
+
+import optparse
+import os
+import sys
+import threading
+import time
+
+try:
+  import cherrypy
+except ImportError:
+  print ('CNS requires CherryPy v3 or higher to be installed. Please install\n'
+         'and try again. On Linux: sudo apt-get install python-cherrypy3\n')
+  sys.exit(1)
+
+
+# Default port to serve the CNS on.
+_DEFAULT_SERVING_PORT = 80

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

Does this script require running as root?
(IWBN not to)

[DaleCurtis, 2011/11/15 18:46:00]

Technically this script doesn't. However shadi's tc scripts will need to be
root. I can think of a few options:

- Start the CNS as root.
- Start the CNS as a user w/ password-less sudo.
- Make a suid wrapper for the TC scripts.

Any others?

On 2011/11/15 17:54:07, Ami Fischman wrote:

[Ami GONE FROM CHROMIUM, 2011/11/16 03:11:22]

I think suid for tc scripts makes most sense.
So defaulting this port to something >1024 would be nice (AFAICT this is the
only thing requiring root by default in this script).

[DaleCurtis, 2011/11/16 20:20:15]

Done. Used 9000 since layout tests run on 8000.

On 2011/11/16 03:11:22, Ami Fischman wrote:

+
+# Default port range for constrained use.
+_DEFAULT_CNS_PORT_RANGE = (50000, 51000)
+
+# Default number of seconds before a port can be torn down.
+_DEFAULT_PORT_EXPIRY_TIME_SECS = 5 * 60
+
+
+class PortAllocator(object):
+  """Dynamically allocates/deallocates ports with a given set of constraints."""
+
+  def __init__(self, port_range, expiry_time_secs=5 * 60):
+    """Sets up initial state for the Port Allocator.
+
+    Args:
+      port_range: Range of ports available for allocation.
+      expiry_time_secs: Amount of time in seconds before constrained ports are
+          cleaned up.
+    """
+    self._port_range = port_range
+    self._expiry_time_secs = expiry_time_secs
+
+    # Keeps track of ports we've used, the creation key, and the last request
+    # time for the port so they can be cached and cleaned up later.
+    self._ports = {}
+
+    # Locks port creation and cleanup. TODO(dalecurtis): If performance becomes
+    # an issue a per-port based lock system can be used instead.
+    self._port_lock = threading.Lock()
+
+  def Get(self, key, **kwargs):
+    """Sets up a constrained port using the requested parameters.
+
+    Requests for the same key and constraints will result in a cached port being
+    returned if possible.
+
+    Args:
+      key: Used to cache ports with the given constraints.
+      **kwargs: Constraints to pass into ***TODO*** ConstraintGenerator...

[scherkus (not reviewing), 2011/11/16 02:25:42]

TODO(dalecurtis)?

[DaleCurtis, 2011/11/16 02:35:56]

Done.

+
+    Returns:
+      None if no port can be setup or the port number of the constrained port.
+    """
+    with self._port_lock:
+      # Check port key cache to see if this port is already setup. Update the
+      # cache time and return the port if so. Performance isn't a concern here,
+      # so just iterate over ports dict for simplicity.
+      full_key = (key,) + tuple(kwargs.values())
+      for port, status in self._ports.iteritems():
+        if full_key == status['key']:
+          self._ports[port]['last_update'] = time.time()
+          return port
+
+      # Cleanup ports on new port requests. Do it after the cache check though
+      # so we don't erase and then setup the same port.
+      self.Cleanup(locked=True)
+
+      # Performance isn't really an issue here, so just iterate over the port
+      # range to find an unused port. If no port is found, None is returned.
+      for port in xrange(self._port_range[0], self._port_range[1]):
+        if port in self._ports:
+          continue
+
+        # TODO(dalecurtis): Integrate with shadi's scripts.
+        # We've found an open port so call the script and set it up.
+        #Port.Setup(port=port, **kwargs)
+
+        self._ports[port] = {'last_update': time.time(), 'key': full_key}
+        return port
+
+  def Cleanup(self, all_ports=False, locked=False):
+    """Cleans up expired ports, or if all=True, all allocated ports.
+
+    By default, ports which haven't been used for _PORT_EXPIRATION_TIME_SECS are
+    torn down. If all=True then they are torn down regardless.
+
+    Args:
+      all_ports: Should all ports be torn down regardless of expiration?
+      locked: Has a lock already been acquired in this context?
+    """
+    if not locked:
+      self._port_lock.acquire()
+
+    try:
+      # Use .items() instead of .iteritems() so we can delete keys w/o error.
+      for port, status in self._ports.items():
+        expired = time.time() - status['last_update'] > self._expiry_time_secs

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

Usually calling time.time() once per function (and assigning to a local "now"
var) makes your life better in the long run as you can reason about the behavior
of the function when looking at a dump of the current state, instead of having
to dump state on each iteration.

[DaleCurtis, 2011/11/15 18:46:00]

Done. You make a good case.

On 2011/11/15 17:54:07, Ami Fischman wrote:

+        if all_ports or expired:
+          cherrypy.log('Cleaning up port %d' % port)
+
+          # TODO(dalecurtis): Integrate with shadi's scripts.
+          #Port.Delete(port=port)
+
+          del self._ports[port]
+    finally:
+      if not locked:
+        self._port_lock.release()

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

In English this stanza looks wrong (if not locked, how can we release the
lock?). 
There's a code smell in the current impl in that locking is a class-internal
concept, so outside callers shouldn't need to know about locking, so they
shouldn't be able to specify the locked param.

Two ways to avoid this are:
- having a public Cleanup & a private _Cleanup_Locked:
  def Cleanup(self, all_ports=False)
    with self._port_lock:
      self._Cleanup_Locked(all_ports)
  def _Cleanup_Locked(self, all_ports=False)
    ...
and then have locked=true callers call self._Cleanup_Locked instead of Cleanup()

- using threading.RLock (reentrant) and always acquire/release'ing (with) in
Cleanup.

Personally I think the first way is better (reentrancy is generally far more
subtle than you care to think through).

[DaleCurtis, 2011/11/15 18:46:00]

Done. Agreed on the first solution.

On 2011/11/15 17:54:07, Ami Fischman wrote:

+
+
+class ConstrainedNetworkServer(object):
+  """A CherryPy-based HTTP server for serving files with network constraints."""
+
+  def __init__(self, options, port_allocator):
+    """Sets up initial state for the CNS.
+
+    Args:
+      options: optparse based class returned by ParseArgs()
+      port_allocator: A port allocator instance.
+    """
+    self._options = options
+    self._port_allocator = port_allocator
+
+  @cherrypy.expose
+  def ServeConstrained(self, file=None, bandwidth=None, latency=None,
+                       loss=None):
+    """Serves the requested file with the requested constraints.
+
+    Subsequent requests for the same constraints from the same IP will share the
+    previously created port. If no constraints are provided the file is served
+    as is.
+
+    Args:
+      file: path relative to http root of file to serve.
+      bandwidth: maximum allowed bandwidth for the provided port (integer
+          in kbit/s).
+      latency: time to add to each request (integer in ms).

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

Latency in my mind was always the ms to add to each TCP packet, not to the
overall request.  Have we been miscommunicating or is this just a poorly chosen
phrasing?

[DaleCurtis, 2011/11/15 18:46:00]

Poor phrasing on my part. I'll clean it up a bit. Latency is controlled via TC,
so to be specific this is what it's doing:

http://www.linuxfoundation.org/collaborate/workgroups/networking/netem#Emulat...

On 2011/11/15 17:54:07, Ami Fischman wrote:

+      loss: percentage of packets to drop (integer, 0-100).
+    """
+    # CherryPy is a bit wonky at detecting parameters, so just make them all
+    # optional and validate them ourselves.
+    if not file:
+      raise cherrypy.HTTPError(400, 'Invalid request. File must be specified.')
+
+    # Sanitize and check the path to prevent www-root escapes.
+    sanitized_path = os.path.abspath(os.path.join(self._options.www_root, file))

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

Personally I'd overwrite |file| at this point, to make sure future lines only
use the sanitized version.

[DaleCurtis, 2011/11/15 18:46:00]

Done.

[DaleCurtis, 2011/11/16 02:17:39]

Actually can't do this because the relative/unsanitized path needs to be passed
back to the redirect link.

On 2011/11/15 18:46:00, DaleCurtis wrote:

+    if not sanitized_path.startswith(self._options.www_root):
+      raise cherrypy.HTTPError(403, 'Invalid file requested.')
+
+    # Check existence early to prevent wasted constraint setup.
+    if not os.path.exists(sanitized_path):
+      raise cherrypy.HTTPError(404, 'File not found.')
+
+    # If there are no constraints, just serve the file.
+    if bandwidth is None and latency is None and loss is None:
+      return cherrypy.lib.static.serve_file(sanitized_path)
+
+    # Validate inputs. isdigit() guarantees a natural number.
+    if bandwidth and not bandwidth.isdigit():
+      raise cherrypy.HTTPError(400, 'Invalid bandwidth constraint.')
+
+    if latency and not latency.isdigit():
+      raise cherrypy.HTTPError(400, 'Invalid latency constraint.')
+
+    if loss and not loss.isdigit() and not int(loss) <= 100:
+      raise cherrypy.HTTPError(400, 'Invalid loss constraint.')
+
+    # Allocate a port using the given constraints. If a port with the requested
+    # key is already allocated, it will be reused.
+    #
+    # TODO(dalecurtis): The key cherrypy.request.remote.ip might not be unique
+    # if build slaves are sharing the same VM.
+    constrained_port = self._port_allocator.Get(
+        cherrypy.request.remote.ip, bandwidth=bandwidth, latency=latency,
+        loss=loss)
+
+    if not constrained_port:
+      raise cherrypy.HTTPError(503, 'Service unavailable. Out of ports.')
+
+    # Build constrained URL. Only pass on the file parameter.
+    constrained_url = '%s?f=%s' % (
+        cherrypy.url().replace(
+            ':%d' % self._options.port, ':%d' % constrained_port),
+        file)
+
+    # Redirect request to the constrained port.
+    cherrypy.lib.cptools.redirect(constrained_url, internal=False)
+
+
+def ParseArgs():
+  """Define and parse the command-line arguments."""
+  parser = optparse.OptionParser()
+
+  parser.add_option('--expiry-time', type='int',
+                    default=_DEFAULT_PORT_EXPIRY_TIME_SECS,
+                    help=('Number of seconds before constrained ports expire '
+                          ' and are cleaned up. Default: %default'))

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

s/' /'/

[DaleCurtis, 2011/11/15 18:46:00]

Done.

[Ami GONE FROM CHROMIUM, 2011/11/16 03:11:22]

Actually you did the opposite of what I suggested.
IMO it's better to leave the inter-word space on the previous line because it
makes indentation look better (rather than having a dangling apostrophe).  (here
and below)
Matter of taste.

[DaleCurtis, 2011/11/16 20:20:15]

Ah, I thought you were just pointing out my accidental double spacing, not a
style nit. Done.

On 2011/11/16 03:11:22, Ami Fischman wrote:

+  parser.add_option('--port', type='int', default=_DEFAULT_SERVING_PORT,
+                    help='Port to serve the API on. Default: %default')
+  parser.add_option('--port-range', default=_DEFAULT_CNS_PORT_RANGE,
+                    help=('Range of ports for constrained serving. Specify as '
+                          ' a comma separated value pair. Default: %default'))

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

s/' /'/

[DaleCurtis, 2011/11/15 18:46:00]

Done.

+  parser.add_option('--local', action='store_true', default=False,

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

Unused?

[DaleCurtis, 2011/11/15 18:46:00]

Currently, yes. I need to talk to shadi to work this out properly. The tc
commands for local vs eth are different.

With a fresh mind this morning, it might actually be better to name this
--interface and accept eth# or lo. I can then lookup the IP for each interface
and set cherrypy to only bind there.

On 2011/11/15 17:54:07, Ami Fischman wrote:

+                    help='Setup server for use with a local client.')
+  parser.add_option('--threads', type='int',
+                    help='Number of threads in the thread pool.')

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

doco omission semantics.

[DaleCurtis, 2011/11/15 18:46:00]

Done. I'll forward the default value from:
cherrypy._cpserver.Server.thread_pool

On 2011/11/15 17:54:07, Ami Fischman wrote:

+  parser.add_option('--www-root', default=os.getcwd(),
+                    help=('Port to serve the API on. Defaults to the current '

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

s/Port to serve the API on./Directory root of servable files./

[DaleCurtis, 2011/11/15 18:46:00]

Done.

+                          'directory: %default'))
+
+  options = parser.parse_args()[0]
+
+  # Convert port range into the desired tuple format.
+  try:
+    if isinstance(options.port_range, str):
+      options.port_range = [int(port) for port in options.port_range.split(',')]
+  except ValueError:
+    parser.error('Invalid port range specified.')
+
+  # Normalize the path to remove any . or ..
+  options.www_root = os.path.normpath(options.www_root)
+
+  return options
+
+
+def Main():
+  """Configure and start the ConstrainedNetworkServer."""
+  options = ParseArgs()
+
+  cherrypy.config.update(
+      {'server.socket_host': '::', 'server.socket_port': options.port})
+
+  if options.threads:
+    cherrypy.config.update({'server.thread_pool': options.threads})
+
+  # Setup port allocator here so we can call cleanup on failures/exit.
+  pa = PortAllocator(options.port_range, expiry_time_secs=options.expiry_time)
+
+  try:
+    cherrypy.quickstart(ConstrainedNetworkServer(options, pa))
+  finally:
+    pa.Cleanup(all_ports=True)

[Ami GONE FROM CHROMIUM, 2011/11/15 17:54:07]

Does this run in case of ctrl-c?  Or in what cases does it run?

[DaleCurtis, 2011/11/15 18:46:00]

Yes, it will always run. Although it doesn't disable Ctrl-C so if you were to
hit it multiple times it will abort the cleanup call. It's a bit messy, but I
can trap Ctrl-C during this phase if necessary.

On 2011/11/15 17:54:07, Ami Fischman wrote:

[Ami GONE FROM CHROMIUM, 2011/11/16 03:11:22]

I don't think that's needed; a ctrl-c that aborted cleanup would also exit the
process, at which point there's no need for more cleanup (unless cherrypy
daemonizes a fork?)

[DaleCurtis, 2011/11/16 20:20:15]

Well if cleanup gets aborted the interface will be left in an unknown state due
to tc/iptables rules. If that's okay I can remove the code I added to trap
Ctrl-C. Cherrypy doesn't fork or daemonize.

On 2011/11/16 03:11:22, Ami Fischman wrote:

[Ami GONE FROM CHROMIUM, 2011/11/16 20:28:27]

That's a good point - we should not let tc/iptables config bleed out after this
script exits.

+
+
+if __name__ == '__main__':
+  Main()