Add build mailer capability to support gatekeeper_ng.

mailer.py

+import datetime
+from google.appengine.ext import db

[Vadim Sh., 2013/08/26 21:30:41]

ndb? 

It will speed up secret fetch by automagically caching it in memcache.

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

Done.

+from google.appengine.api import mail
+import hashlib
+import hmac

[Vadim Sh., 2013/08/26 21:30:41]

nit: reorganize imports order:
1. stdlib.
2. other third party libs (google.* and webapp2 in that case).
3. modules belonging to this app.

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

Done.

+import logging
+import json
+import webapp2
+from webapp2_extras import jinja2
+
+import gatekeeper_mailer
+
+
+class BaseHandler(webapp2.RequestHandler):
+  """Provide a cached Jinja environment to each request."""
+  @webapp2.cached_property
+  def jinja2(self):
+    # Returns a Jinja2 renderer cached in the app registry.
+    return jinja2.get_jinja2(app=self.app)
+
+  def render_response(self, _template, **context):
+    # Renders a template and writes the result to the response.
+    rv = self.jinja2.render_template(_template, **context)
+    self.response.write(rv)
+
+
+class MainPage(BaseHandler):
+  def get(self):
+    context = {'title': 'Chromium Gatekeeper Mailer'}
+    self.render_response('main_mailer.html', **context)
+
+class MailerSecret(db.Model):

[Vadim Sh., 2013/08/26 21:30:41]

nit: Move this class to the top of the page before BaseHandler. I think it sort
of a convention to put models on top...

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

Done.

+  """Model to represent the shared secret for the mail endpoint."""
+  secret = db.StringProperty()
+
+class Email(BaseHandler):
+  @staticmethod
+  def _validate_message(message, secret):
+    # Make this backwards compatable with python 2.6, since total_seconds()
+    # exists only in python 2.7.

[Vadim Sh., 2013/08/26 21:30:41]

Is this comment still relevant?

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

nope

+    utc_now_td = (datetime.datetime.utcnow() -
+        datetime.datetime.utcfromtimestamp(0))
+    utc_now = utc_now_td.days * 86400 + utc_now_td.seconds

[Vadim Sh., 2013/08/26 21:30:41]

And I think all this can be reduced to time.time(). (I hate datetime module
too).

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

Done.

+
+    if (utc_now < message['time']) or (utc_now - message['time'] > 60):

[Vadim Sh., 2013/08/26 21:30:41]

abs(utc_now - message['time']) > 60

clock skew can happen in any direction

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

Done.

+      logging.error('message was rejected due to time')
+      return False
+
+    hasher = hmac.new(secret, '%s:%d:%d' % (message['message'],
+                                            message['time'],
+                                            message['salt']),
+                      hashlib.sha256)
+
+    client_hash = hasher.hexdigest()
+
+    return client_hash == message['sha256']

[Vadim Sh., 2013/08/26 21:30:41]

Usually this comparison is done using constant time algorithm that always runs
same number of steps regardless of actual inputs to prevent timing attacks:

if len(a) != len(b):
  return False
result = 0
for x, y in zip(a, b):
  result |= ord(x) ^ ord(y)
return result == 0


(Though I myself don't believe it's of any difference for python apps on
appengine...)

[iannucci, 2013/08/27 21:45:20]

((Actually it can make a difference. I've used a padding oracle attack to get
into a python appengine app before [part of a crypto class]. Just takes a few
more runs to eliminate timing noise)).

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

Done.

+
+
+  @staticmethod
+  def _verify_json(build_data):
+    fields = ['waterfall_url',
+              'build_url',
+              'project_name',
+              'builderName',
+              'steps',
+              'unsatisfied',
+              'revisions',
+              'blamelist',
+              'result',
+              'number',
+              'changes',
+              'reason',
+              'recipients']
+
+    for field in fields:
+      if field not in build_data:
+        logging.error('build_data did not contain field %s' % field)
+        return False
+
+    stepfields = ['started',

[Vadim Sh., 2013/08/26 21:30:41]

nit: step_fields

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

Done.

+                  'text',
+                  'results',
+                  'name',
+                  'logs',
+                  'urls']
+
+    if not build_data['steps']:
+      logging.error('build_data did not contain any steps')
+      return False
+    for step in build_data['steps']:
+      for field in stepfields:
+        if field not in step:
+          logging.error('build_step did not contain field %s' % field)
+          return False
+
+    return True
+
+  def post(self):
+    blob = self.request.get('json')

[Vadim Sh., 2013/08/26 21:30:41]

I'm not sure what part of the system is sending this.. would it be simpler just
to put json directly into POST body with content-type application/json instead
of application/x-www-form-urlencoded?

Alternatively, send 4 separate POST parameters: 'message', 'time', 'salt',
'sha256'.

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

that's scripts/slave/gatekeeper_ng.py. I'll make a separate CL there, then
update this

+    if not blob:
+      self.response.out.write('no json data sent')
+      logging.error('error no json sent')
+      self.error(400)
+      return
+
+    message = {}
+    try:
+      message = json.loads(blob)
+    except ValueError as e:
+      self.response.out.write('couldn\'t decode json')
+      logging.error('error decoding incoming json: %s' % e)
+      self.error(400)
+      return
+
+    secret = str(MailerSecret.get_or_insert('mailer_secret').secret)

[Vadim Sh., 2013/08/26 21:30:41]

Why 'str' here?
I'm thinking about a possibility of 'secret' to be None. In that case str will
convert it to 'None' which is undesirable.

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

stuff from db came in as unicode, while stuff from the web wasn't. I'll look
into another way of converting it

+    if not secret:
+      self.response.out.write('unauthorized')
+      logging.critical('mailer shared secret has not been set!')
+      self.error(500)
+      return
+
+    if not self._validate_message(message, secret):
+      self.response.out.write('unauthorized')
+      logging.error('incoming message did not validate')
+      self.error(403)
+      return
+
+    build_data = json.loads(message['message'])

[Vadim Sh., 2013/08/26 21:30:41]

Catch ValueError here as well?

[Mike Stip (use stip instead), 2013/08/29 19:56:46]

Done.

+
+    if not self._verify_json(build_data):
+      logging.error('error verifying incoming json: %s' % build_data)
+      self.response.out.write('json build format is incorrect')
+      self.error(400)
+      return
+
+    from_addr = build_data.get('from_addr', 'buildbot@chromium.org')
+    recipients = ', '.join(build_data['recipients'])
+
+    template = gatekeeper_mailer.MailTemplate(build_data['waterfall_url'],
+                                              build_data['build_url'],
+                                              build_data['project_name'],
+                                              from_addr)
+
+
+    text_content, html_content, subject = template.genMessageContent(build_data)
+
+    message = mail.EmailMessage(sender=from_addr,
+                                subject=subject,
+                                #to=recipients,
+                                to=['xusydoc@chromium.org'],
+                                body=text_content,
+                                html=html_content)
+    logging.info('sending email to %s', recipients)
+    message.send()
+    self.response.out.write('email sent')
+
+
+app = webapp2.WSGIApplication([('/mailer', MainPage),
+                               ('/mailer/email', Email)],
+                              debug=True)