[catapult] - Copy Telemetry's gsutilz over to third_party.

third_party/gsutil/gslib/addlhelp/security.py

+# -*- coding: utf-8 -*-
+# Copyright 2015 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Additional help about types of credentials and authentication."""
+
+from __future__ import absolute_import
+
+from gslib.help_provider import HelpProvider
+
+_DETAILED_HELP_TEXT = ("""
+<B>OVERVIEW</B>
+  This help section provides details about various precautions taken by gsutil
+  to protect data security, as well as recommendations for how customers should
+  safeguard security.
+
+
+<B>TRANSPORT LAYER SECURITY</B>
+  gsutil performs all operations using transport-layer encryption (HTTPS), to
+  protect against data leakage over shared network links. This is also important
+  because gsutil uses "bearer tokens" for authentication (OAuth2) as well as for
+  resumable upload identifiers, and such tokens must be protected from being
+  eavesdropped and reused.
+
+  gsutil also supports the older HMAC style of authentication via the XML API
+  (see "gsutil help apis").  While HMAC authentication does not use bearer
+  tokens (and thus is not subject to eavesdropping/replay attacks), it's still
+  important to encrypt data traffic.
+
+  Prior to gsutil release 4.0 it was possible to use HTTP instead of HTTPS by
+  setting the "is_secure" configuration parameter in the [Boto] section of the
+  boto configuration file to False. However, starting with gsutil version 4.0
+  setting is_secure to False is disallowed.  For more details about different
+  credential options, see "gsutil help creds".
+
+
+<B>LOCAL FILE STORAGE SECURITY</B>
+  gsutil takes a number of precautions to protect against security exploits in
+  the files it stores locally:
+
+    - When the gsutil config command runs it sets file protection mode 600
+      ("-rw-------") on the the .boto configuration file it generates, so only
+      the user (or superuser) can read it. This is important because these files
+      contain security-sensitive information, including credentials and proxy
+      configuration.
+
+    - The gsutil config command also uses file protection mode 600 for the
+      private key file stored locally when you create service account
+      credentials.
+
+    - The default level of logging output from gsutil commands does not include
+      security-sensitive information, such as OAuth2 tokens and proxy
+      configuration information. (See the "RECOMMENDED USER PRECAUTIONS" section
+      below if you increase the level of debug output, using the gsutil -D
+      option.)
+
+  Note that protection modes are not supported on Windows, so if you
+  use gsutil on Windows we recommend using an encrypted file system and strong
+  account passwords.
+
+
+<B>SECURITY-SENSITIVE FILES WRITTEN TEMPORARILY TO DISK BY GSUTIL</B>
+    gsutil buffers data in temporary files in several situations:
+
+      - While compressing data being uploaded via gsutil cp -z, gsutil
+        buffers the data in temporary files with protection 600, which it
+        deletes after the upload is complete (similarly for downloading files
+        that were uploaded with gsutil cp -z or some other process that sets the
+        Content-Encoding to "gzip"). However, if you kill the gsutil process
+        while the upload is under way the partially written file will be left
+        in place. See the "CHANGING TEMP DIRECTORIES" section in
+        "gsutil help cp" for details of where the temporary files are written
+        and how to change the temp directory location.
+
+      - When performing a resumable upload gsutil stores the upload ID (which,
+        as noted above, is a bearer token and thus should be safe-guarded) in a
+        file under ~/.gsutil/tracker-files with protection 600, and deletes this
+        file after the upload is complete. However, if the upload doesn't
+        complete successfully the tracker file is left in place so the resumable
+        upload can be re-attempted later. Over time it's possible to accumulate
+        these tracker files from aborted upload attempts, though resumable
+        upload IDs are only valid for 1 week, so the security risk only exists
+        for files less than that old. If you consider the risk of leaving
+        aborted upload IDs in the tracker directory too high you could modify
+        your upload scripts to delete the tracker files; or you could create a
+        cron job to clear the tracker directory periodically.
+
+      - The gsutil rsync command stores temporary files (with protection 600)
+        containing the names, sizes, and checksums of source and destination
+        directories/buckets, which it deletes after the rsync is complete.
+        However, if you kill the gsutil process while the rsync is under way the
+        listing files will be left in place.
+
+    Note that gsutil deletes temporary files using the standard OS unlink system
+    call, which does not perform `data wiping
+    <https://en.wikipedia.org/wiki/Data_erasure>`_. Thus, the content of such
+    temporary files can be recovered by a determined adversary.
+
+
+<B>ACCESS CONTROL LISTS</B>
+  Unless you specify a different ACL (e.g., via the gsutil cp -a option), by
+  default objects written to a bucket use the default object ACL on that bucket.
+  Unless you modify that ACL (e.g., via the gsutil defacl command), by default
+  it will allow all project editors write access to the object and read/write
+  access to the object's metadata; and will allow all project viewers read
+  access to the object.
+
+  The GCS access control system includes the ability to specify that objects are
+  publicly readable. Make sure you intend for any objects you write with this
+  permission to be public. Once "published", data on the Internet can be copied
+  to many places, so it's effectively impossible to regain read control over an
+  object written with this permission.
+
+  The GCS access control system includes the ability to specify that buckets are
+  publicly writable. While configuring a bucket this way can be convenient for
+  various purposes, we recommend against using this permission - it can be
+  abused for distributing illegal content, viruses, and other malware, and the
+  bucket owner is legally and financially responsible for the content stored in
+  their buckets. If you need to make content available to customers who don't
+  have Google accounts consider instead using signed URLs (see
+  "gsutil help signurl").
+
+
+<B>SOFTWARE INTEGRITY AND UPDATES</B>
+  gsutil is distributed as a standalone bundle via tar and zip files stored in
+  the gs://pub bucket, as a PyPi module, and as part of the bundled Cloud
+  SDK release. Each of these distribution methods takes a variety of security
+  precautions to protect the integrity of the software. We strongly recommend
+  against getting a copy of gsutil from any other sources (such as mirror
+  sites).
+
+
+<B>PROXY USAGE</B>
+  gsutil supports access via proxies, such as Squid and a number of commercial
+  products. A full description of their capabilities is beyond the scope of this
+  documentation, but proxies can be configured to support many security-related
+  functions, including virus scanning, Data Leakage Prevention, control over
+  which certificates/CA's are trusted, content type filtering, and many more
+  capabilities. Some of these features can slow or block legitimate gsutil
+  behavior. For example, virus scanning depends on decrypting file content,
+  which in turn requires that the proxy terminate the gsutil connection and
+  establish a new connection - and in some cases proxies will rewrite content in
+  ways that result in checksum validation errors and other problems.
+
+  For details on configuring proxies see the proxy help text in your .boto
+  configuration file (generated by the gsutil config command).
+
+
+<B>ENCRYPTION AT REST</B>
+  All GCS data are stored encrypted. For more information see
+  `Server-Side Encryption
+  <https://cloud.google.com/storage/docs/concepts-techniques#encryption>`_.
+  
+
+<B>DATA PRIVACY FROM GOOGLE EMPLOYEES</B>
+  Google employees will never look at your data unless you first explicitly
+  grant them permission to do so while troubleshooting a specific incident.
+
+  Google will never ask you to share your credentials, password, or other
+  security-sensitive information. Beware of potential phishing scams where
+  someone attempts to impersonate Google and asks for such information.
+
+
+<B>MEASUREMENT DATA</B>
+  The gsutil perfdiag command collects a variety of performance-related
+  measurements and details about your local system and network environment, for
+  use in troubleshooting performance problems. None of this information will be
+  sent to Google unless you choose to send it.
+
+
+<B>RECOMMENDED USER PRECAUTIONS</B>
+  The first and foremost precaution is: Never share your credentials. Each user
+  should have distinct credentials.
+
+  If you run gsutil -D (to generate debugging output) it will include OAuth2
+  refresh and access tokens in the output. Make sure to redact this information
+  before sending this debug output to anyone during troubleshooting/tech support
+  interactions.
+
+  The proxy configuration information in the .boto configuration is
+  security-sensitive, especially if your proxy setup requires user and
+  password information. Even if your proxy setup doesn't require user and
+  password, the host and port number for your proxy is often considered
+  security-sensitive. Protect access to your .boto configuration file.
+
+  If you are using gsutil from a production environment (e.g., via a cron job
+  running on a host in your data center), use service account credentials rather
+  than individual user account credentials. These credentials were designed for
+  such use and, for example, protect you from losing access when an employee
+  leaves your company.
+""")
+
+
+class CommandOptions(HelpProvider):
+  """Additional help about security and privacy considerations using gsutil."""
+
+  # Help specification. See help_provider.py for documentation.
+  help_spec = HelpProvider.HelpSpec(
+      help_name='security',
+      help_name_aliases=['encryption', 'protection', 'privacy', 'proxies',
+                         'proxy'],
+      help_type='additional_help',
+      help_one_line_summary='Security and Privacy Considerations',
+      help_text=_DETAILED_HELP_TEXT,
+      subcommand_help_text={},
+  )