Suspend (rather than shut down) the host upon policy errors.

remoting/host/remoting_me2me_host.cc

Index: remoting/host/remoting_me2me_host.cc
diff --git a/remoting/host/remoting_me2me_host.cc b/remoting/host/remoting_me2me_host.cc
index b66f8376f791bea52a9ff8cc57ce6339324331d2..4e5bffe35dd4c1b83a52a5743e8939eb2f2098a3 100644
--- a/remoting/host/remoting_me2me_host.cc
+++ b/remoting/host/remoting_me2me_host.cc
@@ -147,6 +147,12 @@ const int kShutdownTimeoutSeconds = 15;
 // before continuing normal process shutdown.
 const int kHostOfflineReasonTimeoutSeconds = 10;
 
+// Host offline reasons not associated with shutting down the host process
+// and therefore not expressible through HostExitCodes enum.
+const char kHostOfflineReasonPolicyReadError[] = "POLICY_READ_ERROR";
+const char kHostOfflineReasonPolicyChangeRequiresRestart[] =
+    "POLICY_CHANGE_REQUIRES_RESTART";
+
 }  // namespace
 
 namespace remoting {
@@ -183,34 +189,83 @@ class HostProcess : public ConfigWatcher::Delegate,
 
  private:
   enum HostState {
-    // Host process has just been started. Waiting for config and policies to be
-    // read from the disk.
+    // Waiting for valid config and policies to be read from the disk.
+    // Either the host process has just been started, or it is trying to start
+    // again after temporarily going offline due to policy change or error.
     HOST_INITIALIZING,

[Wez, 2015/02/14 00:49:15]

Suggest renaming this to HOST_STARTING since it can now be hit multiple times.

[Łukasz Anforowicz, 2015/02/17 21:02:27]

Done.

 
     // Host is started and running.
     HOST_STARTED,
 
-    // Host is being stopped and will need to be started again.
-    HOST_STOPPING_TO_RESTART,
+    // Host is sending offline reason, before trying to (re)initialize again.
+    // Used when
+    // 1) policy change (or a config change in the future) requires a restart.

[Wez, 2015/02/14 00:49:15]

How can we detect a config change in the future...?

[Łukasz Anforowicz, 2015/02/17 21:02:27]

I meant that a config change can theoretically also require a restart (i.e.
Sergey's TODO in onConfigUpdate says [emphasis mine]: "[...] Change
ApplyConfig() to detect other changes in the config and *restart* host if
necessary here.") and that such restart would be implemented in the future.

I'll just remove the text in parens.  

+    // 2) policy error puts the host offline until we have a valid policy again.
+    HOST_GOING_OFFLINE_TO_RESTART,

[Wez, 2015/02/14 00:49:15]

Why not simply HOST_RESTARTING or HOST_STOPPING_TO_RESTART?

[Łukasz Anforowicz, 2015/02/17 21:02:27]

I like your suggestion for its brevity, but I think
HOST_GOING_OFFLINE_TO_RESTART and HOST_GOING_OFFLINE_TO_STOP are slightly better
so I will stick with them (at least for now):

1) I see "restarting" and "stopping" as something that begins *after*
OnHostOfflineReasonAck.  The state *before* OnHostOfflineReasonAck needs a
separate name IMO (i.e. "going offline").

2) The meaning of "stopping" and "shutting down" is quite overloaded and
ambiguous (i.e. I dislike the old ShutdownOnNetworkingThread name which
sometimes does a real shutdown via ShutdownOnUiThread and sometimes doesn't). 
Having a separate name for "going offline" helps avoid the ambiguous "shutdown"
or "stopping".

 
-    // Host is being stopped.
-    HOST_STOPPING,
+    // Host is sending offline reason, before shutting down.
+    HOST_GOING_OFFLINE_TO_STOP,

[Wez, 2015/02/14 00:49:15]

Why not simply HOST_STOPPING as it was before?

[Łukasz Anforowicz, 2015/02/17 21:02:27]

Please see my other comment.

 
-    // Host has been stopped.
+    // Host has been stopped (host process will end soon).
     HOST_STOPPED,
 

[Łukasz Anforowicz, 2015/02/13 20:41:00]

Summary of the changes I made to "allowed state transitions":

STOPPED->STARTED is the only transition that I deleted altogether (reading the
code I don't
see how this transition could possibly happen).

I renamed STOPPING_TO_RESTART to GOING_OFFLINE_TO_RESTART and renamed STOPPING
to GOING_OFFLINE_TO_STOP.

There is no direct transition from INITIALIZING to STOPPED.  This always goes
through GOING_OFFLINE_TO_STOP first.

Also - after my changes there is no direct transition from STOPPING_TO_RESTART
to STARTED.  Right now it always (sometimes very briefly/synchronously) goes
through intermediate INITIALIZING state.

And there is one new transition because of OnPolicyError - going from
INITIALIZING to GOING_OFFLINE_TO_RESTART.

-    // Allowed state transitions:
-    //   INITIALIZING->STARTED
-    //   INITIALIZING->STOPPED
-    //   STARTED->STOPPING_TO_RESTART
-    //   STARTED->STOPPING
-    //   STOPPING_TO_RESTART->STARTED
-    //   STOPPING_TO_RESTART->STOPPING
-    //   STOPPING->STOPPED
-    //   STOPPED->STARTED
+    // Allowed state transitions (enforced via DCHECKs in SetState method):
+    //   INITIALIZING->STARTED (once we have valid config + policy)
+    //   INITIALIZING->GOING_OFFLINE_TO_STOP
+    //   INITIALIZING->GOING_OFFLINE_TO_RESTART
+    //   STARTED->GOING_OFFLINE_TO_STOP
+    //   STARTED->GOING_OFFLINE_TO_RESTART
+    //   GOING_OFFLINE_TO_RESTART->GOING_OFFLINE_TO_STOP
+    //   GOING_OFFLINE_TO_RESTART->INITIALIZING (after OfflineReasonAck)
+    //   GOING_OFFLINE_TO_STOP->STOPPED (after OfflineReasonAck)
     //
-    // |host_| must be nullptr in INITIALIZING and STOPPED states and not
-    // nullptr in all other states.
+    // |host_| must be not-null in STARTED state and nullptr in all other states
+    // (although this invariant can be temporarily violated when doing
+    // synchronous processing on the networking thread).
+  };
+
+  void SetState(HostState target_state) {
+    DCHECK(context_->network_task_runner()->BelongsToCurrentThread());
+
+    // DCHECKs below enforce state allowed transitions listed in HostState.
+    switch (state_) {
+      case HOST_INITIALIZING:
+        DCHECK((target_state == HOST_STARTED) ||
+               (target_state == HOST_GOING_OFFLINE_TO_STOP) ||
+               (target_state == HOST_GOING_OFFLINE_TO_RESTART));
+        break;
+      case HOST_STARTED:
+        DCHECK((target_state == HOST_GOING_OFFLINE_TO_STOP) ||
+               (target_state == HOST_GOING_OFFLINE_TO_RESTART));
+        break;
+      case HOST_GOING_OFFLINE_TO_RESTART:
+        DCHECK((target_state == HOST_GOING_OFFLINE_TO_STOP) ||
+               (target_state == HOST_INITIALIZING));
+        break;
+      case HOST_GOING_OFFLINE_TO_STOP:
+        DCHECK_EQ(target_state, HOST_STOPPED);
+        break;
+      case HOST_STOPPED:
+      default:
+        NOTREACHED();  // HOST_STOPPED is a terminal state.
+        break;
+    }
+    state_ = target_state;
+  }
+
+  enum PolicyState {
+    // Cannot start the host, because a valid policy has not been read yet.
+    POLICY_INITIALIZING,
+
+    // Policy was loaded successfully.
+    POLICY_LOADED,
+
+    // Policy error was detected, and we haven't yet sent out a
+    // host-offline-reason (i.e. because we haven't yet read the config).
+    POLICY_ERROR_REPORT_PENDING,
+
+    // Policy error was detected, and we have sent out a host-offline-reason.
+    POLICY_ERROR_REPORTED,
   };
 
   friend class base::RefCountedThreadSafe<HostProcess>;
@@ -245,6 +300,7 @@ class HostProcess : public ConfigWatcher::Delegate,
   // Handles policy updates, by calling On*PolicyUpdate methods.
   void OnPolicyUpdate(scoped_ptr<base::DictionaryValue> policies);
   void OnPolicyError();
+  void ReportPolicyErrorAndRestartHost();
   void ApplyHostDomainPolicy();
   void ApplyUsernamePolicy();
   bool OnHostDomainPolicyUpdate(base::DictionaryValue* policies);
@@ -268,16 +324,12 @@ class HostProcess : public ConfigWatcher::Delegate,
   void OnUnknownHostIdError() override;
   void OnAuthFailed() override;
 
-  void RestartHost();
-
-  // Stops the host and shuts down the process with the specified |exit_code|.
+  void RestartHost(const std::string& host_offline_reason);
   void ShutdownHost(HostExitCodes exit_code);
 
-  // Private helper used by ShutdownHost method to initiate sending of
-  // host-offline-reason before continuing shutdown.
-  void SendOfflineReasonAndShutdownOnNetworkThread(HostExitCodes exit_code);
-
-  void ShutdownOnNetworkThread();
+  // Helper methods doing the work needed by RestartHost and ShutdownHost.
+  void GoOffline(const std::string& host_offline_reason);
+  void OnHostOfflineReasonAck(bool success);
 
 #if defined(OS_WIN)
   // Initializes the pairing registry on Windows. This should be invoked on the
@@ -325,7 +377,7 @@ class HostProcess : public ConfigWatcher::Delegate,
   int64_t frame_recorder_buffer_size_;
 
   scoped_ptr<PolicyWatcher> policy_watcher_;
-  bool policies_loaded_;
+  PolicyState policy_state_;
   std::string host_domain_;
   bool host_username_match_required_;
   bool allow_nat_traversal_;
@@ -381,7 +433,7 @@ HostProcess::HostProcess(scoped_ptr<ChromotingHostContext> context,
       use_service_account_(false),
       enable_vp9_(false),
       frame_recorder_buffer_size_(0),
-      policies_loaded_(false),
+      policy_state_(POLICY_INITIALIZING),
       host_username_match_required_(false),
       allow_nat_traversal_(true),
       allow_relay_(true),
@@ -554,9 +606,8 @@ void HostProcess::OnConfigUpdated(
   if (state_ == HOST_INITIALIZING) {
     StartHostIfReady();
   } else if (state_ == HOST_STARTED) {
-    DCHECK(policies_loaded_);
-
     // Reapply policies that could be affected by a new config.
+    DCHECK_EQ(policy_state_, POLICY_LOADED);
     ApplyHostDomainPolicy();
     ApplyUsernamePolicy();
 
@@ -786,7 +837,6 @@ void HostProcess::ShutdownOnUiThread() {
   // Tear down resources that need to be torn down on the UI thread.
   daemon_channel_.reset();
   desktop_environment_factory_.reset();
-
   policy_watcher_.reset();
 
   // It is now safe for the HostProcess to be deleted.
@@ -981,13 +1031,13 @@ void HostProcess::OnPolicyUpdate(scoped_ptr<base::DictionaryValue> policies) {
   restart_required |= OnPairingPolicyUpdate(policies.get());
   restart_required |= OnGnubbyAuthPolicyUpdate(policies.get());
 
-  policies_loaded_ = true;
+  policy_state_ = POLICY_LOADED;
 
   if (state_ == HOST_INITIALIZING) {
     StartHostIfReady();
   } else if (state_ == HOST_STARTED) {
     if (restart_required)
-      RestartHost();
+      RestartHost(kHostOfflineReasonPolicyChangeRequiresRestart);
   }
 }
 
@@ -998,7 +1048,23 @@ void HostProcess::OnPolicyError() {
     return;
   }
 
-  ShutdownHost(kInvalidHostConfigurationExitCode);
+  if (policy_state_ != POLICY_ERROR_REPORTED) {
+    policy_state_ = POLICY_ERROR_REPORT_PENDING;
+    if (((state_ == HOST_INITIALIZING) || (state_ == HOST_STARTED)) &&
+        (!serialized_config_.empty())) {
+      ReportPolicyErrorAndRestartHost();
+    }
+  }
+}
+
+void HostProcess::ReportPolicyErrorAndRestartHost() {
+  DCHECK(context_->network_task_runner()->BelongsToCurrentThread());
+
+  DCHECK_EQ(policy_state_, POLICY_ERROR_REPORT_PENDING);
+  policy_state_ = POLICY_ERROR_REPORTED;
+
+  LOG(INFO) << "Restarting the host due to policy errors.";
+  RestartHost(kHostOfflineReasonPolicyReadError);
 }
 
 void HostProcess::ApplyHostDomainPolicy() {
@@ -1300,11 +1366,10 @@ scoped_ptr<HostSignalingManager> HostProcess::CreateHostSignalingManager() {
                                              oauth_refresh_token_,
                                              use_service_account_));
 
-  return HostSignalingManager::Create(this, context_->network_task_runner(),
-                                      context_->url_request_context_getter(),
-                                      xmpp_server_config_, talkgadget_prefix_,
-                                      host_id_, key_pair_, directory_bot_jid_,
-                                      oauth_credentials.Pass());
+  return HostSignalingManager::Create(
+      this, context_->url_request_context_getter(), xmpp_server_config_,
+      talkgadget_prefix_, host_id_, key_pair_, directory_bot_jid_,
+      oauth_credentials.Pass());
 }
 
 void HostProcess::StartHostIfReady() {
@@ -1312,8 +1377,13 @@ void HostProcess::StartHostIfReady() {
   DCHECK_EQ(state_, HOST_INITIALIZING);
 
   // Start the host if both the config and the policies are loaded.
-  if (!serialized_config_.empty() && policies_loaded_)
-    StartHost();
+  if (!serialized_config_.empty()) {
+    if (policy_state_ == POLICY_LOADED) {
+      StartHost();
+    } else if (policy_state_ == POLICY_ERROR_REPORT_PENDING) {
+      ReportPolicyErrorAndRestartHost();
+    }
+  }
 }
 
 void HostProcess::StartHost() {
@@ -1321,10 +1391,7 @@ void HostProcess::StartHost() {
   DCHECK(!host_);
   DCHECK(!host_signaling_manager_);
 
-  DCHECK(state_ == HOST_INITIALIZING || state_ == HOST_STOPPING_TO_RESTART ||
-         state_ == HOST_STOPPED)
-      << "state_ = " << state_;
-  state_ = HOST_STARTED;
+  SetState(HOST_STARTED);
 
   host_signaling_manager_ = CreateHostSignalingManager();
 
@@ -1409,22 +1476,12 @@ void HostProcess::OnAuthFailed() {
   ShutdownHost(kInvalidOauthCredentialsExitCode);
 }
 
-void HostProcess::RestartHost() {
+void HostProcess::RestartHost(const std::string& host_offline_reason) {
   DCHECK(context_->network_task_runner()->BelongsToCurrentThread());
-  DCHECK_EQ(state_, HOST_STARTED);
-
-  state_ = HOST_STOPPING_TO_RESTART;
-  ShutdownOnNetworkThread();
-}
+  DCHECK(!host_offline_reason.empty());
 
-void HostProcess::SendOfflineReasonAndShutdownOnNetworkThread(
-    HostExitCodes exit_code) {
-  DCHECK(context_->network_task_runner()->BelongsToCurrentThread());
-  DCHECK(host_signaling_manager_);
-  host_signaling_manager_.release()->SendHostOfflineReasonAndDelete(
-      ExitCodeToString(exit_code),
-      base::TimeDelta::FromSeconds(kHostOfflineReasonTimeoutSeconds));
-  ShutdownOnNetworkThread();
+  SetState(HOST_GOING_OFFLINE_TO_RESTART);
+  GoOffline(host_offline_reason);
 }
 
 void HostProcess::ShutdownHost(HostExitCodes exit_code) {
@@ -1434,41 +1491,66 @@ void HostProcess::ShutdownHost(HostExitCodes exit_code) {
 
   switch (state_) {
     case HOST_INITIALIZING:
-      state_ = HOST_STOPPING;
-      DCHECK(!host_signaling_manager_);
-      host_signaling_manager_ = CreateHostSignalingManager();
-      SendOfflineReasonAndShutdownOnNetworkThread(exit_code);
-      break;
-
     case HOST_STARTED:
-      state_ = HOST_STOPPING;
-      SendOfflineReasonAndShutdownOnNetworkThread(exit_code);
+      SetState(HOST_GOING_OFFLINE_TO_STOP);
+      GoOffline(ExitCodeToString(exit_code));
       break;
 
-    case HOST_STOPPING_TO_RESTART:
-      state_ = HOST_STOPPING;
+    case HOST_GOING_OFFLINE_TO_RESTART:
+      SetState(HOST_GOING_OFFLINE_TO_STOP);
       break;
 
-    case HOST_STOPPING:
+    case HOST_GOING_OFFLINE_TO_STOP:
     case HOST_STOPPED:
       // Host is already stopped or being stopped. No action is required.
       break;
   }
 }
 
-void HostProcess::ShutdownOnNetworkThread() {
+void HostProcess::GoOffline(const std::string& host_offline_reason) {
   DCHECK(context_->network_task_runner()->BelongsToCurrentThread());
+  DCHECK(!host_offline_reason.empty());
+  DCHECK((state_ == HOST_GOING_OFFLINE_TO_STOP) ||
+         (state_ == HOST_GOING_OFFLINE_TO_RESTART));
 
+  // Shut down everything except the HostSignalingManager.
   host_.reset();
   host_event_logger_.reset();
   host_status_logger_.reset();
-  host_signaling_manager_.reset();
   host_change_notification_listener_.reset();
 
-  if (state_ == HOST_STOPPING_TO_RESTART) {
-    StartHost();
-  } else if (state_ == HOST_STOPPING) {
-    state_ = HOST_STOPPED;
+  // Before shutting down HostSignalingManager, send the |host_offline_reason|
+  // if possible (i.e. if we have the config).
+  if (!serialized_config_.empty()) {
+    if (!host_signaling_manager_) {
+      host_signaling_manager_ = CreateHostSignalingManager();
+    }
+
+    host_signaling_manager_->SendHostOfflineReason(
+        host_offline_reason,
+        base::TimeDelta::FromSeconds(kHostOfflineReasonTimeoutSeconds),
+        base::Bind(&HostProcess::OnHostOfflineReasonAck, this));
+    return;  // Shutdown will resume after OnHostOfflineReasonAck.
+  }
+
+  // Continue the shutdown without sending the host offline reason.
+  HOST_LOG << "SendHostOfflineReason( " << host_offline_reason << ") "
+           << "not possible without a config...";
+  OnHostOfflineReasonAck(false);
+}
+
+void HostProcess::OnHostOfflineReasonAck(bool success) {

[Wez, 2015/02/14 00:49:15]

nit: It's weird for this to be OnHostOfflineReasonAck without there being any
function in here called anything relating to sending the host offline reason. :/

[Łukasz Anforowicz, 2015/02/17 21:02:27]

This (presence of OnXxx method, without DoXxx method) is similar to other
methods in HostProcess, like OnPolicyUpdate, OnAuthFailed, OnUnknownHostIdError,
OnHeartbeatSuccessful...

+  DCHECK(context_->network_task_runner()->BelongsToCurrentThread());
+  DCHECK(!host_);  // Assert that the host is really offline at this point.
+
+  HOST_LOG << "SendHostOfflineReason " << (success ? "succeeded." : "failed.");
+  host_signaling_manager_.reset();
+
+  if (state_ == HOST_GOING_OFFLINE_TO_RESTART) {
+    SetState(HOST_INITIALIZING);
+    StartHostIfReady();
+  } else if (state_ == HOST_GOING_OFFLINE_TO_STOP) {
+    SetState(HOST_STOPPED);
 
     shutdown_watchdog_->SetExitCode(*exit_code_out_);
     shutdown_watchdog_->Arm();
@@ -1479,7 +1561,6 @@ void HostProcess::ShutdownOnNetworkThread() {
     context_->ui_task_runner()->PostTask(
         FROM_HERE, base::Bind(&HostProcess::ShutdownOnUiThread, this));
   } else {
-    // This method is only called in STOPPING_TO_RESTART and STOPPING states.
     NOTREACHED();
   }
 }