[NaCl SDK Docs] Check in the generated NaCl SDK Documentation.

native_client_sdk/src/doc/_developer.chrome.com_generated/devguide/coding/audio.html

+{{+bindTo:partials.standard_nacl_article}}
+
+<section id="audio">
+<span id="devguide-coding-audio"></span><h1 id="audio"><span id="devguide-coding-audio"></span>Audio</h1>
+<div class="contents local topic" id="contents">
+<ul class="small-gap">
+<li><a class="reference internal" href="#reference-information" id="id1">Reference information</a></li>
+<li><a class="reference internal" href="#about-the-pepper-audio-api" id="id2">About the Pepper audio API</a></li>
+<li><a class="reference internal" href="#digital-audio-concepts" id="id3">Digital audio concepts</a></li>
+<li><a class="reference internal" href="#setting-up-the-module" id="id4">Setting up the module</a></li>
+<li><p class="first"><a class="reference internal" href="#creating-an-audio-configuration-resource" id="id5">Creating an audio configuration resource</a></p>
+<ul class="small-gap">
+<li><a class="reference internal" href="#resources" id="id6">Resources</a></li>
+<li><a class="reference internal" href="#sample-frame-count" id="id7">Sample frame count</a></li>
+<li><a class="reference internal" href="#supported-audio-configurations" id="id8">Supported audio configurations</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#creating-an-audio-resource" id="id9">Creating an audio resource</a></li>
+<li><p class="first"><a class="reference internal" href="#implementing-a-callback-function" id="id10">Implementing a callback function</a></p>
+<ul class="small-gap">
+<li><a class="reference internal" href="#application-threads-and-real-time-requirements" id="id11">Application threads and real-time requirements</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#starting-and-stopping-playback" id="id12">Starting and stopping playback</a></li>
+</ul>
+</div>
+<p>This chapter describes how to use the Pepper audio API to play an audio
+stream. The Pepper audio API provides a low-level means of playing a stream of
+audio samples generated by a Native Client module. The API generally works as
+follows: A Native Client module creates an audio resource that represents an
+audio stream, and tells the browser to start or stop playing the audio
+resource. The browser calls a function in the Native Client module to fill a
+buffer with audio samples every time it needs data to play from the audio
+stream.</p>
+<p>The code examples in this chapter describe a simple Native Client module that
+generates audio samples using a sine wave with a frequency of 440 Hz. The module
+starts playing the audio samples as soon as it is loaded into the browser. For a
+slightly more sophisticated example, see the <code>audio</code> example (source code in
+the SDK directory <code>examples/api/audio</code>), which lets users specify a frequency
+for the sine wave and click buttons to start and stop audio playback.</p>
+<section id="reference-information">
+<h2 id="reference-information">Reference information</h2>
+<p>For reference information related to the Pepper audio API, see the following
+documentation:</p>
+<ul class="small-gap">
+<li><a class="reference external" href="https://developers.google.com/native-client/peppercpp/classpp_1_1_audio_config">pp::AudioConfig class</a></li>
+<li><a class="reference external" href="https://developers.google.com/native-client/peppercpp/classpp_1_1_audio">pp::Audio class</a></li>
+<li><a class="reference external" href="https://developers.google.com/native-client/peppercpp/audio__config_8h">audio_config.h</a></li>
+<li><a class="reference external" href="https://developers.google.com/native-client/peppercpp/audio_8h">audio.h</a></li>
+<li><a class="reference external" href="https://developers.google.com/native-client/pepperc/group___enums.html#gaee750c350655f2fb0fe04c04029e0ff8">PP_AudioSampleRate</a></li>
+</ul>
+</section><section id="about-the-pepper-audio-api">
+<h2 id="about-the-pepper-audio-api">About the Pepper audio API</h2>
+<p>The Pepper audio API lets Native Client modules play audio streams in a
+browser. To play an audio stream, a module generates audio samples and writes
+them into a buffer. The browser reads the audio samples from the buffer and
+plays them using an audio device on the client computer.</p>
+<img alt="/native-client/images/pepper-audio-buffer.png" src="/native-client/images/pepper-audio-buffer.png" />
+<p>This mechanism is simple but low-level. If you want to play plain sound files in
+a web application, you may want to consider higher-level alternatives such as
+using the HTML <code>&lt;audio&gt;</code> tag, JavaScript, or the new <a class="reference external" href="http://chromium.googlecode.com/svn/trunk/samples/audio/index.html">Web Audio API</a>.</p>
+<p>The Pepper audio API is a good option for playing audio data if you want to do
+audio processing in your web application. You might use the audio API, for
+example, if you want to apply audio effects to sounds, synthesize your own
+sounds, or do any other type of CPU-intensive processing of audio
+samples. Another likely use case is gaming applications: you might use a gaming
+library to process audio data, and then simply use the audio API to output the
+processed data.</p>
+<p>The Pepper audio API is straightforward to use:</p>
+<ol class="arabic simple">
+<li>Your module creates an audio configuration resource and an audio resource.</li>
+<li>Your module implements a callback function that fills an audio buffer with
+data.</li>
+<li>Your module invokes the StartPlayback and StopPlayback methods of the audio
+resource (e.g., when certain events occur).</li>
+<li>The browser invokes your callback function whenever it needs audio data to
+play. Your callback function can generate the audio data in a number of
+ways&#8212;e.g., it can generate new data, or it can copy pre-mixed data into the
+audio buffer.</li>
+</ol>
+<p>This basic interaction is illustrated below, and described in detail in the
+sections that follow.</p>
+<img alt="/native-client/images/pepper-audio-api.png" src="/native-client/images/pepper-audio-api.png" />
+</section><section id="digital-audio-concepts">
+<h2 id="digital-audio-concepts">Digital audio concepts</h2>
+<p>Before you use the Pepper audio API, it&#8217;s helpful to understand a few concepts
+that are fundamental to how digital audio is recorded and played back:</p>
+<dl class="docutils">
+<dt>sample rate</dt>
+<dd>the number of times an input sound source is sampled per second;
+correspondingly, the number of samples that are played back per second</dd>
+<dt>bit depth</dt>
+<dd>the number of bits used to represent a sample</dd>
+<dt>channels</dt>
+<dd>the number of input sources recorded in each sampling interval;
+correspondingly, the number of outputs that are played back simultaneously
+(typically using different speakers)</dd>
+</dl>
+<p>The higher the sample rate and bit depth used to record a sound wave, the more
+accurately the sound wave can be reproduced, since it will have been sampled
+more frequently and stored using a higher level of quantization. Common sampling
+rates include 44,100 Hz (44,100 samples/second, the sample rate used on CDs),
+and 48,000 Hz (the sample rate used on DVDs and Digital Audio Tapes). A common
+bit depth is 16 bits per sample, and a common number of channels is 2 (left and
+right channels for stereo sound).</p>
+<p id="pepper-audio-configurations">The Pepper audio API currently lets Native Client modules play audio streams
+with the following configurations:</p>
+<ul class="small-gap">
+<li><strong>sample rate</strong>: 44,100 Hz or 48,000 Hz</li>
+<li><strong>bit depth</strong>: 16</li>
+<li><strong>channels</strong>: 2 (stereo)</li>
+</ul>
+</section><section id="setting-up-the-module">
+<h2 id="setting-up-the-module">Setting up the module</h2>
+<p>The code examples below describe a simple Native Client module that generates
+audio samples using a sine wave with a frequency of 440 Hz. The module starts
+playing the audio samples as soon as it is loaded into the browser.</p>
+<p>The Native Client module is set up by implementing subclasses of the
+<code>pp::Module</code> and <code>pp::Instance</code> classes, as normal.</p>
+<pre class="prettyprint">
+class SineSynthInstance : public pp::Instance {
+ public:
+  explicit SineSynthInstance(PP_Instance instance);
+  virtual ~SineSynthInstance() {}
+
+  // Called by the browser once the NaCl module is loaded and ready to
+  // initialize.  Creates a Pepper audio context and initializes it. Returns
+  // true on success.  Returning false causes the NaCl module to be deleted
+  // and no other functions to be called.
+  virtual bool Init(uint32_t argc, const char* argn[], const char* argv[]);
+
+ private:
+  // Function called by the browser when it needs more audio samples.
+  static void SineWaveCallback(void* samples,
+                               uint32_t buffer_size,
+                               void* data);
+
+  // Audio resource.
+  pp::Audio audio_;
+
+  ...
+
+};
+
+class SineSynthModule : public pp::Module {
+ public:
+  SineSynthModule() : pp::Module() {}
+  ~SineSynthModule() {}
+
+  // Create and return a SineSynthInstance object.
+  virtual pp::Instance* CreateInstance(PP_Instance instance) {
+    return new SineSynthInstance(instance);
+  }
+};
+</pre>
+</section><section id="creating-an-audio-configuration-resource">
+<h2 id="creating-an-audio-configuration-resource">Creating an audio configuration resource</h2>
+<section id="resources">
+<h3 id="resources">Resources</h3>
+<p>Before the module can play an audio stream, it must create two resources: an
+audio configuration resource and an audio resource. Resources are handles to
+objects that the browser provides to module instances. An audio resource is an
+object that represents the state of an audio stream, including whether the
+stream is paused or being played back, and which callback function to invoke
+when the samples in the stream&#8217;s buffer run out. An audio configuration resource
+is an object that stores configuration data for an audio resource, including the
+sampling frequency of the audio samples, and the number of samples that the
+callback function must provide when the browser invokes it.</p>
+</section><section id="sample-frame-count">
+<h3 id="sample-frame-count">Sample frame count</h3>
+<p>Prior to creating an audio configuration resource, the module should call
+<code>RecommendSampleFrameCount</code> to obtain a <em>sample frame count</em> from the
+browser. The sample frame count is the number of samples that the callback
+function must provide per channel each time the browser invokes the callback
+function. For example, if the sample frame count is 4096 for a stereo audio
+stream, the callback function must provide a 8192 samples (4096 for the left
+channel and 4096 for the right channel).</p>
+<p>The module can request a specific sample frame count, but the browser may return
+a different sample frame count depending on the capabilities of the client
+device. At present, <code>RecommendSampleFrameCount</code> simply bound-checks the
+requested sample frame count (see <code>include/ppapi/c/ppb_audio_config.h</code> for the
+minimum and maximum sample frame counts, currently 64 and 32768). In the future,
+<code>RecommendSampleFrameCount</code> may perform a more sophisticated calculation,
+particularly if there is an intrinsic buffer size for the client device.</p>
+<p>Selecting a sample frame count for an audio stream involves a tradeoff between
+latency and CPU usage. If you want your module to have short audio latency so
+that it can rapidly change what&#8217;s playing in the audio stream, you should
+request a small sample frame count. That could be useful in gaming applications,
+for example, where sounds have to change frequently in response to game
+action. However, a small sample frame count results in higher CPU usage, since
+the browser must invoke the callback function frequently to refill the audio
+buffer. Conversely, a large sample frame count results in higher latency but
+lower CPU usage. You should request a large sample frame count if your module
+will play long, uninterrupted audio segments.</p>
+</section><section id="supported-audio-configurations">
+<h3 id="supported-audio-configurations">Supported audio configurations</h3>
+<p>After the module obtains a sample frame count, it can create an audio
+configuration resource. Currently the Pepper audio API supports audio streams
+with the configuration settings shown <a class="reference internal" href="#pepper-audio-configurations"><em>above</em></a>.
+C++ modules can create a configuration resource by instantiating a
+<code>pp::AudioConfig</code> object. Check <code>audio_config.h</code> for the latest
+configurations that are supported.</p>
+<pre class="prettyprint">
+bool SineSynthInstance::Init(uint32_t argc,
+                             const char* argn[],
+                             const char* argv[]) {
+
+  // Ask the browser/device for an appropriate sample frame count size.
+  sample_frame_count_ =
+      pp::AudioConfig::RecommendSampleFrameCount(PP_AUDIOSAMPLERATE_44100,
+                                                 kSampleFrameCount);
+
+  // Create an audio configuration resource.
+  pp::AudioConfig audio_config = pp::AudioConfig(this,
+                                                 PP_AUDIOSAMPLERATE_44100,
+                                                 sample_frame_count_);
+
+  // Create an audio resource.
+  audio_ = pp::Audio(this,
+                     audio_config,
+                     SineWaveCallback,
+                     this);
+
+  // Start playback when the module instance is initialized.
+  return audio_.StartPlayback();
+}
+</pre>
+</section></section><section id="creating-an-audio-resource">
+<h2 id="creating-an-audio-resource">Creating an audio resource</h2>
+<p>Once the module has created an audio configuration resource, it can create an
+audio resource. To do so, it instantiates a <code>pp::Audio</code> object, passing in a
+pointer to the module instance, the audio configuration resource, a callback
+function, and a pointer to user data (data that is used in the callback
+function).  See the example above.</p>
+</section><section id="implementing-a-callback-function">
+<h2 id="implementing-a-callback-function">Implementing a callback function</h2>
+<p>The browser calls the callback function associated with an audio resource every
+time it needs more samples to play. The callback function can generate new
+samples (e.g., by applying sound effects), or copy pre-mixed samples into the
+audio buffer. The example below generates new samples by computing values of a
+sine wave.</p>
+<p>The last parameter passed to the callback function is generic user data that the
+function can use in processing samples. In the example below, the user data is a
+pointer to the module instance, which includes member variables
+<code>sample_frame_count_</code> (the sample frame count obtained from the browser) and
+<code>theta_</code> (the last angle that was used to compute a sine value in the previous
+callback; this lets the function generate a smooth sine wave by starting at that
+angle plus a small delta).</p>
+<pre class="prettyprint">
+class SineSynthInstance : public pp::Instance {
+ public:
+  ...
+
+ private:
+  static void SineWaveCallback(void* samples,
+                               uint32_t buffer_size,
+                               void* data) {
+
+    // The user data in this example is a pointer to the module instance.
+    SineSynthInstance* sine_synth_instance =
+        reinterpret_cast&lt;SineSynthInstance*&gt;(data);
+
+    // Delta by which to increase theta_ for each sample.
+    const double delta = kTwoPi * kFrequency / PP_AUDIOSAMPLERATE_44100;
+    // Amount by which to scale up the computed sine value.
+    const int16_t max_int16 = std::numeric_limits&lt;int16_t&gt;::max();
+
+    int16_t* buff = reinterpret_cast&lt;int16_t*&gt;(samples);
+
+    // Make sure we can't write outside the buffer.
+    assert(buffer_size &gt;= (sizeof(*buff) * kChannels *
+                           sine_synth_instance-&gt;sample_frame_count_));
+
+    for (size_t sample_i = 0;
+         sample_i &lt; sine_synth_instance-&gt;sample_frame_count_;
+         ++sample_i, sine_synth_instance-&gt;theta_ += delta) {
+
+      // Keep theta_ from going beyond 2*Pi.
+      if (sine_synth_instance-&gt;theta_ &gt; kTwoPi) {
+        sine_synth_instance-&gt;theta_ -= kTwoPi;
+      }
+
+      // Compute the sine value for the current theta_, scale it up,
+      // and write it into the buffer once for each channel.
+      double sin_value(std::sin(sine_synth_instance-&gt;theta_));
+      int16_t scaled_value = static_cast&lt;int16_t&gt;(sin_value * max_int16);
+      for (size_t channel = 0; channel &lt; kChannels; ++channel) {
+        *buff++ = scaled_value;
+      }
+    }
+  }
+
+  ...
+};
+</pre>
+<section id="application-threads-and-real-time-requirements">
+<h3 id="application-threads-and-real-time-requirements">Application threads and real-time requirements</h3>
+<p>The callback function runs in a background application thread. This allows audio
+processing to continue even when the application is busy doing something
+else. If the main application thread and the callback thread access the same
+data, you may be tempted to use a lock to control access to that data. You
+should avoid the use of locks in the callback thread, however, as attempting to
+acquire a lock may cause the thread to get swapped out, resulting in audio
+dropouts.</p>
+<p>In general, you must program the callback thread carefully, as the Pepper audio
+API is a very low level API that needs to meet hard real-time requirements. If
+the callback thread spends too much time processing, it can easily miss the
+real-time deadline, resulting in audio dropouts. One way the callback thread can
+miss the deadline is by taking too much time doing computation. Another way the
+callback thread can miss the deadline is by executing a function call that swaps
+out the callback thread. Unfortunately, such function calls include just about
+all C Run-Time (CRT) library calls and Pepper API calls. The callback thread
+should therefore avoid calls to malloc, gettimeofday, mutex, condvars, critical
+sections, and so forth; any such calls could attempt to take a lock and swap out
+the callback thread, which would be disastrous for audio playback. Similarly,
+the callback thread should avoid Pepper API calls. Audio dropouts due to thread
+swapping can be very rare and very hard to track down and debug&#8212;it&#8217;s best to
+avoid making system/Pepper calls in the first place. In short, the audio
+(callback) thread should use &#8220;lock-free&#8221; techniques and avoid making CRT library
+calls.</p>
+<p>One other issue to be aware of is that the <code>StartPlayback</code> function (discussed
+below) is an asynchronous RPC; i.e., it does not block. That means that the
+callback function may not be called immediately after the call to
+<code>StartPlayback</code>. If it&#8217;s important to synchronize the callback thread with
+another thread so that the audio stream starts playing simultaneously with
+another action in your application, you must handle such synchronization
+manually.</p>
+</section></section><section id="starting-and-stopping-playback">
+<h2 id="starting-and-stopping-playback">Starting and stopping playback</h2>
+<p>To start and stop audio playback, the module simply reacts to JavaScript
+messages.</p>
+<pre class="prettyprint">
+const char* const kPlaySoundId = &quot;playSound&quot;;
+const char* const kStopSoundId = &quot;stopSound&quot;;
+
+void SineSynthInstance::HandleMessage(const pp::Var&amp; var_message) {
+  if (!var_message.is_string()) {
+    return;
+  }
+  std::string message = var_message.AsString();
+  if (message == kPlaySoundId) {
+    audio_.StartPlayback();
+  } else if (message == kStopSoundId) {
+    audio_.StopPlayback();
+  } else if (...) {
+    ...
+  }
+}
+</pre>
+</section></section>
+
+{{/partials.standard_nacl_article}}