Motown in-proc streaming framework used to implement media services.

services/media/framework/stream_type.h

Index: services/media/framework/stream_type.h
diff --git a/services/media/framework/stream_type.h b/services/media/framework/stream_type.h
new file mode 100644
index 0000000000000000000000000000000000000000..73d02032e5afaedfeec1abafb26e04264e168530
--- /dev/null
+++ b/services/media/framework/stream_type.h
@@ -0,0 +1,631 @@
+// Copyright 2016 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#ifndef SERVICES_MEDIA_FRAMEWORK_STREAM_TYPE_H_
+#define SERVICES_MEDIA_FRAMEWORK_STREAM_TYPE_H_

[johngro, 2016/01/26 23:47:30]

top level comment.

This whole file feels like a custom, in-proc-centric C++ implementation of the
media types which you spec'ed in mojom way back when.

Why are we not using the mojom generated classes here?  How are we going to
manage this when we have to start to inter-operate with media services
(potentially implemented in other languages) across mojo transport boundaries?

If we choose to do anything at all about this in the short term, it should
probably be handled with a follow-up change, but I suspect that this will start
to be an issue in the short-to-mid term.

[dalesat, 2016/01/28 18:49:17]

I did this for four reasons:

1) to avoid mojo dependencies
2) because these classes are easier to use than the mojo ones
3) I can implement useful code in these classes
4) to allow internal support for things that aren't supported in mojo.

This last item was important when I was getting the interleaved flag out of the
mojo defs. I've since removed it from here as well. If we want internal support
for e.g. kSigned16In32, reason 4 will become valid again. If none of these
reasons bear out, I will do the work to get rid of this.

I get that there's significant maintenance cost. If this becomes a problem for
you, let me know, and I will do the work. Interop is a non-issue as the things
that get moved across mojo are the mojo-generated things. To the degree that
those are extensible, these must be as well (e.g. when we switch to using
strings instead of enums for some values).

[johngro, 2016/02/01 22:38:16]

Acknowledged.

+
+#include <cstring>
+#include <string>
+#include <vector>
+
+#include "services/media/framework/ptr.h"
+
+namespace mojo {
+namespace media {
+
+class StreamType;
+class StreamTypes;
+class MultiplexedStreamType;
+class LpcmStreamType;
+class CompressedAudioStreamType;
+class VideoStreamType;
+
+typedef UniquePtr<StreamType> StreamTypePtr;
+typedef UniquePtr<StreamTypes> StreamTypesPtr;
+
+class StreamTypes : public std::vector<StreamTypePtr> {
+ public:
+  static StreamTypesPtr New(size_t size) {
+    return StreamTypesPtr(new StreamTypes(size));
+  }
+
+  StreamTypes(size_t size) : std::vector<StreamTypePtr>(size) {}

[johngro, 2016/01/26 23:47:30]

if we are imposing move semantics on this class, this constructor should be
private, and the copy constructor should be removed.  I think that the Mojo
framework already has a handy macro for this, but I'm not sure.

If we are not imposing move semantics on this class, then we should probably
just say...

using StreamTypes = std::vector<StreamTypePtr>;

and leave it at that.  This seems to apply to all of the extensions of
std::vector below.  They all seem to add New and Clone to the mix to force
unique_ptr and move patterns, but they all expose a direct constructor as well.

[dalesat, 2016/01/28 18:49:17]

I'm not enforcing move semantics, but I do support them. There are plenty of
cases in which classes have StreamType members that are only exposed as const
StreamType&.

[johngro, 2016/02/01 22:38:16]

Acknowledged.

+
+  StreamTypesPtr Clone() const;
+};
+
+class Bytes;
+typedef UniquePtr<Bytes> BytesPtr;
+
+class Bytes : public std::vector<uint8_t> {
+ public:
+  static BytesPtr New(size_t size) {
+    return BytesPtr(new Bytes(size));
+  }
+
+  static BytesPtr New(uint8_t* data, size_t size) {

[johngro, 2016/01/26 23:47:30]

DCHECK(data || !size);

memcpy's behavior when passing nullptr for the source is not super consistent
across std C library implementations, but its probably pretty safe to call it
with nullptr, provided that size is zero.

[dalesat, 2016/01/28 18:49:17]

Done.

+    BytesPtr result = New(size);
+    std::memcpy(result->data(), data, size);
+    return result;
+  }
+
+  Bytes(size_t size) : std::vector<uint8_t>(size) {}

[johngro, 2016/01/26 23:47:30]

explicit Bytes(size_t size) ...

also, this should probably be private.  Same for the copy constructor below.

[dalesat, 2016/01/28 18:49:17]

Leaving public, otherwise done.

+
+  Bytes(const Bytes& copy_from) : std::vector<uint8_t>(copy_from) {}
+
+  const uint8_t* data() const { return &(*this)[0]; }

[johngro, 2016/01/26 23:47:30]

Why is the base class's implementation of data (here and below) not good enough
for this job?

[dalesat, 2016/01/28 18:49:17]

Wasn't aware that vector had that. Done.

+
+  uint8_t* data() { return &(*this)[0]; }
+
+  BytesPtr Clone() const;
+};
+
+// Describes the type of a stream.
+class StreamType {
+ public:
+  enum class Scheme {
+    kUnknown,
+    kNone,
+    kAnyElementary,
+    kAnyAudio,
+    kAnyVideo,
+    kAnySubpicture,
+    kAnyText,
+    kAnyMultiplexed,
+    kAny,
+
+    kMultiplexed,
+    kLpcm,
+    kCompressedAudio,
+    kVideo
+  };
+
+  static StreamTypePtr New(Scheme scheme) {
+    return StreamTypePtr(new StreamType(scheme));
+  }
+
+  StreamType(Scheme scheme);
+
+  ~StreamType();
+
+  Scheme scheme() const {
+    return scheme_;
+  }
+
+  virtual const MultiplexedStreamType* multiplexed() const;
+  virtual const LpcmStreamType* lpcm() const;
+  virtual const CompressedAudioStreamType* compressed_audio() const;
+  virtual const VideoStreamType* video() const;
+
+  virtual StreamTypePtr Clone() const;
+
+ private:
+  Scheme scheme_;
+};
+
+template<typename T>
+struct Range {
+  Range(T min_param, T max_param) : min(min_param), max(max_param) {}

[johngro, 2016/01/26 23:47:30]

DCHECK(min <= max);

[dalesat, 2016/01/28 18:49:17]

Done.

+
+  T min;
+  T max;
+
+  bool contains(T t) const {

[johngro, 2016/01/26 23:47:30]

inline, constexpr

Also, this is probably only for primitive data types, by you might want to spec
this as const T& instead of passing T by value.  When the method gets inlined,
the compiler will "do the right thing" for PoDs.

[dalesat, 2016/01/28 18:49:17]

Done.

+    return t >= min && t <= max;
+  }
+};
+
+class StreamTypeSet;
+class StreamTypeSets;
+class MultiplexedStreamTypeSet;
+class LpcmStreamTypeSet;
+class CompressedAudioStreamTypeSet;
+class VideoStreamTypeSet;
+
+typedef UniquePtr<StreamTypeSet> StreamTypeSetPtr;
+typedef UniquePtr<StreamTypeSets> StreamTypeSetsPtr;
+
+class StreamTypeSets : public std::vector<StreamTypeSetPtr> {
+ public:
+  static StreamTypeSetsPtr New(size_t size) {
+    return StreamTypeSetsPtr(new StreamTypeSets(size));
+  }
+
+  StreamTypeSets(size_t size) : std::vector<StreamTypeSetPtr>(size) {}
+
+  StreamTypeSetsPtr Clone() const;
+};
+
+// Describes a set of possible stream types.
+class StreamTypeSet {
+ public:
+  static StreamTypeSetPtr New(StreamType::Scheme scheme) {
+    return StreamTypeSetPtr(new StreamTypeSet(scheme));
+  }
+
+  StreamTypeSet(StreamType::Scheme scheme);
+
+  ~StreamTypeSet();

[johngro, 2016/01/26 23:47:30]

virtual destructor.  I'm pretty sure that style mandates that you always have a
virtual destructor if you have virtual members.

Also in the derived classes.

[dalesat, 2016/01/28 18:49:17]

Done.

+
+  StreamType::Scheme scheme() const {
+    return scheme_;
+  }
+
+  virtual const MultiplexedStreamTypeSet* multiplexed() const;

[johngro, 2016/01/26 23:47:30]

If only we allowed dynamic_cast, you would not need to do this.

[dalesat, 2016/01/28 18:49:17]

Acknowledged.

+  virtual const LpcmStreamTypeSet* lpcm() const;
+  virtual const CompressedAudioStreamTypeSet* compressed_audio() const;
+  virtual const VideoStreamTypeSet* video() const;
+
+  virtual StreamTypeSetPtr Clone() const;
+
+ private:
+  StreamType::Scheme scheme_;
+};
+
+// Describes the type of a multiplexed stream.
+class MultiplexedStreamType : public StreamType {
+ public:
+  static StreamTypePtr New(
+      StreamTypePtr multiplex_type,
+      StreamTypesPtr substream_types) {
+    return StreamTypePtr(
+        new MultiplexedStreamType(
+            std::move(multiplex_type),
+            std::move(substream_types)));
+  }
+
+  MultiplexedStreamType(
+      StreamTypePtr multiplex_type,
+      StreamTypesPtr substream_types);
+
+  ~MultiplexedStreamType();
+
+  const MultiplexedStreamType* multiplexed() const override;
+
+  const StreamTypePtr& multiplex_type() const {
+    return multiplex_type_;
+  }
+
+  const StreamTypesPtr& substream_types() const {
+    return substream_types_;
+  }
+
+  StreamTypePtr Clone() const override;
+
+private:
+  StreamTypePtr multiplex_type_;
+  StreamTypesPtr substream_types_;
+};
+
+// Describes the type of a multiplexed stream.
+class MultiplexedStreamTypeSet : public StreamTypeSet {
+public:
+  static StreamTypeSetPtr New(
+      StreamTypeSetPtr multiplex_type_set,
+      StreamTypeSetsPtr substream_type_sets) {
+    return StreamTypeSetPtr(
+        new MultiplexedStreamTypeSet(
+            std::move(multiplex_type_set),
+            std::move(substream_type_sets)));
+  }
+
+  MultiplexedStreamTypeSet(
+      StreamTypeSetPtr multiplex_type_set,
+      StreamTypeSetsPtr substream_type_sets);
+
+  ~MultiplexedStreamTypeSet();
+
+  const MultiplexedStreamTypeSet* multiplexed() const override;
+
+  const StreamTypeSetPtr& multiplex_type_set() const {
+    return multiplex_type_set_;
+  }
+
+  const StreamTypeSetsPtr& substream_type_sets() const {
+    return substream_type_sets_;
+  }
+
+  StreamTypeSetPtr Clone() const override;
+
+private:
+  StreamTypeSetPtr multiplex_type_set_;
+  StreamTypeSetsPtr substream_type_sets_;
+};
+
+// Describes the type of an LPCM stream.
+class LpcmStreamType : public StreamType {
+ public:
+  enum class SampleFormat {
+    kUnknown,
+    kAny,
+    kUnsigned8,
+    kSigned16,
+    kSigned24In32,
+    kFloat
+  };
+
+  static StreamTypePtr New(
+      SampleFormat sample_format,
+      uint32_t channels,
+      uint32_t frames_per_second) {
+    return StreamTypePtr(new LpcmStreamType(
+        sample_format,
+        channels,
+        frames_per_second));
+  }
+
+  LpcmStreamType(
+      SampleFormat sample_format,
+      uint32_t channels,
+      uint32_t frames_per_second);
+
+  ~LpcmStreamType();
+
+  const LpcmStreamType* lpcm() const override;
+
+  SampleFormat sample_format() const {
+    return sample_format_;
+  }
+
+  uint32_t channels() const {
+    return channels_;
+  }
+
+  uint32_t frames_per_second() const {
+    return frames_per_second_;
+  }
+
+  uint32_t sample_size() const {
+    return sample_size_;
+  }
+
+  uint32_t bytes_per_frame() const {
+    return sample_size_ * channels_;
+  }
+
+  uint64_t min_buffer_size(uint64_t frame_count) const {
+    return frame_count * sample_size_ * channels_;
+  }
+
+  static uint32_t SampleSizeFromFormat(SampleFormat sample_format);
+
+  StreamTypePtr Clone() const override;
+
+ protected:
+  LpcmStreamType(
+      Scheme scheme,
+      SampleFormat sample_format,
+      uint32_t channels,
+      uint32_t frames_per_second);
+
+ private:
+  SampleFormat sample_format_;
+  uint32_t channels_;
+  uint32_t frames_per_second_;
+  uint32_t sample_size_;
+};
+
+// Describes a set of LPCM stream types.
+class LpcmStreamTypeSet : public StreamTypeSet {
+ public:
+  static StreamTypeSetPtr New(
+      LpcmStreamType::SampleFormat sample_format,
+      Range<uint32_t> channels,
+      Range<uint32_t> frames_per_second) {
+    return StreamTypeSetPtr(new LpcmStreamTypeSet(
+        sample_format,
+        channels,
+        frames_per_second));
+  }
+
+  LpcmStreamTypeSet(
+      LpcmStreamType::SampleFormat sample_format,
+      Range<uint32_t> channels,
+      Range<uint32_t> frames_per_second);
+
+  ~LpcmStreamTypeSet();
+
+  const LpcmStreamTypeSet* lpcm() const override;
+
+  LpcmStreamType::SampleFormat sample_format() const {
+    return sample_format_;
+  }
+
+  Range<uint32_t> channels() const {
+    return channels_;
+  }
+
+  Range<uint32_t> frames_per_second() const {
+    return frames_per_second_;
+  }
+
+  bool contains(const LpcmStreamType& type) const;
+
+  StreamTypeSetPtr Clone() const override;
+
+ protected:
+  LpcmStreamTypeSet(
+      StreamType::Scheme scheme,
+      LpcmStreamType::SampleFormat sample_format,
+      Range<uint32_t> channels,
+      Range<uint32_t> frames_per_second);
+
+ private:
+  LpcmStreamType::SampleFormat sample_format_;
+  Range<uint32_t> channels_;
+  Range<uint32_t> frames_per_second_;
+};
+
+// Describes the type of a compressed audio stream.
+class CompressedAudioStreamType : public LpcmStreamType {
+ public:
+  enum class AudioEncoding {
+    kUnknown,
+    kAny,
+    kVorbis
+  };
+
+  static StreamTypePtr New(
+      AudioEncoding encoding,
+      SampleFormat sample_format,
+      uint32_t channels,
+      uint32_t frames_per_second,
+      BytesPtr encoding_details) {
+    return StreamTypePtr(new CompressedAudioStreamType(
+        encoding,
+        sample_format,
+        channels,
+        frames_per_second,
+        std::move(encoding_details)));
+  }
+
+  CompressedAudioStreamType(
+    AudioEncoding encoding,
+    SampleFormat sample_format,
+    uint32_t channels,
+    uint32_t frames_per_second,
+    BytesPtr encoding_details);
+
+  ~CompressedAudioStreamType();
+
+  const CompressedAudioStreamType* compressed_audio() const override;
+
+  AudioEncoding encoding() const {
+    return encoding_;
+  }
+
+  const BytesPtr& encoding_details() const {
+    return encoding_details_;
+  }
+
+  StreamTypePtr Clone() const override;
+
+ private:
+  AudioEncoding encoding_;
+  BytesPtr encoding_details_;
+};
+
+// Describes a set of compressed audio stream types.
+class CompressedAudioStreamTypeSet : public LpcmStreamTypeSet {
+ public:
+  static StreamTypeSetPtr New(
+      CompressedAudioStreamType::AudioEncoding encoding,
+      CompressedAudioStreamType::SampleFormat sample_format,
+      Range<uint32_t> channels,
+      Range<uint32_t> frames_per_second) {
+    return StreamTypeSetPtr(new CompressedAudioStreamTypeSet(
+        encoding,
+        sample_format,
+        channels,
+        frames_per_second));
+  }
+
+  CompressedAudioStreamTypeSet(
+    CompressedAudioStreamType::AudioEncoding encoding,
+    CompressedAudioStreamType::SampleFormat sample_format,
+    Range<uint32_t> channels,
+    Range<uint32_t> frames_per_second);
+
+  ~CompressedAudioStreamTypeSet();
+
+  const CompressedAudioStreamTypeSet* compressed_audio() const override;
+
+  CompressedAudioStreamType::AudioEncoding encoding() const {
+    return encoding_;
+  }
+
+  bool contains(const CompressedAudioStreamType& type) const;
+
+  StreamTypeSetPtr Clone() const override;
+
+ private:
+  CompressedAudioStreamType::AudioEncoding encoding_;
+};
+
+// Describes the type of a video stream.
+class VideoStreamType : public StreamType {

[johngro, 2016/01/26 23:47:30]

I think that we will need to revisit this at some point in time.  I'm not all
that sure that using enums in header files is a long term scale-able solution to
communicating this information (this really starts to show as you start to
enumerate all of the profiles of H.264 below, but have not mixed in their
concept of levels as well)

The pixel packing section has related issues.  In addition to needing to
enumerate all of the combinations in the world (ARGB, ABGR, BGRA, etc...), just
saying that a frame is YUYV can miss hardware imposed line stride requirements
(I420 as issues with both line and plane stride, etc...)

No change requested now, but I think we need to come back to this and sit down
with the graphics team at some point and figure out a final solution.  It may
end up being enums, but I'm not convinced that this will be the proper thing to
do in the long run.

[dalesat, 2016/01/28 18:49:17]

I agree. I dropped these into the mojom in an attempt to get an ffmpeg video
decoder working isolated from the ffmpeg demux. That effort was sidelined to get
the framework ready for the mixer. Many of these should be strings for
extensibility. If we can do without some, that would be great. The bottom line
is that I need to be able to reconstruct an ffmpeg decoder context across mojo
using (the mojom version of) this representation. We could make some aspects
opaque in the mojom definitions.

+ public:
+  enum class VideoEncoding {
+    kUnknown,
+    kAny,
+    kTheora,
+    kVp8,
+  };
+
+  enum class VideoProfile {
+    kUnknown,
+    kNotApplicable,
+    kH264Baseline,
+    kH264Main,
+    kH264Extended,
+    kH264High,
+    kH264High10,
+    kH264High422,
+    kH264High444Predictive,
+    kH264ScalableBaseline,
+    kH264ScalableHigh,
+    kH264StereoHigh,
+    kH264MultiviewHigh
+  };
+
+  enum class PixelFormat {
+    kUnknown,
+    kI420,
+    kYv12,
+    kYv16,
+    kYv12A,
+    kYv24,
+    kNv12,
+    kNv21,
+    kUyvy,
+    kYuy2,
+    kArgb,
+    kXrgb,
+    kRgb24,
+    kRgb32,
+    kMjpeg,
+    kMt21
+  };
+
+  enum class ColorSpace {
+    kUnknown,
+    kNotApplicable,
+    kJpeg,
+    kHdRec709,
+    kSdRec601
+  };
+
+  static StreamTypePtr New(
+      VideoEncoding encoding,
+      VideoProfile profile,
+      PixelFormat pixel_format,
+      ColorSpace color_space,
+      uint32_t width,
+      uint32_t height,
+      uint32_t coded_width,
+      uint32_t coded_height,
+      BytesPtr encoding_details) {
+    return StreamTypePtr(new VideoStreamType(
+        encoding,
+        profile,
+        pixel_format,
+        color_space,
+        width,
+        height,
+        coded_width,
+        coded_height,
+        std::move(encoding_details)));
+  }
+
+  VideoStreamType(

[johngro, 2016/01/26 23:47:30]

again, something to revisit later on, but I am not convinced that these are
necessarily properties of the video stream.  Things like the colorspace, the
width/height, the active video region, the pixel aspect ratio and so on can
actually change on a per-frame basis.

Whether or not we support this at the per-frame level or we support this by
tearing down and rebuilding video sub-streams in a seamless fashion (or through
some completely other means) is something I am not clear on, but it is something
I have been burnt on when attempting to use other media frameworks to create
broadcast video devices in the past.

I tossed out some starting points for how this might be approached in the
video-pipe strawman doc I wrote for James forever ago, but I'm not certain that
anyone ever took the time to read through it.

[dalesat, 2016/01/28 18:49:17]

Acknowledged.

+    VideoEncoding encoding,
+    VideoProfile profile,
+    PixelFormat pixel_format,
+    ColorSpace color_space,
+    uint32_t width,
+    uint32_t height,
+    uint32_t coded_width,
+    uint32_t coded_height,
+    BytesPtr encoding_details);
+
+  ~VideoStreamType();
+
+  const VideoStreamType* video() const override;
+
+  VideoEncoding encoding() const {
+    return encoding_;
+  }
+
+  VideoProfile profile() const {
+    return profile_;
+  }
+
+  PixelFormat pixel_format() const {
+    return pixel_format_;
+  }
+
+  ColorSpace color_space() const {
+    return color_space_;
+  }
+
+  uint32_t width() const {
+    return width_;
+  }
+
+  uint32_t height() const {
+    return height_;
+  }
+
+  uint32_t coded_width() const {
+    return coded_width_;
+  }
+
+  uint32_t coded_height() const {
+    return coded_height_;
+  }
+
+  const BytesPtr& encoding_details() const {
+    return encoding_details_;
+  }
+
+  StreamTypePtr Clone() const override;
+
+ private:
+  VideoEncoding encoding_;
+  VideoProfile profile_;
+  PixelFormat pixel_format_;
+  ColorSpace color_space_;
+  uint32_t width_;
+  uint32_t height_;
+  uint32_t coded_width_;
+  uint32_t coded_height_;
+  BytesPtr encoding_details_;
+};
+
+// Describes a set of video stream types.
+class VideoStreamTypeSet : public StreamTypeSet {
+ public:
+  static StreamTypeSetPtr New(
+      VideoStreamType::VideoEncoding encoding,
+      Range<uint32_t> width,
+      Range<uint32_t> height) {
+    return StreamTypeSetPtr(new VideoStreamTypeSet(
+        encoding,
+        width,
+        height));
+  }
+
+  VideoStreamTypeSet(
+    VideoStreamType::VideoEncoding encoding,
+    Range<uint32_t> width,
+    Range<uint32_t> height);
+
+  ~VideoStreamTypeSet();
+
+  const VideoStreamTypeSet* video() const override;
+
+  VideoStreamType::VideoEncoding encoding() const {
+    return encoding_;
+  }
+
+  Range<uint32_t> width() const {
+    return width_;
+  }
+
+  Range<uint32_t> height() const {
+    return height_;
+  }
+
+  StreamTypeSetPtr Clone() const override;
+
+ private:
+  VideoStreamType::VideoEncoding encoding_;
+  Range<uint32_t> width_;
+  Range<uint32_t> height_;
+};
+
+}  // namespace media
+}  // namespace mojo
+
+#endif // SERVICES_MEDIA_FRAMEWORK_STREAM_TYPE_H_