MoQ for the Curious

Chapter 07

Media and the Browser

On this page

A player turns received objects into sound and pictures. MOQT delivers the objects; the media format and player tell the browser how to interpret, decode, and present them.

We’ll follow a picture from the media description to the screen: choose a track, find a starting picture, decode it, and decide when to display it. Our examples use moq.dev’s hang media format and the browser’s WebCodecs API. We’ll also cover playback permissions and cleaning up when the player stops.

How does the player interpret a track?

The lecture offers an audio track and two video tracks, one at a lower resolution for slower connections. Track names let the player request them, but names alone do not explain how to use their bytes. The player also needs the codec, decoder configuration, and timing information. A codec defines how pictures or sound are encoded and decoded. A container packages the encoded data with information needed to interpret it, such as timestamps.

MOQT carries application payloads without choosing their media format. Our example uses hang, the media layer in the moq.dev stack. Its catalog.json track describes the available renditions and their decoder configuration. A rendition is an alternative encoding of the content, such as the lecture’s lower-resolution picture. See the hang catalog and container documentation.

Suppose the selected video rendition contains H.264 pictures at 1280 × 720 pixels. Its codec identifier tells the player which decoder it needs. Its codec configuration supplies any required setup, either as initialization data or in-band as the format permits, and its container field tells the player how to find the encoded chunks and their timestamps. With this information, the player can check support on the student’s device and subscribe to a suitable track.

flowchart TD
    accTitle: From a lecture catalog to a displayed picture
    accDescr: The player reads the catalog, selects a supported rendition, receives its media objects, parses encoded chunks and timestamps, decodes pictures, then schedules and displays them.
    A[Read the media catalog] --> B[Choose a supported rendition]
    B --> C[Subscribe and receive objects]
    C --> D[Parse chunks and timestamps]
    D --> E[Decode pictures]
    E --> F[Schedule and display]
    A -. Decoder configuration .-> E
    D -. Media timestamps .-> F

The catalog can change after playback starts. The lecturer might add screen sharing, or a rendition might temporarily become unavailable. The player needs to respond to those updates instead of treating the initial list as permanent.

What does container parsing actually do?

Let’s look inside one payload. In hang’s legacy container, the first bytes encode a timestamp in microseconds as a variable-length integer. The encoded media comes after it. The player reads the timestamp, strips off that header, and passes the media to the decoder with its timing information. It chooses this parser from the container field in the catalog. See the hang container formats.

For example, mistaking timestamp bytes for H.264 data can produce a decoder failure even when delivery was perfect. This is why parsing has its own step in the diagram.

Four things that are easy to confuse

Term What it describes In the lecture
Codec How media is compressed and reconstructed H.264 video
Rendition One encoding offered to the viewer A 720p version of the lecturer’s camera
MoQ object An addressed unit delivered by the transport A payload identified by its track, group, and object IDs
Decoded frame A picture the decoder has reconstructed A VideoFrame the player can render

An encoded chunk is compressed media ready to submit to a decoder. The media format tells the player how to extract chunks from MoQ objects. One object might contain one video frame, as in our example, but other formats can package them differently.

Where can decoding begin?

Video compression often describes how one picture differs from earlier pictures. Such a predicted frame is useful only when the decoder has the reference pictures it needs. A keyframe supplies a fresh decoding starting point in the video model we are using.

This explains the group boundaries from the object-model chapter. The publisher starts a video group with a keyframe, then places dependent frames after it. Our late-arriving student can retrieve the beginning of the current group or wait for the next group. Receiving only the middle of a group may leave the decoder without its references, even though every requested byte arrived correctly.

We can’t create a starting point just by cutting the stream every second. The encoder needs to produce a picture that can start the new group, and the packager needs to put the boundary there. The player also needs the decoder configuration for that picture.

The same issue appears when the student switches to higher resolution. The new rendition has its own decoder state. The player changes configuration as needed and starts at a valid decoding point; it cannot simply feed an arbitrary high-resolution predicted frame into the old sequence.

How does WebCodecs decode the media?

WebCodecs gives JavaScript access to media encoders and decoders. For video, the player checks a configuration with VideoDecoder.isConfigSupported(), applies it with configure(), and submits encoded chunks with decode(). Support depends on the actual codec configuration and device, so the presence of the API alone is insufficient.

The decoder calls the player’s output callback with a VideoFrame. The player then holds that picture until it’s time to display it. Your player or media library handles this queue, the playback clock, and rendering. See the VideoDecoder interface for the API.

Each decoded frame holds resources. After rendering or discarding a frame, the player calls VideoFrame.close() when it no longer needs that frame. Waiting for garbage collection can retain scarce resources and stall decoding. Bounded queues are equally important: if rendering slows down, receiving more frames must not cause unlimited memory growth. See the WebCodecs memory model.

When should a frame be presented?

Frames do not necessarily arrive at the spacing at which they were captured. Variable network delay produces jitter. If the player displayed pictures on arrival, that uneven timing would become visible movement on the screen.

Media timestamps tell the player how far apart the frames belong. In the example below, the media timeline and receiver clock start at different points. The table uses milliseconds, while WebCodecs VideoFrame.timestamp uses microseconds. The player needs to account for both the starting points and the units.

Frame Media timestamp Arrival on receiver clock Planned display on receiver clock
A 0 100 160
B 33 145 193
C 67 150 227

The player chooses receiver time 160 as the moment to display media time zero. It adds each subsequent media timestamp to that anchor. Although B and C arrive only five milliseconds apart, the player displays them 34 milliseconds apart, preserving the media timeline.

Arrival time and presentation time

flowchart LR
    accTitle: Uneven arrivals become evenly scheduled presentation
    accDescr: Frame A arrives at receiver time 100 and is displayed at 160. B arrives at 145 and is displayed at 193. C arrives at 150 and is displayed at 227. Each frame waits until its scheduled display time.
    A["A arrives: 100 ms"] -->|"Wait 60 ms"| D["Display A: 160 ms"]
    B["B arrives: 145 ms"] -->|"Wait 48 ms"| E["Display B: 193 ms"]
    C["C arrives: 150 ms"] -->|"Wait 77 ms"| F["Display C: 227 ms"]

Each arrow spans arrival to display, including any parsing and decoding still needed. The example assumes those steps finish before the planned display time. A frame that arrives on time but finishes decoding late can still miss that time. Arrow lengths are not a time scale.

That first 60 ms wait gives later frames some time to arrive. Now suppose B arrives at 205 ms, after its planned display time of 193 ms. The player can skip it or delay playback. Waiting longer at the start would have made room for B, but it would also have delayed every picture.

Keeping audio and video together

Audio follows the same presentation timeline. An audio sample and a picture intended for media time 67 ms can arrive separately; the player schedules them against a shared clock so the lecturer’s voice stays aligned with their face. Over longer sessions it may also need to correct differences between clocks.

To measure how far behind the lecture we are, we also need to relate the lecturer’s capture clock to the viewer’s clock. The playback timestamps above tell us how to space the frames, but don’t give us that relationship.

What else does a browser player need?

Even after media arrives, the browser may require a user action before it allows playback. A publisher capturing a camera or microphone needs capture permission; publishing a supplied file does not require access to those devices. See the media capture permission rules. Devices can disappear, tabs can become hidden, and the page must release resources when the session ends.

The moq.dev project provides higher-level player and publisher packages that handle much of this integration. Start with those packages for the first experiment. If playback fails, check each step: did the player find the media description, receive a decodable starting picture, decode it, and display it?

Sources and further reading