Document initial browser implementation test results.
Initial ProtoXEP submission.
Real-time text is already defined for XMPP by In-Band Real Time Text (XEP-0301)
This specification defines a way to negotiate real-time text as a Jingle content in the same session as audio and video. The text content can be human typed RTT, captions, ASR output, interpreter text, translation text or transcript text. The goal is Total Conversation: audio, video and text presented as one conversational unit.
The motivating implementation problem is simple: a call can exist, text can exist, and yet the text might not be part of the negotiated Jingle session. In that case the receiver cannot reliably treat the text as synchronized conversational media.
This specification is designed to meet the following requirements.
Implementations can support different levels without falsely claiming full synchronization.
| Level | Name | Minimum capability | User-visible promise |
|---|---|---|---|
| 0 | XEP-0301 fallback | Ordinary in-band RTT outside Jingle | Live text, not media synchronized |
| 1 | Jingle co-session text | Text is negotiated by the same Jingle session but does not share a media clock | Belongs to the call, limited synchronization |
| 2 | Session-clock text | Text has timestamps relative to a shared call or session clock | Call-synchronized text |
| 3 | Media-clock text | RTP/T.140 or equivalent media-clock timing with audio/video correlation | Strict synchronized Total Conversation |
An implementation MUST NOT advertise a higher level than it can actually deliver. In particular, a WebRTC data channel that is merely opened during a call is Level 1 unless it can demonstrate a shared session clock or media clock.
An initiator offers audio, video and text contents in one Jingle session. The receiver accepts all three contents and presents them as a single Total Conversation.
A participant starts an audio-video call and later adds captions, ASR or typed text by sending a Jingle content-add action for the text content.
If the peer does not support this specification, a client can fall back to In-Band Real Time Text (XEP-0301)
A Total Conversation call SHOULD contain three Jingle contents:
The text content is not an ordinary XMPP message stream. It is part of the Jingle session and is described by this extension.
The binding key is the Jingle sid plus the content name and the sync-group. A client MUST NOT infer synchronization only from the peer JID, because a user can have multiple simultaneous sessions, devices or fallback chat streams with the same peer.
An entity supporting this specification MUST advertise the following feature:
If the entity supports RTP/T.140, it SHOULD advertise:
If the entity supports WebRTC datachannel T.140, it SHOULD advertise:
If the entity supports fallback to In-Band Real Time Text (XEP-0301)
This specification defines an rtt-sync element qualified by the urn:xmpp:jingle:apps:rtt-sync:0 namespace.
| Attribute | Required | Values | Meaning |
|---|---|---|---|
| role | yes | conversation, caption, transcript, translation, interpreter | Purpose of the text stream |
| source | no | human, asr, captioner, interpreter, translation, system | Origin of the text |
| lang | no | BCP 47 language tag | Language of the text |
| sync-group | yes | token | Group shared by audio, video and text contents |
| sync-reference | no | content name | Content this text is synchronized with, usually audio |
| sync-mode | yes | media-clock, session-clock, co-session, none | Synchronization model |
| max-skew | no | milliseconds | Maximum target presentation difference |
| finality | no | partial, final, mixed | Whether text can change |
The RTP/T.140 profile is the preferred profile when strict synchronization with audio and video is required. The initiator offers a Jingle RTP content with media='text' and payload types for t140 and optionally red.
When sync-mode='media-clock' is negotiated, endpoints SHOULD use the same RTCP CNAME for audio, video and text RTP streams belonging to the same endpoint. Receivers SHOULD use RTP/RTCP timing to align text with audio or video where possible. If timing information is unavailable, the receiver MAY fall back to session arrival time and SHOULD indicate reduced synchronization quality.
The datachannel profile supports browser/WebRTC deployments using T.140 over a reliable, ordered data channel. This profile is useful when a WebRTC implementation naturally uses data channels for RTT. However, data channels do not automatically share the RTP media clock, so the synchronization mode MUST be declared carefully.
The exact Jingle mapping for WebRTC data channel negotiation should be aligned with the relevant Jingle data channel signalling specification. This document does not attempt to replace that signalling.
If the responder does not support urn:xmpp:jingle:apps:rtt-sync:0, the initiator MAY fall back to In-Band Real Time Text (XEP-0301)
Fallback is a state transition, not just a transport choice. If a Jingle text content is rejected but audio and video are accepted, the call MAY continue without synchronized text. If fallback RTT is started for the same conversation, it SHOULD be bound to the Jingle sid and shown as fallback rather than synchronized captions.
A user interface SHOULD distinguish at least these cases: live text, live captions, AI captions, human captions, translation and unsynchronized fallback.
During call setup, a client SHOULD expose whether synchronized text was negotiated, whether live text fallback is active or whether text is unavailable in the call.
This specification is specifically motivated by accessibility and Total Conversation use cases. A deaf or hard-of-hearing user MUST be able to distinguish between typed text, human captions, AI or ASR captions and translated text where this information is known.
A client SHOULD visibly indicate late captions, uncertain ASR captions or unsynchronized fallback text. A client SHOULD allow users to prefer synchronized captions over lowest-latency captions, or lowest-latency captions over strict synchronization.
Text content MUST support Unicode. Language tags SHOULD use BCP 47. Clients SHOULD support multiple simultaneous text streams where translation or interpreter text is provided in addition to original captions.
Synchronized RTT and captions can contain highly sensitive conversation content. Implementations SHOULD use end-to-end encrypted signalling and encrypted media where available.
For RTP/T.140, implementations SHOULD use SRTP or an equivalent encrypted RTP transport, authenticate the sender of the text stream and protect against injection of false captions. Implementations SHOULD prevent downgrade attacks from synchronized RTT to unsynchronized fallback without user indication.
Clients SHOULD avoid misrepresenting AI captions as human or verified text.
Real-time text can reveal text before the sender considers it final. Captions can reveal speech content to captioning, relay or ASR services. A client SHOULD obtain user consent before sending typed RTT and before sending audio to ASR or captioning services.
A client SHOULD not store partial captions or partial RTT as a final transcript unless enabled. A client SHOULD indicate when a third-party captioning, ASR, relay or interpreting service is active.
This document makes no direct IANA request unless future revisions define new SDP attributes or new media types. The RTP/T.140 profile uses existing text/t140 and text/red media formats.
This specification requests registration of the following namespace:
urn:xmpp:jingle:apps:rtt-sync:0
The following service discovery features are requested:
urn:xmpp:jingle:apps:rtt-sync:0
urn:xmpp:jingle:apps:rtt-sync:rtp-t140:0
urn:xmpp:jingle:apps:rtt-sync:dc-t140:0
This document does not replace In-Band Real Time Text (XEP-0301)
RTP/T.140 is the preferred strict synchronization profile. WebRTC datachannel T.140 is useful for browser deployments, but MUST NOT be described as media-clock synchronized unless the implementation can provide the required timing relationship.
An experimental browser implementation has tested the WebRTC datachannel profile at Level 1. Two browser sessions negotiated one Jingle audio-video session plus a text content using urn:xmpp:jingle:apps:rtt-sync:0, opened a reliable ordered data channel labelled rtt, exchanged live RTT updates, and delivered final text bound to the Jingle session. The client presented the call as live text synchronized with the call session.
The same implementation retained In-Band Real Time Text (XEP-0301)
The following schema is an initial sketch.
]]>