XEP-0198: Stream Management

NOTICE: This document is currently within Last Call or under consideration by the XMPP Council for advancement to the next stage in the XSF standards process.

Table of Contents

1. Introduction

XMPP Core [1] defines the fundamental streaming XML technology used by XMPP (i.e., stream establishment and termination including authentication and encryption). However, the core XMPP specification does not provide tools for actively managing a "live" XML stream.

The basic concept behind stream management is that the initiating entity (either a client or a server) and the receiving entity (a server) can exchange "commands" for active management of the stream. (Instead of using XMPP IQ, message, or presence stanzas, stream management uses a series of short XML elements at the root stream level.) In particular, it is expected that the following stream management features will improve network reliability and the end-user experience:

• Acks -- the ability to know if a stanza or series of stanzas has been received by one's peer.
• Resumption -- the ability to quickly resume a stream that has been terminated.
• Throttling -- the ability for the receiving entity to inform the initiating entity that rate limiting is in force.

Detailed descriptions of these features are provided in the remainder of this specification.

Note: To check TCP connectivity for a given stream, it is RECOMMENDED to use XMPP Ping [2], whitespace pings (see Section 5.7.3 of rfc3920bis [3]), or TCP keepalives.

Note: This specification applies at the level of an XML stream between a client and a server or between two servers. By constrast, Advanced Message Processing [4] and Message Receipts [5] define acks that are sent end-to-end over multiple streams; these facilities are useful in special scenarios but are unnecessary for checking of a direct stream between two XMPP entities. (It is also expected that this protocol will revive interest in Advanced Message Processing (AMP), because single-hop acks are necessary for AMP delivery receipts to function properly.)

Note: Examples prepended by "C:" are sent by a client and examples prepended by "S:" are sent by a server. Stream management can be used server-to-server but most of the examples in this specification show its use between a client and a server.

2. Stream Feature

After negotiating use of TLS and authenticating via SASL, the receiving entity returns a new stream header to the intiating entity along with stream features, where the features include an <sm/> element qualified by the 'urn:xmpp:sm:2' namespace (see Namespace Versioning regarding the possibility of incrementing the version number).

The stream management feature MUST NOT be offered unless the initiating entity has been authenticated (e.g., by means of SASL, Non-SASL Authentication [6], or Server Dialback [7]).

Example 1. Server sends new stream header along with stream features

S: <stream:stream
       to='example.com'
       xmlns='jabber:client'
       xmlns:stream='http://etherx.jabber.org/streams'
       version='1.0'>

S: <stream:features>
     <bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'>
       <required/>
     </bind>
     <sm xmlns='urn:xmpp:sm:2'>
       <optional/>
     </sm>
   </stream:features>
  

3. Enabling Stream Management

To enable use of stream management, the initiating entity sends an <enable/> command to the receiving entity.

Example 2. Client enables stream management

C: <enable xmlns='urn:xmpp:sm:2'/>
  

If the initiating entity wants to be allowed to resume the stream, it includes a boolean 'resume' attribute, which defaults to false [8]. For information about resuming a previous session, see the Resumption section of this document.

The <enable/> element MAY include a 'max' attribute to specify the initiating entity's preferred maximum resumption time.

The <enable/> element MAY include a 'stanzas' attribute to specify the initiating entity's preferred number of stanzas between acks.

Upon receiving the enable request, the receiving entity MUST reply with an <enabled/> element or a <failed/> element qualified by the 'urn:xmpp:sm:2' namespace. The <failed/> element indicates that there was a problem establishing the stream management "session". The <enabled/> element indicates successful establishment of the stream management session.

Example 3. Server enables stream management

S: <enabled xmlns='urn:xmpp:sm:2'/>
  

The parties can then the use stream management features defined below.

If the receiving entity allows session resumption, it MUST include a 'resume' attribute set to a value of "true" or "1".

Example 4. Server enables stream management with session resumption

S: <enabled xmlns='urn:xmpp:sm:2' resume='true'/>
  

The <enabled/> element MAY include a 'max' attribute to specify the receiving entity's preferred maximum resumption time.

The <enabled/> element MAY include a 'stanzas' attribute to specify the receiving entity's preferred number of stanzas between acks.

For client-to-server connections, the client SHOULD NOT attempt to enable stream management until after it has completed Resource Binding unless it is resuming a previous session (see Resumption). The server MAY enforce this order and return a <failed/> element in response (see Error Handling).

Example 5. Server returns error if client attempts to enable stream management before resource binding

S: <failed xmlns='urn:xmpp:sm:2'>
     <unexpected-request xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/>
   </failed>
  

4. Acks

After enabling stream management, the initiating or receiving entity can send ack elements at any time over the stream. An ack element is one of the following:

• The <a/> element is used to answer a request for acknowledgement or send an unrequested ack.
• The <r/> element is used to request acknowledgement of received stanzas.

The following attribute is defined:

• The 'h' attribute identifies the last handled stanza (i.e., the last stanza that the receiver will acknowledge as having received).

An <a/> element MUST possess an 'h' attribute.

An <r/> element SHOULD NOT possess any attributes.

Definition: Acknowledging a previously-received ack element indicates that the stanza(s) sent since then have been "handled" by the receiver. By "handled" we mean that the stanzas have been accepted by the receiver, not necessarily that they have been fully processed by the receiver, delivered to a local entity such as another connected client on the same server, or routed to a remote entity at a different server. Therefore value of 'h' starts at one (the first stanza handled) and is incremented with each subsequent stanza handled.

The following example shows a message sent by the client, a request for acknowledgement, and an ack of the stanza.

Example 6. Simple stanza acking

C: <message from='laurence@example.net/churchyard'
            to='juliet@example.com'
            xml:lang='en'>
     <body>
       I'll send a friar with speed, to Mantua, 
       with my letters to thy lord.
     </body>
   </message>

C: <r xmlns='urn:xmpp:sm:2'>

S: <a xmlns='urn:xmpp:sm:2' h='1'>
  

When an <r/> element ("request") is received, the recipient MUST acknowledge it by sending an <a/> element to the sender containing a value of 'h' that is equal to the number of stanzas handled by the recipient of the <r/> element. The response SHOULD be sent as soon as possible after receiving the <r/> element, and MUST NOT be withheld for any condition other than a timeout. For example, a client with a slow connection might want to collect many stanzas over a period of time before acking, and a server might want to throttle incoming stanzas. The sender does not have to wait for an ack to continue sending stanzas. Because acks indicate stanza acceptance, a server that is throttling stanzas MUST delay the response until the client is no longer being penalized (but SHOULD notify the client that it is throttling incoming stanzas, as described under Throttling).

When a party returns an ack in response to an <r/> element or receives such an ack, it SHOULD keep a record of the 'h' value returned as the sequence number of the last handled stanza for the current stream (and discard the previous 'h' value).

If a stream ends and it is not resumed within the time specified in the acknowledgement feature element, the sequence number and any associated state MAY be discarded by both parties. Before the session state is discarded, implementations SHOULD take alternative action regarding any unhandled stanzas (e.g. stanzas sent after the most recent 'h' value):

• A server SHOULD treat unacknowledged stanzas in the same way that it would treat a stanza sent to an unavailable resource, by either returning an error to the sender or committing the stanza to offline storage.
• A user-oriented client SHOULD inform the user of the failure via appropriate user-interface elements.

5. Resumption

It can happen that an XML stream is terminated unexpectedly (e.g., because of network outages). In this case, it is desirable to quickly resume the former stream rather than complete the tedious process of stream establishment, roster retrieval, and presence broadcast.

To request that the stream will be resumable, when enabling stream management the initiating entity MUST add a 'resume' attribute to the <enable/> element with a value of "true" or "1" [9].

Example 7. Client enables stream management

C: <enable xmlns='urn:xmpp:sm:2' resume='true'/>
  

If the receiving entity will allow the stream to be resumed, it MUST include a 'resume' attribute set to "true" or "1" on the <enabled/> element and MUST include an 'id' attribute that specifies an identifier for the stream.

Example 8. Server allows stream resumption

S: <enabled xmlns='urn:xmpp:sm:2' id='some-long-sm-id' resume='true'/>
  

Definition: The 'id' attribute defines a unique identifier for purposes of stream management (an "SM-ID"). The SM-ID MUST be generated by the receiving entity (server). The initiating entity MUST consider the SM-ID to be opaque and therefore MUST NOT assign any semantic meaning to the SM-ID. The receiving entity MAY encode any information it deems useful into the SM-ID, such as the full JID <localpart@domain.tld/resource> of a connected client (e.g., the full JID plus a nonce value). Any characters allowed in an XML attribute are allowed. The SM-ID MUST NOT be reused for simultaneous or subsequent sessions (as long as the receiving entity is available). The SM-ID SHOULD NOT be longer than 4000 bytes.

If the stream is terminated unexpectedly, the initiating entity would then open a TCP connection to the receiving entity. The order of events is as follows:

1. Initiating entity sends initial stream header.
2. Receiving entity sends response stream header.
3. Receiving entity sends stream features.
4. Initiating entity sends STARTTLS request.
5. Receiving entity informs initiating entity to proceed with the TLS negotiation.
6. The parties complete a TLS handshake. (Note: When performing session resumption and also utilizing TLS, it is RECOMMENDED to take advantage of TLS session resumption to further optimize the resumption of the XML stream.)
7. Initiating entity sends new initial stream header.
8. Receiving entity sends response stream header.
9. Receiving entity sends stream features, requiring SASL negotiation and offering appropriate SASL mechanisms. (Note: If the server considers the information provided during TLS session resumption to be sufficient authentication, it MAY offer the SASL EXTERNAL mechanism; for details, refer to draft-cridland-sasl-tls-sessions [10].)
10. The parties complete SASL negotiation.
11. Initiating entity sends new initial stream header.
12. Receiving entity sends response stream header.
13. Receiving entity sends stream features, offering the SM feature.
14. Initiating entity requests resumption of the former stream.

To request resumption of the former stream, the initiating entity sends a <resume/> element qualified by the 'urn:xmpp:sm:2' namespace. The <resume/> element MUST include a 'previd' attribute whose value is the SM-ID of the former stream and MAY include an 'h' attribute that identifies the sequence number of the last handled stanza sent over the former stream from the receiving entity to the initiating entity (if stream management was being used in both directions); if there is no such sequence number for the former stream, the 'h' attribute MUST NOT be included.

Example 9. Stream resumption request

C: <resume xmlns='urn:xmpp:sm:2' previd='some-long-sm-id' h='some-sequence-number'/>
  

If the receiving entity can resume the former stream, it MUST return a <resumed/> element that includes a 'previd' attribute set to the SM-ID of the former stream. The <resumed/> element MAY also include an 'h' attribute set to the sequence number of the last handled stanza sent over the former stream from the initiating entity to the receiving entity; if there is no such sequence number for the former stream, the 'h' attribute MUST NOT be included.

Example 10. Stream resumed

S: <resumed xmlns='urn:xmpp:sm:2' previd='some-long-sm-id' h='another-sequence-number'/>
  

If the receiving entity does not support session resumption, it MUST return a <failed/> element, which SHOULD include an error condition of <feature-not-implemented/>. If the receiving entity does not recognize the 'previd' as an earlier session (e.g., because the former session has timed out), it MUST return a <failed/> element, which SHOULD include an error condition of <item-not-found/>. In both of these failure cases, the receiving entity SHOULD allow the initiating entity to bind a resource at this point rather than forcing the initiating entity to restart the stream and re-authenticate.

If the former stream is resumed and the receiving entity still has the stream for the previously-identified session open at this time, the old stream SHOULD be terminated.

When a session is resumed, the parties SHOULD proceed as follows:

• Both parties SHOULD retransmit any stanzas that were not accepted during the previous session, based on the sequence number reported by the peer.
• A reconnecting client SHOULD NOT request the roster, because any roster changes that occurred while the client was disconnected will be sent to the client after the stream resumes.
• The client SHOULD NOT resend presence stanzas in an attempt to restore its former presence state, since this state will have been retained by the server.
• Both parties SHOULD NOT try to re-establish state information (e.g., Service Discovery [11] information).

6. Throttling

When a server acts as a receiving entity for an XML stream, it might throttle the stream (i.e., impose rate limiting) if the initiating entity (a client or a server) attempts to send too much traffic over the stream (e.g., a very large number of stanzas, or a lesser number of stanzas that are relatively large). The formulas for determining when rate limiting shall be imposed are implementation-specific; however, nearly all XMPP server implementations include support for such throttling (often called "karma"). Therefore it would be helpful if the receiving entity could inform the initiating entity that the stream has been voluntarily throttled by the receiving entity. [12] It can do so by sending a <t/> element to the receiving entity:

Example 11. Throttling notification

S: <t xmlns='urn:xmpp:sm:2'/>
  

Note: Sending a throttling notification to the stream peer does not necessarily indicate that the entity is throwing away all stanzas, only that the entity has voluntarily slowed its processing of incoming stanzas.

The throttling notification MAY include a 'stanzas' attribute so that the receiving entity can inform the initiating entity of changes to the maximum number of stanzas between acks. This enables the receiving entity to dynamically adjust stanza acking in response to network conditions or stream usage.

Example 12. Throttling notification with acking adjustment

S: <n xmlns='urn:xmpp:sm:2' stanzas='5'/>
  

If the number of unacknowledged stanzas is greater than or equal to the value of the 'stanzas' attribute, a peer MUST NOT send any further stanzas.

If an entity sets the value of 'stanzas' to zero, it has stopped reading from the stream entirely.

For as long as the initiating entity is throttled, the receiving entity SHOULD periodically send a throttling notification to the initiating entity (e.g., every 30 seconds) to obviate the need for pings generated by the initating entity (which the receiving entity will ignore because the initiating entity is throttled).

7. Error Handling

If an error occurs with regard to an <enable/> or <resume/> element, the receiving entity MUST return a <failed/> element. This element SHOULD contain an error condition, which MUST be one of the stanza error conditions defined in rfc3920bis [13].

An example follows.

Example 13. Server returns error

S: <failed xmlns='urn:xmpp:sm:2'>
     <unexpected-request xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/>
   </failed>
  

Stream management errors SHOULD be considered recoverable; however, misuse of stream management MAY result in termination of the stream.

8. Scenarios

The following scenarios illustrate several different uses of stream management. The examples are that of a client and a server, but stream management can also be used for server-to-server streams.

C: <enable xmlns='urn:xmpp:sm:2'/>
    

S: <enabled xmlns='urn:xmpp:sm:2'/>
    

C: <iq id='ls72g593' type='get'>
     <query xmlns='jabber:iq:roster'/>
   </iq>

C: <r xmlns='urn:xmpp:sm:2'/>
    

S: <a xmlns='urn:xmpp:sm:2' h='1'/>

S: <iq id='ls72g593' type='result'>
     <query xmlns='jabber:iq:roster'>
       <item jid='juliet@capulet.lit'/>
       <item jid='benvolio@montague.lit'/>
     </query>
   </iq>

    

C: <a xmlns='urn:xmpp:sm:2' h='1'/>

C: <presence/>

C: <r xmlns='urn:xmpp:sm:2'/>
    

S: <a xmlns='urn:xmpp:sm:2' h='2'/>

S: <presence from='romeo@montague.lit/orchard' to='romeo@montague.lit/orchard'/>
    

C: <a xmlns='urn:xmpp:sm:2' h='2'/>

C: <message to='juliet@capulet.lit'>
     <body>ciao!</body>
   </message>

C: <r xmlns='urn:xmpp:sm:2'/>
    

S: <a xmlns='urn:xmpp:sm:2' h='3'/>
    

C: <enable/>
S: <enabled stanzas='5'/>
C: <message/>
C: <message/>
C: <message/>
C: <message/>
C: <message/>
C: <r/>
S: <a h='5'/>
C: <message/>
C: <message/>
C: <message/>
C: <message/>
C: <message/>
C: <r/>
S: <a h='10'/>
    

C: <enable/>
S: <enabled stanzas='10'/>
C: <message/>
C: <message/>
C: <message/>
[throttling kicks in]
S: <t stanzas='3'/>
[client requests an ack for the first three messages, but
does not send any more messages until throttling ends]
C: <r/>
[client still throttled, server ignores <r/> for now]
[30 seconds go by]
S: <t/>
[30 seconds go by]
S: <t/>
[backlog starts to ease, server adjusts 'stanzas' value]
S: <t stanzas='5'/>
[client sends another message just because it can]
C: <message/>
[server has handled the first 3 messages so it finally replies to <r/>]
S: <a h='3'/>
C: <message/>
C: <message/>
C: <message/>
C: <message/>
[client has sent 5 messages so requests an ack]
C: <r/>
[throttling is over, server replies to 2nd <r/> and sets 'stanzas' back to 10]
S: <a h='8'/>
S: <t stanzas='10'/>
    

9. Security Considerations

Intermediate proxies MUST pass through all stanzas, else the parties will go out of sync.

10. IANA Considerations

This XEP requires no interaction with the Internet Assigned Numbers Authority (IANA) [14].

11. Jabber Registrar Considerations

12. XML Schemas

<?xml version='1.0' encoding='UTF-8'?>

<xs:schema
    xmlns:xs='http://www.w3.org/2001/XMLSchema'
    targetNamespace='urn:xmpp:sm:2'
    xmlns='urn:xmpp:sm:2'
    elementFormDefault='qualified'>

  <xs:element name='a' type='ackElementType'/>

  <xs:element name='enable'>
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base='empty'>
          <xs:attribute name='max' 
                        type='xs:positiveInteger' 
                        use='optional'/>
          <xs:attribute name='resume' 
                        type='xs:boolean'
                        use='optional'
                        default='false'/>
          <xs:attribute name='stanzas' 
                        type='xs:unsignedShort' 
                        use='optional'/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>

  <xs:element name='enabled'>
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base='empty'>
          <xs:attribute name='id' 
                        type='xs:string'
                        use='optional'/>
          <xs:attribute name='max' 
                        type='xs:positiveInteger' 
                        use='optional'/>
          <xs:attribute name='resume' 
                        type='xs:boolean'
                        use='optional'
                        default='false'/>
          <xs:attribute name='stanzas' 
                        type='xs:unsignedShort' 
                        use='optional'/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>

  <xs:element name='failed'>
    <xs:complexType>
      <xs:sequence xmlns:err='urn:ietf:params:xml:ns:xmpp-stanzas' minOccurs='0'>
        <xs:group ref='err:stanzaErrorGroup'/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>

  <xs:element name='r' type='ackElementType'/>

  <xs:element name='resume' type='resumptionElementType'>

  <xs:element name='resumed' type='resumptionElementType'>

  <xs:element name='sm'>
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base='empty'>
          <xs:attribute name='id' 
                        type='xs:string' 
                        use='required'/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>

  <xs:element name='t'>
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base='empty'>
          <xs:attribute name='stanzas' 
                        type='xs:unsignedShort' 
                        use='optional'/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>

  <xs:complexType name='ackElementType'>
    <xs:simpleContent>
      <xs:extension base='empty'>
        <xs:attribute name='h' 
                      type='xs:integer' 
                      use='optional'/>
      </xs:extension>
    </xs:simpleContent>
  </xs:complexType>

  <xs:complexType name='resumptionElementType'>
    <xs:simpleContent>
      <xs:extension base='empty'>
        <xs:attribute name='h' 
                      type='xs:positiveInteger' 
                      use='optional'/>
        <xs:attribute name='previd' 
                      type='xs:string' 
                      use='optional'/>
      </xs:extension>
    </xs:simpleContent>
  </xs:complexType>

  <xs:simpleType name='empty'>
    <xs:restriction base='xs:string'>
      <xs:enumeration value=''/>
    </xs:restriction>
  </xs:simpleType>

</xs:schema>
  

13. Acknowledgements

Thanks to Dave Cridland, Jack Erwin, Philipp Hancke, Curtis King, Tobias Markmann, Pedro Melo, Robin Redeker, and Mickaël Rémond for their feedback.

Appendices

Appendix A: Document Information

Series: XEP
Number: 0198
Publisher: XMPP Standards Foundation
Status: Proposed
Type: Standards Track
Version: 0.8
Last Updated: 2009-04-09
Approving Body: XMPP Council
Dependencies: XMPP Core
Supersedes: None
Superseded By: None
Short Name: NOT_YET_ASSIGNED
Source Control: HTML  RSS

Appendix B: Author Information

Appendix C: Legal Notices

Appendix D: Relation to XMPP

The Extensible Messaging and Presence Protocol (XMPP) is defined in the XMPP Core (RFC 3920) and XMPP IM (RFC 3921) specifications contributed by the XMPP Standards Foundation to the Internet Standards Process, which is managed by the Internet Engineering Task Force in accordance with RFC 2026. Any protocol defined in this document has been developed outside the Internet Standards Process and is to be understood as an extension to XMPP rather than as an evolution, development, or modification of XMPP itself.

Appendix E: Discussion Venue

The primary venue for discussion of XMPP Extension Protocols is the <standards@xmpp.org> discussion list.

Discussion on other xmpp.org discussion lists might also be appropriate; see <http://xmpp.org/about/discuss.shtml> for a complete list.

Errata can be sent to <editor@xmpp.org>.

Appendix F: Requirements Conformance

The following requirements keywords as used in this document are to be interpreted as described in RFC 2119: "MUST", "SHALL", "REQUIRED"; "MUST NOT", "SHALL NOT"; "SHOULD", "RECOMMENDED"; "SHOULD NOT", "NOT RECOMMENDED"; "MAY", "OPTIONAL".

Appendix G: Notes

Appendix H: Revision History

END