XEP-0237: Data Sequencing

Copyright (c) 1999 - 2009 XMPP Standards Foundation. See Legal Notices.

This specification defines an XMPP extension that enables a requesting entity to receive a large data set only if the set has changed; the primary use case is sequencing of roster changes for more efficient downloading of the roster information.

WARNING: This Standards-Track document is Experimental. Publication as an XMPP Extension Protocol does not imply approval of this proposal by the XMPP Standards Foundation. Implementation of the protocol described herein is encouraged in exploratory implementations, but production systems should not deploy implementations of this protocol until it advances to a status of Draft.

Table of Contents


1. Introduction
2. Data Format
3. Use With Rosters
    3.1. Roster Get
    3.2. Roster Is Unchanged
    3.3. Roster Has Changed
    3.4. Subsequent Roster Pushes
4. Use With Service Discovery
    4.1. Items Request
    4.2. Disco Items Are Unchanged
    4.3. Disco Items Have Changed
    4.4. Subsequent Notifications
5. Determining Support
    5.1. Stream Feature
    5.2. Service Discovery
6. Security Considerations
7. IANA Considerations
8. XMPP Registrar Considerations
    8.1. Protocol Namespaces
9. XML Schemas
10. Acknowledgements

Appendices
    A: Document Information
    B: Author Information
    C: Legal Notices
    D: Relation to XMPP
    E: Discussion Venue
    F: Requirements Conformance
    G: Notes
    H: Revision History

1. Introduction

Certain XMPP technologies can return large data sets to users (examples are rosters as specified in XMPP IM [1] and item lists as specified in Service Discovery [2]). Although Result Set Management [3] provides a generic way to page through such data sets, it does not provide a way to learn if the data set has changed since it was last retrieved. If the client could cache the data set (e.g., the roster) and retrieve only changes to the data set, certain use cases (e.g., the login process) could be significantly streamlined. This feature might be especially valuable over low-bandwidth connections such as those common in mobile environments. This document defines a method for such streamlining, via the concept of data sequencing.

2. Data Format

This document defines a <seq/> element qualified by the 'urn:xmpp:tmp:seq' namespace (see Protocol Namespaces regarding issuance of one or more permanent namespaces). This element can be included in any IQ request that might result in a large data set. Because only one child element is allowed in an IQ stanza, the <seq/> element MUST be included as a child of the payload element (i.e., as a grandchild of the IQ stanza).

The <seq/> element is defined as empty (except when used to advertise a stream feature). It possesses a single attribute: 'num'.

The value of the 'num' attribute MUST be a non-negative integer representing a strictly increasing sequence number that is increased (but not necessarily incremented-by-one) with any change to the data set.

3. Use With Rosters

3.1 Roster Get

If a client supports data sequencing and knows that the server does so (see Determining Support), it SHOULD include the <seq/> element in its request for the roster, where the 'num' attribute is set to the sequence number associated with its last cache of the roster.

Example 1. Roster get with sequence number

<iq from='romeo@montague.lit/home' id='r1' to='romeo@montague.lit' type='get'>
  <query xmlns='jabber:iq:roster'>
    <seq xmlns='urn:xmpp:tmp:seq' num='305'/>
  </query>
</iq>

If the client has not yet cached the roster or the cache is lost or corrupted, but the client wishes to bootstrap the use of data sequencing, it SHOULD include the <seq/> element with the 'num' attribute set to a value of zero (0).

Naturally, if the client does not support data sequencing or does not wish to bootstrap use of data sequencing, it will behave like an RFC-3921-compliant client by not including the <seq/> element.

3.2 Roster Is Unchanged

If the roster has not changed since the version enumerated by the client, the server MUST return an empty IQ-result.

Example 2. Roster result (unchanged)

<iq from='romeo@montague.lit' id='r1' type='result'/>

3.3 Roster Has Changed

If the roster has changed since the version enumerated by the client, the server MUST return a <query/> element that includes the latest sequence number.

The <query/> element MUST either contain the complete roster (including the sequence number to indicate that the roster has changed) or be empty (indicating that roster changes will be sent as interim roster pushes).

In general, if returning the complete roster would use less bandwidth than sending individual roster pushes to the client (e.g., if the roster contains only a few items), the server SHOULD return the complete roster.

Example 3. Roster result with complete roster

<iq from='romeo@montague.lit' id='r1' to='romeo@montague.lit/home' type='result'>
  <query xmlns='jabber:iq:roster'>
    <item jid='bill@shakespeare.lit' subscription='both'/>
    <item jid='nurse@capulet.lit' name='Nurse' subscription='both'>
      <group>Servants</group>
    </item>
    <seq xmlns='urn:xmpp:tmp:seq' num='317'/>
  </query>
</iq>

However, if returning the complete roster would use more bandwidth than sending individual roster pushes to the client (e.g., if the roster contains many items, only a few of which have changed), the server SHOULD return an empty <query/> element, then send individual roster pushes.

Example 4. Roster result with no items

<iq from='romeo@montague.lit' id='r1' to='romeo@montague.lit/home' type='result'>
  <query xmlns='jabber:iq:roster'>
    <seq xmlns='urn:xmpp:tmp:seq' num='317'/>
  </query>
</iq>

Example 5. Interim roster pushes

<iq from='romeo@montague.lit' id='p1' to='romeo@montague.lit/home' type='set'>
  <query xmlns='jabber:iq:roster'>
    <item jid='shylock@shakespeare.lit' subscription='remove'/>
    <seq xmlns='urn:xmpp:tmp:seq' num='313'/>
  </query>
</iq>

<iq from='romeo@montague.lit' id='p2' to='romeo@montague.lit/home' type='set'>
  <query xmlns='jabber:iq:roster'>
    <item jid='bill@shakespeare.lit' subscription='both'/>
    <seq xmlns='urn:xmpp:tmp:seq' num='317'/>
  </query>
</iq>

The interim roster pushes can be understood as follows:

  1. Imagine that the client had an active presence session for the entire time between its cached roster version (in this case, 305) and the new roster version (317).
  2. During that time, the client might have received roster pushes related to data sequence numbers 306, 307, 310, 311, 313, 314, 315, and 317 (the sequence numbers must be strictly increasing but there is no requirement that the sequence shall be continuous).
  3. However, some of those roster pushes might have contained intermediate updates to the same roster item (e.g., changes in the subscription state for bill@shakespeare.lit from "none" to "to" and from "to" to "both").
  4. The interim roster pushes would not include all of the intermediate steps, only the final result of all changes applied while the client was in fact offline.

The client can determine when the interim roster pushes have ended by comparing the sequence number it received on the empty <query/> element against the sequence number it receives in roster pushes.

3.4 Subsequent Roster Pushes

When the server sends subsequent roster pushes to the client, it MUST include the updated data sequence number. Roster pushes MUST occur in sequence order. The sequence number contained in a roster push MUST be unique. A "change to the roster" is any addition of, update to, or removal of a roster item that would result in a roster push, including changes in subscription states, as described in RFC 3921 or rfc3921bis.

Example 6. Roster push

<iq from='romeo@montague.lit' id='p3' to='romeo@montague.lit/home' type='set'>
  <query xmlns='jabber:iq:roster'>
    <item jid='muse@shakespeare.lit' name='The Muse' subscription='to'/>
    <seq xmlns='urn:xmpp:tmp:seq' num='317'/>
  </query>
</iq>

4. Use With Service Discovery

4.1 Items Request

If the requesting supports data sequencing and knows that another entity does so (see Determining Support), it MAY include the <seq/> element in its disco#items request, where the 'num' attribute is set to the sequence number associated with its last cache of the items.

Example 7. Items request with sequence number

<iq from='bill@shakespeare.lit/globe'
    id='disco2'
    to='chat.shakespeare.lit'
    type='get'>
  <query xmlns='http://jabber.org/protocol/disco#items'>
    <seq xmlns='urn:xmpp:tmp:seq' num='3'/>
  </query>
</iq>

As above, if the requesting entity has not yet cached the data set (or the cache is lost or corrupted) but wishes to bootstrap the use of data sequencing, it SHOULD include the <seq/> element with the 'num' attribute set to a value of zero (0).

4.2 Disco Items Are Unchanged

If the set of disco items has not changed since the version enumerated by the requesting entity, the responding entity MUST return an empty IQ-result.

Example 8. Disco items result (unchanged)

<iq from='chat.shakespeare.lit'
    id='r1'
    to='bill@shakespeare.lit/globe'
    type='result'/>

4.3 Disco Items Have Changed

If the set of disco items has changed since the version enumerated by the client, the server MUST return a <query/> element that includes the latest sequence number.

The <query/> element MUST either contain the complete set of items (including the sequence number to indicate that the set has changed) or be empty (indicating that changes will be sent as notifications as specified in Service Discovery Notifications [4]).

In general, if returning the complete set of items would use less bandwidth than sending individual notifications (e.g., if the set contains only a few items), the server SHOULD return the complete set.

Example 9. Disco items result with complete set

<iq from='chat.shakespeare.lit'
    id='disco2'
    to='bill@shakespeare.lit/globe'
    type='result'>
  <query xmlns='http://jabber.org/protocol/disco#items'>
    <item jid='heath@chat.shakespeare.lit'
          name='A Lonely Heath'/>
    <item jid='darkcave@chat.shakespeare.lit'
          name='A Dark Cave'/>
    <item jid='forres@chat.shakespeare.lit'
          name='The Palace'/>
    <item jid='inverness@chat.shakespeare.lit'
          name='Macbeth&apos;s Castle'/>
    <seq xmlns='urn:xmpp:tmp:seq' num='5'/>
  </query>
</iq>

However, if returning the complete set would use more bandwidth than sending individual notifications (e.g., if the complete set contains many items, only a few of which have changed), the server SHOULD return an empty <query/> element, then send individual notifications.

Example 10. Disco items result with no items

<iq from='chat.shakespeare.lit'
    id='disco2'
    to='bill@shakespeare.lit/globe'
    type='result'>
  <query xmlns='http://jabber.org/protocol/disco#items'>
    <seq xmlns='urn:xmpp:tmp:seq' num='5'/>
  </query>
</iq>

Example 11. Interim notifications

<message from='chat.shakespeare.lit' to='bill@shakespeare.lit' id='foo'>
  <event xmlns='http://jabber.org/protocol/pubsub#event'>
    <items node='http://jabber.org/protocol/disco#items'>
      <item id='ae890ac52d0df67ed7cfdf51b644e901'>
        <item xmlns='http://jabber.org/protocol/disco#items'
              jid='henry-the-fifth@chat.shakespeare.lit'>
          <seq xmlns='urn:xmpp:tmp:seq' num='4'/>
        </item>
      </item>
    </items>
  </event>
</message>

<message from='chat.shakespeare.lit' to='bill@shakespeare.lit' id='bar'>
  <event xmlns='http://jabber.org/protocol/pubsub#event'>
    <items node='http://jabber.org/protocol/disco#items'>
      <retract id='fa890ca52d0df67de7fcfd51b644c701'>
        <item xmlns='http://jabber.org/protocol/disco#items'
              jid='cardenio@chat.shakespeare.lit'>
          <seq xmlns='urn:xmpp:tmp:seq' num='5'/>
        </item>
      </retract>
    </items>
  </event>
</message>

The client can determine when the interim notifications have ended by comparing the sequence number it received on the empty <query/> element against the sequence number it receives in the notifications.

4.4 Subsequent Notifications

When the responding entity sends subsequent notifications to the requesting entity, it MUST include the updated sequence number. Notifications MUST occur in sequence order. The sequence number contained in a notification MUST be unique.

Example 12. A subsequent notification

<message from='chat.shakespeare.lit' to='bill@shakespeare.lit' id='baz'>
  <event xmlns='http://jabber.org/protocol/pubsub#event'>
    <items node='http://jabber.org/protocol/disco#items'>
      <retract id='fa890ca52d0df67de7fcfd51b644c701'>
        <item xmlns='http://jabber.org/protocol/disco#items'
              jid='darkcave@chat.shakespeare.lit'>
          <seq xmlns='urn:xmpp:tmp:seq' num='6'/>
        </item>
      </retract>
    </items>
  </event>
</message>

5. Determining Support

5.1 Stream Feature

If a server supports data sequencing, it MUST inform the connecting entity when returning stream features during the stream negotiation process; at the latest, when informing a client that resource binding is required. This is done by including a <seq/> element qualified by the 'urn:xmpp:tmp:seq' namespace (see Protocol Namespaces regarding issuance of one or more permanent namespaces).

Example 13. Stream features

<stream:features>
  <bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'>
    <required/>
  </bind>
  <seq xmlns='urn:xmpp:tmp:seq'>
    <optional/>
  </seq>
</stream:features>

5.2 Service Discovery

In order for an application to determine whether an entity supports this protocol, where possible it SHOULD use the dynamic, presence-based profile of service discovery defined in Entity Capabilities [5]. However, if an application has not received entity capabilities information from an entity, it SHOULD use explicit service discovery instead.

6. Security Considerations

It is possible that caching of data sets (rather than holding them in memory only for the life of the session) could introduce new vulnerabilities. Implementations are advised to appropriately protect cached data sets.

7. IANA Considerations

This document requires no interaction with the Internet Assigned Numbers Authority (IANA) [6].

8. XMPP Registrar Considerations

8.1 Protocol Namespaces

Until this specification advances to a status of Draft, the associated namespace for its stream feature shall be "urn:xmpp:tmp:seq". Upon advancement of this specification, the XMPP Registrar [7] shall issue a permanent namespace in accordance with the process defined in Section 4 of XMPP Registrar Function [8]; the requested namespace is "urn:xmpp:seq", which is thought to be unique per the XMPP Registrar's requirements.

9. XML Schemas

<?xml version='1.0' encoding='UTF-8'?>

<xs:schema
    xmlns:xs='http://www.w3.org/2001/XMLSchema'
    targetNamespace='urn:xmpp:tmp:seq'
    xmlns='urn:xmpp:tmp:seq'
    elementFormDefault='qualified'>

  <xs:element name='seq'>
    <xs:complexType>
      <xs:choice minOccurs='0'>
        <xs:element name='optional' type='empty' minOccurs='0'/>
        <xs:element name='required' type='empty' minOccurs='0'/>
      </xs:choice>
      <xs:attribute name='num' type='xs:string' use='optional'/>
    </xs:complexType>
  </xs:element>

  <xs:simpleType name='empty'>
    <xs:restriction base='xs:string'>
      <xs:enumeration value=''/>
    </xs:restriction>
  </xs:simpleType>

</xs:schema>

10. Acknowledgements

Thanks to Dave Cridland, Richard Dobson, Fabio Forno, Alexander Gnauck, Juha Hartikainen, Joe Hildebrand, Justin Karneges, and Pedro Melo for their comments.

Appendices

Appendix A: Document Information

Series: XEP
Number: 0237
Publisher: XMPP Standards Foundation
Status: Experimental
Type: Standards Track
Version: 0.4
Last Updated: 2008-09-17
Approving Body: XMPP Council
Dependencies: XMPP Core, XMPP IM
Supersedes: None
Superseded By: None
Short Name: NOT_YET_ASSIGNED
Source Control: HTML  RSS

Appendix B: Author Information

Peter Saint-Andre

JabberID: stpeter@jabber.org
URI: https://stpeter.im/

Appendix C: Legal Notices

Copyright

This XMPP Extension Protocol is copyright (c) 1999 - 2009 by the XMPP Standards Foundation (XSF).

Permissions

Permission is hereby granted, free of charge, to any person obtaining a copy of this specification (the "Specification"), to make use of the Specification without restriction, including without limitation the rights to implement the Specification in a software program, deploy the Specification in a network service, and copy, modify, merge, publish, translate, distribute, sublicense, or sell copies of the Specification, and to permit persons to whom the Specification is furnished to do so, subject to the condition that the foregoing copyright notice and this permission notice shall be included in all copies or substantial portions of the Specification. Unless separate permission is granted, modified works that are redistributed shall not contain misleading information regarding the authors, title, number, or publisher of the Specification, and shall not claim endorsement of the modified works by the authors, any organization or project to which the authors belong, or the XMPP Standards Foundation.

Disclaimer of Warranty

## NOTE WELL: This Specification is provided on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. In no event shall the XMPP Standards Foundation or the authors of this Specification be liable for any claim, damages, or other liability, whether in an action of contract, tort, or otherwise, arising from, out of, or in connection with the Specification or the implementation, deployment, or other use of the Specification. ##

Limitation of Liability

In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall the XMPP Standards Foundation or any author of this Specification be liable for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising out of the use or inability to use the Specification (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if the XMPP Standards Foundation or such author has been advised of the possibility of such damages.

IPR Conformance

This XMPP Extension Protocol has been contributed in full conformance with the XSF's Intellectual Property Rights Policy (a copy of which may be found at <http://xmpp.org/extensions/ipr-policy.shtml> or obtained by writing to XSF, P.O. Box 1641, Denver, CO 80201 USA).

Appendix D: Relation to XMPP

The Extensible Messaging and Presence Protocol (XMPP) is defined in the XMPP Core (RFC 3920) and XMPP IM (RFC 3921) specifications contributed by the XMPP Standards Foundation to the Internet Standards Process, which is managed by the Internet Engineering Task Force in accordance with RFC 2026. Any protocol defined in this document has been developed outside the Internet Standards Process and is to be understood as an extension to XMPP rather than as an evolution, development, or modification of XMPP itself.

Appendix E: Discussion Venue

The primary venue for discussion of XMPP Extension Protocols is the <standards@xmpp.org> discussion list.

Discussion on other xmpp.org discussion lists might also be appropriate; see <http://xmpp.org/about/discuss.shtml> for a complete list.

Errata may be sent to <editor@xmpp.org>.

Appendix F: Requirements Conformance

The following requirements keywords as used in this document are to be interpreted as described in RFC 2119: "MUST", "SHALL", "REQUIRED"; "MUST NOT", "SHALL NOT"; "SHOULD", "RECOMMENDED"; "SHOULD NOT", "NOT RECOMMENDED"; "MAY", "OPTIONAL".

Appendix G: Notes

1. RFC 3921: Extensible Messaging and Presence Protocol (XMPP): Instant Messaging and Presence <http://tools.ietf.org/html/rfc3921>.

2. XEP-0030: Service Discovery <http://xmpp.org/extensions/xep-0030.html>.

3. XEP-0059: Result Set Management <http://xmpp.org/extensions/xep-0059.html>.

4. XEP-0230: Service Discovery Notifications <http://xmpp.org/extensions/xep-0230.html>.

5. XEP-0115: Entity Capabilities <http://xmpp.org/extensions/xep-0115.html>.

6. The Internet Assigned Numbers Authority (IANA) is the central coordinator for the assignment of unique parameter values for Internet protocols, such as port numbers and URI schemes. For further information, see <http://www.iana.org/>.

7. The XMPP Registrar maintains a list of reserved protocol namespaces as well as registries of parameters used in the context of XMPP extension protocols approved by the XMPP Standards Foundation. For further information, see <http://xmpp.org/registrar/>.

8. XEP-0053: XMPP Registrar Function <http://xmpp.org/extensions/xep-0053.html>.

Appendix H: Revision History

Version 0.4 (2008-09-17)

Defined new namespace and generalized to handle service discovery and other use cases in addition to rosters.

(psa)

Version 0.3 (2008-04-21)

Defined protocol solely in terms of full rosters and roster pushes (no more roster diffs); added implementation notes; clarified server behavior if cached version is unavailable.

(psa)

Version 0.2 (2008-03-06)

Renamed to data sequencing; clarified server behavior.

(psa)

Version 0.1 (2008-03-05)

Initial published version; per Council consensus, removed optionality regarding semantics of the version attribute.

(psa)

Version 0.0.3 (2008-03-05)

Corrected semantics of version attribute (should be a strictly increasing sequence number but may be any unique identifier).

(psa)

Version 0.0.2 (2008-03-04)

Clarified description of roster diff; added diff attribute and specified its use in roster results; specified use of version attribute in roster pushes.

(psa)

Version 0.0.1 (2008-03-04)

First draft.

(psa)

END