JEP-0059: Result Set Management

This document defines an XMPP protocol extension to enable entities to page through and otherwise manage the receipt of large result sets.

WARNING: This Standards-Track JEP is Experimental. Publication as a Jabber Enhancement Proposal does not imply approval of this proposal by the Jabber Software Foundation. Implementation of the protocol described herein is encouraged in exploratory implementations, but production systems should not deploy implementations of this protocol until it advances to a status of Draft.

JEP Information

Author Information

Legal Notice

Discussion Venue

Relation to XMPP

The Extensible Messaging and Presence Protocol (XMPP) is defined in the XMPP Core (RFC 3920) and XMPP IM (RFC 3921) specifications contributed by the Jabber Software Foundation to the Internet Standards Process, which is managed by the Internet Engineering Task Force in accordance with RFC 2026. Any protocol defined in this JEP has been developed outside the Internet Standards Process and is to be understood as an extension to XMPP rather than as an evolution, development, or modification of XMPP itself.

Conformance Terms

The following keywords as used in this document are to be interpreted as described in RFC 2119: "MUST", "SHALL", "REQUIRED"; "MUST NOT", "SHALL NOT"; "SHOULD", "RECOMMENDED"; "SHOULD NOT", "NOT RECOMMENDED"; "MAY", "OPTIONAL".

1. Introduction
2. Use Cases
3. Examples
4. Determining Support
5. Security Considerations
6. IANA Considerations
7. Jabber Registrar Considerations
8. XML Schema
9. Acknowledgements
Notes
Revision History

1. Introduction

In Jabber Search [1], Service Discovery [2], Publish-Subscribe [3], Message Archiving [4], and perhaps other future XMPP extensions, it is possible to receive large result sets in response to information requests (e.g., a user directory search on a common first name or a service discovery items request sent to a Multi-User Chat [5] service). Thus it would be helpful to define an XMPP protocol extension that enables the following functionality:

2. Use Cases

2.1 Getting the Item Count

In order to get the item count of a result set without retrieving the items themselves, the requesting entity specifies a request type of "count":

Example 1. Requesting the Item Count

<iq type='get' from='stpeter@jabber.org/roundabout' to='users.jabber.org' id='count1'>
  <query xmlns='jabber:iq:search'>
    <first>Peter</first>
    <count xmlns='http://jabber.org/protocol/rsm'/>
  </query>
</iq>

The responding entity then returns the item count, which MAY be approximate rather than precise (since determining the exact number of items may be resource-intensive):

Example 2. Returning the Item Count

<iq type='result' from='users.jabber.org' to='stpeter@jabber.org/roundabout' id='count1'>
  <query xmlns='jabber:iq:search'>
    <count xmlns='http://jabber.org/protocol/rsm'>800</count>
  </query>
</iq>

If no items match, the responding entity MUST return a response that adheres to the definition of the wrapper protocol (e.g., "jabber:iq:search", "http://jabber.org/protocol/disco#items", or "http://jabber.org/protocol/pubsub"). For both JEP-0055 and JEP-0030, that means the responding entity shall return an empty <query/> element; for JEP-0060, that means the responding entity shall return an empty <pubsub/> element.

2.2 Limiting the Size of a Result Set

In order to limit the number of items to be returned in a result set, the requesting entity specifies a request type of "set" and the maximum size of the desired result set (via XML character data of the <max/> element):

Example 3. Requesting a Limit to the Result Set

<iq type='get' from='stpeter@jabber.org/roundabout' to='users.jabber.org' id='limit1'>
  <query xmlns='jabber:iq:search'>
    <first>Peter</first>
    <set xmlns='http://jabber.org/protocol/rsm'>
      <max>10</max>
    </set>
  </query>
</iq>

The responding entity then returns a result set limited to the requested size:

Example 4. Returning a Limited Result Set

<iq type='result' from='users.jabber.org' to='stpeter@jabber.org/roundabout' id='limit1'>
  <query xmlns='jabber:iq:search'>
    <item jid='stpeter@jabber.org'>
      <first>Peter</first>
      <last>Saint-Andre</last>
      <nick>stpeter</nick>
    </item>
    .
    [8 more items]
    .
    <item jid='peterpan@neverland.lit'>
      <first>Peter</first>
      <last>Pan</last>
      <nick>petie</nick>
    </item>
  </query>
</iq>

2.3 Paging Through a Result Set

Note: The set of items that match a query MAY change over time, even during the time that a requesting entity pages through a result set (e.g., a set of chatrooms, since rooms can be created and destroyed at any time). The paging protocol outlined in this section is designed to make the following features possible: [6]

Each result page is up-to-date at the time it is sent (not just at the time the first page was sent).
No items will be omitted from pages not yet sent (even if, after earlier pages were sent, some of the items they contained were removed from the set).
Duplicate items are never received.
Responding entity maintains no state (or a single minimal state for all requestors containing the positions of all recently deleted items).
Rapid calculation of which items should appear on a requested page by responding entity (even for large result sets).

The request for the first page is the same as when Limiting the Size of a Result Set:

Example 5. Requesting the First Page of a Result Set

<iq type='get' from='stpeter@jabber.org/roundabout' to='users.jabber.org' id='page1'>
  <query xmlns='jabber:iq:search'>
    <first>Peter</first>
    <set xmlns='http://jabber.org/protocol/rsm'>
      <max>10</max>
    </set>
  </query>
</iq>

Responding entity support for paging through a result set is optional. If it does support paging (not just Limiting the Size of a Result Set), then in each result set it returns, the responding entity MUST include a <last/> element that specifies a unique ID (UID) for the last item in the page. The responding entity can generate this UID in any way, as long as the UID is unique in the context of all possible members of the full result set. The UID MAY be based on part of the content of the last item, as shown below, or on an internal table index. Another possible method is to serialize the XML of the last item in the result set and then hash it to generate the UID. Note: The requesting entity MUST treat all UIDs as opaque.

Example 6. Returning the First Page of a Result Set

<iq type='result' from='users.jabber.org' to='stpeter@jabber.org/roundabout' id='page1'>
  <query xmlns='jabber:iq:search'>
    <item jid='stpeter@jabber.org'>
      <first>Peter</first>
      <last>Saint-Andre</last>
      <nick>stpeter</nick>
    </item>
    .
    [8 more items]
    .
    <item jid='peterpan@neverland.lit'>
      <first>Peter</first>
      <last>Pan</last>
      <nick>petie</nick>
    </item>
    <last xmlns='http://jabber.org/protocol/rsm'>peterpan@neverland.lit</last>
    <count xmlns='http://jabber.org/protocol/rsm'>800</count>
  </query>
</iq>

As shown in the previous example, the responding entity MAY include the item count with the first result set (or any future result set, although returning the count with the first result set is probably most helpful).

The requesting entity can then ask for the next page in the result set by including in its request the UID of the last item from the previous page (encapsulated in an <after/> element), and the maximum number of items to return. Note: if no <after/> element is specified then the UID defaults to "before the first item in the result set".

Example 7. Requesting the Second Page of a Result Set

<iq type='get' from='stpeter@jabber.org/roundabout' to='users.jabber.org' id='page2'>
  <query xmlns='jabber:iq:search'>
    <first>Peter</first>
    <set xmlns='http://jabber.org/protocol/rsm'>
      <max>10</max>
      <after>peterpan@neverland.lit</after>
    </set>
  </query>
</iq>

The responding entity MUST begin the next page in the result set with the item that follows the item that the requesting entity indicated it has already received:

Example 8. Returning the Second Page of a Result Set

<iq type='result' from='users.jabber.org' to='stpeter@jabber.org/roundabout' id='page2'>
  <query xmlns='jabber:iq:search'>
    <item jid='peter@pixyland.org'>
      <first>Peter</first>
      <last>Pan</last>
      <nick>peterpan</nick>
    </item>
    .
    [8 more items]
    .
    <item jid='peter@rabbit.lit'>
      <first>Peter</first>
      <last>Rabbit</last>
      <nick>prabbit</nick>
    </item>
    <last xmlns='http://jabber.org/protocol/rsm'>'peter@rabbit.lit</last>
  </query>
</iq>

If the result set contains the last items that meet the provided criteria, the responding entitity SHOULD include an empty <end/> element. In this case, it MAY omit the <last/> element:

Example 9. Returning the Last Page of a Result Set

<iq type='result' from='users.jabber.org' to='stpeter@jabber.org/roundabout' id='lastpage'>
  <query xmlns='jabber:iq:search'>
    [the last items]
    <end xmlns='http://jabber.org/protocol/rsm'/>
  </query>
</iq>

The responding entity MUST reply with an 'item-not-found' error if all the following circumstances apply:

The item specified by the requesting entity via the UID in the <after/> element no longer exists (it was deleted after the previous page was sent)
The UID itself cannot be used to derive directly the next item within the set (e.g. the alphabetical or numerical order of the UIDs do not specify the order of the items)
The responding entity does not remember the position of the deleted item within the full list. (Even if the responding entity bothers to remember the position of each deleted item, it will typically be necessary to expire that 'state' after an implementation-specific period of time)

Example 10. Returning a Page-not-Found Error

<iq type='error' from='users.jabber.org' to='stpeter@jabber.org/roundabout' id='page2'>
  <query xmlns='jabber:iq:search'>
    <first>Peter</first>
    <set xmlns='http://jabber.org/protocol/rsm'>
      <max>10</max>
      <after>peterpan@neverland.lit</after>
    </set>
  </query>
  <error type='cancel'>
    <item-not-found xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/>
  </error>
</iq>

2.4 Requesting a Result Set in Reverse Order

If the requesting entity wants to receive the items of a results set in reverse order then it MUST include an empty <reverse/> element in every page request.

Example 11. Requesting a Page from a Reverse Order Result Set

<iq type='get' from='stpeter@jabber.org/roundabout' to='users.jabber.org' id='reverse2'>
  <query xmlns='jabber:iq:search'>
    <first>Peter</first>
    <set xmlns='http://jabber.org/protocol/rsm'>
      <reverse/>
      <max>10</max>
      <after>stpeter@heaven.com</after>
    </set>
  </query>
</iq>

3. Examples

The foregoing examples show the use of result set management in the context of Jabber Search.. Therefore in the following examples we show the use of result set management in the context of Service Discovery. A future version of this document may also include examples from Publish-Subscribe and Message Archiving and other XMPP protocol extensions.

<iq type='get' from='stpeter@jabber.org/roundabout' to='conference.jabber.org' id='ex1'>
  <query xmlns='http://jabber.org/protocol/disco#items'>
    <count xmlns='http://jabber.org/protocol/rsm'/>
  </query>
</iq>

<iq type='result' from='conference.jabber.org' to='stpeter@jabber.org/roundabout' id='ex1'>
  <query xmlns='http://jabber.org/protocol/disco#items'>
    <count xmlns='http://jabber.org/protocol/rsm'>150</count>
  </query>
</iq>

<iq type='get' from='stpeter@jabber.org/roundabout' to='conference.jabber.org' id='ex2'>
  <query xmlns='http://jabber.org/protocol/disco#items'>
    <set xmlns='http://jabber.org/protocol/rsm'>
      <max>20</max>
    </set>
  </query>
</iq>

<iq type='result' from='conference.jabber.org' to='stpeter@jabber.org/roundabout' id='ex2'>
  <query xmlns='http://jabber.org/protocol/disco#items'>
    <item jid='12@conference.jabber.org'/>
    <item jid='adium@conference.jabber.org'/>
    <item jid='airhitch@conference.jabber.org'/>
    <item jid='alphaville@conference.jabber.org'/>
    <item jid='apache@conference.jabber.org'/>
    <item jid='argia@conference.jabber.org'/>
    <item jid='armagetron@conference.jabber.org'/>
    <item jid='atticroom123@conference.jabber.org'/>
    <item jid='banquise@conference.jabber.org'/>
    <item jid='bar_paradise@conference.jabber.org'/>
    <item jid='beer@conference.jabber.org'/>
    <item jid='blondie@conference.jabber.org'/>
    <item jid='bpnops@conference.jabber.org'/>
    <item jid='brasileiros@conference.jabber.org'/>
    <item jid='bulgaria@conference.jabber.org'/>
    <item jid='cantinalivre@conference.jabber.org'/>
    <item jid='casablanca@conference.jabber.org'/>
    <item jid='chinortpcrew@conference.jabber.org'/>
    <item jid='coffeetalk@conference.jabber.org'/>
    <item jid='council@conference.jabber.org'/>
    <last xmlns='http://jabber.org/protocol/rsm'>4da91d4b330112f683dddaebf93180b1bd25e95f</last>
  </query>
</iq>

<iq type='get' from='stpeter@jabber.org/roundabout' to='conference.jabber.org' id='ex3'>
  <query xmlns='http://jabber.org/protocol/disco#items'>
    <set xmlns='http://jabber.org/protocol/rsm'>
      <max>20</max>
      <after>4da91d4b330112f683dddaebf93180b1bd25e95f</after>
    </set>
  </query>
</iq>

4. Determining Support

In order for a requesting entity to determine if a responding entity supports result set management, it SHOULD send a Service Discovery information request to the responding entity:

Example 17. Requesting entity queries responding entity regarding protocol support

<iq from='stpeter@jabber.org/roundabout'
    to='conference.jabber.org'
    id='disco1'>
  <query xmlns='http://jabber.org/protocol/disco#info'/>
</iq>

<iq from='conference.jabber.org'
    to='stpeter@jabber.org/roundabout'
    id='disco1'>
  <query xmlns='http://jabber.org/protocol/disco#info'>
    ...
    <feature var='http://jabber.org/protocol/rsm'/>
    ...
  </query>
</iq>

A requesting entity SHOULD NOT include result set management extensions if it does not have positive knowledge that the responding entity supports the protocol defined herein. If the responding entity does not understand result set management, it MUST ignore the extension.

Note: Even if a responding entity understands the result set management protocol, its support for result set management in the context of any given using protocol is OPTIONAL (e.g., an implementation could support it in the context of the 'jabber:iq:search' namespace but not in the context of the 'http://jabber.org/protocol/disco#items' namespace). Currently the only way for a requesting entity to determine if a responding entity supports result set management in the context of a given using protocol is to include result set management extensions in its request. If the responding entity does not include result set management extensions in its response, then the requesting entity SHOULD NOT include such extensions in future requests wrapped by the using protocol namespace.

5. Security Considerations

Security considerations are the responsibility of the using ("wrapper") protocol, such as JEP-0030 for the 'http://jabber.org/protocol/disco#items' namespace and JEP-0055 for the 'jabber:iq:search' namespace.

6. IANA Considerations

This JEP requires no interaction with the Internet Assigned Numbers Authority (IANA) [7].

7. Jabber Registrar Considerations

7.1 Protocol Namespaces

The Jabber Registrar [8] shall include 'http://jabber.org/protocol/rsm' in its registry of protocol namespaces.

8. XML Schema

<?xml version='1.0' encoding='UTF-8'?>

<xs:schema
    xmlns:xs='http://www.w3.org/2001/XMLSchema'
    targetNamespace='http://jabber.org/protocol/rsm'
    xmlns='http://jabber.org/protocol/rsm'
    elementFormDefault='qualified'>

  <xs:element name='count' type='xs:string'/>

  <xs:element name='end' type='empty'/>

  <xs:element name='last' type='xs:string'/>

  <xs:element name='set'>
    <xs:complexType>
      <xs:sequence>
        <xs:element name='max' type='xs:int' minOccurs='1' maxOccurs='1'/>
        <xs:element name='after' type='xs:string' minOccurs='0' maxOccurs='1'/>
        <xs:element name='reverse' type='empty' minOccurs='0' maxOccurs='1'/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>

  <xs:simpleType name='empty'>
    <xs:restriction base='xs:string'>
      <xs:enumeration value=''/>
    </xs:restriction>
  </xs:simpleType>

</xs:schema>

9. Acknowledgements

Notes

1. JEP-0055: Jabber Search <http://www.jabber.org/jeps/jep-0055.html>.

2. JEP-0030: Service Discovery <http://www.jabber.org/jeps/jep-0030.html>.

3. JEP-0060: Publish-Subscribe <http://www.jabber.org/jeps/jep-0060.html>.

4. JEP-0136: Message Archiving <http://www.jabber.org/jeps/jep-0136.html>.

5. JEP-0045: Multi-User Chat <http://www.jabber.org/jeps/jep-0045.html>.

6. If a responding entity implements dynamic result sets then receiving entities should be aware that the complete result set received may not correspond to the result set as it existed at any one point in time.

7. The Internet Assigned Numbers Authority (IANA) is the central coordinator for the assignment of unique parameter values for Internet protocols, such as port numbers and URI schemes. For further information, see <http://www.iana.org/>.

8. The Jabber Registrar maintains a list of reserved Jabber protocol namespaces as well as registries of parameters used in the context of protocols approved by the Jabber Software Foundation. For further information, see <http://www.jabber.org/registrar/>.

Revision History

Version 0.9 (2006-08-23)

Eliminated static result sets, justified expanded and clarified dynamic result sets, added page-not-found error, described when minimal state is necessary, added reverse order sets.

(ip)

Version 0.8 (2006-08-22)

Added optional method for handling dynamic result sets.

(psa)

Version 0.7 (2006-08-10)

Updated implementation note to clarify handling of result sets (static vs. dynamic).

(psa)

Version 0.6 (2006-07-12)

Updated implementation note to clarify handling of result sets (static vs. dynamic).

(psa)

Version 0.5 (2006-05-02)

Clarified error handling, determination of support in the context of using protocols, and security considerations.

(psa)

Version 0.4 (2006-04-24)

Specified that an item count may be approximate; specified that an item count may be returned with a page of results.

(psa)

Version 0.3 (2006-04-21)

Added <end/> element to specify last result set; added service discovery information; added more examples.

(psa/vm)

Version 0.2 (2005-12-22)

Revived and renamed the JEP; modified syntax; added use case for getting number of items; defined XML schema.

(psa/vm)

Version 0.1 (2002-11-12)

Initial version.

(jls)

JEP-0059: Result Set Management

JEP Information

Author Information

Jean-Louis Seguineau

Peter Saint-Andre

Valerie Mercier

Ian Paterson

Legal Notice

Discussion Venue

Relation to XMPP

Conformance Terms

Table of Contents

1. Introduction

2. Use Cases

2.1 Getting the Item Count

2.2 Limiting the Size of a Result Set

2.3 Paging Through a Result Set

2.4 Requesting a Result Set in Reverse Order

3. Examples

4. Determining Support

5. Security Considerations

6. IANA Considerations

7. Jabber Registrar Considerations

7.1 Protocol Namespaces

8. XML Schema

9. Acknowledgements

Notes

Revision History

Version 0.9 (2006-08-23)

Version 0.8 (2006-08-22)

Version 0.7 (2006-08-10)

Version 0.6 (2006-07-12)

Version 0.5 (2006-05-02)

Version 0.4 (2006-04-24)

Version 0.3 (2006-04-21)

Version 0.2 (2005-12-22)

Version 0.1 (2002-11-12)