XEP-xxxx: Jingle Remote Control

1. Introduction¶

Thanks to Jingle (XEP-0166) [1], Jingle RTP Sessions (XEP-0167) [2], and associated XEPs, it is now possible to have video calls between devices through XMPP. Several XMPP clients also support desktop sharing, i.e., using screen capture of the current desktop session or of a specific application window instead of the usual webcam stream.

With this in place, we are one step away from having a remote desktop feature: if one "controlling" peer sends data from input peripherals to another "controlled" device and this other device uses them as input, we have a remote control system. This specification proposes ways to transmit input data to achieve remote control, with or without attached video call sessions, so that it can be used to remotely control a whole desktop or specific applications, to use remotely or simulate input devices for another device that may lack them (e.g., single-board computers or IoT devices), and it establishes a framework that can be extended to share data (e.g., clipboard content).

2. Requirements¶

The design goals of this XEP are:

• Allow remote control of desktops, single applications, or any kind of device.
• Work with any input device, with special focus on keyboards, mice, and touch devices.
• Work with or without Audio/Video calls as done by Jingle RTP Sessions (XEP-0167) [2] applications.
• Work between devices of different screen sizes, or even if a device has no screen at all.
• Be as close to existing data formats as possible to make implementation easy.
• Allow bidirectional communication for feature such as haptic feedback, LED status, data polling, clipboard transmission, etc.

3. Glossary¶

• Controlling device: A device used for remotely accessing and controlling another device, typically sending its input peripherals (keyboard, mouse, etc.) to the controlled device.
• Controlling entity: The person or program manipulating the controlling device.
• Controlled device: The target system or device being accessed and managed through a controlling device via remote control functionality; it receives and processes the input data from the controlling device as if it were entered locally.
• Controlled application: The software being controlled on the controlled device. It can be a whole desktop screen, a single monitor, a single desktop application, a non graphic application, or anything making sense in this context.

4. Overview¶

Remote control works by defining a new Jingle application, with the namespace 'urn:xmpp:jingle:apps:remote-control:0'. Once permission is granted, a streaming transport is established between "controlling" and "controlled" device. Input peripheral data are sent using CBOR serialization format (RFC 8949 [3]). Controlled device receive those data, decode them, and use them to simulate inputs.

Audio or video streams can optionally be used in the same session to transmit the audio and/or video content of the controlled application. Additionally, Audio/Video streams can also be used for communication between the controlling entity and the controlled device user; this can be useful for explaining what is currently being done, for teaching, and so on.

5. Jingle Conformance¶

In accordance with Section 12 of XEP-0166, this document specifies the following information related to the Remote Control application type:

1. The application format negotiation process is defined in the Negotiating a Remote Control Session section of this document.

2. The semantics of the <description/> element are defined in the Application Format section of this document.

3. There is no mapping of Remote Control semantics to the Session Description Protocol.

4. A Remote Control session MUST use a streaming transport method, not a datagram transport method.

   Even if in theory any streaming transport could be used, Signaling WebRTC datachannels in Jingle (XEP-0343) [4] SHOULD be used as the preferred transport method, given that Remote Control is often utilized in conjunction with Jingle RTP Sessions (XEP-0167) [2] and should be made accessible for web clients.

5. Transport components are not used in Remote Control.

6. Description of content sending and receiving is describe in the Exchanging Input Data section of this document.

6. Negotiating a Remote Control Session¶

A Remote Control session is requested when one of the Jingle <content/> elements has a child <description/> element qualified by the 'urn:xmpp:jingle:apps:remote-control:0' namespace. This content MAY be the only content of the session, or it MAY be associated with Jingle RTP Sessions (XEP-0167) [2] media contents.

If a Remote Control session content is associated with Jingle RTP Sessions (XEP-0167) [2] and if WebRTC is used as transport (via Jingle ICE-UDP Transport Method (XEP-0176) [5] and Signaling WebRTC datachannels in Jingle (XEP-0343) [4]), the same WebRTC session MUST be used for all contents.

A Remote Control content MAY be added to an existing Jingle RTP Sessions (XEP-0167) [2] session by using a Jingle content-add as specified in XEP-0166's "content-add" section. If this new content appears and uses WebRTC for transport, and a WebRTC session is already used for Jingle RTP Sessions (XEP-0167) [2], the same WebRTC session MUST be used and a data-channel MUST be created inside, using the content's name as the Data Channel label. Permission SHOULD be requested from the user to allow remote control, before accepting the <content/>.

As a rule of thumb, Remote Control sessions, in particular in the remote desktop use case, are generally associated with a single unidirectional video stream (with the <content/> 'senders' attribute set to "responder"). This stream MUST diffuse the content of the controlled application. An additional audio unidirectional stream may also be used to transmit the sounds emitted by the controlled application. If those streams are bidirectional (i.e., with the 'senders' attribute not set, or set to "both"), the content emitted by the controlling device SHOULD be similar to a normal A/V call, i.e., webcam and/or desktop sharing with sound from the microphone. This is notably useful if the Remote Control feature is used for teaching, or explaining what is done on the controlled app. Additional video and/or audio streams may be used to have communication with the controlled device user, resulting in up to 2 video and 2 audio streams. In this case, the first video and audio streams MUST be used for controlled application content, and optionally controlling entity video, while the last 2 MUST have the 'senders' attribute set to "responder" and be used for controlled device user feedback.

The following table summarizes the video stream usage in Remote Control sessions.

<iq id="IQ_1" type="set" from="juliet@capulet.lit/balcony" to="romeo@montague.lit/orchard">
  <jingle xmlns="urn:xmpp:jingle:1" sid="42d6beee-b51d-4a4b-8333-405051a33a10" action="session-initiate" initiator="juliet@capulet.lit/balcony">
    <content creator="initiator" name="1" senders="initiator">
      <description xmlns="urn:xmpp:jingle:apps:remote-control:0">
        <device type="keyboard"/>
        <device type="mouse"/>
        <device type="touch"/>
        <device type="wheel"/>
      </description>
      <transport xmlns="urn:xmpp:jingle:transports:webrtc-datachannel:1" sctp-port="5000" max-message-size="1073741823">
        <!-- XEP-0343 payload -->
      </transport>
    </content>
    <content creator="initiator" name="0" senders="responder">
      <description xmlns="urn:xmpp:jingle:apps:rtp:1" media="video">
        <!-- XEP-0167 payload -->
      </description>
      <transport xmlns="urn:xmpp:jingle:transports:ice-udp:1" ufrag="f0a1620c" pwd="6fb807d5f37ca6f7f248dae57fe3da02">
        <!-- XEP-0176 payload -->
      </transport>
    </content>
  </jingle>
</iq>

<iq to="juliet@capulet.lit/balcony" from="romeo@montague.lit/orchard" id="IQ_1" type="result"/>

<iq id="IQ_2" type="set" from="romeo@montague.lit" to="juliet@capulet.lit/balcony">
  <jingle xmlns="urn:xmpp:jingle:1" sid="42d6beee-b51d-4a4b-8333-405051a33a10" action="session-accept" responder="romeo@montague.lit">
    <content creator="initiator" name="1">
      <description xmlns="urn:xmpp:jingle:apps:remote-control:0">
        <device type="keyboard"/>
        <device type="mouse"/>
        <device type="wheel"/>
      </description>
      <transport xmlns="urn:xmpp:jingle:transports:webrtc-datachannel:1" sctp-port="5000" max-message-size="1073741823">
        <!-- XEP-0343 payload -->
      </transport>
    </content>
    <content creator="initiator" name="0">
      <description xmlns="urn:xmpp:jingle:apps:rtp:1" media="video">
        <!-- XEP-0167 payload -->
      </description>
      <transport xmlns="urn:xmpp:jingle:transports:ice-udp:1" ufrag="UozLzExwo0c2lw2yfCm2CSX0RgPeocCT" pwd="RDMWbP4Rh/nFU54Q5UfuFR27oUQ1oALR">
        <!-- XEP-0176 payload -->
      </transport>
    </content>
  </jingle>
</iq>

<iq to="romeo@montague.lit/orchard" from="juliet@capulet.lit/balcony" id="IQ_2" type="result"/>

7. Application Format¶

The <description/> element contains 0, 1, or more <device> child elements, indicating which devices the controlling entity wishes to access.

When the responder is sending back a Remote Control <description/>, it MUST add a <description/> element for each device that has been accepted (i.e., that the controlling device may control). The responder MUST NOT add a <device> element for a specific device if authorization has not been granted to control it. That means that the initiator's list of <device> elements MAY differ from the one of the responder (if the responder is not willing to let the controller control all devices, or if a specific device is not present on the controlled device).

If no <device> element is specified, the session is a simple screen-sharing session. Note that in this case, at least one Jingle RTP Sessions (XEP-0167) [2] <content/> element MUST be present; otherwise, the Jingle session MUST be terminated.

<device> elements may have child elements specific to this device type and how it should be managed. Those elements may be specified by future specifications extending this one.

Note: the Remote Control content may be unidirectional or bidirectional. All devices described here are unidirectional, meaning that the 'senders' attribute of the Remote Control <content/> SHOULD be set to "initiator". However, future specifications may add devices that communicate in both directions (e.g., for haptic feedback, clipboard transmission, event pulling, LED feedback, etc.). In this case, the 'senders' attribute will be unset or set to "both".

A <device> element has a mandatory 'type' attribute, whose value is the kind of device that is requested by the controlling entity.

The current specification defines 4 <device> elements, described below.

8. Exchanging Input Data¶

Events data are transmitted on the wire using CBOR. The reason is that input data must be sent as fast as possible in an efficient way, and CBOR is a standard, extensible, and efficient binary format well suited for this task. Furthermore, it is easily mapped to JSON, which makes it ideal to work with Web APIs and other APIs using JSON. Each event is encoded as a map, resulting as a stream of CBOR map objects on the wire.

To make understanding of the format easier and this document more readable, JSON is used in examples of this specification, but data needs to be ultimately serialized to/from CBOR to go on the wire.

The input event data format here is inspired by those found in Web APIs. This makes them straightforward to use in web-based applications, but also easy to adapt to underlying platforms as the Web APIs are abstracted and thought to work anywhere.

The general format of an input event looks like this:

    {
        "type": "keyboard",
        "device_id": "device_123",
        "timestamp": 1712678325.0,
        "key": "A",
    }
  

The first three keys ("type", "device_id", and "timestamp") are common to all device events. Other keys are device-specific.

The common keys are specified as follows:

9. Device Events¶

Below is the description of the events for the 4 types of devices specified here. Future specifications may add new types of events.

Note that a controlling device doesn't need to have the device in question to send those events: for instance, a touch device doesn't have to send touch events, and can instead send mouse events to simulate a mouse, or an automation mechanism can simulate input devices.

10. Extending this Specification¶

It is expected that future specifications will extend this one. To make an extension, we recommand to follows those rules:

• Use "Remote Control" in your title to easily find extensions. "Remote Control: Clipboard" or "Remote Control: Gamepad" are good extension titles.
• Describe the <device> element and if necessary any device-specific child.
• Specify the new events in the same way as here: consistency make it easier to follow different specifications.
• Indicate it the event is unidirectional (only send from controlling device to controlled device) or bidirectional (controlled device can send data too).
• For each key of the new event, give a short description, the data type, and indicate wether the data is OPTIONAL or REQUIRED.
• The namespace 'urn:xmpp:jingle:apps:remote-control:xxx:y' should be used for extension of this specification, where xxx is a short name, usually the name of the new managed device, and y is the usual specification version number.

11. Business Rules¶

A controlled device MAY send data back to the controlling device, for instance, to send force haptic feedback such as vibration or force feedback. All devices in this specification are unidirectional, but future specifications may add bidirectional ones. If any bidirectional device is requested, the 'senders' attribute of the Remote Control content MUST be unset or set to "both".

It is up to the controlling entity to optimize or not the data sent with whatever appropriate algorithm. For instance, if a mouse is moved twice without a button or keyboard being pressed in-between, the controlling entity could send a single mouse event that is a combination of the two moves.

If a Remote Control <content/> is used with Jingle RTP Sessions (XEP-0167) [2] <content/>, the session is a Remote Control session, and not a video call; it SHOULD be presented as a Remote Control session to the user, and the media streams are used as feedback and/or support as described above.

12. Discovering Support¶

If a client supports the protocol specified in this XEP, it MUST advertise it by including the "urn:xmpp:jingle:apps:remote-control:0" discovery feature in response to a Service Discovery (XEP-0030) [12] information request.

<iq type='get'
    from='juliet@example.org/balcony'
    to='romeo@example.org/orchard'
    id='disco1'>
  <query xmlns='http://jabber.org/protocol/disco#info'/>
</iq>

<iq type='result'
    from='romeo@example.org/orchard'
    to='juliet@example.org/balcony'
    id='disco1'>
  <query xmlns='http://jabber.org/protocol/disco#info'>
    ...
    <feature var='urn:xmpp:jingle:apps:remote-control:0'/>
    ...
  </query>
</iq>

13. Security Considerations¶

Obviously, allowing an entity to remotely control a device is giving a high and dangerous level of access, similar to having the person using the controlling device sitting in front of this computer.

Before starting the remote control session, the client SHOULD ask for permission from the user of the controlled device in a clear and understandable way, explaining in non-technical terms that the other person will fully control the inputs of the device. A client MAY allow for automatic permission (e.g., to control an IoT device or a work computer from home by well-known entities), but this must be done clearly or stated somewhere, and the controlling entity SHOULD be duly checked.

In addition, a controlled device SHOULD display an obvious, clear, and always visible indicator that the inputs are currently being remotely controlled, with an obvious and easily accessible button to stop remote control immediately.

14. IANA Considerations¶

TODO

15. XMPP Registrar Considerations¶

TODO

16. XML Schema¶

<?xml version='1.0' encoding='UTF-8'?>

<xs:schema
    xmlns:xs='http://www.w3.org/2001/XMLSchema'
    targetNamespace='urn:xmpp:jingle:apps:remote-control:0'
    xmlns='urn:xmpp:jingle:apps:remote-control:0'
    elementFormDefault='qualified'>

  <xs:element name='description'>
    <xs:complexType>
      <xs:all>
        <xs:element name='device' type='deviceElementType' minOccurs='0' maxOccurs='unbounded'/>
      </xs:all>
    </xs:complexType>
  </xs:element>

<xs:complexType name='deviceElementType'>
  <xs:attribute name='type' use='required'>
    <xs:simpleType>
      <xs:restriction base='xs:NMTOKEN'/>
    </xs:simpleType>
  </xs:attribute>
  <xs:any namespace='##other' processContents='lax' minOccurs='0' maxOccurs='unbounded'/>
</xs:complexType>

</xs:schema>

17. Acknowledgements¶

Thanks to NLNet foundation/NGI Assure for funding the work on this specification.

Appendices

Appendix A: Document Information¶

Series

XEP

Number

xxxx

Publisher

XMPP Standards Foundation

Status

ProtoXEP

Type

Standards Track

Version

0.0.1

Last Updated

2024-05-11

Approving Body

XMPP Council

Dependencies

XMPP Core, XEP-0001, XEP-0166, XEP-0343

Supersedes

None

Superseded By

None

Short Name

remote-control

This document in other formats: XML  PDF

Appendix B: Author Information¶

Jérôme Poisson

Email

goffi@goffi.org

JabberID

goffi@jabber.fr

Appendix C: Legal Notices¶

Appendix D: Relation to XMPP¶

The Extensible Messaging and Presence Protocol (XMPP) is defined in the XMPP Core (RFC 6120) and XMPP IM (RFC 6121) specifications contributed by the XMPP Standards Foundation to the Internet Standards Process, which is managed by the Internet Engineering Task Force in accordance with RFC 2026. Any protocol defined in this document has been developed outside the Internet Standards Process and is to be understood as an extension to XMPP rather than as an evolution, development, or modification of XMPP itself.

Appendix E: Discussion Venue¶

The primary venue for discussion of XMPP Extension Protocols is the <standards@xmpp.org> discussion list.

Discussion on other xmpp.org discussion lists might also be appropriate; see <https://xmpp.org/community/> for a complete list.

Errata can be sent to <editor@xmpp.org>.

Appendix F: Requirements Conformance¶

The following requirements keywords as used in this document are to be interpreted as described in RFC 2119: "MUST", "SHALL", "REQUIRED"; "MUST NOT", "SHALL NOT"; "SHOULD", "RECOMMENDED"; "SHOULD NOT", "NOT RECOMMENDED"; "MAY", "OPTIONAL".

Appendix G: Notes¶

Appendix H: Revision History¶

Note: Older versions of this specification might be available at https://xmpp.org/extensions/attic/

1. Version 0.0.1 (2024-05-11)¶

   First draft.

   jp

Appendix I: Bib(La)TeX Entry

@report{poisson2024remote-control,
  title = {Jingle Remote Control},
  author = {Poisson, Jérôme},
  type = {XEP},
  number = {xxxx},
  version = {0.0.1},
  institution = {XMPP Standards Foundation},
  url = {https://xmpp.org/extensions/xep-xxxx.html},
  date = {2024-05-11/2024-05-11},
}

END