Voicebot Applet

Configuring a Bidirectional Stream- Voicebot Applet

Bidirectional Streaming

Bidirectional Streams allow two-way flow of voice data over a websocket. Exotel would send the voice data of the caller to a websocket endpoint. The endpoint can return back voice data back on the websocket and Exotel would play it out to the caller. The primary use case for this is to enable building intelligent conversational bots that will help you optimize your workforce WebSocket

You can enable bidirectional streaming on a call flow using the Voicebot Applet.

The applet takes 6 parameters:

1. URL- This is the URL to which Exotel will stream the voice media. You can either specify a wss endpoint or a https endpoint. If you specify a http/https endpoint, Exotel expects the https endpoint to return a wss url in its response. This is to allow

a. Dynamic endpoints for the same call flow b. Have dynamic custom parameters that can be passed to the websocket endpoint to handle any application-specific customisation.

There are two ways to put this: a. Static method: ws(s) endpoint can be entered here, but it will remain the same for every call that you are going to make using this flow. eg: ws://127.0.0.1:5001/media b. Dynamic method: We can enter http(s) URL which can return different ws(s) endpoints based on the use case. eg: https://yourdomain.com this URL must return a ws(s) endpoint

2. Authentication Options for WSS Endpoints- When configuring your WSS endpoint for the Voicebot Applet, Exotel supports the following authentication methods:

IP Whitelisting (Basic Setup) Secure your WSS connections by whitelisting Exotel’s outbound IP ranges. This avoids the need for credential exchange.

Reach out to [email protected] to get the range of IPs.

Basic Authentication Developers specify credentials in the WSS URL, but Exotel transmits them securely in headers during the connection.

This approach ensures compatibility and prevents credentials from being exposed in transit URLs.

3. WSS Sample Rate Parameters-Your WSS endpoint should support configurable sample rates. Exotel’s Voicebot Applet can define the required rate as a query parameter.

Default Behaviour: If no sample rate is explicitly defined, Exotel will use 8 kHz as the default.

Best Practice: Use 16 kHz for most voicebot integrations (balanced quality vs. bandwidth). Use 8 kHz only when interworking with legacy PSTN and 24 kHz for HD audio experiences.

4. Custom parameters(Optional) - Custom params along with the endpoint. There are some validations that need to follow while providing custom params.

a. Maximum number of custom params that are allowed is 3.

b. Format of these params will like :

ws://127.0.0.1:5001/media?param1=value1&param2=value2&param3=value3 (In Dynamic case, http(s) should return ws(s) URL in above format)

c. Total length of the params( bold part in above url ) shouldn’t be more than 256 characters.

5. Record - The checkbox gives an option to record the conversation and generate a recording URL available in passthru applet after voicebot applet.

6. Next Applet - In the case of a Bi-Directional Stream. The stream can end if the call is disconnected or the websocket is closed or the stream is explicitly stopped by the client. In the case of Bi-Directional Stream you do not need to add a explicit “Stop” Stream applet since the stream is automatically closed before executing the next Applet. The call flow proceeds to the next applet configured.

Video WalkThrough

You can find a quick walkthrough of a sample flow here.

Protocol

Communication between Exotel and customer endpoint happens over websocket connection.

Websocket messages - From Exotel

Each message in the websocket will be sent/received as a JSON string. Following are the types of messages that are sent:

Connected message:

After websocket connection is established, this message will be sent.

Start message:

Start message will contain information about the stream parameters. It will be sent only once, right after the connected message. The custom parameters are picked from the URL configured in the Stream Applet. If you had mentioned the URL as

wss://yourstream.service.com?queuename=premium&product=radio

queuename and product would be passed in as keys with premium and radio as values.

Media message:

This message encapsulates the audio packets.

media.chunk : chunk of the message media.timestamp : Timestamp in milliseconds from the start of the stream.

Media in the payloads are sent in raw/slin (16-bit, 8kHz, mono PCM (little-endian)) encoded in base64. The same is expected from the client in the case of bi-directional streams (In the case of a Voice Bot) to be played back to the human.

DTMF message:

DTMF message is sent when the digits are pressed by the user once the connection with websocket is established. This is supported only for bidirectional streaming in voicebot applet.

Stop message:

Stop message is sent when the stream is stopped or the call has ended.

Mark Message:

Mark message is used only in bidirectional streaming to track media when it is completed.

Websocket messages - To Exotel

These messages will be used only in bidirectional streaming.

Mark Message:

Mark message is used only in bidirectional streaming to track media when it is completed. You can send a mark event message after sending a media message to request a notification when the audio that you have sent has been processed. You'll receive a mark event message with a matching name from Exotel when the audio is processed.

Media message:

This message encapsulates the audio packets.

Clear message:

Clear message is used to clear the audio data that was sent before but not yet played. An example situation wherein it will be useful,

Developing human-like bots which can guess what he/she is going to say and send audio accordingly even before he/she completes it. When the guess goes wrong, we can clear that audio using a clear message.

{ "event": "clear", "stream_sid": "<stream sid>",}

Note: For Clear Event to work effectively, it is advisable to send media in smaller chunks. For instance, if two media messages of 5 seconds are sent to us, followed by Clear Event at the 3rd second, and the first message has already been picked up and played, the Clear event will only apply to the second media message. Therefore, sending media in smaller chunks can help prevent confusion and ensure that Clear Event works as intended.

Media Format

Media in the payloads are sent in raw/slin (16-bit, 8kHz, mono PCM (little-endian)) encoded in base64. The same is expected from the client in the case of bi directional streams to be played back to the caller.

Event Struct - Field Reference Table

Sample Code

https://github.com/exotel/Agent-Stream and

https://github.com/exotel/Agent-Stream-echobot

Simulator to make streaming calls with dummy bot



Chunk size window for Bidirectional streaming use case:

Minimum chunk size: 3.2k [100ms data]

Maximum chunk size: 100k

Chunk size should always be in multiple of 320 bytes.

1. If the size is less than the minimum size, we may face audio issues due to network jitters.

2. If the size is greater than 100k, it might result in timeouts

3. if the size X [for ex 4096] which is not in multiple of 320 Bytes: In this case, the last packet will be of lesser size than 320 Bytes, & platform will wait for 20ms before sending next chunk. i.e. sending less amount of data than wait time, then this might result into audio gaps in between.



Limitations

1) When you start a Unidirectional Stream, the stream is forked immediately. In case you have a Connect applet post this, the audio stream, even when Exotel dials out multiple agent,s would be relayed (ringing). The client will have to handle this to filter only relevant parts of the stream. This limitation will be fixed in future releases

2) The number of Custom Parameters that can be passed in the START message is 3

3) The stream is sent as a mono-channel raw audio format, and the client will have to handle speaker diarization

If you have any questions or concerns, please connect with us using the chat widget on your Exotel Dashboard or WhatsApp us on 08088919888.