Enabling live captions

The live captions feature converts live meeting audio into text captions which are visible to all web app participants who have elected to view them. This topic describes how to configure Pexip Infinity and the AI Media Server (AIMS) to enable the live captions feature in Pexip Infinity VMRs, Virtual Auditoriums and gateway calls, including how to enable different languages for different VMRs in your Pexip Infinity deployment.

Diagram showing connections between Pexip Infinity and AIMS

When live captions are enabled by a participant in a VMR (or other service), the audio from the call is sent from the Conferencing Node that is handling the call, to a Conferencing Node in a dedicated live captions location. That Conferencing Node forwards the audio to AIMS, which transcribes it into text. The text is then returned back to the Conferencing Node in the live captions location, and on to the originating Conferencing Node, which then supplies it as a text overlay or transcript to all web app users in the VMR who have enabled live captions.

For AIMS v1 the supported languages are en-US, es-US, de-DE and fr-FR.

When implementing this feature note that:

• The captions are only shown to a web app user who has elected to view the captions using the Show live captions toolbar button. However, all of the other meeting participants are notified that the call is being transcribed.
• The Show live captions button is only displayed in the web app when live captions are enabled on Pexip Infinity via global settings or on the relevant VMR or Call Routing Rule.
• The feature is not available for Microsoft Teams, Google Meet, or Skype for Business / Lync gateway calls.

• Participants have the option to view captions either as ephemeral text overlaid on the main video, or from the Live Captions History panel, which provides a continuously updating view of all captions received while the participant has live captions enabled and is connected to the meeting. In the latter case, if a participant leaves and then rejoins a call, they will only see the captions shown since they rejoined.

• Pexip Infinity does not log or retain the contents of any live captions transcripts.

The sections below describe how to configure Pexip Infinity (including adding AIMS as a media processing server, enabling the live captions feature, setting up the necessary system locations, and optionally setting the language for a VMR), and how to configure AIMS with information about the Pexip Infinity deployment connecting to it.

Infinity configuration

In these steps you configure your Pexip Infinity deployment to use the AIMS server for media processing, and ensure that your VMRs and system locations are set up appropriately.

Adding a media processing server

To configure Pexip Infinity with information about an AIMS server to use to process media, go to Platform > Media Processing servers and select Add media processing server.

Enter the following information:

Option	Description
FQDN	The FQDN of the AIMS server used to process media.

After you have added the server's FQDN, select Save. Pexip Infinity then assigns the server a unique Application ID (visible from the main page) and JWT public key (visible when the server is selected). These are the values required when configuring AIMS.

For Pexip Infinity versions v34.x and earlier, this file will not be in the correct format for uploading to AIMS. See Pexip Infinity v34.x and earlier for information on how to reformat it. Alternatively, contact your Pexip authorized support representative for assistance.

Enabling live captions on VMRs and other services

By default, all Virtual Meeting Rooms, Virtual Auditoriums and Call Routing Rules are able to offer live captions, so any Webapp3 participant using these services can enable them. You can change the availability of live captions globally, and override this on a per-service basis.

To change the default availability of live captions for all services:

1. From your Pexip Infinity Management Node, go to Platform > Global Settings.
2. In the Service configuration section, select or clear the Live captions checkbox.

To configure the availability of live captions for an individual service (such as a VMR):

1. From Services > Virtual meeting rooms select the VMR.
2. Scroll down to and expand the Advanced Options section. From the Live Captions drop down, select one of the following options:
   • Use global live captions setting: live captions are only available for this VMR if the Platform > Global Settings > Service configuration > Live captions checkbox is selected.
   • Yes: live captions are always available for this VMR.
   • No: live captions are not available and cannot be enabled by participants in this VMR.

About system locations and AIMS

Each system location in your Pexip Infinity deployment has one or more nominated Live captions outgoing locations. When live captions are enabled on a VMR, the Conferencing Node that is handling the call will connect to the live caption location configured for the system location that it is in. A Conferencing Node in that live caption location will in turn connect to the AIMS server.

We recommend using a dedicated pool of Conferencing Nodes in their own system location to handle connections to AIMS. All other system locations should then have their Live captions outgoing locations set to these AIMS-specific locations.

Live captions locations should not be used to handle calls, or as overflow locations.

Example

For example, if your deployment has two existing system locations, London and Oslo, you may wish to create two new system locations, Live captions London and Live Captions Oslo. You then add one or more Transcoding Conferencing Nodes to each of these live captions locations.

You then configure the London location to use Live captions London as its first choice and Live Captions Oslo as its second choice for its live captions locations. Likewise, you'd configure the Oslo location to use Live captions Oslo as its first choice and Live Captions London as its second choice.

This would keep live captions traffic geographically local while allowing fallback to a secondary live captions location.

Alternatively, you could create a single Live captions location and configure all other system locations to use it as their Live captions outgoing locations first choice.

Creating and configuring a new live captions location

To create a new system location specifically for use as a live captions outgoing location:

1. From Platform > Locations, select Add system location.
2. Configure the NTP server and DNS server fields as appropriate for your environment. All other fields should be left blank, or with their default settings.
3. In the Live captions outgoing locations section, ensure that all choices are left blank.
4. Select Save.

Configuring existing system locations

To configure each existing system location in your deployment to connect to the live captions outgoing location:

1. From Platform > Locations, select the location and scroll down to Live captions outgoing locations.
2. Select the new live captions-specific location created above as the 1st choice. If you have created other live captions locations, select these as the 2nd choice and 3rd choice.
3. Repeat for all system locations in your deployment.

Setting the language on an individual VMR

AIMS needs to know the language of the incoming audio stream in order to apply the appropriate language model. For AIMS v1, generated live captions text are provided in the same language as the incoming (source) audio stream. Later releases of AIMS will be able to provide live captions text in a different language to the source language.

When configuring AIMS, you can set the default source language for all audio streams from the Pexip Infinity instance. If a default source language is not set, en-US is used.

From Pexip Infinity v36 and later, you can override the source language on a per-VMR / Virtual Auditorium / Call Routing Rule basis using external or local policy in combination with the service's service tag. An example of how to do this using local policy is described below.

Using local policy

The example below uses a combination of a VMR's service tag to specify the source language, and local policy to apply the selected language to the VMR's configuration.

We recommend that you apply the same local policy script to all locations in your Pexip Infinity deployment, so that you can use the service tag to specify the source language for any service.

Before you start, ensure you are familiar with the use of local policy as described in Enabling local policy.

1. Ensure that you have enabled the selected language in AIMS. See Setting the loaded language models.

2. From the Pexip Infinity Administrator interface, go to Call Control > Policy Profiles and select Add Policy profile.

3. Enter a Name and optional Description.

4. In the Service configuration policy section, enable Apply local policy.

5. In the Script section that appears, enter the following (or incorporate this code into an existing local policy script):

   Copy to clipboard

   {
   {% if service_config %}
       "action" : "continue",
       {% if service_config.service_tag.startswith("show_captions:") %}
           "result" : {{service_config|pex_update({
           "source_language":service_config.service_tag.split(":")[1],
           "live_captions_language":service_config.service_tag.split(":")[2]
            })|pex_to_json}}
       {% else %}
           "result" : {{service_config|pex_to_json}}
       {% endif %}
   {% else %}
       "action" : "reject",
       "result" : {}
   {% endif %}
   }

6. Select Save.
7. From Platform > Locations select a Location.
8. From the Policy profile field, select the policy profile created above and select Save.

9. Repeat the above steps to apply the policy profile to all other locations in your deployment.

10. From the Services menu, select the VMR for which you want to set the language.
11. Ensure that live captions are enabled for the VMR. To do this, from the Advanced options section, ensure that the Live Captions field shows either:
    • Yes, or
    • Use global live captions setting, in which case Platform > Global settings > Live captions > Enable live captions on VMRs and routing rules by default must be enabled.

12. From the Advanced options section, in the Service tag field enter the following:

    show_captions:<source-lang>:<captions-lang>

    where <source-lang> is the source language and <captions-lang> is the live captions language. For the current release of AIMS these parameters must be set to the same language.

    This method has been designed to support future releases of AIMS, where the source language and the live captions language may be different.

    For example, if the source language is French:

    show_captions:fr-FR:fr-FR

    For AIMS v1 the supported languages are en-US, es-US, de-DE and fr-FR.

13. Select Save.

To set the source language for any other VMR, Virtual Auditorium or Call Routing Rule in your deployment, ensure that live captions are enabled for the service and repeat the Service tag configuration. You can use different source languages for different services in the same Pexip Infinity deployment.

AIMS configuration

In this step you configure the AIMS server with details of the Pexip Infinity deployment that it will be processing requests from. The process differs depending on the version of Pexip Infinity; please select from the appropriate option below.