Monitoring and troubleshooting Microsoft Teams and Pexip Infinity integrations

This topic provides further information about the Teams Connector installation, how to monitor calls, describes any limitations and provides troubleshooting guidance when integrating Microsoft Teams with Pexip Infinity.

Installation issues

This table describes issues that might occur when running the Teams Connector installation script.

Issue / error message Description / resolution
Invalid Bot data error message If the PowerShell script output reports an error stating "InvalidBotData - Bot is not valid - Id is already in use", this means that the name of the bot channel registration you are attempting to create (via the Register-PexTeamsCviApplicationBot commands) is already in use elsewhere in Azure (most likely in another company's tenant).
Web space already exists error message If the scripts fail to create a resource group with an error message in the style "Web space with name <name> already exists for subscription <id>", this typically means that a previous resource group of the same name had its contents deleted, but the resource group itself was not deleted. When removing resources from Azure e.g. prior to an upgrade or redeployment, ensure that the resource group itself is deleted.
Certificate error: CNG Key Storage Provider If you see the error message "The private key of certificate '<thumbprint>' is stored in a CNG Key Storage Provider, which is currently not supported", this means that the RSA certificate has a private key in a CNG KSP (Cryptographic API: Next Generation Key Storage Provider), which is not supported. The private key must be stored in a Cryptographic Service Provider (CSP) of the CryptoAPI.
Certificate error: ECDSA certificate If you see the error message "The certificate '<thumbprint>' is an ECDSA certificate, which is currently not supported", this means that you have an ECDSA certificate. The Teams Connector certificate must use RSA keys.
Failed to add 'Storage Account Key Operator Service Role' to Storage Account.

If the reported error includes "Failed to add 'Storage Account Key Operator Service Role' to Storage Account" or "Add-AzKeyVaultManagedStorageAccount : Operation returned an invalid status code 'Forbidden'" this means that the person performing the installation does not have an Owner role for the static resource group used for the Azure Key Vault.

You must assign the Owner role as described in Create the resource group for static resources and re-run the installation.

Invalid addresses Ensure that any CIDR-style addresses are valid. For example, an "MCU-signalling-endpoint has invalid Address prefix" error would be reported if an invalid CIDR address, such as "91.90.42.96/26", was specified for the $PxNodesSourceAddressPrefixes variable.
Other errors reported when running the create_vmss_deployment script Many errors that are reported when you run create_vmss_deployment.ps1 are a result of typos or formatting errors in the variables initialization script, such as missing quotes around an IP address. Check the error messages produced by the script for more information about the nature of the problem.

Useful PowerShell commands

The following PowerShell commands are available when troubleshooting or maintaining your system:

Requirement PowerShell command
List the existing interop service provider Get-CsVideoInteropServiceProvider
Modify an interop service provider (see Changing the alternative dialing instructions or IVR address for more information) Set-CsVideoInteropServiceProvider
Remove an interop provider Remove-CsVideoInteropServiceProvider
List the existing service policies Get-CsTeamsVideoInteropServicePolicy -Filter "*enabled*"
Return user information Get-CsOnlineUser -Identity username@example.com
Enable interop for a named user Grant-CsTeamsVideoInteropServicePolicy -PolicyName PexipServiceProviderEnabled -Identity username@example.com
Remove interop for a named user Grant-CsTeamsVideoInteropServicePolicy -PolicyName PexipServiceProviderDisabled -Identity username@example.com
List all active CVI users Get-CsOnlineUser -Filter {TeamsVideoInteropServicePolicy -eq "PexipServiceProviderEnabled"} | Select-Object DisplayName, UserPrincipalName, TeamsVideoInteropServicePolicy

Obtaining Teams Connector logs

To help troubleshoot issues, your Pexip authorized support representative may ask you for log files from your Teams Connector in addition to a snapshot from your Pexip Infinity Management Node.

Note that logs and incident reports are kept for 30 days on the Teams Connector.

To get the logs and snapshot:

  1. Get the Teams Connector logs:

    1. In the Azure portal, for your subscription, select the Teams Connector static resource group that contains resources such as the key vault and public IP address. (Prior to version 23 the logs were stored in the VM's resource group.)
    2. Select the Storage account in that resource group.

      Note that the logs themselves are stored in a logs container within the storage account (typically there are more logs folders than actual VM instances). However, you do not need to access these individual logs as you will generate a URI that grants restricted access to those logs.

    3. In Settings, select Shared access signature.
    4. In the Allowed permissions, clear (untick) the following boxes: Write, Delete, Add, Create, Update.
    5. Set the Allowed IP addresses to 185.35.201.95 — this is the address of the Pexportal support site.
    6. Leave everything else selected according to the default settings unless requested otherwise.
    7. Select Generate SAS and connection string.

      A list of strings and URLs are generated.

    8. Copy the Blob service SAS URL.

  2. Get a snapshot from the Pexip Infinity Management Node:

    1. On the Pexip Infinity Administrator interface, go to Utilities > Diagnostic snapshot.
    2. To download all available data, select Download full snapshot.

      or

      To download a subset of the snapshot, containing just the most recent data:

      1. Type in, or adjust the slider to the minimum number of hours of diagnostic data to be downloaded. The number of hours available will vary depending on the logs available on the system.
      2. If requested by your Pexip authorized support representative, select Include diagnostic metrics from all Conferencing Nodes.

        If this is selected, then metrics from the Management Node and all Conferencing Nodes are included with the snapshot.

        If this is not selected, then metrics from the Management Node only are included with the snapshot.

      3. Select Download limited duration snapshot.

      Wait while the snapshot file is prepared — do not navigate away from the page until the file has been generated.

    3. Follow your browser's prompts to save the file.
  3. Upload the Azure logs and the Management Node snapshot to Pexportal:

    1. Go to pexportal.pex.me and sign in.
    2. Select Upload files.
    3. Fill in the page details:

      Ticket number Enter your support ticket number.
      Azure SAS URL Paste in here the Blob service SAS URL from the Azure portal and select Fetch dates.
      Select a file to upload Select Choose file and select the diagnostic snapshot file you downloaded from the Management Node.
    4. Select Upload.

      You see a message that the snapshot file has been uploaded and the Azure logs are queued for download.

      You can now exit Pexportal.

Data stored/processed by the Teams Connector

Information related to the processing of calls into Microsoft Teams meetings is logged and stored by the Teams Connector. This includes, but is not limited to, Conferencing Node IP address information and FQDNs, VTC display names / caller aliases, call IDs, Teams meeting IDs, Teams meeting participant names / aliases, application IDs, trusted call status, participant lobby status, and participant presentation status.

These logs are stored in an Azure Storage account (owned by the customer) and are rotated daily with expiry after 30 days.

In addition to the logged data:

  • The Teams Connector stores the TLS certificate used to verify connections to the Teams Connector.
  • All call signaling and media between Conferencing Nodes and the Microsoft Teams backend passes through the Teams Connector.

Viewing call and participant status via the Pexip Infinity Administrator interface

When using the Pexip Infinity Administrator interface to monitor calls that are placed into Teams conferences, you should note that:

  • Each participant who is gatewayed into a Teams conference is listed as a separate gateway call. However, if multiple participants are connected to the same Teams conference, the Live View (Status > Live View) will show them as connected to the same external conference — which is identified as "Teams meeting <meeting code>".
  • When viewing the status of the gateway call (Status > Conferences), the Participants tab also lists the other participants in the conference. The format of their alias indicates the type of participant:

    • <name>@<domain> (email address) is used for any Teams clients in the call
    • trusted:<id> represents another gatewayed participant who joined as a trusted participant
    • guest:<id> represents another gatewayed participant who joined as an untrusted participant

    These other participants do not have any associated media stream information.

    Note that only the gatewayed participant is shown as consuming a port license. The outbound leg of the gateway call (into the Teams Connector), which consumes the second license of each gateway call, is not represented in the participant list.

    The media streams associated with the call into the Teams meeting are shown against the conference backplane:

    • There is one audio stream and multiple video streams. You also see a presentation stream if any participant is sharing content.
    • Multiple video streams are set up to receive video from the Teams Connector to support the Pexip conference layout seen by gatewayed participants; if there are fewer participants than streams then the currently unused streams are shown as "Off stage".
    • Pexip Infinity may simultaneously send up to 4 video streams and 4 presentation streams at different resolutions and framerates to the Teams Connector, as requested by Teams.

  • You cannot control (e.g. disconnect, mute or transfer) any of the other participants connected to the Teams conference.

Call failures (invalid conference ID and rejected calls)

This section provides some guidance on how to troubleshoot failed calls to the Teams Connector.

As with all troubleshooting scenarios, reviewing the Pexip Infinity administrator log or the support log (where you can search for "support.teams" for specific Teams-related issues) may help you identify the possible cause of some failure scenarios.

Invalid conference ID failures when dialing via a Virtual Reception

The following table provides a set of typical causes of call failures and their associated solutions when the caller sees a "Conference ID invalid" or "Cannot connect to this extension" message when attempting to connect to a conference via a Virtual Reception.

Symptom Possible cause Resolution
Intermittent call failures: no unusual failures or error codes are in the logs. The user is entering the wrong conference ID. Ensure that the correct 9 to 12-digit VTC conference ID is being entered.
Persistent call failures: no unusual failures or error codes are in the logs. There is no Teams Connector configured against the Virtual Reception or the Virtual Reception's Lookup location, or the Conferencing Node cannot reach the nominated Teams Connector. Ensure that a Teams Connector is configured against the Virtual Reception or the Lookup location, and that the Conferencing Nodes in that location can reach the nominated Teams Connector.
Support log entries report "Teams API request failed" and Error="401".

There is a problem with the certificate on either the Teams Connector or the Conferencing Node, for example it may have expired, been revoked, or the certificate subject names may not match against expected values.

Ensure that the Teams Connector and the Conferencing Nodes have a valid certificate that is signed by an external trusted CA. See Network and certificate requirements for information about certificate subject name requirements.

The Conferencing Node is communicating with the wrong Teams Connector (if, for example, you have other Teams Connectors configured in different regions, or a lab system etc).

Ensure that the Virtual Reception is nominating the appropriate Outgoing location (and thus Conferencing Nodes) for the associated Teams Connector (i.e. the nodes that were referenced in the $PxNodeFqdns variable in the initialization script for that Teams Connector).

A chain of intermediate CA certificates installed on the Management Node (to provide the chain of trust for the Conferencing Node's certificate) includes a HTTP-to-HTTPS redirect in the AIA (Authority Information Access) portion of one of those intermediate certificates.

Obtain your certificates from a different Certificate Authority.

Support log entries report "Teams API request failed" and Error="500". The wrong Azure tenant ID is configured in Pexip Infinity.

Check the Azure tenant ID configured in Pexip Infinity (Call control > Microsoft Teams Tenants) and which tenant ID is associated with the Teams Connector (Call control > Microsoft Teams Connectors).

This should be the tenant ID that was used during the process to authorize/consent the Pexip app.

Call is not connecting (direct or indirect dialing)

If the outbound leg of a gateway call to the Teams Connector fails, you typically see "Gateway dial out failed" or "Call rejected" in the participant history in Pexip Infinity. In these scenarios, Pexip Infinity has failed to place a call to the Teams Connector. This covers direct dialing (where the dialed alias includes the conference ID) and indirect dialing where the caller has successfully entered a valid conference ID into the Virtual Reception, but Pexip Infinity has then failed to connect the call.

The following table shows some possible causes and solutions for such failures:

Symptom Possible cause Resolution
Intermittent failures: disconnect reason is "Gateway dial out failed" or "Call rejected". The dialed alias includes the wrong conference ID (direct dialing). Ensure that the dialed alias contains the correct 9 to 12-digit VTC conference ID.
The Teams Connector was at maximum capacity and thus unable to take the call. Consider increasing the number of instances in your Teams Connector (see Changing the call capacity of a Teams Connector).
Persistent failures: disconnect reason is "Gateway dial out failed" or "Call rejected". The Call Routing Rule has an incorrect regex replace string. Ensure that the Call Routing Rule regex replace string is extracting only the 9 to 12-digit meeting code.
There is no Teams Connector configured against the Call Routing Rule or the rule's Outgoing location, or the Conferencing Node cannot reach the nominated Teams Connector. Ensure that a Teams Connector is configured against the Call Routing Rule or the rule's Outgoing location, and that the Conferencing Nodes in that location can reach the nominated Teams Connector.
The wrong tenant ID is configured in Pexip Infinity.

Check the Azure tenant ID configured in Pexip Infinity (Call control > Microsoft Teams Tenants) and which tenant ID is associated with the Teams Connector (Call control > Microsoft Teams Connectors).

This should be the tenant ID that was used during the process to authorize/consent the Pexip app.

Disconnect reason is "Gateway dial out failed" and the support log entries for the associated Call-id reports "Teams API request failed" and Error="401".

There is a problem with the certificate on either the Teams Connector or the Conferencing Node, for example it may have expired, been revoked, or the certificate subject names may not match against expected values.

Ensure that the Teams Connector and the Conferencing Nodes have a valid certificate that is signed by an external trusted CA. See Network and certificate requirements for information about certificate subject name requirements.

The Conferencing Node is communicating with the wrong Teams Connector (if, for example, you have other Teams Connectors configured in different regions, or a lab system etc).

Ensure that the Call Routing Rules that are handling the calls are nominating the appropriate Outgoing location (and thus Conferencing Nodes) for the associated Teams Connector (i.e. the nodes that were referenced in the $PxNodeFqdns variable in the initialization script for that Teams Connector).

A chain of intermediate CA certificates installed on the Management Node (to provide the chain of trust for the Conferencing Node's certificate) includes a HTTP-to-HTTPS redirect in the AIA (Authority Information Access) portion of one of those intermediate certificates.

Obtain your certificates from a different Certificate Authority.

Discovering your Azure tenant ID

To retrieve your tenant ID from the Azure portal:

  1. Select Azure Active Directory.
  2. Under Manage, select Properties.
  3. The tenant ID is shown in the Directory ID field. You can use the Click to copy option to copy it.

You can also check your tenant ID at https://www.whatismytenantid.com/.

Viewing your tenant's app registrations

Your Pexip CVI application needs to be granted permission to enable access to Microsoft Teams in an Office 365 tenant, as described in Authorize Pexip CVI application to join Teams meetings.

You can check the status of your apps by viewing your app registrations and your list of enterprise applications.

Viewing current app registrations

To see the current app registrations for your Azure tenant:

  1. Go to the Azure portal (https://portal.azure.com) and sign in.
  2. Go to Azure Active Directory > App registrations.
  3. Search for the prefix you used when deploying your Teams Connector, and it will list your app registrations as shown below (your app name will be different).

    Note that if you have deployed the Teams Connector several times, old app registrations will also show — even if you have deleted the botchannelregistration resource group. If required, you can delete these old registrations.

  4. To check if consent has been granted in your Tenant, select each registration from the list.

    If it shows Create Service Principal (as shown below) then consent has not been granted in your own Azure AD. (Note that this is expected if Teams is in a different Azure AD from your Bot Channel Registration, however in most enterprises it will be in the same Azure AD.)

    If consent has been granted in your tenant, the registration will show a reference to the enterprise app ("pexipTeamsConn" in the example below).

View authorized enterprise apps

To view authorized enterprise apps i.e. where consent has been granted:

  1. From the Azure portal go to Azure Active Directory > Enterprise Applications.
  2. Search for the prefix you used when deploying your Teams Connector.

    In the example screenshot below there are currently no apps where consent has been granted:

    If the app has had consent granted, you will see the app as shown below (your app name will be different):

  3. If your Teams Connector app is not shown, you must provide consent to it as described in Authorize Pexip CVI applications.

    Your app will then appear in the list of enterprise applications as shown in the second example screenshot above.

    Also when you view the app via Azure Active Directory > App registrations, each registration will show a reference to the enterprise app.

Azure notification: Denial of service attack detected

You may receive notifications from Azure that it has detected potential outgoing denial-of-service attacks from your Teams Connector. They take the form:

Outgoing denial-of-service:
Azure Security Center detected unusual network activity originating from the virtual machine providers <component list>. An unusually large number of connections were made from this machine. This anomalous activity may indicate that your virtual machine was compromised and is now engaged in denial-of-service (DoS) attacks against external endpoints.

This occurs when Azure detects the media traffic from your Teams Connector as potential UDP flooding. These messages can be safely ignored as the Teams Connector is behaving correctly, and Azure has not blocked your media connection.

Azure is developing an "Alert suppression" feature that in the future will allow you to disable these alerts for your subscription.