Troubleshooting Microsoft Teams and Pexip Infinity integrations

This topic provides further information about the Teams Connector installation, describes any limitations and provides troubleshooting guidance when integrating Microsoft Teams with Pexip Infinity.

For information about Teams Connector status and troubleshooting failed calls, see Viewing Teams Connector instance, call and participant status.

Installation issues

This table describes issues that might occur when running the Teams Connector installation script.

Issue / error message Description / resolution
Invalid Bot data error message If the PowerShell script output reports an error stating "InvalidBotData - Bot is not valid - Id is already in use", this means that the name of the bot channel registration you are attempting to create (via the Register-PexTeamsCviApplicationBot commands) is already in use elsewhere in Azure (most likely in another company's tenant).
Web space already exists error message If the scripts fail to create a resource group with an error message in the style "Web space with name <name> already exists for subscription <id>", this typically means that a previous resource group of the same name had its contents deleted, but the resource group itself was not deleted. When removing resources from Azure e.g. prior to an upgrade or redeployment, ensure that the resource group itself is deleted.
Certificate error: CNG Key Storage Provider If you see the error message "The private key of certificate '<thumbprint>' is stored in a CNG Key Storage Provider, which is currently not supported", this means that the RSA certificate has a private key in a CNG KSP (Cryptographic API: Next Generation Key Storage Provider), which is not supported. The private key must be stored in a Cryptographic Service Provider (CSP) of the CryptoAPI.
Certificate error: ECDSA certificate If you see the error message "The certificate '<thumbprint>' is an ECDSA certificate, which is currently not supported", this means that you have an ECDSA certificate. The Teams Connector certificate must use RSA keys.
Failed to add 'Storage Account Key Operator Service Role' to Storage Account.

If the reported error includes "Failed to add 'Storage Account Key Operator Service Role' to Storage Account" or "Add-AzKeyVaultManagedStorageAccount : Operation returned an invalid status code 'Forbidden'" this most likely means that the person performing the installation does not have an Owner role for the static resource group used for the Azure Key Vault.

You must assign the Owner role as described in Create the resource group for static resources and re-run the installation.

This error can also occur if the culture/locale is not set to en-US. You can run Get-Culture to check your current culture. If it is not en-US:

  1. Run Set-Culture -CultureInfo en-US to set the culture to en-US.
  2. You must then open a new PowerShell window and rerun the .\create_vmss_deployment.ps1 command.
Get-AzADServicePrincipal : Insufficient privileges to complete the operation. This error occurs if a guest user in the tenant attempts to deploy a Teams Connector, but the guest user does not have the Directory Readers role.
Invalid addresses Ensure that any CIDR-style addresses are valid. For example, an "MCU-signalling-endpoint has invalid Address prefix" error would be reported if an invalid CIDR address, such as "91.90.42.96/26", was specified for the $PxNodesSourceAddressPrefixes variable.
Other errors reported when running the create_vmss_deployment script Many errors that are reported when you run create_vmss_deployment.ps1 are a result of typos or formatting errors in the variables initialization script, such as missing quotes around an IP address. Check the error messages produced by the script for more information about the nature of the problem.
More Teams Connector instances than expected are seen during the deployment process Scale sets default to overprovisioning VMs. With overprovisioning turned on, the scale set spins up more VMs than requested, then deletes the extra VMs when the requested number of VMs are successfully provisioned. See https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-design-overview#overprovisioning for more information.

Useful PowerShell commands

The following PowerShell commands are available when troubleshooting or maintaining your system:

Requirement PowerShell command
List the existing interop service provider Get-CsVideoInteropServiceProvider
Modify an interop service provider (see Changing the alternative dialing instructions or IVR address for more information) Set-CsVideoInteropServiceProvider
Remove an interop provider Remove-CsVideoInteropServiceProvider
List the existing service policies Get-CsTeamsVideoInteropServicePolicy -Filter "*enabled*"
Return user information Get-CsOnlineUser -Identity username@example.com
Enable interop for a named user Grant-CsTeamsVideoInteropServicePolicy -PolicyName PexipServiceProviderEnabled -Identity username@example.com
Remove interop completely for a named user Grant-CsTeamsVideoInteropServicePolicy -PolicyName ServiceProviderDisabled -Identity username@example.com
Remove any explicit interop policy for a named user i.e. revert to using the global default policy Grant-CsTeamsVideoInteropServicePolicy -PolicyName $null -Identity username@example.com
List all active CVI users Get-CsOnlineUser -Filter {TeamsVideoInteropServicePolicy -eq "PexipServiceProviderEnabled"} | Select-Object DisplayName, UserPrincipalName, TeamsVideoInteropServicePolicy

Obtaining Teams Connector logs

To help troubleshoot issues, your Pexip authorized support representative may ask you for log files from your Teams Connector in addition to a snapshot from your Pexip Infinity Management Node.

Note that logs and incident reports are kept for 30 days on the Teams Connector.

To get the logs and snapshot:

  1. Get the Teams Connector logs:

    1. In the Azure portal, for your subscription, select the Teams Connector static resource group that contains resources such as the key vault and public IP address. (Prior to version 23 the logs were stored in the VM's resource group.)
    2. Select the Storage account in that resource group.

      Note that the logs themselves are stored in a logs container within the storage account (typically there are more logs folders than actual VM instances). However, you do not need to access these individual logs as you will generate a URI that grants restricted access to those logs.

    3. In Settings, select Shared access signature.
    4. In the Allowed resource types, select (tick) all of the boxes: Service, Container, Object.
    5. In the Allowed permissions, clear (untick) the following boxes: Write, Delete, Add, Create, Update.
    6. Set the Allowed IP addresses to 185.35.201.95 — this is the address of the Pexportal support site.
    7. Leave everything else selected according to the default settings unless requested otherwise.
    8. Select Generate SAS and connection string.

      A list of strings and URLs are generated.

    9. Copy the Blob service SAS URL.

  2. Get a snapshot from the Pexip Infinity Management Node:

    1. On the Pexip Infinity Administrator interface, go to Utilities > Diagnostic snapshot.
    2. To download all available data, select Download full snapshot.

      or

      To download a subset of the snapshot, containing a specified time period:

      1. Type in, or adjust the sliders to the start and end times (in terms of how many hours ago from the current time) of the diagnostic data to be downloaded. The number of hours available will vary depending on the logs available on the system.
      2. If requested by your Pexip authorized support representative, select Include diagnostic metrics from all Conferencing Nodes.

        If this is selected, then metrics from the Management Node and all Conferencing Nodes are included with the snapshot.

        If this is not selected, then metrics from the Management Node only are included with the snapshot.

      3. Select Download limited duration snapshot.

      Wait while the snapshot file is prepared — do not navigate away from the page until the file has been generated.

    3. Follow your browser's prompts to save the file.
  3. Upload the Azure logs and the Management Node snapshot to Pexportal:

    1. Go to pexportal.pex.me and sign in.
    2. Select Upload files.
    3. Fill in the page details:

      Ticket number Enter your support ticket number.
      Azure SAS URL Paste in here the Blob service SAS URL from the Azure portal and select Fetch dates.
      Select a file to upload Select Choose file and select the diagnostic snapshot file you downloaded from the Management Node.
    4. Select Upload.

      You see a message that the snapshot file has been uploaded and the Azure logs are queued for download.

      You can now exit Pexportal.

Data stored/processed by the Teams Connector

Information related to the processing of calls into Microsoft Teams meetings is logged and stored by the Teams Connector. This includes, but is not limited to, Conferencing Node IP address information and FQDNs, VTC display names / caller aliases, call IDs, Teams conference IDs, Teams meeting participant names / aliases, application IDs, trusted call status, participant lobby status, and participant presentation status.

These logs are stored in an Azure Storage account (owned by the customer) and are rotated daily with expiry after 30 days.

In addition to the logged data:

  • The Teams Connector stores the TLS certificate used to verify connections to the Teams Connector.
  • All call signaling and media between Conferencing Nodes and the Microsoft Teams backend passes through the Teams Connector.

Discovering your Azure tenant ID

To retrieve your tenant ID from the Azure portal:

  1. Select Azure Active Directory.
  2. Under Manage, select Properties.
  3. The tenant ID is shown in the Directory ID field. You can use the Click to copy option to copy it.

You can also check your tenant ID at https://www.whatismytenantid.com/.

Viewing your tenant's app registrations

Your Pexip CVI application needs to be granted permission to enable access to Microsoft Teams in an Office 365 tenant, as described in Authorize Pexip CVI application to join Teams meetings.

You can check the status of your apps by viewing your app registrations and your list of enterprise applications.

Viewing current app registrations

To see the current app registrations for your Azure tenant:

  1. Go to the Azure portal (https://portal.azure.com) and sign in.
  2. Go to Azure Active Directory > App registrations.
  3. Search for the prefix you used when deploying your Teams Connector, and it will list your app registrations as shown below (your app name will be different).

    Note that if you have deployed the Teams Connector several times, old app registrations will also show — even if you have deleted the botchannelregistration resource group. If required, you can delete these old registrations.

  4. To check if consent has been granted in your Tenant, select each registration from the list.

    If it shows Create Service Principal (as shown below) then consent has not been granted in your own Azure AD. (Note that this is expected if Teams is in a different Azure AD from your Bot Channel Registration, however in most enterprises it will be in the same Azure AD.)

    If consent has been granted in your tenant, the registration will show a reference to the enterprise app ("pexipTeamsConn" in the example below).

View authorized enterprise apps

To view authorized enterprise apps i.e. where consent has been granted:

  1. From the Azure portal go to Azure Active Directory > Enterprise Applications.
  2. Search for the prefix you used when deploying your Teams Connector.

    In the example screenshot below there are currently no apps where consent has been granted:

    If the app has had consent granted, you will see the app as shown below (your app name will be different):

  3. If your Teams Connector app is not shown, you must provide consent to it as described in Authorize Pexip CVI applications.

    Your app will then appear in the list of enterprise applications as shown in the second example screenshot above.

    Also when you view the app via Azure Active Directory > App registrations, each registration will show a reference to the enterprise app.

Azure notification: Denial of service attack detected

You may receive notifications from Azure that it has detected potential outgoing denial-of-service attacks from your Teams Connector. They take the form:

Outgoing denial-of-service:
Azure Security Center detected unusual network activity originating from the virtual machine providers <component list>. An unusually large number of connections were made from this machine. This anomalous activity may indicate that your virtual machine was compromised and is now engaged in denial-of-service (DoS) attacks against external endpoints.

This occurs when Azure detects the media traffic from your Teams Connector as potential UDP flooding. These messages can be safely ignored as the Teams Connector is behaving correctly, and Azure has not blocked your media connection.

Azure is developing an "Alert suppression" feature that in the future will allow you to disable these alerts for your subscription.