Troubleshooting Microsoft Teams and Pexip Infinity integrations

This topic provides further information about the Teams Connector installation and describes any limitations and provides troubleshooting guidance when integrating Microsoft Teams with Pexip Infinity.

Installation issues

This section describes issues that might occur when running the Teams Connector installation script.

Invalid Bot data error message

If the PowerShell script output reports an error stating "InvalidBotData - Bot is not valid - Id is already in use", this means that the name of the bot channel registration you are attempting to create (via the Register-PexTeamsCviApplicationBot commands) is already in use elsewhere in Azure (most likely in another company's tenant).

Web space already exists error message

If the scripts fail to create a resource group with an error message in the style "Web space with name <name> already exists for subscription <id>", this typically means that a previous resource group of the same name had its contents deleted, but the resource group itself was not deleted. When removing resources from Azure e.g. prior to an upgrade or redeployment, ensure that the resource group itself is deleted.

Certificate error: CNG Key Storage Provider

If you see this error message: The private key of certificate '<thumbprint>' is stored in a CNG Key Storage Provider, which is currently not supported.

this means that the RSA certificate has a private key in a CNG KSP (Cryptographic API: Next Generation Key Storage Provider), which is not supported. The private key must be stored in a Cryptographic Service Provider (CSP) of the CryptoAPI.

Certificate error: ECDSA certificate

If you see this error message: The certificate '<thumbprint>' is an ECDSA certificate, which is currently not supported.

this means that you have an ECDSA certificate. The Teams Connector certificate must use RSA keys.

Other errors reported when running create_vmss_deployment.ps1

Many errors that are reported when you run create_vmss_deployment.ps1 are a result of typos or formatting errors in the variables initialization script, such as missing quotes around an IP address.

Using the Azure activity log

Some of the reported installation issues can be difficult to interpret. In many cases, extra information that is available in the Azure activity log can help to pinpoint the underlying problem.

For example, here is an installation validation error:

New-AzureRmResourceGroupDeployment : 14:35:44 - Error: Code=InvalidTemplateDeployment; Message=The template deployment 'azuredeploy' is not valid according to the validation procedure. The tracking id is
'c2c6943d-3825-43cb-9a77-d2c0094ecd95'. See inner errors for details. Please see https://aka.ms/arm-deploy for usage details.
At C:\Teams Connector\Pexip_Infinity_Connector_for_Microsoft_Teams_v21.1_48372\create_vmss_deployment.ps1:884 char:23
+   $deploymentResult = New-AzureRmResourceGroupDeployment `
+                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   + CategoryInfo          : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
   + FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceGroupDeploymentCmdlet

To find out more information, you need to use the Azure portal and look at the activity log:

  1. In the Azure portal, enter "activity log" in the search box and select the Activity log service.
  2. Add a filter and select a Resource group of Event initiated by and enter the name of the user who made the deployment.

    Azure activity log filter

  3. Select the log item, in this case the Validate Deployment operation. However, the summary information often does not provide the root cause of the issue.
  4. Select the JSON option.

    Azure activity log summary

  5. Scroll to the bottom and look at the statusMessage item.

    In this case we can see that the problem is overlapping subnets in the NSG rules.

    Azure activity log JSON view

Obtaining Teams Connector logs

To help troubleshoot issues, your Pexip authorized support representative may ask you for log files from your Teams Connector in addition to a snapshot from your Pexip Management Node.

Note that logs and incident reports are kept for 30 days on the Teams Connector.

To get the logs and snapshot:

  1. Get the Teams Connector logs:

    1. In the Azure portal, for your subscription, select the Teams Connector resource group that contains your Virtual machine scale set, Load balancer, Network security group and so on.
    2. Select the Storage account in that resource group.

      Note that the logs themselves are stored as Blobs in a Logs container within the storage account (typically there are more logs folders than actual VM instances). However, you do not need to access these individual logs as you will generate a URI that grants restricted access to those logs.

    3. In Settings, select Shared access signature.
    4. In the Allowed permissions, clear (untick) the following boxes: Write, Delete, Add, Create, Update.
    5. Set the Allowed IP addresses to 185.35.201.95 — this is the address of the Pexportal support site.
    6. Leave everything else selected according to the default settings unless requested otherwise.
    7. Select Generate SAS and connection string.

      A list of strings and URLs are generated.

    8. Copy the Blob service SAS URL.

  2. Get a snapshot from the Pexip Management Node:

    1. On the Pexip Infinity Administrator interface, go to Utilities > Diagnostic snapshot.
    2. To download all available data, select Download full snapshot.

      or

      To download a subset of the snapshot, containing just the most recent data:

      1. Type in, or adjust the slider to the minimum number of hours of diagnostic data to be downloaded. The number of hours available will vary depending on the logs available on the system.
      2. If requested by your Pexip authorized support representative, select Include diagnostic metrics from all Conferencing Nodes.

        If this is checked, then metrics from the Management Node and all Conferencing Nodes are included with the snapshot.

        If this is not checked, then metrics from the Management Node only are included with the snapshot.

      3. Select Download limited duration snapshot.

      Wait while the snapshot file is prepared — do not navigate away from the page until the file has been generated.

    3. Follow your browser's prompts to save the file.
  3. Upload the Azure logs and the Management Node snapshot to Pexportal:

    1. Go to pexportal.pex.me and sign in.
    2. Select Upload files.
    3. Fill in the page details:

      Ticket number Enter your support ticket number.
      Azure SAS URL Paste in here the Blob service SAS URL from the Azure portal and select Fetch dates.
      Select a file to upload Select Choose file and select the diagnostic snapshot file you downloaded from the Pexip Management Node.
    4. Select Upload.

      You see a message that the snapshot file has been uploaded and the Azure logs are queued for download.

      You can now exit Pexportal.

Data stored/processed by the Teams Connector

Information related to the processing of calls into Microsoft Teams meetings is logged and stored by the Teams Connector. This includes, but is not limited to, Conferencing Node IP address information and FQDNs, VTC display names / caller aliases, call IDs, Teams meeting IDs, Teams meeting participant names / aliases, application IDs, trusted call status, participant lobby status, and participant presentation status.

These logs are stored in an Azure Storage account (owned by the customer) and are rotated daily with expiry after 30 days.

In addition to the logged data:

  • The Teams Connector stores the TLS certificate used to verify connections to the Teams Connector.
  • All call signaling and media between Conferencing Nodes and the Microsoft Teams backend passes through the Teams Connector.

Call failures (invalid conference ID and rejected calls)

This section provides some guidance on how to troubleshoot failed calls to the Teams Connector.

As with all troubleshooting scenarios, reviewing the Pexip Infinity administrator log or the support log (where you can search for "support.teams" for specific Teams-related issues) may help you identify the possible cause of some failure scenarios.

Invalid conference ID failures when dialing via a Virtual Reception

The following table provides a set of typical causes of call failures and their associated solutions when the caller sees a "Conference ID invalid" or "Cannot connect to this extension" message when attempting to connect to a conference via a Virtual Reception.

Symptom Possible cause Resolution
Intermittent call failures: no unusual failures or error codes are in the logs. The user is entering the wrong conference ID. Ensure that the correct 9 to 12-digit VTC conference ID is being entered.
Persistent call failures: no unusual failures or error codes are in the logs. There is no Teams Connector configured against the Virtual Reception or the Virtual Reception's Lookup location, or the Conferencing Node cannot reach the nominated Teams Connector. Ensure that a Teams Connector is configured against the Virtual Reception or the Lookup location, and that the Conferencing Nodes in that location can reach the nominated Teams Connector.
Support log entries report "Teams API request failed" and Error="401".

There is a problem with the certificate on either the Teams Connector or the Conferencing Node, for example it may have expired, been revoked, or the certificate subject names may not match against expected values.

Ensure that the Teams Connector and the Conferencing Nodes have a valid certificate that is signed by an external trusted CA. See Network and certificate requirements for information about certificate subject name requirements.

The Conferencing Node is communicating with the wrong Teams Connector (if, for example, you have other Teams Connectors configured in different regions, or a lab system etc).

Ensure that the Virtual Reception is nominating the appropriate Outgoing location (and thus Conferencing Nodes) for the associated Teams Connector (i.e. the nodes that were referenced in the $PxNodeFqdns variable in the initialization script for that Teams Connector).

A chain of intermediate CA certificates installed on the Management Node (to provide the chain of trust for the Conferencing Node's certificate) includes a HTTP-to-HTTPS redirect in the AIA (Authority Information Access) portion of one of those intermediate certificates.

Obtain your certificates from a different Certificate Authority.

Support log entries report "Teams API request failed" and Error="500". The wrong Azure tenant ID is configured in Pexip Infinity.

Check the Azure tenant ID configured in Pexip Infinity (Call control > Microsoft Teams Tenants) and which tenant ID is associated with the Teams Connector (Call control > Microsoft Teams Connectors).

This should be the tenant ID that was used during the process to authorize/consent the Pexip app.

Call is not connecting (direct or indirect dialing)

If the outbound leg of a gateway call to the Teams Connector fails, you typically see "Gateway dial out failed" or "Call rejected" in the participant history in Pexip Infinity. In these scenarios, Pexip Infinity has failed to place a call to the Teams Connector. This covers direct dialing (where the dialed alias includes the conference ID) and indirect dialing where the caller has successfully entered a valid conference ID into the Virtual Reception, but Pexip Infinity has then failed to connect the call.

The following table shows some possible causes and solutions for such failures:

Symptom Possible cause Resolution
Intermittent failures: disconnect reason is "Gateway dial out failed" or "Call rejected". The dialed alias includes the wrong conference ID (direct dialing). Ensure that the dialed alias contains the correct 9 to 12-digit VTC conference ID.
The Teams Connector was at maximum capacity and thus unable to take the call. Consider increasing the number of instances in your Teams Connector (see Changing the call capacity of a Teams Connector).
Persistent failures: disconnect reason is "Gateway dial out failed" or "Call rejected". The Call Routing Rule has an incorrect regex replace string. Ensure that the Call Routing Rule regex replace string is extracting only the 9 to 12-digit meeting code.
There is no Teams Connector configured against the Call Routing Rule or the rule's Outgoing location, or the Conferencing Node cannot reach the nominated Teams Connector. Ensure that a Teams Connector is configured against the Call Routing Rule or the rule's Outgoing location, and that the Conferencing Nodes in that location can reach the nominated Teams Connector.
The wrong tenant ID is configured in Pexip Infinity.

Check the Azure tenant ID configured in Pexip Infinity (Call control > Microsoft Teams Tenants) and which tenant ID is associated with the Teams Connector (Call control > Microsoft Teams Connectors).

This should be the tenant ID that was used during the process to authorize/consent the Pexip app.

Disconnect reason is "Gateway dial out failed" and the support log entries for the associated Call-id reports "Teams API request failed" and Error="401".

There is a problem with the certificate on either the Teams Connector or the Conferencing Node, for example it may have expired, been revoked, or the certificate subject names may not match against expected values.

Ensure that the Teams Connector and the Conferencing Nodes have a valid certificate that is signed by an external trusted CA. See Network and certificate requirements for information about certificate subject name requirements.

The Conferencing Node is communicating with the wrong Teams Connector (if, for example, you have other Teams Connectors configured in different regions, or a lab system etc).

Ensure that the Call Routing Rules that are handling the calls are nominating the appropriate Outgoing location (and thus Conferencing Nodes) for the associated Teams Connector (i.e. the nodes that were referenced in the $PxNodeFqdns variable in the initialization script for that Teams Connector).

A chain of intermediate CA certificates installed on the Management Node (to provide the chain of trust for the Conferencing Node's certificate) includes a HTTP-to-HTTPS redirect in the AIA (Authority Information Access) portion of one of those intermediate certificates.

Obtain your certificates from a different Certificate Authority.

Discovering your Azure tenant ID

To retrieve your tenant ID from the Azure portal:

  1. Select Azure Active Directory.
  2. Under Manage, select Properties.
  3. The tenant ID is shown in the Directory ID field. You can use the Click to copy option to copy it.

You can also check your tenant ID at https://www.whatismytenantid.com/.

Viewing your tenant's app registrations

Your Pexip CVI application needs to be granted permission to enable access to Microsoft Teams in an Office 365 tenant, as described in Authorize Pexip CVI application to join Teams meetings.

You can check the status of your apps by viewing your app registrations and your list of enterprise applications.

Viewing current app registrations

To see the current app registrations for your Azure tenant:

  1. Go to the Azure portal (https://portal.azure.com) and sign in.
  2. Go to Azure Active Directory > App registrations.
  3. Search for the prefix you used when deploying your Teams Connector, and it will list your app registrations as shown below (your app name will be different).

    Note that if you have deployed the Teams Connector several times, old app registrations will also show — even if you have deleted the botchannelregistration resource group. If required, you can delete these old registrations.

  4. To check if consent has been granted in your Tenant, select each registration from the list.

    If it shows Create Service Principal (as shown below) then consent has not been granted in your own Azure AD. (Note that this is expected if Teams is in a different Azure AD from your Bot Channel Registration, however in most enterprises it will be in the same Azure AD.)

    If consent has been granted in your tenant, the registration will show a reference to the enterprise app ("pexipTeamsConn" in the example below).

View authorized enterprise apps

To view authorized enterprise apps i.e. where consent has been granted:

  1. From the Azure portal go to Azure Active Directory > Enterprise Applications.
  2. Search for the prefix you used when deploying your Teams Connector.

    In the example screenshot below there are currently no apps where consent has been granted:

    If the app has had consent granted, you will see the app as shown below (your app name will be different):

  3. If your Teams Connector app is not shown, you must provide consent to it as described in Authorize Pexip CVI application to join Teams meetings.

    Your app will then appear in the list of enterprise applications as shown in the second example screenshot above.

    Also when you view the app via Azure Active Directory > App registrations, each registration will show a reference to the enterprise app.

Azure notification: Denial of service attack detected

You may receive notifications from Azure that it has detected potential outgoing denial-of-service attacks from your Teams Connector. They take the form:

Outgoing denial-of-service:
Azure Security Center detected unusual network activity originating from the virtual machine providers <component list>. An unusually large number of connections were made from this machine. This anomalous activity may indicate that your virtual machine was compromised and is now engaged in denial-of-service (DoS) attacks against external endpoints.

This occurs when Azure detects the media traffic from your Teams Connector as potential UDP flooding. These messages can be safely ignored as the Teams Connector is behaving correctly, and Azure has not blocked your media connection.

Azure is developing an "Alert suppression" feature that in the future will allow you to disable these alerts for your subscription.