Troubleshooting Microsoft Teams and Pexip Infinity integrations

This topic provides further information about the Teams Connector installation, describes any limitations and provides troubleshooting guidance when integrating Microsoft Teams with Pexip Infinity.

For information about Teams Connector status and troubleshooting failed calls, see Viewing Teams Connector instance, call and participant status.

Installation issues

This table describes issues that might occur when running the Teams Connector installation script.

Issue / error message Description / resolution
Invalid Bot data error message If the PowerShell script output reports an error stating "InvalidBotData - Bot is not valid - Id is already in use", this means that the name of the Azure Bot you are attempting to create (via the Register-PexTeamsCviApplicationBot commands) is already in use elsewhere in Azure (most likely in another company's tenant).
Web space already exists error message If the scripts fail to create a resource group with an error message in the style "Web space with name <name> already exists for subscription <id>", this typically means that a previous resource group of the same name had its contents deleted, but the resource group itself was not deleted. When removing resources from Azure e.g. prior to an upgrade or redeployment, ensure that the resource group itself is deleted.
Certificate error: CNG Key Storage Provider If you see the error message "The private key of certificate '<thumbprint>' is stored in a CNG Key Storage Provider, which is currently not supported", this means that the RSA certificate has a private key in a CNG KSP (Cryptographic API: Next Generation Key Storage Provider), which is not supported. The private key must be stored in a Cryptographic Service Provider (CSP) of the CryptoAPI.
Certificate error: ECDSA certificate If you see the error message "The certificate '<thumbprint>' is an ECDSA certificate, which is currently not supported", this means that you have an ECDSA certificate. The Teams Connector certificate must use RSA keys.
Get-AzADServicePrincipal: Insufficient privileges to complete the operation. This error occurs if a guest user in the tenant attempts to deploy a Teams Connector, but the guest user does not have the Directory Readers role.
Invalid addresses Ensure that any CIDR-style addresses are valid. For example, an "MCU-signalling-endpoint has invalid Address prefix" error would be reported if an invalid CIDR address, such as "91.90.42.96/26", was specified for the $PxNodesSourceAddressPrefixes variable.
Other errors reported when running the create_vmss_deployment script Many errors that are reported when you run create_vmss_deployment.ps1 are a result of typos or formatting errors in the variables initialization script, such as missing quotes around an IP address. Check the error messages produced by the script for more information about the nature of the problem.
More Teams Connector instances than expected are seen during the deployment process Scale sets default to overprovisioning VMs. With overprovisioning turned on, the scale set spins up more VMs than requested, then deletes the extra VMs when the requested number of VMs are successfully provisioned. See https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-design-overview#overprovisioning for more information.
Connect-AzAccount WARNING: Unable to acquire token for tenant 'organizations' WARNING: Please run 'Connect-AzAccount -DeviceCode' if browser is not supported in this session.

Try to Connect with: Connect-AzAccount -DeviceCode

If there are still issues then set default web proxy credentials and retry executing Connect-AzAccount:

[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials

If that doesn't resolve the issue:

  1. Close all Powershell windows.
  2. Open an Administrator command prompt.
  3. Execute: powershell -NoProfile -Command "Uninstall-Module Az.Accounts"
  4. Close the Administrator command prompt.
  5. Open a PowerShell window.
  6. Execute: Install-Module Az.Accounts -MinimumVersion 1.9.5
  7. Retry Connect-AzAccount
The remote server returned an error: (407) Proxy Authentication Required

Set default web proxy credentials and retry the deployment:

[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials

Module being installed is not catalog signed

This can happen when you run the Install-Module command if you have downgraded a package that was signed in the newer version.

To overcome it you need to add the -SkipPublisherCheck switch to the Install-Module command.

For more information see this article.

WARNING: The names of some imported commands from the module 'Microsoft.Azure.PowerShell.Cmdlets.Network' include unapproved verbs that might make them less discoverable.

and

WARNING: The names of some imported commands from the module 'Az.Network' include unapproved verbs that might make them less discoverable.

These warnings may appear and can be safely ignored.

Errors like:

Get-AzAccessToken : You must use multi-factor authentication to access resource MicrosoftGraphEndpointResourceId,
please rerun 'Connect-AzAccount' with additional parameter '-AuthScope MicrosoftGraphEndpointResourceId'.
At line:1 char:16
+ ... cessToken = Get-AzAccessToken -ResourceTypeName MSGraph -TenantId $ct ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Get-AzAccessToken], AzPSAuthenticationFailedException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.GetAzureRmAccessTokenCommand

 

or a warning:

WARNING: Unable to acquire token for tenant {tenantId} with error 'You must use multi-factor authentication to access tenant {tenantId}, please rerun 'Connect-AzAccount' with additional parameter '-TenantId {tenantId}'.'

You're probably connecting to a tenant as a guest user. Additionally the tenant has 2FA set up.

The solution is to specify the TenantId parameter (where TenantId is an id of the tenant at which your user is a guest) when running Connect-AzAccount:

Connect-AzAccount -TenantId {tenantId}

Errors like:

Get-MgApplication : Method not found: 'Void
Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider...'.
At line:1 char:1
+ Get-MgApplication
+ ~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-MgApplication_List], MissingMethodException
+ FullyQualifiedErrorId : Microsoft.Graph.PowerShell.Cmdlets.GetMgApplication_List

or

WARNING: Unable to acquire token for tenant 'organizations' with error 'Method not found:
'System.Threading.Tasks.Task`1<Azure.Identity.AuthenticationRecord>...'
Connect-AzAccount : Method not found: 'System.Threading.Tasks.Task`1<Azure.Identity...'.
At line:1 char:1
+ Connect-AzAccount
+ ~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Connect-AzAccount], MissingMethodException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand

These occur if you are using Microsoft.Graph module version 2.0.0. (See https://github.com/microsoftgraph/msgraph-sdk-powershell/issues/2148 for more information.)

You should uninstall version 2.0.0 and install version 2.2.0 instead.

Call failures (invalid conference ID, rejected calls and disconnections)

This section provides some guidance on how to troubleshoot failed calls to the Teams Connector.

As with all troubleshooting scenarios, reviewing the Pexip Infinity administrator log or the support log (where you can search for "support.teams" for specific Teams-related issues) may help you identify the possible cause of some failure scenarios.

Invalid conference ID failures when dialing via a Virtual Reception

The following table provides a set of typical causes of call failures and their associated solutions when the caller sees a "Conference ID invalid" or "Cannot connect to this extension" message when attempting to connect to a conference via a Virtual Reception.

Symptom Possible cause Resolution
Intermittent call failures: no unusual failures or error codes are in the logs. The user is entering the wrong conference ID. Ensure that the correct 9 to 12-digit conference ID is being entered.
Persistent call failures: no unusual failures or error codes are in the logs. There is no Teams Connector configured against the Virtual Reception or the Virtual Reception's Lookup location, or the Conferencing Node cannot reach the nominated Teams Connector. Ensure that a Teams Connector is configured against the Virtual Reception or the Lookup location, and that the Conferencing Nodes in that location can reach the nominated Teams Connector.
Support log entries report "Teams API request failed" and Error="401".

There is a problem with the certificate on either the Teams Connector or the Conferencing Node, for example it may have expired, been revoked, or the certificate subject names may not match against expected values.

Ensure that the Teams Connector and the Conferencing Nodes have a valid certificate that is signed by an external trusted CA. See Network and certificate requirements for information about certificate subject name requirements.

The Conferencing Node is communicating with the wrong Teams Connector (if, for example, you have other Teams Connectors configured in different regions, or a lab system etc).

Ensure that the Virtual Reception is nominating the appropriate Outgoing location (and thus Conferencing Nodes) for the associated Teams Connector (i.e. the nodes that were referenced in the $PxNodeFqdns variable in the initialization script for that Teams Connector).

A chain of intermediate CA certificates installed on the Management Node (to provide the chain of trust for the Conferencing Node's certificate) includes a HTTP-to-HTTPS redirect in the AIA (Authority Information Access) portion of one of those intermediate certificates.

Obtain your certificates from a different Certificate Authority.

Support log entries report "Teams API request failed" and Error="500". The wrong Azure tenant ID is configured in Pexip Infinity.

Check the Azure tenant ID configured in Pexip Infinity (Call control > Microsoft Teams Tenants) and which tenant ID is associated with the Teams Connector (Call control > Microsoft Teams Connectors).

This should be the tenant ID that was used during the process to authorize/consent the Pexip CVI app.

CVI caller gets an invalid conference ID message when trying to join a scheduled meeting, but Teams clients can connect to the meeting.

The meeting may have expired. This can happen if the meeting invitation was created a long time ago (typically more than 60 days ago). In this scenario, when the Teams user clicks on their meeting link, Teams automatically creates a new meeting with a new conference ID, and it appears as though they have joined the original meeting. The CVI caller cannot join as they do not have the new conference ID.

If you have PSTN enabled on your meetings, you can see if a meeting has expired by joining the meeting via the Teams client and comparing the PSTN Conference ID shown in the Meeting Options join details against the PSTN Conference ID shown in the original invitation. If the two IDs are different then the original meeting has expired.

See https://docs.microsoft.com/en-us/microsoftteams/limits-specifications-teams#meeting-expiration for more information.

The CVI caller cannot join that meeting.

The only option is to create a new scheduled or instant meeting.

Support log entries report "Failed to create call leg: Failed to resolve VTC short key". You are using an invald conference ID. Ensure you are using the correct conference ID.

Call is not connecting (direct or indirect dialing)

If the outbound leg of a gateway call to the Teams Connector fails, you typically see "Gateway dial out failed" or "Call rejected" in the participant history in Pexip Infinity. In these scenarios, Pexip Infinity has failed to place a call to the Teams Connector. This covers direct dialing (where the dialed alias includes the conference ID) and indirect dialing where the caller has successfully entered a valid conference ID into the Virtual Reception, but Pexip Infinity has then failed to connect the call.

The following table shows some possible causes and solutions for such failures:

Symptom Possible cause Resolution
Intermittent failures: disconnect reason is "Gateway dial out failed" or "Call rejected". The dialed alias includes the wrong conference ID (direct dialing). Ensure that the dialed alias contains the correct 9 to 12-digit conference ID.
The Teams Connector was at maximum capacity and thus unable to take the call. Consider increasing the number of instances in your Teams Connector (see Changing the call capacity of a Teams Connector).
Persistent failures: disconnect reason is "Gateway dial out failed" or "Call rejected". The Call Routing Rule has an incorrect regex replace string. Ensure that the Call Routing Rule regex replace string is extracting only the 9 to 12-digit conference ID.
There is no Teams Connector configured against the Call Routing Rule or the rule's Outgoing location, or the Conferencing Node cannot reach the nominated Teams Connector. Ensure that a Teams Connector is configured against the Call Routing Rule or the rule's Outgoing location, and that the Conferencing Nodes in that location can reach the nominated Teams Connector.
The wrong tenant ID is configured in Pexip Infinity.

Check the Azure tenant ID configured in Pexip Infinity (Call control > Microsoft Teams Tenants) and which tenant ID is associated with the Teams Connector (Call control > Microsoft Teams Connectors).

This should be the tenant ID that was used during the process to authorize/consent the Pexip CVI app.

Disconnect reason is "Gateway dial out failed" and the support log entries for the associated Call-id reports "Teams API request failed" and Error="401".

There is a problem with the certificate on either the Teams Connector or the Conferencing Node, for example it may have expired, been revoked, or the certificate subject names may not match against expected values.

Ensure that the Teams Connector and the Conferencing Nodes have a valid certificate that is signed by an external trusted CA. See Network and certificate requirements for information about certificate subject name requirements.

The Conferencing Node is communicating with the wrong Teams Connector (if, for example, you have other Teams Connectors configured in different regions, or a lab system etc).

Ensure that the Call Routing Rules that are handling the calls are nominating the appropriate Outgoing location (and thus Conferencing Nodes) for the associated Teams Connector (i.e. the nodes that were referenced in the $PxNodeFqdns variable in the initialization script for that Teams Connector).

A chain of intermediate CA certificates installed on the Management Node (to provide the chain of trust for the Conferencing Node's certificate) includes a HTTP-to-HTTPS redirect in the AIA (Authority Information Access) portion of one of those intermediate certificates.

Obtain your certificates from a different Certificate Authority.

You get this audio message: "You can’t talk to the bot just yet, but we’re working on it" when calling from a Teams Room. Port 4277 is blocked from the Teams Connector to the Conferencing Node. Allow port 4277. See Enabling a Microsoft Teams Room to call a VTC endpoint or VMR for more information.

Disconnections from ongoing conferences

The following table shows some possible causes and solutions if participants are disconnected from ongoing conferences:

Symptom Possible cause Resolution
Calls are disconnected from a conference and "Running conference has been forcibly terminated due to MCU shutdown" messages are in the Pexip Infinity administrator log. This can occur if the Adaptive Composition layout is in use and there is a large number of active participants. This can be mitigated by assigning more RAM to your Conferencing Node processor cores.

Useful PowerShell commands

The following PowerShell commands are available when troubleshooting or maintaining your system:

Requirement PowerShell command
List the existing interop service provider Get-CsVideoInteropServiceProvider
Modify an interop service provider (see Changing the alternative dialing instructions or IVR address for more information) Set-CsVideoInteropServiceProvider
Remove an interop provider Remove-CsVideoInteropServiceProvider
List the existing service policies Get-CsTeamsVideoInteropServicePolicy -Filter "*enabled*"
Return user information Get-CsOnlineUser -Identity username@example.com
Enable interop for a named user Grant-CsTeamsVideoInteropServicePolicy -PolicyName PexipServiceProviderEnabled -Identity username@example.com
Enable interop for an AD group Grant-CsTeamsVideoInteropServicePolicy -PolicyName PexipServiceProviderEnabled -Group <name of the Azure AD group>
Remove interop completely for a named user Grant-CsTeamsVideoInteropServicePolicy -PolicyName ServiceProviderDisabled -Identity username@example.com
Remove any explicit interop policy for a named user i.e. revert to using the global default policy Grant-CsTeamsVideoInteropServicePolicy -PolicyName $null -Identity username@example.com
Revert the global interop policy to its default configuration Grant-CsTeamsVideoInteropServicePolicy -PolicyName ServiceProviderDisabled
List all active CVI users Get-CsOnlineUser -Filter {TeamsVideoInteropServicePolicy -eq "PexipServiceProviderEnabled"} | Select-Object DisplayName, UserPrincipalName, TeamsVideoInteropServicePolicy

Obtaining Teams Connector logs

To help troubleshoot issues, your Pexip authorized support representative may ask you for log files from your Teams Connector in addition to a snapshot from your Pexip Infinity Management Node.

As of version 30, logs are stored in a diag container (previously logs container), and access is now via user delegated SAS tokens (previously SAS tokens signed by the account keys).

Note that logs and incident reports are kept for 30 days on the Teams Connector.

To get the logs and snapshot:

  1. Get the Teams Connector logs:

    1. In the Azure portal, for your subscription, select the Teams Connector static resource group that contains resources such as the key vault and public IP address.
    2. Select the Storage account in that resource group.
    3. Select Access Control (IAM) and assign the Storage Blob Data Reader role to the storage account for the user (you would need to do this even for yourself) who is going to generate the user delegated SAS token in order to access the diag container.
    4. If you have VNet integration enabled you need to add both your client IP address and "176.125.235.150" in the storage account’s Networking settings (Firewall > Address range).
    5. Wait for 5 minutes for the access permission to propagate.
    6. Generate a user delegated SAS token:

      1. Go to Data storage > Containers.
      2. Click the context menu next to the diag container and select Generate SAS.

    7. On the Generate SAS panel:

      1. Select User delegation key.
      2. For Permissions, select Read and List.
      3. Set the Allowed IP addresses to 176.125.235.150 and HTTPS only.
      4. Click Generate SAS token and URL.
      5. Copy the Blob SAS URL that appears. You will use this in Pexportal.

  2. Get a snapshot from the Pexip Infinity Management Node:

    1. On the Pexip Infinity Administrator interface, go to Utilities > Diagnostic snapshot.
    2. To download all available data, select Download full snapshot.

      or

      To download a subset of the snapshot, containing a specified time period:

      1. Type in, or adjust the sliders to the start and end times (in terms of how many hours ago from the current time) of the diagnostic data to be downloaded. The number of hours available will vary depending on the logs available on the system.
      2. If requested by your Pexip authorized support representative, select Include diagnostic metrics from all Conferencing Nodes.

        If this is selected, then metrics from the Management Node and all Conferencing Nodes are included with the snapshot.

        If this is not selected, then metrics from the Management Node only are included with the snapshot.

      3. Select Download limited duration snapshot.

      Wait while the snapshot file is prepared — do not navigate away from the page until the file has been generated.

    3. Follow your browser's prompts to save the file.
  3. Upload the Azure logs and the Management Node snapshot to Pexportal:

    1. Go to pexportal.pex.me and sign in.
    2. You have to upload the Azure logs and the Pexip Infinity snapshot as two separate steps. First, upload the Azure logs:

      1. Select Upload files.
      2. Fill in the page details:

        Ticket number Enter your support ticket number.
        Azure SAS URL Paste in here the Blob SAS URL from the Azure portal and select Fetch dates.
        Select a file to upload Ignore this button when uploading the Azure logs.

      3. Select Upload.

        You see a message that the Azure logs are queued for download.

    3. Next, upload the Pexip Infinity snapshot:

      1. Select Upload files.
      2. Fill in the page details:

        Ticket number Enter your support ticket number.
        Azure SAS URL Leave blank
        Select a file to upload Select Choose file and select the diagnostic snapshot file you downloaded from the Management Node.

      3. Select Upload.

        You see a message that the that the snapshot file has been uploaded.

    You can now exit Pexportal.

Data stored/processed by the Teams Connector

Information related to the processing of calls into Microsoft Teams meetings is logged and stored by the Teams Connector. This includes, but is not limited to, Conferencing Node IP address information and FQDNs, VTC display names / caller aliases, call IDs, Teams conference IDs, Teams meeting participant names / aliases, application IDs, trusted call status, participant lobby status, and participant presentation status.

These logs are stored in an Azure Storage account (owned by the customer) and are rotated daily with expiry after 90 days.

In addition to the logged data:

  • The Teams Connector stores the TLS certificate used to verify connections to the Teams Connector.
  • All call signaling and media between Conferencing Nodes and the Microsoft Teams backend passes through the Teams Connector.

Discovering your Azure tenant ID

To retrieve your tenant ID from the Azure portal:

  1. Select Azure Active Directory.
  2. Under Manage, select Properties.
  3. The tenant ID is shown in the Directory ID field. You can use the Click to copy option to copy it.

You can also check your tenant ID at https://www.whatismytenantid.com/.

Viewing your tenant's app registrations

Your Pexip CVI application needs to be granted permission to enable access to Microsoft Teams in an Office 365 tenant, as described in Authorize Pexip CVI application to join Teams meetings.

You can check the status of your apps by viewing your app registrations and your list of enterprise applications.

Viewing current app registrations

To see the current app registrations for your Azure tenant:

  1. Go to the Azure portal (https://aad.portal.azure.com) and sign in.
  2. Go to Azure Active Directory > App registrations.
  3. Search for the prefix you used when deploying your Teams Connector, and it will list your app registrations as shown below (your app names will be different).

    Note that if you have deployed the Teams Connector several times, old app registrations will also show — even if you have deleted the Azure Bot resource group. If required, you can delete these old registrations.

  4. To check if consent has been granted in your Tenant, select the CVI app registration from the list (the one named "<prefix>TeamsConn").

    If it shows Create Service Principal (as shown below) then consent has not been granted in your own Microsoft Entra ID. (Note that this is expected if Teams is in a different Entra ID from your Azure Bot, however in most enterprises it will be in the same Entra ID.)

    If consent has been granted in your tenant, the registration will show a reference to the enterprise app ("pexipTeamsConn" in the example below).

View authorized enterprise apps

To view authorized enterprise apps i.e. where consent has been granted:

  1. From the Azure portal go to Azure Active Directory > Enterprise Applications.
  2. Search for the prefix you used when deploying your Teams Connector.

    In the example screenshot below there are currently no apps where consent has been granted:

    If the app has had consent granted, you will see the app as shown below (your app name will be different):

  3. If your Teams Connector CVI app is not shown, you must provide consent to it as described in Authorize Pexip CVI applications.

    Your app will then appear in the list of enterprise applications as shown in the second example screenshot above.

    Also when you view the app via Azure Active Directory > App registrations, each registration will show a reference to the enterprise app.

Azure notification: Denial of service attack detected

You may receive notifications from Azure that it has detected potential outgoing denial-of-service attacks from your Teams Connector. They take the form:

Outgoing denial-of-service:
Azure Security Center detected unusual network activity originating from the virtual machine providers <component list>. An unusually large number of connections were made from this machine. This anomalous activity may indicate that your virtual machine was compromised and is now engaged in denial-of-service (DoS) attacks against external endpoints.

This occurs when Azure detects the media traffic from your Teams Connector as potential UDP flooding. These messages can be safely ignored as the Teams Connector is behaving correctly, and Azure has not blocked your media connection.

Azure is developing an "Alert suppression" feature that in the future will allow you to disable these alerts for your subscription.