Monitoring and troubleshooting Microsoft Teams and Pexip Infinity integrations

This topic provides further information about the Teams Connector installation, how to monitor calls, describes any limitations and provides troubleshooting guidance when integrating Microsoft Teams with Pexip Infinity.

• Installation issues
• Useful PowerShell commands
• Obtaining Teams Connector logs
• Data stored/processed by the Teams Connector
• Viewing call and participant status via the Pexip Infinity Administrator interface
• Call failures (invalid conference ID and rejected calls)
• Discovering your Azure tenant ID
• Viewing your tenant's app registrations
• Azure notification: Denial of service attack detected

Installation issues

This table describes issues that might occur when running the Teams Connector installation script.

Issue / error message	Description / resolution
Invalid Bot data error message	If the PowerShell script output reports an error stating "InvalidBotData - Bot is not valid - Id is already in use", this means that the name of the bot channel registration you are attempting to create (via the Register-PexTeamsCviApplicationBot commands) is already in use elsewhere in Azure (most likely in another company's tenant).
Web space already exists error message	If the scripts fail to create a resource group with an error message in the style "Web space with name <name> already exists for subscription <id>", this typically means that a previous resource group of the same name had its contents deleted, but the resource group itself was not deleted. When removing resources from Azure e.g. prior to an upgrade or redeployment, ensure that the resource group itself is deleted.
Certificate error: CNG Key Storage Provider	If you see the error message "The private key of certificate '<thumbprint>' is stored in a CNG Key Storage Provider, which is currently not supported", this means that the RSA certificate has a private key in a CNG KSP (Cryptographic API: Next Generation Key Storage Provider), which is not supported. The private key must be stored in a Cryptographic Service Provider (CSP) of the CryptoAPI.
Certificate error: ECDSA certificate	If you see the error message "The certificate '<thumbprint>' is an ECDSA certificate, which is currently not supported", this means that you have an ECDSA certificate. The Teams Connector certificate must use RSA keys.
Failed to add 'Storage Account Key Operator Service Role' to Storage Account.	If the reported error includes "Failed to add 'Storage Account Key Operator Service Role' to Storage Account" or "Add-AzKeyVaultManagedStorageAccount : Operation returned an invalid status code 'Forbidden'" this most likely means that the person performing the installation does not have an Owner role for the static resource group used for the Azure Key Vault. You must assign the Owner role as described in Create the resource group for static resources and re-run the installation. This error can also occur if the culture/locale is not set to en-US. You can run Get-Culture to check your current culture. If it is not en-US: Run Set-Culture -CultureInfo en-US to set the culture to en-US. You must then open a new PowerShell window and rerun the .\create_vmss_deployment.ps1 command.
Get-AzADServicePrincipal : Insufficient privileges to complete the operation.	This error occurs if a guest user in the tenant attempts to deploy a Teams Connector, but the guest user does not have the Directory Readers role.
Invalid addresses	Ensure that any CIDR-style addresses are valid. For example, an "MCU-signalling-endpoint has invalid Address prefix" error would be reported if an invalid CIDR address, such as "91.90.42.96/26", was specified for the $PxNodesSourceAddressPrefixes variable.
Other errors reported when running the create_vmss_deployment script	Many errors that are reported when you run create_vmss_deployment.ps1 are a result of typos or formatting errors in the variables initialization script, such as missing quotes around an IP address. Check the error messages produced by the script for more information about the nature of the problem.
Upgrade fails with "Template parameter JToken type is not valid"	This occurs if you have version 4.6+ of Az installed. You have to downgrade the Az module to 4.5.0. Downgrade script This script can be used to downgrade your Az module from 4.6.1 to 4.5.0. Copy to clipboard function Uninstall-AzModule {   [CmdletBinding(SupportsShouldProcess)]   param(     [ValidateNotNullOrEmpty()]     [ValidateSet('Az','AzureRM','Azure')]     [string]$Name = 'Az',     [Parameter(Mandatory)]     [string]$Version,     [switch]$AllowPrerelease   )   $Params = @{}   if ($PSBoundParameters.AllowPrerelease) {     $Params.AllowPrerelease = $true   }   $IsAdmin = ([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] 'Administrator')   if (-not(Get-InstalledModule -Name $Name -RequiredVersion $Version -ErrorAction SilentlyContinue -OutVariable RootModule @Params)) {     Write-Warning -Message "Uninstall aborted. $Name version $Version not found."   } elseif (($RootModule.InstalledLocation -notlike "*$env:USERPROFILE*") -and ($IsAdmin -eq $false)) {     Write-Warning -Message "Uninstall aborted. $Name version $Version exists in a system path. PowerShell must be run elevated as an admin to remove it."   } elseif ((Get-Process -Name powershell, pwsh -OutVariable Sessions -ErrorAction SilentlyContinue).Count -gt 1) {     Write-Warning -Message "Uninstall aborted. Please close all other PowerShell sessions before continuing. There are currently $($Sessions.Count) PowerShell sessions running."   } else {     Write-Verbose -Message 'Creating list of dependencies...'     $target = Find-Module -Name $Name -RequiredVersion $Version @Params     $AllModules = @([pscustomobject]@{       Name = $Name       Version = $Version     })     $AllModules += foreach ($dependency in $target.Dependencies) {       switch ($dependency.keys) {         {$_ -contains 'RequiredVersion'} {$UninstallVersion = $dependency.RequiredVersion; break}         {$_ -contains 'MinimumVersion'} {$UninstallVersion = $dependency.MinimumVersion; break}       }       [pscustomobject]@{         Name = $dependency.Name         Version = $UninstallVersion       }     }     [int]$i = 100 / $AllModules.Count     foreach ($module in $AllModules) {       Write-Progress -Activity 'Uninstallation in Progress' -Status "$i% Complete:" -PercentComplete $i       $i++       if (Get-InstalledModule -Name $module.Name -RequiredVersion $module.Version -ErrorAction SilentlyContinue @Params) {         Write-Verbose -Message "Uninstalling $($module.Name) version $($module.Version)"         Remove-Module -FullyQualifiedName @{ModuleName=$module.Name;ModuleVersion=$module.Version} -ErrorAction SilentlyContinue         try {           if ($PSCmdlet.ShouldProcess("$($module.Name) version $($module.Version)")) {             Uninstall-Module -Name $module.Name -RequiredVersion $module.Version -Force -ErrorAction Stop @Params           }           $State = 'Uninstalled'         } Catch {           $State = 'Manual uninstall required'           Write-Verbose -Message "$($module.Name) version: $($module.Version) may require manual uninstallation."         }       } else {         $State = 'Not found'         Write-Verbose -Message "$($module.Name) version: $($module.Version) not found."       }       if (-not $PSBoundParameters.WhatIf) {         [pscustomobject]@{           ModuleName = $module.Name           Version = $module.Version           State = $State         }       }     }   } } Uninstall-AzModule -Name Az -version 4.6.1 Install-Module -Name Az -MinimumVersion 3.0.0 -MaximumVersion 4.5.0 -AllowClobber -Scope AllUsers

Useful PowerShell commands

The following PowerShell commands are available when troubleshooting or maintaining your system:

Requirement	PowerShell command
List the existing interop service provider	Get-CsVideoInteropServiceProvider
Modify an interop service provider (see Changing the alternative dialing instructions or IVR address for more information)	Set-CsVideoInteropServiceProvider
Remove an interop provider	Remove-CsVideoInteropServiceProvider
List the existing service policies	Get-CsTeamsVideoInteropServicePolicy -Filter "*enabled*"
Return user information	Get-CsOnlineUser -Identity username@example.com
Enable interop for a named user	Grant-CsTeamsVideoInteropServicePolicy -PolicyName PexipServiceProviderEnabled -Identity username@example.com
Remove interop for a named user	Grant-CsTeamsVideoInteropServicePolicy -PolicyName ServiceProviderDisabled -Identity username@example.com
List all active CVI users	Get-CsOnlineUser -Filter {TeamsVideoInteropServicePolicy -eq "PexipServiceProviderEnabled"} | Select-Object DisplayName, UserPrincipalName, TeamsVideoInteropServicePolicy

Obtaining Teams Connector logs

To help troubleshoot issues, your Pexip authorized support representative may ask you for log files from your Teams Connector in addition to a snapshot from your Pexip Infinity Management Node.

Note that logs and incident reports are kept for 30 days on the Teams Connector.

To get the logs and snapshot:

1. Get the Teams Connector logs:

   1. In the Azure portal, for your subscription, select the Teams Connector static resource group that contains resources such as the key vault and public IP address. (Prior to version 23 the logs were stored in the VM's resource group.)

   2. Select the Storage account in that resource group.

      Note that the logs themselves are stored in a logs container within the storage account (typically there are more logs folders than actual VM instances). However, you do not need to access these individual logs as you will generate a URI that grants restricted access to those logs.

   3. In Settings, select Shared access signature.
   4. In the Allowed resource types, select (tick) all of the boxes: Service, Container, Object.
   5. In the Allowed permissions, clear (untick) the following boxes: Write, Delete, Add, Create, Update.
   6. Set the Allowed IP addresses to 176.125.235.150 — this is the address of the Pexportal support site.
   7. Leave everything else selected according to the default settings unless requested otherwise.

   8. Select Generate SAS and connection string.

      

      A list of strings and URLs are generated.

   9. Copy the Blob service SAS URL.

      

2. Get a snapshot from the Pexip Infinity Management Node:

   1. On the Pexip Infinity Administrator interface, go to Utilities > Diagnostic snapshot.

   2. To download all available data, select Download full snapshot.

      or

      To download a subset of the snapshot, containing a specified time period:

      1. Type in, or adjust the sliders to the start and end times (in terms of how many hours ago from the current time) of the diagnostic data to be downloaded. The number of hours available will vary depending on the logs available on the system.

      2. If requested by your Pexip authorized support representative, select Include diagnostic metrics from all Conferencing Nodes.

         If this is selected, then metrics from the Management Node and all Conferencing Nodes are included with the snapshot.

         If this is not selected, then metrics from the Management Node only are included with the snapshot.

      3. Select Download limited duration snapshot.

      Wait while the snapshot file is prepared — do not navigate away from the page until the file has been generated.

   3. Follow your browser's prompts to save the file.

3. Upload the Azure logs and the Management Node snapshot to Pexportal:

   1. Go to pexportal.pex.me and sign in.
   2. Select Upload files.

   3. Fill in the page details:

      Ticket number	Enter your support ticket number.
      Azure SAS URL	Paste in here the Blob service SAS URL from the Azure portal and select Fetch dates.
      Select a file to upload	Select Choose file and select the diagnostic snapshot file you downloaded from the Management Node.

   4. Select Upload.

      

      You see a message that the snapshot file has been uploaded and the Azure logs are queued for download.

      

      You can now exit Pexportal.

Data stored/processed by the Teams Connector

Information related to the processing of calls into Microsoft Teams meetings is logged and stored by the Teams Connector. This includes, but is not limited to, Conferencing Node IP address information and FQDNs, VTC display names / caller aliases, call IDs, Teams meeting IDs, Teams meeting participant names / aliases, application IDs, trusted call status, participant lobby status, and participant presentation status.

These logs are stored in an Azure Storage account (owned by the customer) and are rotated daily with expiry after 30 days.

In addition to the logged data:

• The Teams Connector stores the TLS certificate used to verify connections to the Teams Connector.
• All call signaling and media between Conferencing Nodes and the Microsoft Teams backend passes through the Teams Connector.

Viewing call and participant status via the Pexip Infinity Administrator interface

When using the Pexip Infinity Administrator interface to monitor calls that are placed into Teams conferences, you should note that:

• Each participant who is gatewayed into a Teams conference is listed as a separate gateway call. However, if multiple participants are connected to the same Teams conference, the Live View (Status > Live View) will show them as connected to the same external conference — which is identified as "Teams meeting <meeting code>".

• When viewing the status of the gateway call (Status > Conferences), the Participants tab also lists the other participants in the conference. The format of their alias indicates the type of participant:

  • <name>@<domain> (email address) is used for any Teams clients in the call
  • trusted:<id> represents another gatewayed participant who joined as a trusted participant
  • guest:<id> represents another gatewayed participant who joined as an untrusted participant

  These other participants do not have any associated media stream information.

  Note that only the gatewayed participant is shown as consuming a port license. The outbound leg of the gateway call (into the Teams Connector), which consumes the second license of each gateway call, is not represented in the participant list.

  

  

  The media streams associated with the call into the Teams meeting are shown against the conference backplane:

  • There is one audio stream and multiple video streams. You also see a presentation stream if any participant is sharing content.
  • Multiple video streams are set up to receive video from the Teams Connector to support the Pexip conference layout seen by gatewayed participants; if there are fewer participants than streams then the currently unused streams are shown as "Off stage".
  • Pexip Infinity may simultaneously send up to 4 video streams and 4 presentation streams at different resolutions and frame rates to the Teams Connector, as requested by Teams.

  

• You cannot control (e.g. disconnect, mute or transfer) any of the other participants connected to the Teams conference.

Discovering your Azure tenant ID

To retrieve your tenant ID from the Azure portal:

1. Select Azure Active Directory.
2. Under Manage, select Properties.
3. The tenant ID is shown in the Directory ID field. You can use the Click to copy option to copy it.

You can also check your tenant ID at https://www.whatismytenantid.com/.

Viewing your tenant's app registrations

Your Pexip CVI application needs to be granted permission to enable access to Microsoft Teams in an Office 365 tenant, as described in Authorize Pexip CVI application to join Teams meetings.

You can check the status of your apps by viewing your app registrations and your list of enterprise applications.

Viewing current app registrations

To see the current app registrations for your Azure tenant:

1. Go to the Azure portal (https://portal.azure.com) and sign in.
2. Go to Azure Active Directory > App registrations.

3. Search for the prefix you used when deploying your Teams Connector, and it will list your app registrations as shown below (your app name will be different).

   

   Note that if you have deployed the Teams Connector several times, old app registrations will also show — even if you have deleted the botchannelregistration resource group. If required, you can delete these old registrations.

4. To check if consent has been granted in your Tenant, select each registration from the list.

   If it shows Create Service Principal (as shown below) then consent has not been granted in your own Azure AD. (Note that this is expected if Teams is in a different Azure AD from your Bot Channel Registration, however in most enterprises it will be in the same Azure AD.)

   

   If consent has been granted in your tenant, the registration will show a reference to the enterprise app ("pexipTeamsConn" in the example below).

   

View authorized enterprise apps

To view authorized enterprise apps i.e. where consent has been granted:

1. From the Azure portal go to Azure Active Directory > Enterprise Applications.

2. Search for the prefix you used when deploying your Teams Connector.

   In the example screenshot below there are currently no apps where consent has been granted:

   

   If the app has had consent granted, you will see the app as shown below (your app name will be different):

   

3. If your Teams Connector app is not shown, you must provide consent to it as described in Authorize Pexip CVI applications.

   Your app will then appear in the list of enterprise applications as shown in the second example screenshot above.

   Also when you view the app via Azure Active Directory > App registrations, each registration will show a reference to the enterprise app.

Azure notification: Denial of service attack detected

You may receive notifications from Azure that it has detected potential outgoing denial-of-service attacks from your Teams Connector. They take the form:

Outgoing denial-of-service:
Azure Security Center detected unusual network activity originating from the virtual machine providers <component list>. An unusually large number of connections were made from this machine. This anomalous activity may indicate that your virtual machine was compromised and is now engaged in denial-of-service (DoS) attacks against external endpoints.

This occurs when Azure detects the media traffic from your Teams Connector as potential UDP flooding. These messages can be safely ignored as the Teams Connector is behaving correctly, and Azure has not blocked your media connection.

Azure is developing an "Alert suppression" feature that in the future will allow you to disable these alerts for your subscription.