Troubleshooting the AI Media Server
This section covers how to troubleshoot issues you may experience with the AI Media Server (AIMS) and its features and functionality.
This section includes:
- Gathering logs from the AIMS VM
- Setting and viewing the log retention period
- Taking a packet capture
- Checking the time on the AIMS VM
- Outputting the current GPU information
- Outputting the version of the AIMS VM
- Outputting the status of the AIMS VM
- Outputting support information
- Common issues
Gathering logs from the AIMS VM
To gather logs from the AIMS VM:
-
Log in to the AIMS VM as the user admin, either through SSH or the VMware console.
-
At the command prompt, run the following command:
sudo tar --exclude='/var/log/journal' -czvf /home/admin/aims_logs.gz /etc/pexip/pexaims /var/log
-
Export the /home/admin/aims_logs.gz file, either using a local SCP client (i.e.
scp admin@aims-appliance.domain.com:/home/admin/aims_logs.gz) or, for Windows, using WinSCP or similar.
Setting and viewing the log retention period
You can view and set the maximum number of days for which logging information will be retained. Note that logs have a minimum and maximum size limit, so in busy deployments log information may cover less than the maximum number of days while in very quiet deployments it may cover more.
To view the current maximum retention period for logging:
-
Log in to the AIMS VM as the user admin, either through SSH or the VMware console.
-
At the command prompt, run the command
aims-config log-retention-getThis will output the following:
admin@aims:~$ aims-config log-retention-get logs_max_age: 0
To set the maximum retention period for logging, in days:
-
Log in to the AIMS VM as the user admin, either through SSH or the VMware console.
-
At the command prompt, run the command
aims-config log-retention-set <days>For example:
aims-config log-retention-set 10
Taking a packet capture
To take a packet capture by running tcpdump on the AIMS appliance:
-
Log in to the AIMS VM as the user admin, either through SSH or the VMware console.
-
At the command prompt, run the command
sudo tcpdump -s0 -w /tmp/capture.pcap -
To stop the capture, press or.
You can download the capture via SCP from the /tmp folder.
Checking the time on the AIMS VM
To check the time on the AIMS VM:
-
Log in to the AIMS VM as the user admin, either through SSH or the VMware console.
-
At the command prompt, run the command
date && ntpq -pn && sudo hwclock -r
The following output can be shared with your Pexip authorized support representative, for example:
Fri Jun 21 06:44:53 AM UTC 2024 remote refid st t when poll reach delay offset jitter ======================================================================================================= +91.189.xxx.xx 194.58.xxx.xx 2 u 59 64 337 0.4440 7.9776 2.9072 +152.65.xxx.xx 129.24.xxx.xx 2 u 42 64 377 8.1303 7.1930 2.4215 2024-06-21 06:44:52.997148+00:00
Outputting the current GPU information
To get the GPU information:
-
Log in to the AIMS VM as the user admin, either through SSH or the VMware console.
-
At the command prompt, run the command
aims-config show-gpus
This will output information on the current GPU that is passed through to the AIMS appliance, and you should expect to see an output similar to the below:
AIMS GPU InfoAIMS GPU Info ============= GPU,Device,UUID,Variant,Memory gpu0,/dev/nvidia0,GPU-aa0b1111-a111-00a1-abcd-0aa1111b0a1b,adalovelace,23034 CDI Info ========= CDI Device count: 1 CDI Device nodes: /dev/nvidia-uvm,/dev/nvidiactl Mounts: HostPath,ContainerPath,Type,Options .... NVIDIA GPU Info ================ GPU 0: NVIDIA L4 (UUID: GPU-aa0b1111-a111-00a1-abcd-0aa1111b0a1b)
You can also include the verbose flag to include the NVSMI log in the output aims-config -v show-gpus.
If the AIMS appliance does not detect the GPU, you will see the following output:
AIMS GPU Info ============= E: Could not determine AIMS status CDI Info ========= E: Could not determine CDI status NVIDIA GPU Info ================ E: Could not determine NVIDIA status
You will need to ensure that you have correctly configured PCI pass-through of the GPU device in the hypervisor.
Outputting the version of the AIMS VM
To output the AIMS system version:
-
Log in to the AIMS VM as the user admin, either through SSH or the VMware console.
-
At the command prompt, run the command
aims-config show-version
For example:
AIMS Version Info
=================
{'major': '0', 'minor': '1', 'maintenance': '0', 'pseudo-version': '100.0.0', 'revision': '48457ce5a0aa426f926baa0ab481feaa'}
Outputting the status of the AIMS VM
To show AIMS system status:
-
Log in to the AIMS VM as the user admin, either through SSH or the VMware console.
-
At the command prompt, run the command aims-config show-status
You will see the running services here, the number of services will depend on the hardware passed through to the AIMS appliance. For example:
CTM Instance Summary ==================== Livecaption instances: ctm-livecaption0.service Riva instances: ctm-riva0.service Instance Details ================= ctm-livecaption0.service: loaded, active, running ctm-riva0.service: loaded, active, running
Outputting support information
To show detailed AIMS system status:
-
Log in to the AIMS VM as the user admin, either through SSH or the VMware console.
-
At the command prompt, run the command
aims-config show-all
This provides a detailed output which you can send to your Pexip authorized support representative, for example:
AIMS Version Info
=================
{'major': '0', 'minor': '1', 'maintenance': '0', 'pseudo-version': '100.0.0', 'revision': '48457ce5a0aa426f926baa0ab481feaa'}
CTM Instance Summary
====================
Livecaption instances: ctm-livecaption0.service
Riva instances: ctm-riva0.service
Instance Details
=================
ctm-livecaption0.service: loaded, active, running
ctm-riva0.service: loaded, active, running
AIMS GPU Info
=============
GPU,Device,UUID,Variant,Memory
gpu0,/dev/nvidia0,GPU-aa0b1111-a111-00a1-abcd-0aa1111b0a1b,adalovelace,23034
CDI Info
=========
CDI Device count: 1
CDI Device nodes: /dev/nvidia-uvm,/dev/nvidiactl
Mounts:
HostPath,ContainerPath,Type,Options
...
NVIDIA GPU Info
================
GPU 0: NVIDIA L4 (UUID: GPU-aa0b1111-a111-00a1-abcd-0aa1111b0a1b)
Name,AppID,DefaultLanguage
Pexip1,00abc000-111a-1234-1abc-a11122b3c444,en-US
loaded_services: en-US-parakeet
You can also include the verbose flag to include the NVSMI log in the output aims-config -v show-all.
Common issues
| Symptom | Possible cause | Resolution |
|---|---|---|
| Nothing happens when the closed captions button is pressed in a conference. |
This could be caused by a number of issues:
|
|