Livestream freezing

I am using a 5 sensor setup. Whenever I start streaming, the software freezes and so does the stream until I stop streaming and have to start over only for it to freeze again after a minute or so. But when I use a 3 sensor setup, everything works well.
Another issue is that the sensors aren’t lined up correctly when I view the stream in Unity, I’ve attached a screenshot

@Telmo when changing the number of sensors you are livestreaming with, Depthkit will update the “livestream_multicam_meta.txt” metadata file in the “_Exports” folder of your Depthkit bin. In order for Unity to render the scene with the proper geometry, the metadata file in the Unity project linked to your Depthkit asset also needs to be updated.

Some other questions about your livestream issue:

  • Can you describe the freezing behavior in more detail? Is the Depthkit app unresponsive, or do the sensor streams just update slowly / fail to update at all? How long after stating to stream does it freeze?
  • What resolution have you set to be the maximum livestream width/height constraints in Depthkit Preferences? Have you tried any different resolutions?
  • Also, what does your GPU usage look like in the Performance tab of Task Manager while livestreaming? If the GPU usage is at or near 100%, you can follow this recently-published guide to improving performance in Unity.

@Telmo Also, the name of your Depthkit object in the Hierarchy seems to indicate that you’re using the Depthkit Clip + Core Built-In Look for your Depthkit asset, which is intended for single-sensor captures. If this is the case, remove the Depthkit Core Mesh Source and Built-in Look components, and replace them with their Depthkit Studio equivelants. (Or simply import the prefabs found in the Depthkit Studio plugin.

The freezing behaviour happens within Depthkit, it starts happens during the calibration process throughout streaming and recording. Only the viewport freezes while the rest of the software is running. We usually have to stop calibration for a couple of seconds and continue again, then it freezes after a minute or so. Same thing during recording or livestreaming.

We’ve noticed that this only happens when we have more than 3 sensors connected, as it runs smoothly with 3 or less sensors.

@Telmo, which color and depth resolution are your sensors set to during calibration and livestreaming? For calibration, we recommend 720p color and either of the “raw” depth modes.

I saw in an earlier post that you were using a PC with the following spec’s:

Windows 10
12th Gen i9-12900KF
Mag z690 Tomahawk (MS-7D32)
Renesas USB 4 port Startech
NVIDIA GeForce RTX 3080 Ti
Cable Matter Extenders

Are you still using this 3080 Ti GPU? If so, this is likely contributing to the issue. Nvidia’s consumer GPU’s support up to 3 simultaneous NVENC sessions which is suitable for 3 sensor capture/livestreaming, but for more sensors, you’ll need a professional GPU like the RTX A4000. You can find more information here.

EDIT: A professional GPU is only required for RECORDING more than 3 sensors, and shouldn’t be necessary for calibration and livestreaming.

Can you also confirm in Device Manager what USB chipset the 5th sensor is plugged into? (See the device tree shown in our hardware documentation.)

@Telmo Just following up:

  • Can you confirm which color and depth resolution the sensors are set to when you are calibrating and streaming? Have you tried 720p color and 640x576 NFOV Raw depth?
  • Can you confirm in Device Manager which USB controller each of the 5 sensors is plugged into, and which driver is installed for each controller?

Hi Cory

Yes, we calibrated with the recommended 720p color and raw depth modes.

We tried various streaming settings, of course the higher you go the harder it is on our system.
For streaming we had a 3 sensor setup and 1080 color seems to work the best. Adding more sensors and higher resolution took strain on our PC making the streaming window crash.

Ill take a look at the USB controller and driver settings once the system is back up and running, and will let you know. However, they are set to how we had them previously when we last met. Will let you know asap.

@Telmo Since we have been discussing the sensor depth issue in your other thread, I wanted to check in on this issue - Are you still seeing this viewport freezing issue? If so:

  • Were you able to confirm the exact type of USB controller that the sensors are plugged into?
  • What do Windows Task Manager’s performance graphs for the CPU and GPU look like when the freezing occurs?
  • Do you see the a continuous stream of red “dropped frame” alerts in the console once the stream freezes? Some number of dropped frames is expected when Depthkit is running normally during calibration, but if the computer can’t keep up with the streams at all, you’ll see a constant stream of “dropped frame” alerts.

Hi Cory, we are still experiencing the freezing issue. At times it does allow us to record at 1080p for about a minute or so before it freezes. And if you try any higher res, it freezes in less that 10 seconds.

-We have 4 sensors plugged into the Graphics Card and the 5th one in plugged into a USB 3 port on the motherboard.
-The graphs shoot all the way up when the freezing occurs, until I quit the program.
-Yes, the “dropped frame” alerts keep coming in a constant stream until I quit the program or stop calibration