Studio-Lite, Quest Pro RenderStreaming Visual Quality Advice

Depthkit Integrations
1

We have been working with the new livestreaming setup over the past month; we’ve got everything up and running, and streaming to a Quest Pro headset. However, we haven’t been able to achieve the same visual quality as seen in your video: Depthkit Livestreaming Tutorial - Part 3: Peer-to-Peer Livestreaming with WebRTC - YouTube (the clip in the beginning with both of you, and also segment towards the end where Cory steps into the volume during the walkthrough).

In our setup, the system is picking up walls, floors etc. but we noticed that in your videos you have none of these issues. Any advice on how to achieve this on our end? We are aware of the limitations of the Lite Renderer and would also love to hear about anything in the works that might help us out.

Thanks!

2

@CornellTech You have a keen eye - We accomplished the background removal through a slight modification to the standard Depthkit Livestream workflow.

image
image1459×823 93.5 KB

Using TouchDesigner (with a commercial license to avoid resolution constraints), we

  • intercepted the Spout stream provided by Depthkit
  • cached a few frames of the stream of just the empty stage
  • blended the frames together
  • subtracted the result from the Depthkit stream’s depth pixels
  • fed that out via Spout to the Unity RenderStreaming sender in place of the original stream

You can find a TouchDesigner project configured to do the above here.

Additionally, we have found that materials like vinyl marley floor covering absorbs the IR signal from the sensors, voiding the depth data from any surfaces it covers.

3

@CornellTech You caught our little hack :slight_smile:

I’ll chime in with a little more detail from a product roadmap perspective on how we intend to address in the future.

We are tackling this issue in two ways:

Livestreaming Bounding Box in Depthkit
The new Depthkit 0.7.0 Pre-release introduces a bounding box to the Editor context to allow you to remove floor geometry at that stage, before export. The next step is to bring the bounding box into the Studio Calibration & Streaming context, which will allow floor removal for the livestreams right within Depthkit

Mobile optimizations
We are also doing R&D to improve the performance of our full-fusion renderer on mobile XR devices, to reduce the need for Studio Lite specifically Quest 2 / Quest Pro. The full fusion in Unity allows for the bounding-box based floor removal.

Let us know if the TouchDesigner and/or material solutions are workable for you in the meantime. Thanks for your patience!

4

@CornellTech - Depthkit 0.7.0 is now fully released, and includes the bounding box feature to remove the floor from recorded clips.

You can read more about some of these features in our latest blog post, and find more info in the announcement for the new update.