I am asking for a college development project about the DepthKit rendering a one-camera video on the HoloLens2 device.
The integration to Unity is fully functional, however, the DepthKit render does not appear when in the HoloLens device. We can run in the editor all of the animations and even interact with them when given a box collider. Then when deploying the Unity build to the HoloLens 2 device, we see no video appearing as we do in our Oculus environment.
I have the assumption that maybe certain settings are not loaded properly, or possibly the HoloLens 2 is having issues with rendering DepthKit, but I thought I would ask for some assistance. We should be seeing, in the image below, a render from our DepthKit integration on Unity. The render would be beside the breathing ring and would show the upper half of an individual but it seems to either not activate or doesn’t load into the HoloLens 2 device. I will note again that it does load and can be interacted with in the Unity editor but not the device.
Hi, @HILABMSU. Can you share more information about your project?
Which type of Depthkit assets are you trying to render? (Studio? Core/Cinema?)
Are you getting any errors, warnings, or other messages in the Unity console when running the scene in Play Mode on the computer?
Which version of Unity are you using?
Which render pipeline are you using?
Which Look component is applied to your Depthkit asset?
Which version of AVPro are you using as the video player for the Depthkit asset?
Are you able to get the Depthkit video to play as a 2D quad in the scene (bypassing the Depthkit renderer), or does that also not show up in the build?
For this testing, are you able to use Microsoft’s Holographic Remoting to render on the PC, and pixel stream to the Hololens? We have seen other teams succesfully integrate this into their Depthkit projects to performantly render Depthkit Studio assets on a Hololens.
Let me know the answers to the above, and we’ll see if we can narrow down the issue.
We receive one error but that is from the mixed reality toolkit and is due to a pathway that is too long for a cached item.
We are working with Unity 2020.3.21f1
We are using the 3D render pipeline, no URP or HDRP.
I believe it is Core Photo Look Built-in RP
I do not know which version as of now, I do not know entirely how I would verify what version we are using for the project.
I did not know we could render it as a 2D quad, but when trying the direct mp4 file it does not properly setup for our project. I did go for another way of setting up the DepthKit clip system and tested it in the hololens, but they do not seem to be appearing.
I did the debug mode with Holographic Remoting and still did not see anything appear, even if it did I feel that remote linking isn’t going to help since we do want the system on the HoloLens 2.
I tried some of the things you mentioned but we are still not seeing results of our Depthkit render appearing. I hope the info I provided helps.
Can you confirm which video player component is in your Depthkit object? Is it the standard Unity Video Player? If so, have you tried installing our depthkit.avpro package, and separately, AVPro (free trial and full verison available in the Unity Store) and using that as the Depthkit object’s video player type instead?
The goal of getting the Depthkit video play on a quad is to lets us know if the issue is occurring within video playback, or if it’s within the renderer. If we don’t know whether or not the video plays in your scene on a quad (bypassing the Depthkit renderer), then we cannot narrow down where the issue is occurring.
Also, do you have any more details about the Depthkit video? What is the resolution? Is it an h264 MP4 that was directly exported from Depthkit, or have you encoded/transcoded it to another file type and/or codec?
I do not know where to verify if the video is played as quad at this point and I do not know where to verify the resolution and codec of the DepthKit video. If that is in the editor I will relay to my team where to verify that information.
Do you guys have tutorials for the components of the video player and where these components are so I can get the right information to your side?
The team was recently on a break due to the college research atmosphere, sorry for the very late response.
@HILABMSU, no worries on the delay - We simply want to get you unstuck as quickly as is convenient for you.
The Depthkit video can used to texture a Unity object by assigning it to a render texture and material, and then connecting that material to the object, as in this tutorial. We recommend following this guide to assign the video as the texture for a quad. Doing so verifies that the video player is either working, or needs troubleshooting.
To know which video player component you are using, which Player is specified in your Depthkit Clip component? The default is Video Player (Unity) - see below.
If the Depthkit video was produced as a result of exporting from Depthkit, it is an MP4 container with an h.264 codec. If it was encoded any other way, you will have to have specified the container and codec format.
Other details about the video, like resolution, can be found by selecting it in Windows’ File Explorer and looking at the Details Pane:
I am going to try an use the AVPro Video player, but I can say that we are using the Unity Video Player and when loading onto the device the Quad View functions. We can see the MP4 play with the guided instructions you provided.
I also re-imported all the files and then checked the video MP4 we are using, our video sample is an MP4 file that comes with codec and everything, which means it is exported from DepthKit properly by the examples you provided.
Update Each time we try to use the AVPro Video we get grayed out selections in the upper window options, and Unity does not give AVPro as a video player options. We are trying the trial version with the packages provided and it should be working but seems to not be functional even following the steps from the DepthKit documentation for a re-imported setup.
