Multicam Depthkit alignment issues only in Unity

Depthkit Depthkit Expansion Package for Unity
, ,
1

I’m using a 5 kinect azure setup which is working fine for streaming. The calibration looks reasonable, but when I export to unity, the textures and mesh look very distorted. I’m using Depthkit 0.7.1, and the Unity Expansion Package with Depthkit Core Shader Graph Look.

From Unity
Screenshot 2023-11-14 210300

My unity project seems okay and it renders the sample footage from AutumnExport just fine. If it is my calibration file, why does it show up okay in Studio?

Any help would be appreciated!
Thanks!
~Sky

{
“_versionMajor”: 0,
“_versionMinor”: 5,
“boundsCenter”: {
“x”: 0.0,
“y”: -1.0,
“z”: 0.0
},
“boundsSize”: {
“x”: 1.0,
“y”: 2.0,
“z”: 1.0
},
“defaultReconstructionSettings”: {
“depthBiasAdjustment”: 0.0,
“edgeMaskBlurAmount”: 0.009999999776482582,
“edgeMaskSobelMultiplier”: 10.0,
“surfaceNormalColorBlendPower”: 2.0,
“viewDependentColorBlendPower”: 0.0010000000474974513
},
“format”: “perpixel”,
“numAngles”: 5,
“numColumns”: 3,
“numRows”: 2,
“perspectives”: [
{
“clipEpsilon”: 0.002180099952965975,
“crop”: {
“w”: 0.7326955795288086,
“x”: 0.13219758868217468,
“y”: 0.035388629883527756,
“z”: 0.6778416633605957
},
“depthFocalLength”: {
“x”: 961.8482055664063,
“y”: 961.5028076171875
},
“depthImageSize”: {
“x”: 2048.0,
“y”: 1536.0
},
“depthPrincipalPoint”: {
“x”: 1022.4246826171875,
“y”: 775.9476318359375
},
“extrinsics”: {
“e00”: 0.32978564500808716,
“e01”: -0.7017371654510498,
“e02”: -0.6315112113952637,
“e03”: -1.1290602684020996,
“e10”: -0.8146204948425293,
“e11”: 0.1265416145324707,
“e12”: -0.5660218000411987,
“e13”: -2.090745449066162,
“e20”: 0.47711098194122314,
“e21”: 0.7011078000068665,
“e22”: -0.5299178957939148,
“e23”: -0.7921983003616333,
“e30”: 0.0,
“e31”: 0.0,
“e32”: 0.0,
“e33”: 1.0
},
“farClip”: 2.8969361782073975,
“nearClip”: 0.6034635305404663
},
{
“clipEpsilon”: 0.0022770597133785486,
“crop”: {
“w”: 0.736912727355957,
“x”: 0.1822064220905304,
“y”: 0.26308730244636536,
“z”: 0.6660498380661011
},
“depthFocalLength”: {
“x”: 960.4542236328125,
“y”: 959.8101806640625
},
“depthImageSize”: {
“x”: 2048.0,
“y”: 1536.0
},
“depthPrincipalPoint”: {
“x”: 1014.5922241210938,
“y”: 776.8858642578125
},
“extrinsics”: {
“e00”: -0.02178417332470417,
“e01”: -0.9764485955238342,
“e02”: 0.21464768052101135,
“e03”: 0.4154072403907776,
“e10”: -0.7848621010780334,
“e11”: -0.11628890037536621,
“e12”: -0.6086611747741699,
“e13”: -2.0707054138183594,
“e20”: 0.6192874908447266,
“e21”: -0.18172800540924072,
“e22”: -0.7638441920280457,
“e23”: -1.2060859203338623,
“e30”: 0.0,
“e31”: 0.0,
“e32”: 0.0,
“e33”: 1.0
},
“farClip”: 2.7600317001342773,
“nearClip”: 0.5642176866531372
},
{
“clipEpsilon”: 0.0022342398297041655,
“crop”: {
“w”: 0.7237358689308167,
“x”: 0.1444859504699707,
“y”: 0.20954139530658722,
“z”: 0.703752875328064
},
“depthFocalLength”: {
“x”: 972.2930908203125,
“y”: 972.3392333984375
},
“depthImageSize”: {
“x”: 2048.0,
“y”: 1536.0
},
“depthPrincipalPoint”: {
“x”: 1024.6275634765625,
“y”: 777.5380249023438
},
“extrinsics”: {
“e00”: -0.4390701651573181,
“e01”: 0.610372006893158,
“e02”: 0.659290075302124,
“e03”: 1.1165591478347778,
“e10”: -0.8547919988632202,
“e11”: -0.057819705456495285,
“e12”: -0.5157397985458374,
“e13”: -1.9297479391098022,
“e20”: -0.27667316794395447,
“e21”: -0.7900018095970154,
“e22”: 0.5471280217170715,
“e23”: 0.9779639840126038,
“e30”: 0.0,
“e31”: 0.0,
“e32”: 0.0,
“e33”: 1.0
},
“farClip”: 2.869664430618286,
“nearClip”: 0.6317668557167053
},
{
“clipEpsilon”: 0.002484434051439166,
“crop”: {
“w”: 0.6172139048576355,
“x”: 0.09190216660499573,
“y”: 0.2402353733778,
“z”: 0.684151291847229
},
“depthFocalLength”: {
“x”: 969.7317504882813,
“y”: 969.3528442382813
},
“depthImageSize”: {
“x”: 2048.0,
“y”: 1536.0
},
“depthPrincipalPoint”: {
“x”: 1025.77685546875,
“y”: 779.6158447265625
},
“extrinsics”: {
“e00”: 0.04932315647602081,
“e01”: 0.9367123246192932,
“e02”: -0.3466082215309143,
“e03”: -0.4836212694644928,
“e10”: -0.9053907990455627,
“e11”: -0.10459019243717194,
“e12”: -0.41149529814720154,
“e13”: -1.946414589881897,
“e20”: -0.4217045307159424,
“e21”: 0.33411213755607605,
“e22”: 0.8429319858551025,
“e23”: 1.4360320568084717,
“e30”: 0.0,
“e31”: 0.0,
“e32”: 0.0,
“e33”: 1.0
},
“farClip”: 2.773815155029297,
“nearClip”: 0.761284351348877
},
{
“clipEpsilon”: 0.0033333436585962772,
“crop”: {
“w”: 0.9978224635124207,
“x”: 0.0,
“y”: 0.0,
“z”: 1.0
},
“depthFocalLength”: {
“x”: 971.4751586914063,
“y”: 971.258544921875
},
“depthImageSize”: {
“x”: 2048.0,
“y”: 1536.0
},
“depthPrincipalPoint”: {
“x”: 1024.15234375,
“y”: 777.4072875976563
},
“extrinsics”: {
“e00”: -0.021854449063539505,
“e01”: -0.9467461109161377,
“e02”: -0.32123851776123047,
“e03”: -0.3327024281024933,
“e10”: -0.9907816648483276,
“e11”: 0.0634777843952179,
“e12”: -0.11967542767524719,
“e13”: -1.151632308959961,
“e20”: 0.13369375467300415,
“e21”: 0.31566181778907776,
“e22”: -0.9394059777259827,
“e23”: -1.0430679321289063,
“e30”: 0.0,
“e31”: 0.0,
“e32”: 0.0,
“e33”: 1.0
},
“farClip”: 1.854885458946228,
“nearClip”: 0.3548901379108429
}
],
“textureHeight”: 2048,
“textureWidth”: 2016
}

2

Hi, @SkyRolnick - Sorry to hear you’re getting poor results.

First, ensure you’re using the Depthkit Studio (not Core) Shader Graph Look in Unity to render any clips captured with multiple sensors. Follow this dependency chart to ensure you’re importing the packages in the correct sequence.

If that doesn’t solve the issue, can you confirm that the sensors have been in the same positions for the calibration and every capture in that project?

Are you able to share the Combined per Pixel Video itself as well as its metadata? We can take a look and see if we spot any causes of the misalignment.

3

@SkyRolnick - Following up to see if you’re still experiencing this issue. If so, please answer the questions above. Otherwise, I will close the ticket.

4

Hi Cory,
Thanks for the response and sorry about my slow reply. Indeed I was using the Core and not the Studio.

I’ll also just make a note here in case it helps anyone else, when I initially tested the Studio prefabs, they did not show for me but Core did, leading me to mistakenly assume I should be using Core. In order to see the Studio prefabs, I also needed to switch Render Pipelines to Standard.

In general, I find the use of Shader Graph, VFX, and different render pipelines very confusing. For example, sometimes VFX don’t show because they have not been compiled, etc. Is there any helpful documentation you can share in best practices for working with depthkit in different render pipelines?

Also, I see that Studio only provides shaders for Built-in RP. Is there a workflow to get special VFX working with Studio?

Thanks for your help!
~Sky

5

Hi, Sky.

When using the Built-in Render Pipeline, you use the prefab contained in the standard Depthkit Studio package, which includes a photo look for that render pipeline.

For either of the scriptable render pipelines (URP or HDRP), you’ll need to use the prefabs found in the Depthkit Studio Shader Graph (for photo looks) and/or Depthkit Studio VFX Graph (for VFX looks) packages. If those are not compiling, be sure that you have added the Core version of each package before adding the Studio, as there are dependencies.