I’d like to ask you about a huge issue. So far, during all this time working with depthkit studio I have never been able to get fingers of a human captured in full-body scale. Sometimes it’s acceptable with some projects, but recently it became unacceptable for some AR lectures we’re making today.
So maybe I’m doing something wrong with refinement in Depthkit or in Unity. But have you ever got the good fingers of a full-body human?
Please check out a full-res CPP poster. Ten cameras placed as close as possible. As you see, all fingers are clean and nicely separated. When I decrease Weight Unknown, volumetric fingers do appear, but then all this junk appears too. So what would you recommend to do to get nice fingers of a full-body capture: maybe camera placements, or refinements? The rest of the quality is good and acceptable, it’s only fingers.
@NikitaShokhov Thanks for sharing. The quality of the capture and the mattes looks very good from the still frame, so you should be able to get details like fingers back.
I noticed in the screen recording you sent us that the Volume Density for this asset was set to 70, which makes sense for performance purposes when rendering in AR on mobile devices, but be aware that lower volume densities like this make it less likely that small details like fingers and facial features will be rendered properly. One alternative workflow that gets around these performance challenges is our Studio Lite renderer, which if using the Built-in RP, may be worth testing.
The Surface Sensitivity was also set quite high. If decreasing the Weight Unknown values is producing lots of stray geometry, bring the Surface Sensitivity down somewhat to carve away some of those artifacts. Also adding a small amount of Surface Smoothing will help get rid of the smallest floating bits, but setting this too high will also round off fine details and fingers.
Can you share the metadata for this asset? I can take a quick pass at the volume settings to see if I can find a good combination of settings to get the fingers back and share here.
I tried slightly increasing Volume Density and decreasing Surface Sensitivity while decreasing Weight Unknown. It helps indeed very much. And facial features became more developed too. I haven’t tried this performance on a mobile device yet, but at least it looks good in Unity.
As you see, fingertips are still barely elaborated when fingers are apart from each other, and there’re lots of artifacts around the fingers. I sent you a screen capture with these settings, a video clip, and metadata privately for you to try. So maybe you know how to eliminate some of these artifacts and better develop fingertips while keeping the cleanliness of the rest of the body?
@NikitaShokhov I took a pass at the reconstruction settings, and it’s definitely tough to get back high-frequency details like fingers at this volume density. See the screenshot below for the settings that look the best to my eye. Of note:
I increased the Depth Bias from the default of 0.7 to 1.5 (as the depth data of skin, including hands, is prone to error in the Azure Kinect)
I adjusted the Weight Unseen Falloff Power along with Surface Sensitivity and Weight Unknown.
Smoothing is just shaving the fingers off, so I would either keep it at it’s minimum setting, or disable it.
(Compare to a higher density like 150, which has an easier time with fine details with some adjusted setting, but at the cost of performance - 170,000 triangles vs 45,000)
I’ll also worth noting that what looks good in a still frame doesn’t necessarily look good in motion, and that if you haven’t already, setting up the clip on a timeline to be able to quickly queue up and play back specific sections of the clip is extremely helpful.
I hope this helps. Let me know if there’s anything else I can help with.
