and set the zoom lens to 16 mm (because the sensor is 36*24 mm).
Then we created succesfully the lens correction profile which we could succesfully use in the calibration step inside of Depthkit, when doing the pairing process.
We have tried the pairing three times. Two times with another rig construction (where the camera lens and the sensor lens were exactly vertically above each other, but quite far away. Then, because the pairing result was always “BAD”, we changed the rig-construction to attached photo.
We again started the calibration: always covering all four quadrants of the image, at three different distances from the calibration chart. But still the result says “BAD”.
The calibration chart stood always at the same position and rotation. Untouched. We only moved and rotated the camera rig in space. Maybe we should have kept the camera rig untouched and just move the calibration chart in space, covering both lenses and quadrants?
Also we had a hard time seeing the white alignment indicators, because the contrast and exposure may have been too high. But that worked best for detection the chart.
In 13 of the 19 samples, the calibration board is clipped by the edge of the depth camera’s frame/vignette, leaving only 6 valid samples. The entire pattern needs to be in frame for it to successfully be detected and registered. In the Pairing documentation, we recommend setting the depth sensor to the WFOV Raw mode to avoid this during pairing, then switching it to NFOV when it comes time to record a capture.
The Pairing Sample images do indeed look a bit overexposed, and many of them are not sharply focused on the calibration pattern. If the corners of the squares are not in sharp focus, Depthkit cannot detect their position, so make sure you re-focus the lens each time you move or rotate the camera/chart before capturing that sample.
The Sony video files in the CLIP folder - which appear to be the video clips from which the Pairing samples were pulled - are 1920x1080 resolution. Not only is this a lower resolution than the RGB video from the Orbbec sensor, but it is also harder for Depthkit to detect the calibration markers at 1080p than at a higher resolution like 2160p. If this is how the camera recorded them, I recommend retrying the entire Lens Profile & Cinema Pairing process using a higher-resolution codec like XAVC S-I 4K. If the camera recorded at a higher resolution like 4K, it’s possible that this downscaling took place during transcoding, so check your transcode settings to ensure that the original resolution is preserved.
Not critical in this step, but I also noticed the Sony video files are recording at 59.94fps. For best results with recordings, select a framerate which most closely matches the sensor’s 30fps framerate, like 29.97fps.
Making these adjustments to resolution, exposure, focus, and framing should provide much better result on the next attempt. If you continue to get poor results, feel free to send an updated project folder with all of the materials you sent this time, but also include:
The LCP/XML lens profile file
The images used to generate the lens profile
The Cinema camera video which accompanies any recorded takes.
We rebuilt the entire camera rig, so it’s mor similar to yours and the sensor and cinema camera are very close.
Recalibrated everything. The scores are now better.
But lens coverage will most likely never be better, due to the fisheye-lens. The lens coverage of the rgb femto bolt will be almost 100%. But the cinema camera lens coverage (fisheye) will not be. See overlayed image of the fisheye-videos:
These are all improvements, even if some metrics seem to indicate otherwise. Most importantly, how does the texture from the camera align with the mesh when you enable Cinema on a recording? How it looks visually is more important than any of these metrics, so if things align to your eye, stick with that pairing.
Apologies for not answering your question in the initial post - The breakdown is as follows:
Pairing Sample 14 - the ID of the sample, in this case 14 - gives a quick visual ranking of the error score of that sample based on the current solution. Brown/Red is bad; Yellow/Green is good. 9.7616 - the numerical error score of that sample based on the current solution. Lower is better. 65 - The number of grid crosspoints detected in the sample. Higher is better. Low indicates the sample was too far/blurry/distorted/bright/dark to be fully detected. - For any samples with a particularly high error score and/or low crosspoint count, you can manually exclude each of them by unchecking this checkbox, then clicking ‘Pair Camera’ again to see if the scores for the remaining samples improve.
- gives a quick visual ranking of the error score of that sample based on the current solution. Brown/Red is bad; Yellow/Green is good.
- For any samples with a particularly high error score and/or low crosspoint count, you can manually exclude each of them by unchecking this checkbox, then clicking ‘Pair Camera’ again to see if the scores for the remaining samples improve.