Hi DK!
Say I have a clip of a character idling and a clip of a character saying something, and I want to go into a sequence like:
Idle > Action > Idle
Or even
Idle > Action > Walk somewhere > Idle
If those were shot as 3 or 4 separate clips, is there a way I could morph them together to create the illusion of a single shot? The aesthetic is already pretty abstract so I’m not concerned with photorealism or melting faces at this point.
Hi Max! Great question.
There are a few ways to create transitions between two Depthkit clips in Unity.
If you’re intending to seamlessly stitch two clips together so that you can load them as a single clip in Unity you are best off doing that in a NLE like After Effects or Premiere. There are three primary ways to do that.
Always be aware (Edit: Updated link) of your export settings when re-encoding Depthkit clips to avoid introducing artifacts in Unity.
If your goal is not necessarily to stitch them together, there is a dynamic way to do it that makes use of Depthkit Looks. If your idle loop uses the photo Look, when you need to trigger a transition into an action, you fade the Depthkit clip into particles then reduce the opacity, while simultaneously starting your second clip and fading it in using the inverse transition. This is the more technically complex approach because it would require you to build the transitions in code.
These techniques are useful for loop-closure in order to create an idle loop. In order to do that, you would put a single clip end to end in the timeline and create the transition in and out of itself.
These techniques can also be used if you’re trying to morph in or out of an idle loop to a specific action like you asked (e.g. Idle => Action => Idle) where each of those is a different Depthkit video clip. In that situation, you’ll not always be guaranteed that the action will match if you trigger it dynamically – especially if you have a very lively idle loop – but it can allow you to film your actions piecemeal and stitch them together dynamically in the scene. To do that, you would create a transition on your Action that starts in a general/central pose in your Idle loop and morphs into the Action, then back into the idle loop again at the end.
The way you would accomplish that is to lay your Loop clip, then your Action clip, then your Loop clip again in a timeline and create transitions at the cuts. Then you export only the Action clip which should now have transitions baked into the beginning and end.
I’d like to do the first way described here but it’s missing the next steps to be able to export the longer clip from Depthkit. How do you do that? If I modify the videos (one video per sensor), Depthkit won’t recognize the new clips because they won’t fit the length of the depth files… maybe it’s no longer relevant to the latest version?
Or am I missing something?
@Julien-RobertLegault , I discussed this a bit with Patrice yesterday, but here’s a brief recap:
First, combining/editing clips together only works with image sequences which have been exported with the same calibration and near- and far-plane settings per sensor, as the resulting sequence can only reference one metadata file.
FFMPEG is the only only workflow we have validated to preserve the color in the depth map accurately to avoid distorted geometry in Unity. The only way to “edit” with FFMPEG is with the source image sequences (e.g. combining image sequences from two recordings into one folder, and batch renaming one of the sequences so that they share the same prefix, and that the suffix is one unterupted sequence of numbers - tedious and grueling, but it works). You can use this technique to combine multiple clips with hard cuts, extend one clip, and even reverse sections (by reversing the sequence numbers with the renamer), but that’s about it.
Alternatively, some NLE’s (Premiere, Resolve, Media Composer, etc.) may have the export options necessary to preserve the color and resolution, though we haven’t investigated or validated them ourselves. If you would like to test any of these workflows yourself, import your CPP Image Sequence into the software of your choice and put it on a timeline which:
When exporting, look for options that specify the following:
To verify the color space of your exported video, use the ffprobe command in our documentation.
If your exported video meets all of the required specifications, feel free to use that NLE with those export settings to edit your footage, add audio, use effects (be careful with the depth maps!) including transiotions like dissolves and morphs, etc. Be sure to share any of these workflows so we can validate them and add them to our documentation.