Read on Twitter
This stuff blows me away and I definitely think it’ll play a role in the future of game production. That said, I think it’s worth talking about the challenges of implementing full body 4D capture in a game pipeline right now. 1/ https://twitter.com/8Infinite8/status/1333752668679008257
One of the big things that gives 4D captures a sense of reality is that, at least as I understand it, you’re storing geometry and texture data for each frame. 2/
Storing corresponding texture data is especially important with surfaces like skin, where skin stretches and blood gets pushed to different areas, changing the look of the surface as the character moves. 3/
So instead of storing a single texture (or small number of textures), per character, it’s like storing a video, per animation. That’s a lot of data to store and process, and historically one of the big challenges with 4D capture in games has been how to compress that data. 4/
Another big challenge is the capture area. You’ll note in all the examples you’ll see of this, that the motion only takes place in a relatively small area. That’s likely to keep the cameras close to the performer to maintain a good resolution on the scan. 5/
Also, it could just be that covering a large area with expensive cameras and lighting is crazy expensive. 6/
With traditional mocap, the size of the capture volume is important. We need to be able to capture fast locomotion moves and that requires space. You can use a treadmill, but it doesn’t give the same effect and doesn’t work well for starts, stops, plant and turns, etc. /7
Another huge challenge with 4D captures is that they aren’t skeleton based. In games, we’re heavily reliant on the skeleton for a lot of things: When we blend from one animation to another, you need an idea of the skeleton structure so that the verts can interpolate correctly. /8
With skeletal animation, bones rotate relative to their parent joint, and vertices then move relative to the bones. Without that structure, the vertices would each interpolate independently, and you wouldn’t have a cohesive mesh during blends. /9
We also use the skeleton for animation layering, runtime constraints, procedural motion adjustments (IK, head look, slope adjust, foot locking), motion matching pose features, reference points for gameplay, (e.g. the origin for raycasts), etc. /10
You could potentially decompose this scan data and use it to build an animatable rig, which is how a lot of AAA character rigs work these days. /11
You extract the extreme poses and save the mesh shapes and texture information relative to the pose, then blend between those shapes at runtime, as the character changes poses. /12
That misses a lot of the nuance of the scan though, as it's just blending between poses and doesn’t save motion. You’re moving from an accurate point A to an accurate point B, but the motion in between wouldn’t be right. /13
E.g. You wouldn’t see the same follow through on the coat, or jiggling of muscle and fat, or see exactly at what point clothing folds start to come in as a body part is bent. /14
Another option would be to use a skinning converter to convert the mesh motion to skeleton motion ( https://vimeo.com/123883474 ). /15
This would work well during any non-interactive moments, but would lose some of the nuance when you start deviating away from the initial scan (e.g. during blends, layering, etc). /16
Another way to go would be to use Machine Learning. Feed a neural network with enough of this data, and it might be able to figure out how to convert runtime motion into an output mesh and texture for the next frame. /17
That's not an easy problem to solve though, and would take a LOT of input data. We'd need to get the scanning process to where it's fast enough and consistent enough, to acquire enough data for training. /18
So again: Super, super cool stuff, and it makes me excited for the future, but I think at least for now, this stuff will likely be of more use for vfx companies or other offline character motion, rather than games. /19
Should also note, there's a good chance there are people reading this who know way more about it than I do, so if any of what I posted above is wrong, feel free to set me straight. /20
