Vision-based Modeling of dynamic Appearance and Deformation of Human Actors

Abstract:

The goal of the proposed research is to capture a concise representation of
human deformation and appearance in a low-cost environment that can be seamlessly integrated
into entertainment and Virtual Reality applications–applications that often require
relighting, high levels of realism, and rendering from novel viewpoints. Geometric modeling
techniques exist in the current state of the field, but they focus solely on
geometric deformation and ignore the appearance component. I propose
a combined model of both geometric and appearance deformation, leading
to a more compact representation while still achieving high levels of realism.
Furthermore, many existing methods require input data from artists or expensive
Laser scanners. The proposed vision-based method is less expensive
and requires only limited manual intervention.

In a controlled environment, several cameras will be used to acquire a dense
static geometry and a basic appearance model of a street-clothed human (e.g., a texture mapped model).
A skeletal model is manually aligned with the existing geometry,
and the actor performs a sequence of predefined motions. Multi-image vision-based
tracking techniques are used to extract the kinematic motion parameters that account for the
gross motion of the object. Multi-view stereo and silhouette cues, and temporal
coherence are used to extract the time-varying residual geometric deformation. These time-dependent
geometric quantities, and the time and view-dependent photometric quantities are used to create
the final compact model of appearance and geometric variation. Following several graphics techniques
for modeling articulated deformation and leveraging existing tools for image-based rendering,
the deformation model is built on top of linear blend skinning. Appearance and geometric variation
are modeled together as a function of abstract interpolating parameters
that include, e.g., relative joint angles and view angles. Examples of deformation include muscle bulging
and some forms of cloth budging. We model this compactly using a dimensionality
reduction and sparse data interpolation, where several low-dimensional subspaces and corresponding
interpolation spaces are used, effectively clustering
portions of the surface that are affected by the same abstract interpolators. Contrast to many of the
vision-based human modeling techniques, our complete model can be re-rendered under novel viewpoint,
novel animation, and even novel illumination when the illumination during capture is calibrated

The specified model also has potential applications in the use of appearance and
deformation transfer between different subjects. Furthermore, the combined geometric and appearance
model will almost directly transfer to similar domains, such as modeling hand deformations, or
related domains, as facial deformations. An improved model of geometry and appearance could also
be used to improve markerless vision-based motion capture. This work also hopes to better identify the
range of clothing and deformations that can be both captured and satisfactory modeled as a function
of relative joint angles neighboring rigid body velocities and other related external factors, such as direction
of gravity, and wind.