WuMap

The WuMap project applies the techniques of Isomap to a variety of problems in Computer Vision and Robotics. Isomap is a technique vaguely similar to PCA, but supports non-linear dimensionality reduction. Isomap has been used to discover the meaningful degrees of freedom for complicated natural data sets, including images of faces from different viewpoints and human handwriting samples. Because is uses only similarity measures between input data points, it is ideally suited to problems in multi-sensor fusion.

Live interactive Demos!

Test Data Archive

Video Analysis with WuMap

Here we propose to consider a video as two parts -- a low dimensional image space and a trajectory through that space. Analysis of the shape of the video trajectory gives new tools for video analysis.

Sample Trajectories

The following shows the automatically created trajectories for a number of video sequences.

Any completely periodic sequence (Cyclic) Completely repetitive video sequences have trajectories which are embedded as ellipses in a two-dimensional space. Images which are seperated arbitrarily in time may be identical because the scene in view is changing in a periodic manner.
A bird flying across the sky (sample 10Mbytes AVI) (Helical) A periodic action being viewed by a moving camera is characterized by similarity between local images separated by the period of the action, but a drift over time in appearance of the object even in the same phase. This leads to a helical structure in the trajectory.
A bird flying then gliding (sample 10Mbytes AVI)) Video sequences may have smooth transformations between pieces that fit cleanly into the one of the above categories.
Fountain Sequence (used "original" from Dynamic Textures work at UCLA) (Knotted) Dynamic textures (fountains, smoke, flames, and natural motions of trees in the wind) are characterized by a non-periodic sampling from a limited image space.

Applications of Video Analysis with WuMap

The video trajectory is a representation of a video as (1) an image space and (2) a trajectory through that image space. One application of this representation is "Temporal Super-Resolution", or, adding new frames between frames in the original sequence by intelligent resampling of the rest of the video. The algorithm works as follows:

(1) Make the video trajectory:
  (a) directly compare the images.
  (b) Use Isomap to embed the images 
      in a low (say, 3) dimensional space.
  (c) Draw the spline curve between the 
      images in order.

(2) Then, as illustrated to the right, add extra points
    along the spline curve.  Shown here are 4 extra
    points between original frame 61 and 62.  For each
    extra point, find the closest original point.
    Those original points correspond to other images in
    the video that can be swapped in to smoothly
    interpolate between frames 61 and 62.

Results: From a sequence of a bird flying, the following shows 19 images inserted between two frames of the original sequence (some inserted frames use the SAME original image). This is contrasted with a smooth blurring between the original images (the bottom layer).

Additional examples (WuMap interpolation on left, linear interpolation on right):
A video with 5 frames inserted between each original frame: (download 10Mbytes, AVI).
A video with 20 frames inserted between each original frame: (download 10Mbytes, AVI).
A video with 20 frames inserted between each original frame: (download 1Mbytes, AVI (Cinepak)).

Video Trajectories from movie scenes.

(download 0.8Mbytes, AVI).
Earlier projects: Pose estimation from image data sets.

Open offer

We are in the process of creating and releasing a set of matlab routines which fully automate the process of creating a video trajectory. In the meantime, we will gladly create video trajectories or temporal super-resolution from you video, send e-mail to pless@cs.wustl.edu to inquire.