Update Report

Summary of Work to Date:

Setting off to implement a Cinemagraph generator using the two papers [1] [2] as primary references, following is a list of work done:

• Going through the papers' methods, two main steps have been outlined for generating semi-automatic / automatic cinemagraphs, namely, Warping and Compositing. The former step (also the one that I majorly studied hitherto) requires use of KLT feature tracking. To understand the working of the tracker, I read through topics including Corner Detection, Homography, RANSAC and Optical Flow. I experimented with sample codes provided by OpenCV to understand how they work and how would I use this existing implementation towards the project goal.

Optical Flow example (1):
The lines denote the direction of motion of pixel under the dot

Optical Flow example (2):
HSV flow visualization

KLT Tracking example:
Detects good-tracking-points and tracks them across frames.

As a part of the intermediate tasks, I was able to immobilize a hard-coded region in a video to create the Cinemagraphs shown below using OpenCV-Python and MoviePy. MoviePy is a Python module for video editing, which can be used for basic operations (e.g. cuts, concatenations, title insertions), video compositing, processing, or to create advanced effects. It can read and write the most common video formats. 

These are composed by cropping off hard-coded portions of the input videos and compositing the video frames along with a masked snapshot of a still frame (at t=0). As assumed in the beginning of the project, the input videos are stabilized. 

Only the water reflections are mobilized
[YouTube]

Only the region near the rock is mobilized
[YouTube]

Analysis of Work:

Following is the task board [see here] marked with sub-tasks completed and to-be-done to put the progress in perspective:  

Plan for Completion:

So far, I am off to a rough start due to underestimation of prerequisites for the implementation of the methods described in the paper. As mentioned earlier, I was caught up with the concepts taken as known in the papers thus keeping me from completion of all the intermediate goals. 

Final Tasks until December 12th:

Before wrestling the final goals, I may work the backlog task to immobilize / mobilize the user selected area on a reference frame while generating cinemagraph. This should be a good intermediate step before the real implementation, which will be gone though if time permits.

Nonetheless, here's a list of final goals:

Warping:

• Implement a spatially varying warp to have the motion of a region of video removed. The region can be defined either:
• apriori, or
• by user input (de-animating strokes).

• This will be done in two stages:
• track the "anchor" tracks (the KLT tracks that fall under the defined region on the reference frame) and warp the video.
• track the "floating" tracks (the tracks that fall under the defined region, but not on the reference frame) and do a warp refinement over the first step.

Composition:

• The previous step is bound to introduce unwanted warping in the regions not selected for immobilizing. To keep the pixels other than in ROI intact, we need to composite the warped video with static still frames from input video. 

• The paper uses graph cuts to perform Markov Random Field (MRF) optimization, to seamlessly composite dynamic regions from the warped video with static regions from input frames into a single output video.
• To tackle the composition in the given time frame can be challenging. Thus, I plan to composite the warped video with a hard-coded static frame from the video for final generation of cinemagraph.
• Additionally, if this plan works out better than expected, I plan to study the MRF optimization for seamless composition, still frame candidate selection, energy function and the Seam Cost in it, and how to selectively assign the source of a pixel (warped video or static frame) in the output video that respects the user defined strokes.

Kbeznak Parmatonic