May 31, 2010

Combine Multiple Videos Together

Sometimes you want to combine 2 video files into 1, either for general video editing, or to compare two videos side-by-side, or to play one video and then another video when it has finished. While you can use professional video editing software to do these and many more functions, it is nice to be able to do this easily, or from a command-line script, or from within your own programs. Therefore I have created two open-source (freeware) programs that will combine videos, either next to one another, or after one another. Note that these programs can not process audio, so you will loose the soundtrack when you use these.

"CombineVids" will put the videos next to each other at the same time (also called Picture-In-Picture).


"AppendVids" will attach the second video onto the end of the first video.


How to combine 2 videos next to each other (PiP):

Here is a sample OpenCV program that puts 2 videos next to each other, so you can see both of them playing at the same time. The inputs may be AVI videos or MPG/MPEG videos.

The command line syntax is:
    CombineVids <input_video1> <input_video2> [output_video] [--VERTICAL] [--FPS <fps>]

The source-code to my program is available here (opensource freeware), to use as you wish for non-military projects only.

Click here to download "CombineVids": combineVids.7z

(19kB 7-Zip file, including C/C++ source code, VS2005 project files and the compiled Win32 program, created 27th Apr 2010).

If you dont have the freeware 7-Zip program (better than WinZip and WinRar in my opinion), you can download it at http://www.7-zip.org/download.html.

And if you dont have the OpenCV 2.0 SDK then you can just get the Win32 DLLs for running this program here: openCV200_Win32DLLs.7z (1.3MB 7-Zip file).

How the videos are combined next to each other:

The basic method of rendering both videos next to each other is shown here:
// Open the input video files or camera stream.
CvCapture* capture1 = cvCaptureFromAVI( argv[1] );
CvCapture* capture2 = cvCaptureFromAVI( argv[2] );
// Create a new video file for output.
CvSize combinedSize = [Big enough to fit both video frames next to each other, plus a border]
CvVideoWriter* videoWriter = cvCreateVideoWriter(outputFilename, CV_FOURCC('D','I','V','3'), FPS, combinedSize, TRUE);
IplImage *frameOut = cvCreateImage(combinedSize, 8, 3);  // Create an empty RGB image for storing the combined frame.
IplImage *frame1, *frame2;

// Process both video streams while atleast one is still running (AVI or MPG file or camera stream).
frame1 = (IplImage*)1; // Enter the loop.
frame2 = (IplImage*)1; // Enter the loop.
while (frame1 || frame2) {
 // Get the next video frames.
 frame1 = cvQueryFrame( capture1 );
 frame2 = cvQueryFrame( capture2 );
 if (frame1 || frame2) {
  // Combine the 2 image frames into 1 big frame. 
  frameOut = combineImages(frame1, frame2);
  // Store the combined video frame into the new video file.
  cvWriteFrame(videoWriter, frameOut);
 }
}

Append a video on the end of another video:

Here is a sample OpenCV program that attaches one video onto the end of another video, so you can watch one video after the first one has finished. The inputs may be AVI files, MPG files, folder of JPG images, or even a camera stream.

The command line syntax is:
    AppendVids <input_video1>|<image_folder1> [<input_video2>] [output_video] [--FPS <fps>]

Note that if you are using image folder mode, the syntax is in the printf() format. For example, "AppendVids frames1\img%03d.jpg frames2\img%d.jpg out.mpg --FPS 25" will generate the video "out.mpg" from the files "frames1\img001.jpg", "frames1\img002.jpg", ..., and "frames2\img1.jpg", "frames2\img2.jpg", ...

The source-code to my program is available here (opensource freeware), to use as you wish for non-military projects only.

Click here to download "AppendVids": appendVids.7z

(19kB 7-Zip file, including C/C++ source code, VS2005 project files and the compiled Win32 program, created 27th Apr 2010).

If you dont have the freeware 7-Zip program (better that WinZip and WinRar in my opinion), you can download it at http://www.7-zip.org/download.html.

And if you dont have the OpenCV 2.0 SDK then you can just get the DLLs for running this program here: openCV200_Win32DLLs.7z.

How the videos are joined one after the other:

The basic method of using the first video and then using the second video is shown here:
// Create a new video file for output.
CvVideoWriter* videoWriter = cvCreateVideoWriter(outputFilename, CV_FOURCC('D','I','V','3'), FPS, size, TRUE);
// Open the input video files or camera stream.
CvCapture* capture1 = cvCaptureFromAVI( argv[1] );
CvCapture* capture2 = cvCaptureFromAVI( argv[2] );
IplImage *frame;

// Process the first video stream (AVI or MPG file or camera stream).
frame = (IplImage*)1; // Enter the loop.
while (frame) {
 // Get the next video frame.
 frame = cvQueryFrame( capture1 );
 // Store the video frame into the new video file.
 if (frame) {
  cvWriteFrame(videoWriter, frame);
 }
}

// Process the first video stream (AVI or MPG file or camera stream).
frame = (IplImage*)1; // Enter the loop.
while (frame) {
 // Get the next video frame.
 frame = cvQueryFrame( capture2 );
 // Store the video frame into the new video file.
 if (frame) {
  cvWriteFrame(videoWriter, frame);
 }
}

Detect a person's t-shirt color, to aid Face Recognition or People Tracking

The task of "Face Recognition", where you want a computer to figure out who a person is from a photo of them, is a very difficult function to do well. But in some cases such as robotics, it may not be so important to figure out exactly who it is, you might just want to get an idea of where the people are, or to notice when the camera sees the same person again after the camera has moved around in the room. These are cases when Shirt Detection can be used as a simple method to keep track of who is in the room. For example, a robot could just keep track that there is a person wearing a red shirt to its left, a person wearing a blue shirt in front, and a person wearing a yellow shirt on its right side, so that it can track where these people are when they move around or when the robot moves around.

Shirt Detection can be performed quite easily compared to Face Recognition, by using OpenCV's very reliable Face Detection. Once the program knows where a person's face is, it can look in a small region below the face (where a person's shirt would be), and determine the approximate color of their shirt in that shirt region. This can be performed even without a complex method of determining the exact contour region of a person's shirt, since all you need to know is the color of a region below their face, as opposed to the color of their entire shirt or body.

Here you can find a program I have created (named "ShirtDetection") that detects the shirt color of multiple people in a photo. As a demo, here is the input photo:

The program will convert each pixel into an approximate color type (eg: Green or Orange or Purple or Black, etc), based on its HSV color components. In the image below, you can see that some pixels are converted to red, some to orange, some to green, etc.

It will then perform Face Detection to find where the faces are within the original image. For each face that it finds, it will look at the approximate color types in a small rectangle region below the detected face. It will then see if there is a certain color type that is most dominant in that small region, such as if 60% of the pixels in the shirt rectangle are detected as purple pixels, then it will classify the person's shirt as Purple, with a 60% confidence rating. You can see the final image below, where the detected shirt colors are overlayed on top of the person's shirt.

In most photos, it is reliable enough for simple uses such as Human-Robot-Interaction, where a false detection is not such a problem. Note that even though it appears as if image segmentation was performed, there was no segmentation or contour detection being done. It is simply the result of converting each pixel into an approximate color type.

ShirtDetection:

The source-code to my program is available here (opensource freeware), to use as you wish for non-military projects only.

The shirt detection process only takes a few milliseconds to execute, but also does face detection, which can take upto 1 second, depending on the image size. Also, it generates a "Color Image" window so you can see the detected colors of all pixels, for debugging. You can disable that feature by commenting out "SHOW_DEBUG_IMAGE" in the source-code, to run faster.

Click here to download "ShirtDetection": shirtDetection.zip

(182kB Zip file, including C/C++ source code, VS2008 project files and the compiled Win32 program, created 9th Nov 2009).

And if you dont have the OpenCV 2.0 SDK then you can just get the Win32 DLLs for running this program here: openCV200_Win32DLLs.zip (1.9MB Zip file).