The basic process flow goes like this. Clicking the ‘Capture’ button on the user interface calls an IBAction called -saveAnimation. It prompts the user to select an output file for the movie. When the user has selected the file, the animations are created and added to the layer. Next we create a timer that is going to call a function that will grab the current frame and place it into the QTMovie object using -addImage at a specified interval. We set our AppDelegate to also be the delegate for the animation group so that when the animation completes we get notified and can then write our QTMovie object data to disk.

Use An Interesting Animation

First you are going to need an animation that is worth recording. Of course any old animation will do, but we’ll keep it interesting by adding multiple animations to a single layer. I have created four different keyframe animations that we will add to an animation group. The keypaths are “backgroundColor”, “borderWidth”, “position”, and “bounds”. Check the sample code to see how these animations are constructed.

We set the duration for all of the animations to five seconds. We also need to make sure that we set the duration for the group itself, otherwise it will override the five second duration we set for the animations themselves and run in the default 0.25 seconds. The code below shows how the animations are added to the layer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

- (void)loadAnimations;
{
  CAAnimationGroup *group = [CAAnimationGroup animation];
 
  [group setAnimations:[NSArray arrayWithObjects:[self backgroundColorAnimation],
                          [self borderWidthAnimation],
                          [self positionAnimation],
                          [self boundsAnimation], nil]];
 
  [group setValue:@"mainGroup" forKey:@"name"];
  [group setDuration:5.0];
  [group setAutoreverses:YES];
  [group setDelegate:self];
 
  [layer addAnimation:group forKey:@"group"];  
}

Notice that we have used KVC here to set a name for the animation group. We will use this as a tag later to make sure the animation that triggers our -animationDidStop:finished animation delegate is the correct one. More on that later.

Get Our Movie Ready

Prior to loading the animations, we prompt the user to select a file to write the movie file to. After loading the animations, we start our timer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

- (IBAction)saveAnimation:(id)sender;
{
  NSSavePanel *savePanel;
 
  savePanel = [NSSavePanel savePanel];
  [savePanel setExtensionHidden:YES];
  [savePanel setCanSelectHiddenExtension:NO];
  [savePanel setTreatsFilePackagesAsDirectories:NO];
 
  if( [savePanel runModal] == NSOKButton )
  {
    movie = [[QTMovie alloc] initToWritableFile:[savePanel filename] error:nil];    
  }
 
  [self loadAnimations];
 
  timer = [NSTimer scheduledTimerWithTimeInterval:1.0/(NSTimeInterval)10.0
                                           target:self
                                         selector:@selector(updateTime:)
                                         userInfo:NULL
                                          repeats:YES];    
 
}

Our -updateTime selector will get called every 1/10th of a second and will grab the current frame to save it to the QTMoive object. Here is the -updateTime code.

1
2
3
4
5
6
7
8
9
10
11

- (void)updateTime:(NSTimer*)theTimer;
{
  NSBitmapImageRep *image = [self getCurrentFrame];
 
  QTTime time = QTMakeTime(1, 10);
  NSDictionary *attrs = [NSDictionary dictionaryWithObject:@"png " forKey:QTAddImageCodecType];
  NSImage *img = [[NSImage alloc] initWithData:[image TIFFRepresentation]];
  [movie addImage:img forDuration:time withAttributes:attrs];
 
  [image release];
}

Obtaining the Current Frame

The code to obtain the current frame is somewhat lengthy, but the concepts are pretty simple. We need to create a graphics context that we can draw into and then draw into it using the presentationLayer of the contenView’s root layer. If you’re not familiar, the presentationLayer provides the current state of the animated fields while “in-flight”.

Core Animation doesn’t provide any callbacks for when a frame is ready to be displayed which is why we are using a timer. This means that we may be capturing more frames than we need to, so getting the right frame rate takes a bit of trial and error, which I have to confess I wasn’t able to get nailed down completely. I’m still working on it and will update here when I get that part figured out. Meanwhile, here is the code for -getCurrentFrame

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

- (NSBitmapImageRep*)getCurrentFrame;
{
  CGContextRef    context = NULL;
  CGColorSpaceRef colorSpace;
  int bitmapByteCount;
  int bitmapBytesPerRow;
 
  int pixelsHigh = (int)[[[window contentView] layer] bounds].size.height;
  int pixelsWide = (int)[[[window contentView] layer] bounds].size.width;
 
  bitmapBytesPerRow   = (pixelsWide * 4);
  bitmapByteCount     = (bitmapBytesPerRow * pixelsHigh);
 
  colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);
 
  context = CGBitmapContextCreate (NULL,
                                   pixelsWide,
                                   pixelsHigh,
                                   8,
                                   bitmapBytesPerRow,
                                   colorSpace,
                                   kCGImageAlphaPremultipliedLast);
  if (context== NULL)
  {
    NSLog(@"Failed to create context.");
    return nil;
  }
 
  CGColorSpaceRelease( colorSpace );
 
  [[[[window contentView] layer] presentationLayer] renderInContext:context];
 
  CGImageRef img = CGBitmapContextCreateImage(context);
  NSBitmapImageRep *bitmap = [[NSBitmapImageRep alloc] initWithCGImage:img];
  CFRelease(img);
 
  return bitmap;
 
}

Notice that we are calling -renderInContext on the presentationLayer of the window’s contentView’s root layer. If we were only to render our animated layer, we wouldn’t be able to see the animation as it will only render the containing rectangle of the animating layer.

Finishing Up

Finally we need to write the movie data out to disk. The QTMovie object provides a single call to do so, but we need a way to know when the animation has finished so we can make this call. When we created our animation group, we set its delegate to our AppDelegate which will cause the delegate method -animationDidStop:finished to get called. Remember that setting the delegate for each of the individual animations gets ignored when you are using an animation group. We implement the -animationDidStop:finished delegate as shown below.

1
2
3
4
5
6
7
8
9
10
11
12

- (void)animationDidStop:(CAAnimation *)theAnimation finished:(BOOL)flag;
{
  NSLog(@"Animation stopped: %@", theAnimation);
 
  id name = [theAnimation valueForKey:@"name"];
  if( name )
    if( [name isEqualToString:@"mainGroup"] )
    {
      [movie updateMovieFile];
      [timer invalidate];
    }
}

The first thing we do is check the animation tag we set when creating the animation group. This really isn’t necessary in this code example since there is only one animation that is going to use it, but this code shows you how to differentiate if you were to use multiple animations or groups and wanted to know when each of them finished animating.

The call to -updateMovieFile writes the data to disk and we now have a QuickTime movie that will play our animation. Open the resulting file in QuickTime or just invoke QuickLook to see the result.

Conclusion

Maybe you can think of a use for this kind of thing. I haven’t yet–other than for writing a blog post of course. Shoot me your thoughts and comments in the comments section. Until next time.

CA Animation Capture Demo Project

Getting two technologies to work together can be a challenge especially when the bridges between the two are not necessarily clearly laid out in documentation. As I pointed out Apple provides some excellent sample code to help you along, but there is no hand-holding approach to any of it. I actually appreciate that having come from the Windows world as it seems that there all you get sometimes is hand-holding where Microsoft doesn’t trust you, the developer to figure things out and really own what you’re doing. But I digress (wouldn’t be a CIMGF post if I didn’t dig on MS a bit).

Like peanut butter and chocolate, what you get when you put together QuickTime and OpenGL is something greater than either of them left on their own (ok, this is subjective. Not everyone likes peanut butter and chocolate together, but again, I digress).

If you read the Core Video Programming Guide from Apple, you see they provide the reasons for using Core Video:

CoreVideo is necessary only if you want to manipulate individual video frames. For example, the following types of video processing would require CoreVideo:

• Color correction or other filtering, such as provided by Core Image filters
• Physical transforms of the video images (such as warping, or mapping on to a surface)
• Adding video to an OpenGL scene
• Adding additional information to frames, such as a visible timecode
• Compositing multiple video streams

If all you need to do is display a movie, you should simply use either a QTMovieView or, if you want to stick with the Core Animation route, use a QTMovieLayer. They both function similarly, however, the view provides a lot of features that you won’t have to implement in the UI yourself such as a scrubber or play/pause buttons. Plus the view is very fast and efficient. I’m in the process of exploring performance differences between the two, but I will save my comments about that for another post.

For our example code we are most interested in the third point above–adding video to an OpenGL scene. It seems that new QuickTime developers often want to know how to manipulate movie images before displaying them. Often this leads them to pursue adding sub-views to the movie view which can become a big mess. Because we are using OpenGL, doing other drawing on the scene is very fast. I won’t kid you. OpenGL is a pain. I don’t know anybody who loves it, but everybody respects it because of its raw speed.

Point number five above–compositing multiple video streams–is also interesting. While I won’t be covering it in this post, I will say that it makes a world of difference performance wise if you composite the movies into a OpenGL scene. If you’ve ever tried to run multiple videos simultaneously in two different views or layers, it can get pretty herky jerky. You can see why it is necessary to use OpenGL instead.

The OpenGL QuickTime Two Step

Ok, it actually will take more than two steps, however, when you are working with Core Animation layers things get a whole lot easier than they are for rendering a movie in an NSOpenGLView. Here is what you get for free, as the kids say.

• You don’t have to set up the OpenGL context. It is already available for you to send your OpenGL calls to.
• The viewport for display is already configured
• You don’t need to set up a display link callback

What took over 400 lines of code when rendering a QuickTime movie with no filters to an NSOpengGLView now only takes around 150 lines. Any time you can reduce code to something simpler, it makes life easier. It also makes it much easier to grok, in my opinion.

There really are two primary steps you take when using a CAOpenGLLayer. First you check to see if you should draw. Then, depending upon the answer, drawInCGLContext gets called or doesn’t. Really thats it. Determining whether or not you should draw depends upon what you are trying to do. In our case, we only want to draw if all of the following are true:

• The movie is actually playing back
• The visual context for the movie has been initialized
• The visual context has a new image ready to be rendered

If all of these are true, then our call to canDrawInCGLContext returns YES. Here is the code I use to check these contraints in canDrawInCGLContext:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

- (BOOL)canDrawInCGLContext:(CGLContextObj)glContext 
                pixelFormat:(CGLPixelFormatObj)pixelFormat 
               forLayerTime:(CFTimeInterval)timeInterval 
                displayTime:(const CVTimeStamp *)timeStamp
{ 
    // There is no point in trying to draw anything if our
    // movie is not playing.
    if( [movie rate] <= 0.0 )
        return NO;
 
    if( !qtVisualContext )
    {
        // If our visual context for our QTMovie has not been set up
        // we initialize it now
        [self setupVisualContext:glContext withPixelFormat:pixelFormat];
    }
 
    // Check to see if a new frame (image) is ready to be draw at
    // the time specified.
    if(QTVisualContextIsNewImageAvailable(qtVisualContext,timeStamp))
    {
        // Release the previous frame
        CVOpenGLTextureRelease(currentFrame);
 
        // Copy the current frame into our image buffer
        QTVisualContextCopyImageForTime(qtVisualContext,
                                        NULL,
                                        timeStamp,
                                        &currentFrame);
 
        // Returns the texture coordinates for the part of the image that should be displayed
        CVOpenGLTextureGetCleanTexCoords(
                            currentFrame, 
                            lowerLeft, 
                            lowerRight, 
                            upperRight, upperLeft);
        return YES;
    }
 
    return NO;
}

The call to setup the visual context is where we are associating the QuickTime movie itself with a QTVisualContextRef object which is what OpenGL needs to draw the current frame. We will then use this object to load image data into a CVImageBufferRef which can be used for rendering with OpenGL. Here is the code to set up the visual context.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

- (void)setupVisualContext:(CGLContextObj)glContext 
           withPixelFormat:(CGLPixelFormatObj)pixelFormat;
{
    OSStatus			    error;
 
    NSDictionary	    *attributes = nil;
    attributes = [NSDictionary dictionaryWithObjectsAndKeys:
                  [NSDictionary dictionaryWithObjectsAndKeys:
                   [NSNumber numberWithFloat:[self frame].size.width],
                   kQTVisualContextTargetDimensions_WidthKey,
                   [NSNumber numberWithFloat:[self frame].size.height],
                   kQTVisualContextTargetDimensions_HeightKey, nil], 
                  kQTVisualContextTargetDimensionsKey, 
                  [NSDictionary dictionaryWithObjectsAndKeys:
                   [NSNumber numberWithFloat:[self frame].size.width], 
                   kCVPixelBufferWidthKey, 
                   [NSNumber numberWithFloat:[self frame].size.height], 
                   kCVPixelBufferHeightKey, nil], 
                  kQTVisualContextPixelBufferAttributesKey,
                  nil];
 
    // Create our quicktimee visual context
    error = QTOpenGLTextureContextCreate(NULL,
                                         glContext,
                                         pixelFormat,
                                         (CFDictionaryRef)attributes,
                                         &qtVisualContext);
 
    // Associate it with our movie.
    SetMovieVisualContext([movie quickTimeMovie],qtVisualContext);
}

Next we check to see if there is an image ready using:

1

if(QTVisualContextIsNewImageAvailable(qtVisualContext,timeStamp))

And then we copy the image to our CVImageBufferRef with:

1
2
3
4
5

// Copy the current frame into our image buffer
QTVisualContextCopyImageForTime(qtVisualContext,
                                NULL,
                                timeStamp,
                                &currentFrame);

Now it’s all a matter of rendering the frame for the current time stamp.

But Wait! What TimeStamp?

If you asked this question, then you are a very astute reader. In order to obtain the next image, we simply passed the CVTimeStamp parameter, timeStamp to our call to QTVisualContextCopyImageForTime. But how do we even have a timestamp? Isn’t that something we need to get from a display link? If you’re asking what is a display link at this point, take a look at the Core Video Programming Guide which states:

To simplify synchronization of video with a display’s refresh rate, Core Video provides a special timer called a display link. The display link runs as a separate high priority thread, which is not affected by interactions within your application process. In the past, synchronizing your video frames with the display’s refresh rate was often a problem, especially if you also had audio. You could only make simple guesses for when to output a frame (by using a timer, for example), which didn’t take into account possible latency from user interactions, CPU loading, window compositing and so on. The Core Video display link can make intelligent estimates for when a frame needs to be output, based on display type and latencies.

I will provide a more complete answer to the question in the future as I am still studying it myself, however, I will mention that a display link callback is unnecessary in this context as the CAOpenGLLayer is providing this for us. The timestamp field is all we need in order to get the current frame assuming that the movie is playing back.

Drawing The Frame

There is a special group of people who really get OpenGL. I salute all of you to whom this applies. You are amazing. I, however, only write as much of it as necessary and you’ll see that most of the code I have here is simply a copy and paste from sample code I got from Apple. I am starting to understand it more and more, however, it makes my brain hurt. Here is my drawing code for when a frame is ready to be rendered.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

- (void)drawInCGLContext:(CGLContextObj)glContext 
             pixelFormat:(CGLPixelFormatObj)pixelFormat 
            forLayerTime:(CFTimeInterval)interval 
             displayTime:(const CVTimeStamp *)timeStamp
{
    NSRect    bounds = NSRectFromCGRect([self bounds]);
 
    GLfloat 	minX, minY, maxX, maxY;        
 
    minX = NSMinX(bounds);
    minY = NSMinY(bounds);
    maxX = NSMaxX(bounds);
    maxY = NSMaxY(bounds);
 
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    glOrtho( minX, maxX, minY, maxY, -1.0, 1.0);
 
    glClearColor(0.0, 0.0, 0.0, 0.0);	     
    glClear(GL_COLOR_BUFFER_BIT);
 
    CGRect imageRect = [self frame];
    // Enable target for the current frame
    glEnable(CVOpenGLTextureGetTarget(currentFrame));
    // Bind to the current frame
    // This tells OpenGL which texture we are wanting 
    // to draw so that when we make our glTexCord and 
    // glVertex calls, our current frame gets drawn
    // to the context.
    glBindTexture(CVOpenGLTextureGetTarget(currentFrame), 
                  CVOpenGLTextureGetName(currentFrame));
    glMatrixMode(GL_TEXTURE);
    glLoadIdentity();
    glColor4f(1.0, 1.0, 1.0, 1.0);
    glBegin(GL_QUADS);
 
    // Draw the quads
    glTexCoord2f(upperLeft[0], upperLeft[1]);
    glVertex2f  (imageRect.origin.x, 
                 imageRect.origin.y + imageRect.size.height);
    glTexCoord2f(upperRight[0], upperRight[1]);
    glVertex2f  (imageRect.origin.x + imageRect.size.width, 
                 imageRect.origin.y + imageRect.size.height);
    glTexCoord2f(lowerRight[0], lowerRight[1]);
    glVertex2f  (imageRect.origin.x + imageRect.size.width, 
                 imageRect.origin.y);
    glTexCoord2f(lowerLeft[0], lowerLeft[1]);
    glVertex2f  (imageRect.origin.x, imageRect.origin.y);
 
    glEnd();
 
    // This CAOpenGLLayer is responsible to flush
    // the OpenGL context so we call super
    [super drawInCGLContext:glContext 
                pixelFormat:pixelFormat 
               forLayerTime:interval 
                displayTime:timeStamp];
 
    // Task the context
    QTVisualContextTask(qtVisualContext);
 
}

If you’re not familiar with OpenGL it would help you to know that it’s all about the current state. What does this mean? Well, simply put, it means that the call you are making right now, applies to whatever state the context is in. These two calls are what are the most important for our purposes.

1
2
3

glEnable(CVOpenGLTextureGetTarget(currentFrame));
glBindTexture(CVOpenGLTextureGetTarget(currentFrame), 
                  CVOpenGLTextureGetName(currentFrame));

With these two calls we have told OpenGL which texture to use. Now, every subsequent call applies to this texture until the state is changed to something else. So now, when we set a color or draw a quad, it applies to the texture that has been set here.

Conclusion

I love both of these technologies, QuickTime and OpenGL. They are so powerful. It’s harnessing the power that’s the trick. I’ve got some other ideas for some related posts that I plan to cover in the weeks to come, but this was a real breakthrough for me. With the help of John Clayton, Jean-Daniel Dupas, and David Duncan on the cocoa-dev list, I was able to get the sample code put together for this post. Feel free to ask questions in the comments. I will do my best to answer, but I’m still pretty new to these technologies. Write some code yourself and have fun. This is really exciting stuff. Until next time.

About The Demo Code

John Clayton has some issues getting the code to work on his Mac Pro. I successfully ran it on my MacBook Pro, and the family iMac without any issues. Anyhow, we’re not sure what the problem is, so if you do run into trouble, let me know. Maybe we can figure it out. Meanwhile we’re investigating it as we have time.

Update: John Clayton figured it out. Apparently the visual context needs to be reset because the pixel format is not correct on the first run. We just reset the visual context now in the call to -copyCGLContextForPixelFormat and everything seems happy. The demo code has been updated to reflect the change.

Quicktime CAOpenGLLayer Demo

You can download the demo project for this post here: Audio Scrub Demo Application

You may be wondering why you might want something like this. Here’s my reasoning. I sometimes set up to record audio through a high-end microphone while also video taping an event. The high end microphone doesn’t plug into my camcorder, so I just run it into another audio recording device. This way, I have a high-quality audio track to go along with the video. The problem then is that I need a way to merge the audio track with the video track. In my resulting video, I only want one audio track, so I need to overwrite the original audio track from the video clip with the audio track from the high-quality audio recording. It is necessary to match the track times to get it to sound/look right.

I will not go into any further depth as to what my application will do. Suffice it to say that to merge the high-quality audio track with the video, I have to set a marker in both tracks that will allow me to align them. That’s where the need for an audio scrubber comes in. I need to find the same marker (hand clap, other loud noise) in both files and set the two tracks to have that marker as their matching starting points.

Timers and Selectors with Timeout

QTKit provides this great little abstraction called QTMovie. It allows you to load up any media file and call play: on it and just starts to playback. What I wanted was for the audio to start playing back when the slider changed, but then to stop after a period of time so that it was just playing a quick sample on each change. My first approach was to use an NSTimer, but then I found something a little simpler.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

- (IBAction)scrubAudio:(id)sender;
{
    if( !movie )
        return;
 
    QTTime currentTime;
    NSTimeInterval sliderTime = [sender floatValue];
 
    currentTime.timeValue = movieDuration.timeValue * sliderTime;
    currentTime.timeScale = movieDuration.timeScale;
    currentTime.flags = 0;
 
    // Set the current time of the movie based upon the slider
    // position
    [movie setCurrentTime:currentTime];
 
    // Update our time code field.
    [self updateTimeTextField];
 
    // Start playback
    [movie setRate:1.0];
 
    isPlaying = YES;
 
    // Let the audio play for nearly a second and then call checkPlayback which
    // will stop playback. This gives the user a chance to hear a sample of the 
    // audio clip at the current time.
    [self performSelector:@selector(checkPlayback:) withObject:nil afterDelay:0.95];
 
}

This IBAction is connected to the slider in Interface Builder. Notice specifically the performSelector call. We’ll get into more depth with that call in a minute.

Srubbing Without Bubbles

I’m sure there are other ways to achieve an audio scrubbing feature, however, I was looking for the path of least resistance and using a QTMoive object seemed the best way to do that.

Now, here is what I figured was the most logical way to solve this problem.

• When the scrubber moves, an audio sample should be played from current position in the audio file


• When the audio sample starts to play, it should only play for a moment and then stop playing


• When the next or previous buttons are clicked, the sample at that point should be played and then stopped, but the playhead should be positioned at the next position though playback rolled ahead.

To keep track of whether the audio is playing currently, I use a boolean flag called isPlaying. This gets toggled when the audio sample is set to play. It then gets toggled again when the playback gets stopped.

Perform Selector After Delay

When playback gets initiated, I call performSelector:withobject:afterDelay. This is a really handy little function that allows us to call another function after the specified delay. In the case of using the slider, we simply call checkPlayback: without any arguments. In the case of clicking the next or previous buttons, we pass in a QTTime object as an argument as we want the current time of playback to be set back to this time though playback has actually rolled ahead. Here is the implementation of checkPlayback:.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

- (void)checkPlayback:(id)previousTime;
{
    if( movie )
    {
        // If it's still playing back...
        if( [movie rate] > 0.0 )
        {
           // stop playback
            [movie stop];
 
            // If the previousTime object was passed in, set
            // our currentTime in the audio file (movie) to use it.
            if( previousTime )
                [movie setCurrentTime:[previousTime QTTimeValue]];
            [self updateSlider];
            isPlaying = NO;
        }
    }
}

NSSlider Considerations

The demo project has this set up already, but you need to understand that there are a few important details when creating your slider in Interface Builder. The first thing is that you need to make sure you minimum and maximum values are set to 0.0 and 1.0 respectively. This will give you the fractional number you need to calculate the position in the audio track where the current time is. Next, you need to make sure that continuous updating is enabled by checking the Continuous property. See the screenshot below. Continuous updating means that the IBAction for the slider gets called while you are dragging the slider. If you left this unchecked, it would only be called when you released your mouse button.

Conclusion

If you know of a better way to achieve this affect, please let me know. I feel this method I’m using works adequately well, however, I’m sure there is room for improvement.

Getting Ready For The Image

The focus of this tutorial is Core Animation and specifically how to use a QTMovieLayer as well as animating an image across the screen using a CALayer. I, therefore, will not spend any time addressing NSOperation and I think we’ve covered this adequately on this site. I would suggest, however, that you familiarize yourself with Marcus’ post on NSOperation as he discusses calling back into the main thread once the NSOperation has completed. I employ Marcus’ technique in this project so that I can be notified when the image I’m wanting to animate is ready.

You can download the demo project for this post from here: Movie Image Flick Demo Project

Download the project and run it. It will prompt you to select a QuickTime movie to play. Once the movie starts playing, tap your spacebar and watch the images pop off of the movie and animate to random places on the screen. Tap the spacebar quickly to see the effect work in rapid succession.

As in my NSOperation post, I want the video playback to be completely flicker free. I achieve this by obtaining a reference to a copy of the QTMovie object from which I obtain the image at the given time. If you take a look at my main: function in my NSOperation derived class, you’ll see how I am notifying the main thread that it may now do something with the image I’ve obtained.

ImageGrabber.m:main

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

- (void)main;
{
    if( movie )
    {
        NSImage* image;
        image = [movie frameImageAtTime:time 
                      withAttributes:imageAttrs error:nil];
        [image retain];
 
        [[AppDelegate shared] performSelectorOnMainThread:@selector(imageReady:)
                                               withObject:image
                                            waitUntilDone:YES];
        [image release];
    }
 
}

In the second last line of the function, notice I am calling back into my main thread and letting the app delegate know that the image is ready. I am also handing the image to the imageReady: function. This is very convenient for what we need to do next. But first, let’s discuss drawing the layers with no backing view or window.

Drawing Layers In Thin Air

This project employs a trick that makes it appear as if the layers are being drawn without a backing view. This is, of course, not really possible, so instead we do a little trick. I achieved this effect by creating a borderless window with a transparent background. As usual, I want to give credit where credit is due. Lucas Newman answered a question I posted on Scott Stevenson’s blog to find out how Lucas did this in his demo project a the Silicon Valley CocoaHeads meeting on Core Animation. You simply create your own NSWindow derived class and override the initWithContentRect: method. Here is the resulting initialization code I came up with from his suggestion.

BorderlessWindow.m:initWithContentRect

1
2
3
4
5
6
7
8
9
10
11
12
13
14

- (id) initWithContentRect: (NSRect) contentRect
                 styleMask: (unsigned int) aStyle
                   backing: (NSBackingStoreType) bufferingType
                     defer: (BOOL) flag
{
    if (![super initWithContentRect: contentRect 
                 styleMask: NSBorderlessWindowMask 
                   backing: bufferingType 
                   defer: flag]) return nil;
    [self setBackgroundColor: [NSColor clearColor]];
    [self setOpaque:NO];
 
    return self;
}

The only issue with using a transparent backing window/view is that you cannot click through it even though it looks like you ought to be able to. There is probably a way to pass the click on to the next front most window, but I haven’t yet researched this. I’ll let you know when I find the answer, or if you know the answer, post it in the comments.

Rapid Fire Image Animation

I created a method to initialize my NSOperation derived objects and hand them off to the NSOperationQueue. I hooked it up to the spacebar and can then tap the spacebar as quickly as possible and watch the images just pop off the playing movie. Here is my flickIt: function.

AppDelegate.m:flickIt

1
2
3
4
5
6
7
8
9
10
11
12

- (IBAction)flickIt:(id)sender;
{
    ImageGrabber *grabber = [[ImageGrabber alloc] init];
 
    [grabber setTime:[movie currentTime]];
    // Hand off the background movie to the NSOperation derived
    // object
    [grabber setMovie:movieBak];
 
    // Add the oepration to the queue. It will be run immediately.
    [queue addOperation:grabber];
}

This function creates the NSOpeartion derived objects and passes a reference to the current time in the playing movie as well as a reference to the background movie that I use to pull the image data from. Remember that this object is copied from the original QTMovie object and is used to keep from needing to pull the image data from the QTMovie object that is actually playing in the QTMovieLayer. Pulling the image from the QTMovie object that is being used for playback would cause the playback to stutter.

As stated earlier, when my image is ready, the NSOperation derived object, ImageGrabber, calls back into the main thread calling the function called imageReady:. Here is its implementation.

AppDelegate.m:imageReady

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

- (void)imageReady:(NSImage*)image;
{    
    [[NSAnimationContext currentContext] setDuration:1.5f];
 
    // Save off the rectangle of the ready layer before
    // it gets animated. We will use this to reset the
    // ready layer.
    CGRect prevRect = [readyLayer frame];
 
    // Convert the NSImage to a CGImageRef
    CGImageRef imageRef = [self nsImageToCGImageRef:image];
    CFRetain(imageRef);
    readyLayer.contents = (id)imageRef;
    readyLayer.backgroundColor = CGColorCreateGenericRGB(0.0f,0.0f,0.0f,1.0f);
    readyLayer.borderColor = CGColorCreateGenericRGB(0.45f,0.45f,0.45f,1.0f);
    readyLayer.borderWidth = 4.0f;
 
    // Grab a random rectangle on the screen where our image
    // layer will animate to.
    CGRect randRect = [self getRandomRect:prevRect];
    [readyLayer setFrame:randRect];
    [readyLayer setOpacity:1.0f];
 
    // Reset the ready layer so that it is ready for the next image
    readyLayer = [[[CALayer alloc] init] retain];
    readyLayer.frame = prevRect;
    readyLayer.opacity = 0.01f;
 
    [[[window contentView] layer] addSublayer:readyLayer];
}

Notice that we are converting the NSImage that gets passed to this function into a CGImageRef which is what our Core Animation layer requires to set it contents. Here is the code to convert an NSImage to a CGImageRef.

AppDelegate.m:nsImageToCGImageRef

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

- (CGImageRef)nsImageToCGImageRef:(NSImage*)image;
{
    NSData * imageData = [image TIFFRepresentation];
    CGImageRef imageRef;
    if(imageData)
    {
        CGImageSourceRef imageSource = 
                     CGImageSourceCreateWithData(
                           (CFDataRef)imageData,  NULL);
 
        imageRef = CGImageSourceCreateImageAtIndex(
                          imageSource, 0, NULL);
    }
    return imageRef;
}

Preloading A Layer to Animate

Instead of creating the layer, setting its contents, and then animating it all at once, I instead set an empty layer with a near zero opacity to overlay the QTMovieLayer when the movie first loads. Then, when the user taps the spacebar, I populate that invisible layer’s contents with the image data and then animate it. This keeps me from having to create a more complicated animation to get the desired effect.

Once the image layer has it’s new frame set, I re-initialize the ready layer. You see the ready layer get reset with this code from the imageReady: function.

1
2
3
4
5
6

// Reset the ready layer so that it is ready for the next image
readyLayer = [[[CALayer alloc] init] retain];
readyLayer.frame = prevRect;
readyLayer.opacity = 0.01f;
 
[[[window contentView] layer] addSublayer:readyLayer];

Note: I set the opacity to near zero as when it is set to zero it won’t animate the opacity change properly. I’m looking into why this is and will update this post with what I find when I find it.

 

Random Points

When I animate the layers, I simply grab a random point and the screen and then scale the image down to 35% of the original. When I call setFrame: the magic ensues and my screen gets really cluttered with lots of images. As I said before, not very useful, but excellent. Here is the random rectangle calculation function, getRandomRect:.

AppDelegate.m:getRandomRect

1
2
3
4
5
6
7
8
9
10

- (CGRect)getRandomRect:(CGRect)startRect;
{    
    CGPoint point = CGPointMake(random() % 
                          (NSUInteger)NSWidth([window frame]), 
                          random() % 
                           (NSUInteger)NSHeight([window frame]));
 
    return CGRectMake(point.x, point.y, 
                 startRect.size.width * 0.35f, startRect.size.height * 0.35f);
}

Conclusion

The nuances of Core Animation are quite interesting and provide a fun challenge when trying to solve problems. In other words, the devil is in the details. There is probably a better way to achieve the animation code, but I felt that the solution I came up with here works pretty well. There are details about Core Animation that I am still learning, but this type of effect demonstrates exactly the kinds of interesting things that are possible.

In his post, Marcus introduced how to use NSOperation to greatly simplify multi-threading tasks in your application. I am now going to provide a step-by-step walk-through sample application that uses NSOperation, NSOperationQueue, and QTKit.

While looking around the Internet, I noticed that the only examples of using NSOperation available were related to scientific applications. I wanted something that I could relate to a little better and since I’ve been working with QTKit a lot lately, I figured it would be a good framework to build from. This application simply grabs images of a movie while it is playing back and saves them out to a file. It’s pretty simple, but it shows how to do something fairly practical.

Grabbing Movie Image Frames

The application displays a movie and provides a button for the user to click whenever the user wants to save the current frame of the movie to disk while it is playing. Here are some basic requirements I’ve given the app.

• Images are only saved when the movie is playing back
• Clicking the button and the subsequent saving of the current image must not cause the movie to flicker
• Clicking in rapid succession should not cause the application to crash. For each click, a new task (operation) should be generated

Here is a screenshot of the final application.

Follow along in XCode to create the application called QT Image Grab or download the sample project here.

Note If you are already comfortable with setting up an AppDelegate class and making connections in Interface Builder, you can jump down to the section titled Derive A Class From NSOperation

Create The Application

Create a Cocoa Application Using the following steps:

1. Select File > New Project… and choose Cocoa Application in the ensuing dialog. Click Next
2. Enter ‘QTImageGrab’ as the project name. Click Finish

You should see a project workspace like the following:

Create A Controller/Delegate Class

The next thing you need to do is create an application controller, or delegate. To do so, use the following steps:

1. Option-Click on the Classes folder in your workspace and select ‘Add File…’
2. Choose Object-C Class in the ensuing dialog and click Next
3. Name the file ‘AppDelegate.m’ and click Finish
4. A new code window will display with your ‘AppDelegate’ interface code (.h file). Edit the code so that it looks like this:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

   #import <Cocoa/Cocoa.h> #import <QTKit/QTKit.h> #import <QuartzCore/CoreAnimation.h>   @interface AppDelegate : NSObject { IBOutlet QTMovieView *movieView; IBOutlet NSTextField *outputPath; IBOutlet NSWindow *window;   NSOperationQueue *operationQueue; QTMovie *imagesMovie; }   - (IBAction)grab:(id)sender; - (IBAction)openMovie:(id)sender; - (IBAction)selectOutputDir:(id)sender;   @end

5. Now we need to add the QTKit framework to your project. Option-Click the Frameworks folder in your project and select Add > Existing Frameworks… In the ensuing dialog, navigate to /System/Library/Frameworks and select QTKit.framework. Click Add
6. Now switch over to your implementation file for ‘AppDelegate’ (.m file). Add the implementation code so that the file looks like this:
   

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

   @implementation AppDelegate   - (void)awakeFromNib; { // We're going to add some initialization code here }   - (IBAction)grab:(id)sender; { // This is where will will start the process for grabbing the current image.   }   - (IBAction)openMovie:(id)sender; { // Here we load our movie using an open panel dialog.   }   - (IBAction)selectOutputDir:(id)sender; { // Here we will select an output directory where the current frame // image will be saved to. }   - (void)dealloc; { // We're going to have some cleanup to do. }   @end

   
   We will actually add some code to do something in each of these methods shortly, but first we need to hook it up to the user interface in Interface Builder.

Make The Interface Builder Connections

Now that we’ve specified all of our actions and outlets in the AppDelegate code, let’s make the connections in our NIB file. To do so, use the following steps.

1. In the tree view of your project in XCode, expand the smart folder labeled NIB Files and double click the file called MainMenu.nib Interface Builder will load the NIB file.
2. In Interface Builder click and drag an NSObject from the Library palette into your NIB file and rename the object to AppDelegate as shown:
3. With the AppDelegate selected, select Tools > Inspector from the menu bar. In the inspector, select the Object Identity tab (tab 6) and in the Class field, enter AppDelegate.
4. Now we need to tell our application to use the AppDelegate object as its delegate. Control-Click on the File’s Owner object in the MainMenu.nib window and drag a connection to the AppDelegate object and select delegate in the ensuing pop-up menu.
5. All we have left now is to drag the controls onto the window and connect the actions and outlets. Edit the main window dragging controls to it so as to make it look like this. Note: The black area of the window is a QTMovieView which you’ll be able to find in the object library. It’s dimensions are set to a standard DV video format of 720 x 480 pixels.
6. Drag a connection (Control-Click) from the Grab button to the AppDelegate and select grab: from the dropdown.
7. Drag a connection from the File | Open… menu item to the AppDelegate and select openMovie: from the dropdown.
8. Drag a connection from the Select Output Folder Path button to the AppDelegate and select selectOutputDir: from the dropdown.
9. Drag a connection from the AppDelegate object to the Output Folder Path text field and select the outputPath NSTextField from the dropdown.
10. Drag a connection from the AppDelegate object to the QTMovieView and select movieView from the dropdown.

Now we are finished in Interface Builder. Let’s return to XCode and start populating our event handlers with code. Don’t forget to save your NIB file in Interface Builder.

Adding Event Handling Code

For most of our event handlers, you can simply copy and paste the code from this section as their use is simply to set up the application and not directly related to the use of NSOperation or NSOperationQueue.

- (void)awakeFromNib

1
2
3
4
5

- (void)awakeFromNib;
{
    // Initialize the queue
    operationQueue = [[NSOperationQueue alloc] init];
}

- (IBAction)selectOutputDir:(id)sender;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

- (IBAction)selectOutputDir:(id)sender;
{
    NSOpenPanel *openPanel;
 
    openPanel = [NSOpenPanel openPanel];
    [openPanel setCanChooseDirectories:YES];
    [openPanel setAllowsMultipleSelection:NO];
    [openPanel setResolvesAliases:YES];
    [openPanel setCanChooseFiles:NO];
 
    if ([openPanel runModalForTypes:nil] == NSOKButton)
    {
        NSString *directory = [openPanel directory];
        [outputPath setStringValue:directory];
    }
 
}

- (void)dealloc

1
2
3
4
5
6

- (void)dealloc;
{
    [super dealloc];
    if( imagesMovie )
        [imagesMovie release];
}

There are two other event handlers that we need to fill in, but they will require a little bit of explanation, so first let’s go and create our NSOpeation derived class that will do all of the work we need.

Derive A Class From NSOperation

In our application, we want to get the current image of the movie while it is playing back and save it out to disk. This is the operation that we want our NSOoperation to complete in a separate thread. In order to do this, we need to create a new Objective-C class that derives from NSOperation. To do so, complete the following steps.

1. In the XCode treeview, Option-Click the Classes folder and select Add | New File…. In the ensuing dialog, select Objective-C Class and click next. Enter the name ‘GragQTImageOperation.m’ making sure that Also create “GrabQTImageOperation.h” is checked and click Finish
2. In the header file for the new Objective-C class you just created, change the interface so that our object inherits from NSOperation.

   1 2 3

   @interface GrabQTImageOperation : NSOperation {   }

Give Our NSOperation Something To Do

Our new class now needs some objects to work with and a method to do the work. To do work, an NSOperation derived class, must override the method called main:. Edit the file GrabQTImageOperation.m and provide the following function declaration.

1
2
3
4

- (void)main
{
 
}

In order to do the work, we need the following objects:

• QTMovie*: a pointer to a QTMovie view from which we will obtain our image to save.
• NSDictionary*: a dictionary to hold our attributes we will use when exporting to an image format
• NSLock*: a lock object to synchronize access to our QTMovie object.
• NSString*: a pointer to the output folder path we will use to save the files to.
• QTTime: a QTTime object representing the point in time as the movie was playing at which the grab button was clicked.

Change the GrabQTImageOperation header file to look like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#import <Cocoa/Cocoa.h>
#import <QTKit/QTKit.h>
 
@interface GrabQTImageOperation : NSOperation {
    QTMovie* movie;
    NSDictionary *imageAttrs;
    NSLock *lock;
    NSString *outputPath;
    QTTime time;
 
}
 
@property (assign) QTMovie *movie;
@property (assign) NSString *outputPath;
@property (assign) QTTime time;
 
- (void)main;
- (void)saveImage:(NSImage*)image;
@end

The method saveImage: actually saves the image to disk. Go ahead and add its declaration in the header as shown above. We’ll get back to that in a minute.

Implement main:

Now we need to switch to our implementation file (GrabQTImageOperation.m) and enter the following code into the main: function.

1
2
3
4
5
6
7
8
9
10
11

- (void)main
{
    if( movie )
    {
        NSImage* image;
        [lock lock];
        image = [movie frameImageAtTime:time withAttributes:imageAttrs error:nil];
        [lock unlock];
        [self saveImage:image];
    }
}

All we are doing at this point is getting the image at the specified time from our QTMovie reference and passing it on to our saveImage: method. Now fill we can populate the rest of the methods with the code we need to finish our NSOperation derived class.

- (id)init

1
2
3
4
5
6
7
8
9
10
11

- (id)init
{
    lock = [[NSLock alloc] init];
    // Specify that we want to save out a high-quality image.
    imageAttrs = [NSDictionary dictionaryWithObjectsAndKeys:
                       QTMovieFrameImageTypeNSImage, QTMovieFrameImageType,
                       [NSNumber numberWithBool:YES], QTMovieFrameImageHighQuality, nil];
    [imageAttrs retain];
    [super init];
    return self;
}

- (void)saveImage:(NSImage*)image

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

- (void)saveImage:(NSImage*)image;
{
    NSArray *representations;
    NSData *bitmapData;
 
    representations = [image representations];
 
    // Specify that we want to save the file as a JPG
    bitmapData = [NSBitmapImageRep representationOfImageRepsInArray:representations 
                                usingType:NSJPEGFileType 
                                properties:[NSDictionary dictionaryWithObject:[NSDecimalNumber numberWithFloat:1.0] 
                                forKey:NSImageCompressionFactor]];
 
    // Creates a long filename, but at least it's unique.
    NSString *filename = [[NSProcessInfo processInfo] globallyUniqueString];
    filename = [filename stringByAppendingString:@".jpg"];
    NSString *imgOut = [outputPath stringByAppendingString:@"/"];
    imgOut = [imgOut stringByAppendingString:filename];
    [bitmapData writeToFile:imgOut  atomically:YES];
}

And finally, don’t forget to add our synthesize calls at the top of the implementation.

1
2
3
4
5
6
7

@implementation GrabQTImageOperation
 
@synthesize movie;
@synthesize time;
@synthesize images;
@synthesize outputPath;
...

Finish Implementing AppDelegate

Now that we’ve finished creating our NSOperation derived class, we can finish the implementation of our AppDelegate. First add the following code to the openMovie: event handler.

- (IBAction)openMovie:(id)sender

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

- (IBAction)openMovie:(id)sender;
{
    NSOpenPanel *openPanel;
 
    openPanel = [NSOpenPanel openPanel];
    [openPanel setCanChooseDirectories:NO];
    [openPanel setAllowsMultipleSelection:NO];
    [openPanel setResolvesAliases:YES];
    [openPanel setCanChooseFiles:YES];
 
    if ([openPanel runModalForTypes:nil] == NSOKButton)
    {
        NSArray *filesToOpen = [openPanel filenames];
        NSString *theFilePath = [filesToOpen objectAtIndex:0];
        QTMovie *movie = [QTMovie movieWithFile:theFilePath error:nil];
        imagesMovie = [QTMovie movieWithFile:theFilePath error:nil];
        [imagesMovie retain];
        [movieView setMovie:movie];		
    }
 
}

Take note of these two lines:

1
2

QTMovie *movie = [QTMovie movieWithFile:theFilePath error:nil];
imagesMovie = [QTMovie movieWithFile:theFilePath error:nil];

Notice that we are loading the same movie into memory twice. The first is for the object called movie which will be used in our QTMovieView for actual movie playback. The other movie is an instance variable called imagesMovie which will act as a background movie. It will be passed to our NSOperation derived class and it is the object from which the images will be obtained for saving to disk.

We use two movie objects representing the same file on disk in order to satisfy one of our requirements which was to ensure that the movie doesn’t flicker or stutter when the operation is happening. One movie reference is for display, the other for processing.

And finally, we now need to populate our grab: method which initializes our NSOperation derived objects and passes them off to the NSOperationQueue which runs them immediately. First, we need to import our header file at the top of AppDelegate.h like this:

1

#import "GrabQTImageOperation.h"

And now, here is the implementation of grab: in AppDelegate.m.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

- (IBAction)grab:(id)sender;
{
    NSString *path = [outputPath stringValue];
    if( !path || [path length] <= 0 )
    {
        NSAlert *alert = [NSAlert alertWithMessageText:@"Set Output Directory" defaultButton:@"OK" alternateButton:nil otherButton:nil informativeTextWithFormat:@"You must first select an output directory before grabbing images."];
 
        [alert beginSheetModalForWindow:window
            modalDelegate:self
            didEndSelector:nil
            contextInfo:NULL];
 
        return;
    }
 
    // Check to see if the movie is playing	
    if( [[movieView movie] rate] > 0.0f )
    {
        // If so, create our NSOperation derived object and hand it off to the queue
        GrabQTImageOperation *op = [[GrabQTImageOperation alloc] init];
        [op setMovie:imagesMovie];
        [op setTime:[[movieView movie] currentTime]];
        [op setOutputPath:path];
 
        // As soon as this method finishes, the operation is set to work
        [operationQueue addOperation:op];
    }
}

The first thing we do is make sure our output path is set. If it is not, we alert the user and return. Otherwise, we check to see if our movie is playing by checking the movie’s current rate. If it is playing, then we are going to spin off one of our new GrabQTImageOperation objects to grab the image at the currentTime. Once we hand the image operation off to the NSOperationQueue we can watch the specified folder in the Finder and images will start to display.

To see the application run, click Build and Go

Conclusion

Developing with NSOperation and NSOperationQueue is a very pleasant experience to me so far. This was a fun application to build and the results are quite pleasing. I was able to achieve all of my project requirements with very little effort. It’s a great addition to the toolset now available in OS X 10.5 Leopard.