The basic process flow goes like this. Clicking the ‘Capture’ button on the user interface calls an IBAction called -saveAnimation. It prompts the user to select an output file for the movie. When the user has selected the file, the animations are created and added to the layer. Next we create a timer that is going to call a function that will grab the current frame and place it into the QTMovie object using -addImage at a specified interval. We set our AppDelegate to also be the delegate for the animation group so that when the animation completes we get notified and can then write our QTMovie object data to disk.

Use An Interesting Animation

First you are going to need an animation that is worth recording. Of course any old animation will do, but we’ll keep it interesting by adding multiple animations to a single layer. I have created four different keyframe animations that we will add to an animation group. The keypaths are “backgroundColor”, “borderWidth”, “position”, and “bounds”. Check the sample code to see how these animations are constructed.

We set the duration for all of the animations to five seconds. We also need to make sure that we set the duration for the group itself, otherwise it will override the five second duration we set for the animations themselves and run in the default 0.25 seconds. The code below shows how the animations are added to the layer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

- (void)loadAnimations;
{
  CAAnimationGroup *group = [CAAnimationGroup animation];
 
  [group setAnimations:[NSArray arrayWithObjects:[self backgroundColorAnimation],
                          [self borderWidthAnimation],
                          [self positionAnimation],
                          [self boundsAnimation], nil]];
 
  [group setValue:@"mainGroup" forKey:@"name"];
  [group setDuration:5.0];
  [group setAutoreverses:YES];
  [group setDelegate:self];
 
  [layer addAnimation:group forKey:@"group"];  
}

Notice that we have used KVC here to set a name for the animation group. We will use this as a tag later to make sure the animation that triggers our -animationDidStop:finished animation delegate is the correct one. More on that later.

Get Our Movie Ready

Prior to loading the animations, we prompt the user to select a file to write the movie file to. After loading the animations, we start our timer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

- (IBAction)saveAnimation:(id)sender;
{
  NSSavePanel *savePanel;
 
  savePanel = [NSSavePanel savePanel];
  [savePanel setExtensionHidden:YES];
  [savePanel setCanSelectHiddenExtension:NO];
  [savePanel setTreatsFilePackagesAsDirectories:NO];
 
  if( [savePanel runModal] == NSOKButton )
  {
    movie = [[QTMovie alloc] initToWritableFile:[savePanel filename] error:nil];    
  }
 
  [self loadAnimations];
 
  timer = [NSTimer scheduledTimerWithTimeInterval:1.0/(NSTimeInterval)10.0
                                           target:self
                                         selector:@selector(updateTime:)
                                         userInfo:NULL
                                          repeats:YES];    
 
}

Our -updateTime selector will get called every 1/10th of a second and will grab the current frame to save it to the QTMoive object. Here is the -updateTime code.

1
2
3
4
5
6
7
8
9
10
11

- (void)updateTime:(NSTimer*)theTimer;
{
  NSBitmapImageRep *image = [self getCurrentFrame];
 
  QTTime time = QTMakeTime(1, 10);
  NSDictionary *attrs = [NSDictionary dictionaryWithObject:@"png " forKey:QTAddImageCodecType];
  NSImage *img = [[NSImage alloc] initWithData:[image TIFFRepresentation]];
  [movie addImage:img forDuration:time withAttributes:attrs];
 
  [image release];
}

Obtaining the Current Frame

The code to obtain the current frame is somewhat lengthy, but the concepts are pretty simple. We need to create a graphics context that we can draw into and then draw into it using the presentationLayer of the contenView’s root layer. If you’re not familiar, the presentationLayer provides the current state of the animated fields while “in-flight”.

Core Animation doesn’t provide any callbacks for when a frame is ready to be displayed which is why we are using a timer. This means that we may be capturing more frames than we need to, so getting the right frame rate takes a bit of trial and error, which I have to confess I wasn’t able to get nailed down completely. I’m still working on it and will update here when I get that part figured out. Meanwhile, here is the code for -getCurrentFrame

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

- (NSBitmapImageRep*)getCurrentFrame;
{
  CGContextRef    context = NULL;
  CGColorSpaceRef colorSpace;
  int bitmapByteCount;
  int bitmapBytesPerRow;
 
  int pixelsHigh = (int)[[[window contentView] layer] bounds].size.height;
  int pixelsWide = (int)[[[window contentView] layer] bounds].size.width;
 
  bitmapBytesPerRow   = (pixelsWide * 4);
  bitmapByteCount     = (bitmapBytesPerRow * pixelsHigh);
 
  colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);
 
  context = CGBitmapContextCreate (NULL,
                                   pixelsWide,
                                   pixelsHigh,
                                   8,
                                   bitmapBytesPerRow,
                                   colorSpace,
                                   kCGImageAlphaPremultipliedLast);
  if (context== NULL)
  {
    NSLog(@"Failed to create context.");
    return nil;
  }
 
  CGColorSpaceRelease( colorSpace );
 
  [[[[window contentView] layer] presentationLayer] renderInContext:context];
 
  CGImageRef img = CGBitmapContextCreateImage(context);
  NSBitmapImageRep *bitmap = [[NSBitmapImageRep alloc] initWithCGImage:img];
  CFRelease(img);
 
  return bitmap;
 
}

Notice that we are calling -renderInContext on the presentationLayer of the window’s contentView’s root layer. If we were only to render our animated layer, we wouldn’t be able to see the animation as it will only render the containing rectangle of the animating layer.

Finishing Up

Finally we need to write the movie data out to disk. The QTMovie object provides a single call to do so, but we need a way to know when the animation has finished so we can make this call. When we created our animation group, we set its delegate to our AppDelegate which will cause the delegate method -animationDidStop:finished to get called. Remember that setting the delegate for each of the individual animations gets ignored when you are using an animation group. We implement the -animationDidStop:finished delegate as shown below.

1
2
3
4
5
6
7
8
9
10
11
12

- (void)animationDidStop:(CAAnimation *)theAnimation finished:(BOOL)flag;
{
  NSLog(@"Animation stopped: %@", theAnimation);
 
  id name = [theAnimation valueForKey:@"name"];
  if( name )
    if( [name isEqualToString:@"mainGroup"] )
    {
      [movie updateMovieFile];
      [timer invalidate];
    }
}

The first thing we do is check the animation tag we set when creating the animation group. This really isn’t necessary in this code example since there is only one animation that is going to use it, but this code shows you how to differentiate if you were to use multiple animations or groups and wanted to know when each of them finished animating.

The call to -updateMovieFile writes the data to disk and we now have a QuickTime movie that will play our animation. Open the resulting file in QuickTime or just invoke QuickLook to see the result.

Conclusion

Maybe you can think of a use for this kind of thing. I haven’t yet–other than for writing a blog post of course. Shoot me your thoughts and comments in the comments section. Until next time.

CA Animation Capture Demo Project

Why are these topics linked?

Generally, there are a few ways to add plug-ins to a Cocoa application.

Scriptable plug-ins

Scriptable plug-ins are probably the easiest to add to an application. They exist in a known location where the application can load them and run them in the appropriate virtual machine. This makes them fairly easy to construct even though the code to run them may be rather complicated.

Publishing a Formal Protocol Plug-ins Interface

This one is a bit more complicated. The developer (you and me) publish a formal protocol that the plug-ins must adhere to. This design works rather well but does not allow us, the application developer, to impart any functionality to the plug-ins easily.

Publishing an Informal Protocol Plug-ins Interface

This is very similar to the formal plug-ins above but a little more tricky. Since the protocol is informal, we have to test for each method that we want to use in the plug-ins before we can use it. Again, it does not make it very easy for us to impart functionality to the plug-ins.

Publishing a Base Class Plug-ins Interface

This is the hardest option I have found so far. Not only do we have to give our plug-ins developers a header file but something to actually compile against. However, with a base class we can easily impart functionality to our plug-ins without the plug-ins developer having to jump through too many hoops.

Creating a base class solution is where frameworks come in. It makes it fairly trivial to hand off a base class to plug-ins developers when we give them a framework. Therefore, we are going to add one class to the plug-ins we wrote previously with the following header:

#import <Cocoa/Cocoa.h>
 
@interface AbstractPlugin : NSObject 
{
 
}
 
- (NSString*)name;
 
- (IBAction)fire:(id)sender;
 
@end

And since I like to make it stupidly easy for people, we are going to use the following implementation of the base “abstract” class:

#import "AbstractPlugin.h"
 
#define kErrFormat @"%s not implemented in subclass %@"
#define kExceptName @"CIMGF"
 
@implementation AbstractPlugin
 
- (NSString*)name;
{
  NSString *reason = [NSString stringWithFormat:kErrFormat, 
    _cmd, [self class]];
  @throw [NSException exceptionWithName:kExceptName 
                                 reason:reason 
                               userInfo:nil];
}
 
- (IBAction)fire:(id)sender;
{
  NSString * reason = [NSString stringWithFormat:kErrFormat, 
    _cmd, [self class]];
  @throw [NSException exceptionWithName:kExceptName 
                                 reason:reason 
                               userInfo:nil];
}
 
@end

For every method that the subclass must override the abstract class will throw an exception if it is not overwritten with a polite message explaining what method in which class is missing.

Building the plug-ins

A plug-ins is effectively just a bundle, like a framework or an application. Inside of the bundle is a Contents directory, MacOS directory (where the binary goes) and a Resources directory.

Xcode has a template for building plug-ins that we can use to get started. For this first plug-ins, we are going to just add it to our existing project as a new target.

This new target is very similar to the Framework target that we built previously. We want to drag it up into the Application’s target so that it is a dependency; this can also be done from the Get Info window. We will also want to add the Framework product as a linked framework in the plug-ins. The end result looks like this.

Once we have the target set up, it is time to add some code. Plug-ins needs to have either a main class defined or a main nib defined. It is this definition that helps determine how to launch the bundle. In our case, we will want to define a main class which will be explained below. To define a main class (or nib), we need to look at the properties tab of the target.

The next step is to create the main class for our plug-ins. Since we are using a base class design for our plug-ins, our main class must extend the AbstractPlugin class and it needs to import the framework directly in the header. For our example, our plug-ins is going to display a new window on the screen, therefore the header must also include a NSWindow attribute and since we want to programmatically close the window, a -closeWindow: IBAction.

#import <Cocoa/Cocoa.h>
#import <Example/Example.h>
 
@interface PluginMainClass : AbstractPlugin 
{
  IBOutlet NSWindow *mainWindow;
}
 
- (IBAction)closeWindow:(id)sender;
 
@end

Note that we are importing the framework which contains the AbstractPlugin super class we built earlier.

Once we have the main class defined it is time to build the user interface to go along with this plug-ins. It is not a requirement that plug-ins have a user interface, they can be pure code instead. However for the sake of completeness, we are going to add one for this example.

The structure of the plug-ins nib is simple. The file owner is the PluginMainClass we wrote previously and there is one window which we bind to the file owner.

The window is also equally simple, just one button that binds back to the file owner’s -closeWindow: method.

The last step is to add a copy phase to our application. Like the framework copy phase we built last time, this is a “Copy Files” build phase and we need to configure it to copy files to the “PlugIns” directory. This is another top level directory next to Resources and Frameworks. Once the build phase is added, drag the plug-ins product into that build phase.

With the window and the main class built we can compile the plug-ins and make sure everything is ready for inclusion into our application. Once the app is built, we should see a copy of the plug-ins inside of the application bundle. Now it is time to dynamically load the bundle.

The flow of loading bundles dynamically is as follows:

1. Find the paths of the bundles.
2. Instantiate NSBundle objects for the paths.
3. Execute the -load method on the bundle.
4. Retrieve the principal class from the bundle.
5. Instantiate an object from the principal class.

And the code for this is as follows:

- (NSArray*)loadPlugins
{
  NSBundle *main = [NSBundle mainBundle];
  NSArray *all = [main pathsForResourcesOfType:@"bundle" 
                                   inDirectory:@"../PlugIns"]];
 
  NSMutableArray *availablePlugins = [NSMutableArray array];
 
  id plugin = nil;
  NSBundle *pluginBundle = nil;
 
  for (NSString *path in all) {
    pluginBundle = [NSBundle bundleWithPath:path];
    [pluginBundle load];
 
    Class prinClass = [pluginBundle principalClass];
    if (![prinClass isSubclassOfClass:[AbstractPlugin class]]) {
      continue;
    }
 
    plugin = [[prinClass alloc] init];
    [availablePlugins addObject:plugin];
    [plugin release], plugin = nil;
    pluginBundle = nil;
  }
  return availablePlugins;
}

In this new method that we have added to the AppDelegate we first get a reference to the main bundle and we then request a path for every file in the PlugIns directory that has an extension of bundle. With that array in hand we then start to loop over the array and construct a new NSBundle for each path. Once the bundle is constructed we ask for it to load. After the load, we grab its principal class, check to make sure we can use it, and construct an object from it.

Since we want to have a list of the loaded plug-ins, we throw this new object into an array that will be returned back from the method. Once this method is complete we now have every plug-ins dynamically loaded and ready to use. So lets use them!

We really want to call this method before the application has finished loading, therefore we need to add an -init method to our AppDelegate and load the plug-ins there. This will guarantee that they are loaded before the UI with some obvious benefits.

- (id)init
{
  if (![super init]) return nil;
 
  [self setPlugins:[self loadPlugins]];
 
  return self;
}

Here are we calling our new -loadPlugins method and setting the returned array into a ivar. This ivar will then be used by our UI once it loads to display a list of the plug-ins.

The main window has a pop up button with a list of the plug-ins using their inherited -name method for display. When the run button is clicked, our application will find out which plug-ins were selected via the NSArrayController and call the inherited -run: method on the plug-ins with predictable results.

Conclusion

For clarity we did leave a few things out of this project. For example, there is no code for unloading the plug-ins nor do we search the Application Support directory for user added plug-ins. However these are easy to add based on the code that was included.

I should also note that we used the default extension for these plug-ins. I would recommend changing this extension to something unique to your application and add that extension (or UTI) to your main application as an accepted file type. See previous articles here on CIMGF to see how to do each of those.

Example Project

Getting two technologies to work together can be a challenge especially when the bridges between the two are not necessarily clearly laid out in documentation. As I pointed out Apple provides some excellent sample code to help you along, but there is no hand-holding approach to any of it. I actually appreciate that having come from the Windows world as it seems that there all you get sometimes is hand-holding where Microsoft doesn’t trust you, the developer to figure things out and really own what you’re doing. But I digress (wouldn’t be a CIMGF post if I didn’t dig on MS a bit).

Like peanut butter and chocolate, what you get when you put together QuickTime and OpenGL is something greater than either of them left on their own (ok, this is subjective. Not everyone likes peanut butter and chocolate together, but again, I digress).

If you read the Core Video Programming Guide from Apple, you see they provide the reasons for using Core Video:

CoreVideo is necessary only if you want to manipulate individual video frames. For example, the following types of video processing would require CoreVideo:

• Color correction or other filtering, such as provided by Core Image filters
• Physical transforms of the video images (such as warping, or mapping on to a surface)
• Adding video to an OpenGL scene
• Adding additional information to frames, such as a visible timecode
• Compositing multiple video streams

If all you need to do is display a movie, you should simply use either a QTMovieView or, if you want to stick with the Core Animation route, use a QTMovieLayer. They both function similarly, however, the view provides a lot of features that you won’t have to implement in the UI yourself such as a scrubber or play/pause buttons. Plus the view is very fast and efficient. I’m in the process of exploring performance differences between the two, but I will save my comments about that for another post.

For our example code we are most interested in the third point above–adding video to an OpenGL scene. It seems that new QuickTime developers often want to know how to manipulate movie images before displaying them. Often this leads them to pursue adding sub-views to the movie view which can become a big mess. Because we are using OpenGL, doing other drawing on the scene is very fast. I won’t kid you. OpenGL is a pain. I don’t know anybody who loves it, but everybody respects it because of its raw speed.

Point number five above–compositing multiple video streams–is also interesting. While I won’t be covering it in this post, I will say that it makes a world of difference performance wise if you composite the movies into a OpenGL scene. If you’ve ever tried to run multiple videos simultaneously in two different views or layers, it can get pretty herky jerky. You can see why it is necessary to use OpenGL instead.

The OpenGL QuickTime Two Step

Ok, it actually will take more than two steps, however, when you are working with Core Animation layers things get a whole lot easier than they are for rendering a movie in an NSOpenGLView. Here is what you get for free, as the kids say.

• You don’t have to set up the OpenGL context. It is already available for you to send your OpenGL calls to.
• The viewport for display is already configured
• You don’t need to set up a display link callback

What took over 400 lines of code when rendering a QuickTime movie with no filters to an NSOpengGLView now only takes around 150 lines. Any time you can reduce code to something simpler, it makes life easier. It also makes it much easier to grok, in my opinion.

There really are two primary steps you take when using a CAOpenGLLayer. First you check to see if you should draw. Then, depending upon the answer, drawInCGLContext gets called or doesn’t. Really thats it. Determining whether or not you should draw depends upon what you are trying to do. In our case, we only want to draw if all of the following are true:

• The movie is actually playing back
• The visual context for the movie has been initialized
• The visual context has a new image ready to be rendered

If all of these are true, then our call to canDrawInCGLContext returns YES. Here is the code I use to check these contraints in canDrawInCGLContext:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

- (BOOL)canDrawInCGLContext:(CGLContextObj)glContext 
                pixelFormat:(CGLPixelFormatObj)pixelFormat 
               forLayerTime:(CFTimeInterval)timeInterval 
                displayTime:(const CVTimeStamp *)timeStamp
{ 
    // There is no point in trying to draw anything if our
    // movie is not playing.
    if( [movie rate] <= 0.0 )
        return NO;
 
    if( !qtVisualContext )
    {
        // If our visual context for our QTMovie has not been set up
        // we initialize it now
        [self setupVisualContext:glContext withPixelFormat:pixelFormat];
    }
 
    // Check to see if a new frame (image) is ready to be draw at
    // the time specified.
    if(QTVisualContextIsNewImageAvailable(qtVisualContext,timeStamp))
    {
        // Release the previous frame
        CVOpenGLTextureRelease(currentFrame);
 
        // Copy the current frame into our image buffer
        QTVisualContextCopyImageForTime(qtVisualContext,
                                        NULL,
                                        timeStamp,
                                        &currentFrame);
 
        // Returns the texture coordinates for the part of the image that should be displayed
        CVOpenGLTextureGetCleanTexCoords(
                            currentFrame, 
                            lowerLeft, 
                            lowerRight, 
                            upperRight, upperLeft);
        return YES;
    }
 
    return NO;
}

The call to setup the visual context is where we are associating the QuickTime movie itself with a QTVisualContextRef object which is what OpenGL needs to draw the current frame. We will then use this object to load image data into a CVImageBufferRef which can be used for rendering with OpenGL. Here is the code to set up the visual context.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

- (void)setupVisualContext:(CGLContextObj)glContext 
           withPixelFormat:(CGLPixelFormatObj)pixelFormat;
{
    OSStatus			    error;
 
    NSDictionary	    *attributes = nil;
    attributes = [NSDictionary dictionaryWithObjectsAndKeys:
                  [NSDictionary dictionaryWithObjectsAndKeys:
                   [NSNumber numberWithFloat:[self frame].size.width],
                   kQTVisualContextTargetDimensions_WidthKey,
                   [NSNumber numberWithFloat:[self frame].size.height],
                   kQTVisualContextTargetDimensions_HeightKey, nil], 
                  kQTVisualContextTargetDimensionsKey, 
                  [NSDictionary dictionaryWithObjectsAndKeys:
                   [NSNumber numberWithFloat:[self frame].size.width], 
                   kCVPixelBufferWidthKey, 
                   [NSNumber numberWithFloat:[self frame].size.height], 
                   kCVPixelBufferHeightKey, nil], 
                  kQTVisualContextPixelBufferAttributesKey,
                  nil];
 
    // Create our quicktimee visual context
    error = QTOpenGLTextureContextCreate(NULL,
                                         glContext,
                                         pixelFormat,
                                         (CFDictionaryRef)attributes,
                                         &qtVisualContext);
 
    // Associate it with our movie.
    SetMovieVisualContext([movie quickTimeMovie],qtVisualContext);
}

Next we check to see if there is an image ready using:

1

if(QTVisualContextIsNewImageAvailable(qtVisualContext,timeStamp))

And then we copy the image to our CVImageBufferRef with:

1
2
3
4
5

// Copy the current frame into our image buffer
QTVisualContextCopyImageForTime(qtVisualContext,
                                NULL,
                                timeStamp,
                                &currentFrame);

Now it’s all a matter of rendering the frame for the current time stamp.

But Wait! What TimeStamp?

If you asked this question, then you are a very astute reader. In order to obtain the next image, we simply passed the CVTimeStamp parameter, timeStamp to our call to QTVisualContextCopyImageForTime. But how do we even have a timestamp? Isn’t that something we need to get from a display link? If you’re asking what is a display link at this point, take a look at the Core Video Programming Guide which states:

To simplify synchronization of video with a display’s refresh rate, Core Video provides a special timer called a display link. The display link runs as a separate high priority thread, which is not affected by interactions within your application process. In the past, synchronizing your video frames with the display’s refresh rate was often a problem, especially if you also had audio. You could only make simple guesses for when to output a frame (by using a timer, for example), which didn’t take into account possible latency from user interactions, CPU loading, window compositing and so on. The Core Video display link can make intelligent estimates for when a frame needs to be output, based on display type and latencies.

I will provide a more complete answer to the question in the future as I am still studying it myself, however, I will mention that a display link callback is unnecessary in this context as the CAOpenGLLayer is providing this for us. The timestamp field is all we need in order to get the current frame assuming that the movie is playing back.

Drawing The Frame

There is a special group of people who really get OpenGL. I salute all of you to whom this applies. You are amazing. I, however, only write as much of it as necessary and you’ll see that most of the code I have here is simply a copy and paste from sample code I got from Apple. I am starting to understand it more and more, however, it makes my brain hurt. Here is my drawing code for when a frame is ready to be rendered.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

- (void)drawInCGLContext:(CGLContextObj)glContext 
             pixelFormat:(CGLPixelFormatObj)pixelFormat 
            forLayerTime:(CFTimeInterval)interval 
             displayTime:(const CVTimeStamp *)timeStamp
{
    NSRect    bounds = NSRectFromCGRect([self bounds]);
 
    GLfloat 	minX, minY, maxX, maxY;        
 
    minX = NSMinX(bounds);
    minY = NSMinY(bounds);
    maxX = NSMaxX(bounds);
    maxY = NSMaxY(bounds);
 
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    glOrtho( minX, maxX, minY, maxY, -1.0, 1.0);
 
    glClearColor(0.0, 0.0, 0.0, 0.0);	     
    glClear(GL_COLOR_BUFFER_BIT);
 
    CGRect imageRect = [self frame];
    // Enable target for the current frame
    glEnable(CVOpenGLTextureGetTarget(currentFrame));
    // Bind to the current frame
    // This tells OpenGL which texture we are wanting 
    // to draw so that when we make our glTexCord and 
    // glVertex calls, our current frame gets drawn
    // to the context.
    glBindTexture(CVOpenGLTextureGetTarget(currentFrame), 
                  CVOpenGLTextureGetName(currentFrame));
    glMatrixMode(GL_TEXTURE);
    glLoadIdentity();
    glColor4f(1.0, 1.0, 1.0, 1.0);
    glBegin(GL_QUADS);
 
    // Draw the quads
    glTexCoord2f(upperLeft[0], upperLeft[1]);
    glVertex2f  (imageRect.origin.x, 
                 imageRect.origin.y + imageRect.size.height);
    glTexCoord2f(upperRight[0], upperRight[1]);
    glVertex2f  (imageRect.origin.x + imageRect.size.width, 
                 imageRect.origin.y + imageRect.size.height);
    glTexCoord2f(lowerRight[0], lowerRight[1]);
    glVertex2f  (imageRect.origin.x + imageRect.size.width, 
                 imageRect.origin.y);
    glTexCoord2f(lowerLeft[0], lowerLeft[1]);
    glVertex2f  (imageRect.origin.x, imageRect.origin.y);
 
    glEnd();
 
    // This CAOpenGLLayer is responsible to flush
    // the OpenGL context so we call super
    [super drawInCGLContext:glContext 
                pixelFormat:pixelFormat 
               forLayerTime:interval 
                displayTime:timeStamp];
 
    // Task the context
    QTVisualContextTask(qtVisualContext);
 
}

If you’re not familiar with OpenGL it would help you to know that it’s all about the current state. What does this mean? Well, simply put, it means that the call you are making right now, applies to whatever state the context is in. These two calls are what are the most important for our purposes.

1
2
3

glEnable(CVOpenGLTextureGetTarget(currentFrame));
glBindTexture(CVOpenGLTextureGetTarget(currentFrame), 
                  CVOpenGLTextureGetName(currentFrame));

With these two calls we have told OpenGL which texture to use. Now, every subsequent call applies to this texture until the state is changed to something else. So now, when we set a color or draw a quad, it applies to the texture that has been set here.

Conclusion

I love both of these technologies, QuickTime and OpenGL. They are so powerful. It’s harnessing the power that’s the trick. I’ve got some other ideas for some related posts that I plan to cover in the weeks to come, but this was a real breakthrough for me. With the help of John Clayton, Jean-Daniel Dupas, and David Duncan on the cocoa-dev list, I was able to get the sample code put together for this post. Feel free to ask questions in the comments. I will do my best to answer, but I’m still pretty new to these technologies. Write some code yourself and have fun. This is really exciting stuff. Until next time.

About The Demo Code

John Clayton has some issues getting the code to work on his Mac Pro. I successfully ran it on my MacBook Pro, and the family iMac without any issues. Anyhow, we’re not sure what the problem is, so if you do run into trouble, let me know. Maybe we can figure it out. Meanwhile we’re investigating it as we have time.

Update: John Clayton figured it out. Apparently the visual context needs to be reset because the pixel format is not correct on the first run. We just reset the visual context now in the call to -copyCGLContextForPixelFormat and everything seems happy. The demo code has been updated to reflect the change.

Quicktime CAOpenGLLayer Demo

Let us now assume that we have a requirement to parse an xml document quickly and without loading the entire tree into memory in a object structure. In a situation like this libxml comes in handy. Unfortunately it is quite a bit more complicated than calling alloc init on NSXMLDocument.

libxml is a C library that is included with all current releases of OS X. With this library we can quickly read in a document, scrape the information we need out of that document and avoid loading the entire tree into memory at once. In addition, libxml (and more specifically xmlReader) does this very quickly, far faster than NSXMLDocument which is very useful when you have a lower end CPU. In this project we are going to create a simple application that reads in an xml file containing a list of people, their names and their ages. For the purposes of demonstration we are going to load that data into an array of NSDictionary objects and display it in a standard Cocoa window.

The raw XML file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<root>
  <person>
    <name>John Doe</name>
    <age>14</age>
  </person>
  <person>
    <name>Mary Doe</name>
    <age>14</age>
  </person>
  <person>
    <name>John Smith</name>
    <age>14</age>
  </person>
</root>

Besides being a C library, another difference in the way that we are going to read in this xml file is that the reading is iterative. For each element in the xml file we will be looping over the reader. Therefore, unlike our old friend NSXMLDocument, we will have to actually walk the file one element at a time and evaluate those elements as we receive them. In the xml file listed above, we would first read in the opening root tag, then the opening person tag, the opening name tag, the first name, etc. until we were done with the file.

To accomplish our goal, I created a new Xcode project which can be downloaded below. In this Xcode project, I added the libxml “framework” to the project by right clicking on the target, clicking on the add button, narrowing the focus to libraries and selecting the libxml2.dylib file.

This will link libxml to our project at compile time. However, unlike an Objective-C framework, we do not get access to its headers for free. Therefore the next step is to add the header to our search path. To do so, open the target’s properties and select the build tab. Narrow its focus to “header” and select the header search path entry.

Double click on this row and add /usr/include/libxml to the list of paths. NOTE: Some people have had issues with this path and if you do then you will need to find another copy of these headers in your Xcode installation directory.

Once this is done, we will be able to compile against this library with no errors or warnings. Next, after we add an application delegate and wire it up in our xib file, we need to import libxml in our AppDelegate.h. While we are in here we are going to add a reference for our records array that we will be loading the data into.

#import <Cocoa/Cocoa.h>
#include <libxml/xmlreader.h>
 
@interface AppDelegate : NSObject {
  NSArray *_records;
}
 
@property (retain) NSArray *records;
 
@end

Since xmlReader is so quick, our data set is so small and this is a demonstration only, we are going to read the xml file directly as the application starts. BE WARNED: for a production application THIS IS A BAD IDEA^tm. Fortunately I can get away with a lot in demos. :-)

The first thing we need to do in the -applicationDidFinishLaunching: method is to load the xml data and initialize the xml reader.

  NSString *path = [[NSBundle mainBundle] pathForResource:@"xmlExample" ofType:@"xml"];
  NSData *xmlData = [NSData dataWithContentsOfFile:path];
  xmlTextReaderPtr reader = xmlReaderForMemory([xmlData bytes], 
                                               [xmlData length], 
                                               [path UTF8String], nil, 
    (XML_PARSE_NOBLANKS | XML_PARSE_NOCDATA | XML_PARSE_NOERROR | XML_PARSE_NOWARNING));
  if (!reader) {
    NSLog(@"Failed to load xmlreader");
    return;
  }

In this section, again for demonstration purposes, we are reading in the included xml file into an NSData object. Normally we would be retrieving this data from an NSURLConnection or some other external source. Once the data is loaded, we pass the raw bytes off to the C function xmlReaderForMemory(). This function will return a pointer reference to our xmlReader. This reader is now ready to walk, one way and one time, through our xml file. Now we need to declare some local variables to store our state information while we walk the xml file.

  NSString *currentTagName = nil;
  NSDictionary *currentPerson = nil;
  NSString *currentTagValue = nil;
  NSMutableArray *people = [NSMutableArray array];
  while (true) {
    if (!xmlTextReaderRead(reader)) break;
    switch (xmlTextReaderNodeType(reader)) {

Since this xml file is very simple we do not need to hold onto a lot of state information. Basically when we hit a person tag we will create a new person dictionary and then every time we hit a text node we will store that text node’s value into the dictionary along with the tag name. Finally we will store each of those person dictionaries into an array.

Once our variables are declared and initialized it is time to start walking the xml file. Since we do not know how long it is going to be we will simply loop until we hit the end. Therefore we have a while(true) to start the loop. The first thing we do inside of the loop is to tell the reader to read the next element. If that returns back a 0 we are done with the file and break. If it returns back a positive value then we know the reader has advanced to the next node. Since the node types are stored as integer values, we can use a switch statement to quickly sort them as we are looping. The xmlTextReaderNodeType() function will return that type integer.

For our purposes, we really only care about two of the node types, opening tags and text values. For a more complex xml file we would watch for more information but fortunately we are working in a controlled environment. The tags we are going to watch for are XML_READER_TYPE_ELEMENT and XML_READER_TYPE_TEXT.

When we hit an element we will grab its name, store it temporarily and check to see if it is a person, if it isn’t then we loop. If it is then we create a person and add it to our array.

When we hit a text node then we want to stick it in the current person object. Since we previously stored the name of this element (remember, an element and its text value are two separate nodes), we can grab the value of the text and store it in the dictionary.

Any other type of node is ignored and we continue on the default: tag of the switch.

case XML_READER_TYPE_ELEMENT:
  //We are starting an element
  temp =  (char*)xmlTextReaderConstName(reader);
  currentTagName = [NSString stringWithCString:temp 
                                      encoding:NSUTF8StringEncoding];
  if ([currentTagName isEqualToString:@"person"]) {
    currentPerson = [NSMutableDictionary dictionary];
    [people addObject:currentPerson];
  } 
  continue;
case XML_READER_TYPE_TEXT:
  //The current tag has a text value, stick it into the current person
  temp = (char*)xmlTextReaderConstValue(reader);
  currentTagValue = [NSString stringWithCString:temp 
                                       encoding:NSUTF8StringEncoding];
  if (!currentPerson) return;
  [currentPerson setValue:currentTagValue forKey:currentTagName];
  currentTagValue = nil;
  currentTagName = nil;
default: continue;

Since we check for the completion of the file a the beginning of the loop that is all that we need to do for this example. Once the while loop exits we pass the array to -setRecords: and we are done. Cocoa bindings will take care of the rest for us.

NSError Tutorial

You can download the demo project for this post here: Audio Scrub Demo Application

You may be wondering why you might want something like this. Here’s my reasoning. I sometimes set up to record audio through a high-end microphone while also video taping an event. The high end microphone doesn’t plug into my camcorder, so I just run it into another audio recording device. This way, I have a high-quality audio track to go along with the video. The problem then is that I need a way to merge the audio track with the video track. In my resulting video, I only want one audio track, so I need to overwrite the original audio track from the video clip with the audio track from the high-quality audio recording. It is necessary to match the track times to get it to sound/look right.

I will not go into any further depth as to what my application will do. Suffice it to say that to merge the high-quality audio track with the video, I have to set a marker in both tracks that will allow me to align them. That’s where the need for an audio scrubber comes in. I need to find the same marker (hand clap, other loud noise) in both files and set the two tracks to have that marker as their matching starting points.

Timers and Selectors with Timeout

QTKit provides this great little abstraction called QTMovie. It allows you to load up any media file and call play: on it and just starts to playback. What I wanted was for the audio to start playing back when the slider changed, but then to stop after a period of time so that it was just playing a quick sample on each change. My first approach was to use an NSTimer, but then I found something a little simpler.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

- (IBAction)scrubAudio:(id)sender;
{
    if( !movie )
        return;
 
    QTTime currentTime;
    NSTimeInterval sliderTime = [sender floatValue];
 
    currentTime.timeValue = movieDuration.timeValue * sliderTime;
    currentTime.timeScale = movieDuration.timeScale;
    currentTime.flags = 0;
 
    // Set the current time of the movie based upon the slider
    // position
    [movie setCurrentTime:currentTime];
 
    // Update our time code field.
    [self updateTimeTextField];
 
    // Start playback
    [movie setRate:1.0];
 
    isPlaying = YES;
 
    // Let the audio play for nearly a second and then call checkPlayback which
    // will stop playback. This gives the user a chance to hear a sample of the 
    // audio clip at the current time.
    [self performSelector:@selector(checkPlayback:) withObject:nil afterDelay:0.95];
 
}

This IBAction is connected to the slider in Interface Builder. Notice specifically the performSelector call. We’ll get into more depth with that call in a minute.

Srubbing Without Bubbles

I’m sure there are other ways to achieve an audio scrubbing feature, however, I was looking for the path of least resistance and using a QTMoive object seemed the best way to do that.

Now, here is what I figured was the most logical way to solve this problem.

• When the scrubber moves, an audio sample should be played from current position in the audio file


• When the audio sample starts to play, it should only play for a moment and then stop playing


• When the next or previous buttons are clicked, the sample at that point should be played and then stopped, but the playhead should be positioned at the next position though playback rolled ahead.

To keep track of whether the audio is playing currently, I use a boolean flag called isPlaying. This gets toggled when the audio sample is set to play. It then gets toggled again when the playback gets stopped.

Perform Selector After Delay

When playback gets initiated, I call performSelector:withobject:afterDelay. This is a really handy little function that allows us to call another function after the specified delay. In the case of using the slider, we simply call checkPlayback: without any arguments. In the case of clicking the next or previous buttons, we pass in a QTTime object as an argument as we want the current time of playback to be set back to this time though playback has actually rolled ahead. Here is the implementation of checkPlayback:.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

- (void)checkPlayback:(id)previousTime;
{
    if( movie )
    {
        // If it's still playing back...
        if( [movie rate] > 0.0 )
        {
           // stop playback
            [movie stop];
 
            // If the previousTime object was passed in, set
            // our currentTime in the audio file (movie) to use it.
            if( previousTime )
                [movie setCurrentTime:[previousTime QTTimeValue]];
            [self updateSlider];
            isPlaying = NO;
        }
    }
}

NSSlider Considerations

The demo project has this set up already, but you need to understand that there are a few important details when creating your slider in Interface Builder. The first thing is that you need to make sure you minimum and maximum values are set to 0.0 and 1.0 respectively. This will give you the fractional number you need to calculate the position in the audio track where the current time is. Next, you need to make sure that continuous updating is enabled by checking the Continuous property. See the screenshot below. Continuous updating means that the IBAction for the slider gets called while you are dragging the slider. If you left this unchecked, it would only be called when you released your mouse button.

Conclusion

If you know of a better way to achieve this affect, please let me know. I feel this method I’m using works adequately well, however, I’m sure there is room for improvement.

Getting Ready For The Image

The focus of this tutorial is Core Animation and specifically how to use a QTMovieLayer as well as animating an image across the screen using a CALayer. I, therefore, will not spend any time addressing NSOperation and I think we’ve covered this adequately on this site. I would suggest, however, that you familiarize yourself with Marcus’ post on NSOperation as he discusses calling back into the main thread once the NSOperation has completed. I employ Marcus’ technique in this project so that I can be notified when the image I’m wanting to animate is ready.

You can download the demo project for this post from here: Movie Image Flick Demo Project

Download the project and run it. It will prompt you to select a QuickTime movie to play. Once the movie starts playing, tap your spacebar and watch the images pop off of the movie and animate to random places on the screen. Tap the spacebar quickly to see the effect work in rapid succession.

As in my NSOperation post, I want the video playback to be completely flicker free. I achieve this by obtaining a reference to a copy of the QTMovie object from which I obtain the image at the given time. If you take a look at my main: function in my NSOperation derived class, you’ll see how I am notifying the main thread that it may now do something with the image I’ve obtained.

ImageGrabber.m:main

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

- (void)main;
{
    if( movie )
    {
        NSImage* image;
        image = [movie frameImageAtTime:time 
                      withAttributes:imageAttrs error:nil];
        [image retain];
 
        [[AppDelegate shared] performSelectorOnMainThread:@selector(imageReady:)
                                               withObject:image
                                            waitUntilDone:YES];
        [image release];
    }
 
}

In the second last line of the function, notice I am calling back into my main thread and letting the app delegate know that the image is ready. I am also handing the image to the imageReady: function. This is very convenient for what we need to do next. But first, let’s discuss drawing the layers with no backing view or window.

Drawing Layers In Thin Air

This project employs a trick that makes it appear as if the layers are being drawn without a backing view. This is, of course, not really possible, so instead we do a little trick. I achieved this effect by creating a borderless window with a transparent background. As usual, I want to give credit where credit is due. Lucas Newman answered a question I posted on Scott Stevenson’s blog to find out how Lucas did this in his demo project a the Silicon Valley CocoaHeads meeting on Core Animation. You simply create your own NSWindow derived class and override the initWithContentRect: method. Here is the resulting initialization code I came up with from his suggestion.

BorderlessWindow.m:initWithContentRect

1
2
3
4
5
6
7
8
9
10
11
12
13
14

- (id) initWithContentRect: (NSRect) contentRect
                 styleMask: (unsigned int) aStyle
                   backing: (NSBackingStoreType) bufferingType
                     defer: (BOOL) flag
{
    if (![super initWithContentRect: contentRect 
                 styleMask: NSBorderlessWindowMask 
                   backing: bufferingType 
                   defer: flag]) return nil;
    [self setBackgroundColor: [NSColor clearColor]];
    [self setOpaque:NO];
 
    return self;
}

The only issue with using a transparent backing window/view is that you cannot click through it even though it looks like you ought to be able to. There is probably a way to pass the click on to the next front most window, but I haven’t yet researched this. I’ll let you know when I find the answer, or if you know the answer, post it in the comments.

Rapid Fire Image Animation

I created a method to initialize my NSOperation derived objects and hand them off to the NSOperationQueue. I hooked it up to the spacebar and can then tap the spacebar as quickly as possible and watch the images just pop off the playing movie. Here is my flickIt: function.

AppDelegate.m:flickIt

1
2
3
4
5
6
7
8
9
10
11
12

- (IBAction)flickIt:(id)sender;
{
    ImageGrabber *grabber = [[ImageGrabber alloc] init];
 
    [grabber setTime:[movie currentTime]];
    // Hand off the background movie to the NSOperation derived
    // object
    [grabber setMovie:movieBak];
 
    // Add the oepration to the queue. It will be run immediately.
    [queue addOperation:grabber];
}

This function creates the NSOpeartion derived objects and passes a reference to the current time in the playing movie as well as a reference to the background movie that I use to pull the image data from. Remember that this object is copied from the original QTMovie object and is used to keep from needing to pull the image data from the QTMovie object that is actually playing in the QTMovieLayer. Pulling the image from the QTMovie object that is being used for playback would cause the playback to stutter.

As stated earlier, when my image is ready, the NSOperation derived object, ImageGrabber, calls back into the main thread calling the function called imageReady:. Here is its implementation.

AppDelegate.m:imageReady

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

- (void)imageReady:(NSImage*)image;
{    
    [[NSAnimationContext currentContext] setDuration:1.5f];
 
    // Save off the rectangle of the ready layer before
    // it gets animated. We will use this to reset the
    // ready layer.
    CGRect prevRect = [readyLayer frame];
 
    // Convert the NSImage to a CGImageRef
    CGImageRef imageRef = [self nsImageToCGImageRef:image];
    CFRetain(imageRef);
    readyLayer.contents = (id)imageRef;
    readyLayer.backgroundColor = CGColorCreateGenericRGB(0.0f,0.0f,0.0f,1.0f);
    readyLayer.borderColor = CGColorCreateGenericRGB(0.45f,0.45f,0.45f,1.0f);
    readyLayer.borderWidth = 4.0f;
 
    // Grab a random rectangle on the screen where our image
    // layer will animate to.
    CGRect randRect = [self getRandomRect:prevRect];
    [readyLayer setFrame:randRect];
    [readyLayer setOpacity:1.0f];
 
    // Reset the ready layer so that it is ready for the next image
    readyLayer = [[[CALayer alloc] init] retain];
    readyLayer.frame = prevRect;
    readyLayer.opacity = 0.01f;
 
    [[[window contentView] layer] addSublayer:readyLayer];
}

Notice that we are converting the NSImage that gets passed to this function into a CGImageRef which is what our Core Animation layer requires to set it contents. Here is the code to convert an NSImage to a CGImageRef.

AppDelegate.m:nsImageToCGImageRef

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

- (CGImageRef)nsImageToCGImageRef:(NSImage*)image;
{
    NSData * imageData = [image TIFFRepresentation];
    CGImageRef imageRef;
    if(imageData)
    {
        CGImageSourceRef imageSource = 
                     CGImageSourceCreateWithData(
                           (CFDataRef)imageData,  NULL);
 
        imageRef = CGImageSourceCreateImageAtIndex(
                          imageSource, 0, NULL);
    }
    return imageRef;
}

Preloading A Layer to Animate

Instead of creating the layer, setting its contents, and then animating it all at once, I instead set an empty layer with a near zero opacity to overlay the QTMovieLayer when the movie first loads. Then, when the user taps the spacebar, I populate that invisible layer’s contents with the image data and then animate it. This keeps me from having to create a more complicated animation to get the desired effect.

Once the image layer has it’s new frame set, I re-initialize the ready layer. You see the ready layer get reset with this code from the imageReady: function.

1
2
3
4
5
6

// Reset the ready layer so that it is ready for the next image
readyLayer = [[[CALayer alloc] init] retain];
readyLayer.frame = prevRect;
readyLayer.opacity = 0.01f;
 
[[[window contentView] layer] addSublayer:readyLayer];

Note: I set the opacity to near zero as when it is set to zero it won’t animate the opacity change properly. I’m looking into why this is and will update this post with what I find when I find it.

 

Random Points

When I animate the layers, I simply grab a random point and the screen and then scale the image down to 35% of the original. When I call setFrame: the magic ensues and my screen gets really cluttered with lots of images. As I said before, not very useful, but excellent. Here is the random rectangle calculation function, getRandomRect:.

AppDelegate.m:getRandomRect

1
2
3
4
5
6
7
8
9
10

- (CGRect)getRandomRect:(CGRect)startRect;
{    
    CGPoint point = CGPointMake(random() % 
                          (NSUInteger)NSWidth([window frame]), 
                          random() % 
                           (NSUInteger)NSHeight([window frame]));
 
    return CGRectMake(point.x, point.y, 
                 startRect.size.width * 0.35f, startRect.size.height * 0.35f);
}

Conclusion

The nuances of Core Animation are quite interesting and provide a fun challenge when trying to solve problems. In other words, the devil is in the details. There is probably a better way to achieve the animation code, but I felt that the solution I came up with here works pretty well. There are details about Core Animation that I am still learning, but this type of effect demonstrates exactly the kinds of interesting things that are possible.

In his post, Marcus introduced how to use NSOperation to greatly simplify multi-threading tasks in your application. I am now going to provide a step-by-step walk-through sample application that uses NSOperation, NSOperationQueue, and QTKit.

While looking around the Internet, I noticed that the only examples of using NSOperation available were related to scientific applications. I wanted something that I could relate to a little better and since I’ve been working with QTKit a lot lately, I figured it would be a good framework to build from. This application simply grabs images of a movie while it is playing back and saves them out to a file. It’s pretty simple, but it shows how to do something fairly practical.

Grabbing Movie Image Frames

The application displays a movie and provides a button for the user to click whenever the user wants to save the current frame of the movie to disk while it is playing. Here are some basic requirements I’ve given the app.

• Images are only saved when the movie is playing back
• Clicking the button and the subsequent saving of the current image must not cause the movie to flicker
• Clicking in rapid succession should not cause the application to crash. For each click, a new task (operation) should be generated

Here is a screenshot of the final application.

Follow along in XCode to create the application called QT Image Grab or download the sample project here.

Note If you are already comfortable with setting up an AppDelegate class and making connections in Interface Builder, you can jump down to the section titled Derive A Class From NSOperation

Create The Application

Create a Cocoa Application Using the following steps:

1. Select File > New Project… and choose Cocoa Application in the ensuing dialog. Click Next
2. Enter ‘QTImageGrab’ as the project name. Click Finish

You should see a project workspace like the following:

Create A Controller/Delegate Class

The next thing you need to do is create an application controller, or delegate. To do so, use the following steps:

1. Option-Click on the Classes folder in your workspace and select ‘Add File…’
2. Choose Object-C Class in the ensuing dialog and click Next
3. Name the file ‘AppDelegate.m’ and click Finish
4. A new code window will display with your ‘AppDelegate’ interface code (.h file). Edit the code so that it looks like this:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

   #import <Cocoa/Cocoa.h> #import <QTKit/QTKit.h> #import <QuartzCore/CoreAnimation.h>   @interface AppDelegate : NSObject { IBOutlet QTMovieView *movieView; IBOutlet NSTextField *outputPath; IBOutlet NSWindow *window;   NSOperationQueue *operationQueue; QTMovie *imagesMovie; }   - (IBAction)grab:(id)sender; - (IBAction)openMovie:(id)sender; - (IBAction)selectOutputDir:(id)sender;   @end

5. Now we need to add the QTKit framework to your project. Option-Click the Frameworks folder in your project and select Add > Existing Frameworks… In the ensuing dialog, navigate to /System/Library/Frameworks and select QTKit.framework. Click Add
6. Now switch over to your implementation file for ‘AppDelegate’ (.m file). Add the implementation code so that the file looks like this:
   

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

   @implementation AppDelegate   - (void)awakeFromNib; { // We're going to add some initialization code here }   - (IBAction)grab:(id)sender; { // This is where will will start the process for grabbing the current image.   }   - (IBAction)openMovie:(id)sender; { // Here we load our movie using an open panel dialog.   }   - (IBAction)selectOutputDir:(id)sender; { // Here we will select an output directory where the current frame // image will be saved to. }   - (void)dealloc; { // We're going to have some cleanup to do. }   @end

   
   We will actually add some code to do something in each of these methods shortly, but first we need to hook it up to the user interface in Interface Builder.

Make The Interface Builder Connections

Now that we’ve specified all of our actions and outlets in the AppDelegate code, let’s make the connections in our NIB file. To do so, use the following steps.

1. In the tree view of your project in XCode, expand the smart folder labeled NIB Files and double click the file called MainMenu.nib Interface Builder will load the NIB file.
2. In Interface Builder click and drag an NSObject from the Library palette into your NIB file and rename the object to AppDelegate as shown:
3. With the AppDelegate selected, select Tools > Inspector from the menu bar. In the inspector, select the Object Identity tab (tab 6) and in the Class field, enter AppDelegate.
4. Now we need to tell our application to use the AppDelegate object as its delegate. Control-Click on the File’s Owner object in the MainMenu.nib window and drag a connection to the AppDelegate object and select delegate in the ensuing pop-up menu.
5. All we have left now is to drag the controls onto the window and connect the actions and outlets. Edit the main window dragging controls to it so as to make it look like this. Note: The black area of the window is a QTMovieView which you’ll be able to find in the object library. It’s dimensions are set to a standard DV video format of 720 x 480 pixels.
6. Drag a connection (Control-Click) from the Grab button to the AppDelegate and select grab: from the dropdown.
7. Drag a connection from the File | Open… menu item to the AppDelegate and select openMovie: from the dropdown.
8. Drag a connection from the Select Output Folder Path button to the AppDelegate and select selectOutputDir: from the dropdown.
9. Drag a connection from the AppDelegate object to the Output Folder Path text field and select the outputPath NSTextField from the dropdown.
10. Drag a connection from the AppDelegate object to the QTMovieView and select movieView from the dropdown.

Now we are finished in Interface Builder. Let’s return to XCode and start populating our event handlers with code. Don’t forget to save your NIB file in Interface Builder.

Adding Event Handling Code

For most of our event handlers, you can simply copy and paste the code from this section as their use is simply to set up the application and not directly related to the use of NSOperation or NSOperationQueue.

- (void)awakeFromNib

1
2
3
4
5

- (void)awakeFromNib;
{
    // Initialize the queue
    operationQueue = [[NSOperationQueue alloc] init];
}

- (IBAction)selectOutputDir:(id)sender;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

- (IBAction)selectOutputDir:(id)sender;
{
    NSOpenPanel *openPanel;
 
    openPanel = [NSOpenPanel openPanel];
    [openPanel setCanChooseDirectories:YES];
    [openPanel setAllowsMultipleSelection:NO];
    [openPanel setResolvesAliases:YES];
    [openPanel setCanChooseFiles:NO];
 
    if ([openPanel runModalForTypes:nil] == NSOKButton)
    {
        NSString *directory = [openPanel directory];
        [outputPath setStringValue:directory];
    }
 
}

- (void)dealloc

1
2
3
4
5
6

- (void)dealloc;
{
    [super dealloc];
    if( imagesMovie )
        [imagesMovie release];
}

There are two other event handlers that we need to fill in, but they will require a little bit of explanation, so first let’s go and create our NSOpeation derived class that will do all of the work we need.

Derive A Class From NSOperation

In our application, we want to get the current image of the movie while it is playing back and save it out to disk. This is the operation that we want our NSOoperation to complete in a separate thread. In order to do this, we need to create a new Objective-C class that derives from NSOperation. To do so, complete the following steps.

1. In the XCode treeview, Option-Click the Classes folder and select Add | New File…. In the ensuing dialog, select Objective-C Class and click next. Enter the name ‘GragQTImageOperation.m’ making sure that Also create “GrabQTImageOperation.h” is checked and click Finish
2. In the header file for the new Objective-C class you just created, change the interface so that our object inherits from NSOperation.

   1 2 3

   @interface GrabQTImageOperation : NSOperation {   }

Give Our NSOperation Something To Do

Our new class now needs some objects to work with and a method to do the work. To do work, an NSOperation derived class, must override the method called main:. Edit the file GrabQTImageOperation.m and provide the following function declaration.

1
2
3
4

- (void)main
{
 
}

In order to do the work, we need the following objects:

• QTMovie*: a pointer to a QTMovie view from which we will obtain our image to save.
• NSDictionary*: a dictionary to hold our attributes we will use when exporting to an image format
• NSLock*: a lock object to synchronize access to our QTMovie object.
• NSString*: a pointer to the output folder path we will use to save the files to.
• QTTime: a QTTime object representing the point in time as the movie was playing at which the grab button was clicked.

Change the GrabQTImageOperation header file to look like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#import <Cocoa/Cocoa.h>
#import <QTKit/QTKit.h>
 
@interface GrabQTImageOperation : NSOperation {
    QTMovie* movie;
    NSDictionary *imageAttrs;
    NSLock *lock;
    NSString *outputPath;
    QTTime time;
 
}
 
@property (assign) QTMovie *movie;
@property (assign) NSString *outputPath;
@property (assign) QTTime time;
 
- (void)main;
- (void)saveImage:(NSImage*)image;
@end

The method saveImage: actually saves the image to disk. Go ahead and add its declaration in the header as shown above. We’ll get back to that in a minute.

Implement main:

Now we need to switch to our implementation file (GrabQTImageOperation.m) and enter the following code into the main: function.

1
2
3
4
5
6
7
8
9
10
11

- (void)main
{
    if( movie )
    {
        NSImage* image;
        [lock lock];
        image = [movie frameImageAtTime:time withAttributes:imageAttrs error:nil];
        [lock unlock];
        [self saveImage:image];
    }
}

All we are doing at this point is getting the image at the specified time from our QTMovie reference and passing it on to our saveImage: method. Now fill we can populate the rest of the methods with the code we need to finish our NSOperation derived class.

- (id)init

1
2
3
4
5
6
7
8
9
10
11

- (id)init
{
    lock = [[NSLock alloc] init];
    // Specify that we want to save out a high-quality image.
    imageAttrs = [NSDictionary dictionaryWithObjectsAndKeys:
                       QTMovieFrameImageTypeNSImage, QTMovieFrameImageType,
                       [NSNumber numberWithBool:YES], QTMovieFrameImageHighQuality, nil];
    [imageAttrs retain];
    [super init];
    return self;
}

- (void)saveImage:(NSImage*)image

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

- (void)saveImage:(NSImage*)image;
{
    NSArray *representations;
    NSData *bitmapData;
 
    representations = [image representations];
 
    // Specify that we want to save the file as a JPG
    bitmapData = [NSBitmapImageRep representationOfImageRepsInArray:representations 
                                usingType:NSJPEGFileType 
                                properties:[NSDictionary dictionaryWithObject:[NSDecimalNumber numberWithFloat:1.0] 
                                forKey:NSImageCompressionFactor]];
 
    // Creates a long filename, but at least it's unique.
    NSString *filename = [[NSProcessInfo processInfo] globallyUniqueString];
    filename = [filename stringByAppendingString:@".jpg"];
    NSString *imgOut = [outputPath stringByAppendingString:@"/"];
    imgOut = [imgOut stringByAppendingString:filename];
    [bitmapData writeToFile:imgOut  atomically:YES];
}

And finally, don’t forget to add our synthesize calls at the top of the implementation.

1
2
3
4
5
6
7

@implementation GrabQTImageOperation
 
@synthesize movie;
@synthesize time;
@synthesize images;
@synthesize outputPath;
...

Finish Implementing AppDelegate

Now that we’ve finished creating our NSOperation derived class, we can finish the implementation of our AppDelegate. First add the following code to the openMovie: event handler.

- (IBAction)openMovie:(id)sender

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

- (IBAction)openMovie:(id)sender;
{
    NSOpenPanel *openPanel;
 
    openPanel = [NSOpenPanel openPanel];
    [openPanel setCanChooseDirectories:NO];
    [openPanel setAllowsMultipleSelection:NO];
    [openPanel setResolvesAliases:YES];
    [openPanel setCanChooseFiles:YES];
 
    if ([openPanel runModalForTypes:nil] == NSOKButton)
    {
        NSArray *filesToOpen = [openPanel filenames];
        NSString *theFilePath = [filesToOpen objectAtIndex:0];
        QTMovie *movie = [QTMovie movieWithFile:theFilePath error:nil];
        imagesMovie = [QTMovie movieWithFile:theFilePath error:nil];
        [imagesMovie retain];
        [movieView setMovie:movie];		
    }
 
}

Take note of these two lines:

1
2

QTMovie *movie = [QTMovie movieWithFile:theFilePath error:nil];
imagesMovie = [QTMovie movieWithFile:theFilePath error:nil];

Notice that we are loading the same movie into memory twice. The first is for the object called movie which will be used in our QTMovieView for actual movie playback. The other movie is an instance variable called imagesMovie which will act as a background movie. It will be passed to our NSOperation derived class and it is the object from which the images will be obtained for saving to disk.

We use two movie objects representing the same file on disk in order to satisfy one of our requirements which was to ensure that the movie doesn’t flicker or stutter when the operation is happening. One movie reference is for display, the other for processing.

And finally, we now need to populate our grab: method which initializes our NSOperation derived objects and passes them off to the NSOperationQueue which runs them immediately. First, we need to import our header file at the top of AppDelegate.h like this:

1

#import "GrabQTImageOperation.h"

And now, here is the implementation of grab: in AppDelegate.m.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

- (IBAction)grab:(id)sender;
{
    NSString *path = [outputPath stringValue];
    if( !path || [path length] <= 0 )
    {
        NSAlert *alert = [NSAlert alertWithMessageText:@"Set Output Directory" defaultButton:@"OK" alternateButton:nil otherButton:nil informativeTextWithFormat:@"You must first select an output directory before grabbing images."];
 
        [alert beginSheetModalForWindow:window
            modalDelegate:self
            didEndSelector:nil
            contextInfo:NULL];
 
        return;
    }
 
    // Check to see if the movie is playing	
    if( [[movieView movie] rate] > 0.0f )
    {
        // If so, create our NSOperation derived object and hand it off to the queue
        GrabQTImageOperation *op = [[GrabQTImageOperation alloc] init];
        [op setMovie:imagesMovie];
        [op setTime:[[movieView movie] currentTime]];
        [op setOutputPath:path];
 
        // As soon as this method finishes, the operation is set to work
        [operationQueue addOperation:op];
    }
}

The first thing we do is make sure our output path is set. If it is not, we alert the user and return. Otherwise, we check to see if our movie is playing by checking the movie’s current rate. If it is playing, then we are going to spin off one of our new GrabQTImageOperation objects to grab the image at the currentTime. Once we hand the image operation off to the NSOperationQueue we can watch the specified folder in the Finder and images will start to display.

To see the application run, click Build and Go

Conclusion

Developing with NSOperation and NSOperationQueue is a very pleasant experience to me so far. This was a fun application to build and the results are quite pleasing. I was able to achieve all of my project requirements with very little effort. It’s a great addition to the toolset now available in OS X 10.5 Leopard.