Steps to compositing A:M Alpha rendered images in Photoshop

If you used Filter Factory to do the above step, you have observed that Photoshop internally uses alpha channels to store transparency information for each layer. But Photoshop does not work with premultiplied bitmaps because it always wants to keep the most accurate color information available for further processing later.

Because of that and because A:M renders its alpha images as premultiplied, trying to composite an A:M rendered alpha image within Photoshop always produces a thin dark halo around the masked portion of the image.

Unfortunately, there are no easy ways to circumvent this. But there is a solution.

The idea is to try to un-multiply the information stored in the color channels with the help of the known transparency information stored in the alpha channel.

The process will not produce perfect results since some original color information was lost during the pre-multiplying process. But still, enough restoring can be done so that the halo will disappear. Fortunately, the most information is lost where the transparency is the highest anyway.

So this is done with the help of a Photoshop Action and another custom made Filters using Filter Factory. You may download a Windows version of the d-multip Photoshop filter here.

The general steps are those:

• Load the A:M rendered alpha image;
• Duplicate the "Background" layer;
• Select the "Background Copy" layer;
• Menu "Select", "Load Selection", Check "Invert" select "Alpha 1" and "OK";
• Delete (This delete operation actually deletes nothing. It simply copies the "Alpha 1" channel information into the layer's own alpha channel);
• Menu "Filter", "Alpha Channel", "Demultiply" (This step restores as best as possible the original colors);

Voilà! You now have an image which may be composited over any background image with Photoshop without producing a black halo.

It is easy to build a Photoshop action that will produce the above steps and use this action to batch convert a sequence of A:M rendered images.

The Filter Factory setup to do the demultiply operation is (note that Uppercase and Lowercase have very different meanings in Filter Factory. They must be entered as is):
   R: r*A/a
   G: g*A/a
   B: b*A/a
   A: a

There is an alternative way to produce the same result using regular Photoshop operations. This alternative way uses the "Color Dodge" technique that was communicated to me by Ian Rickard after I posted my tutorial. Here are the steps:

• Load the A:M rendered alpha image;
• Switch to Channels palette, Select the "Alpha 1" channel, Ctrl-A to select all and Ctrl-C to copy its content (Note that Ctrl-CLick on the layer name will not make the correct selection here);
• Switch to Layers palette and Ctrl-V to paste the alpha content onto a new layer;
• Invert the colors in the new layer and set it to "Color Dodge";
• Flatten the image;
• Duplicate the "Background" layer;
• Select the "Background Copy" layer;
• Menu "Select", "Load Selection", Check "Invert" select "Alpha 1" and "OK";
• Delete (This delete operation actually deletes nothing. It simply copies the "Alpha 1" channel information into the layer's own alpha channel);

With this information, you now know all there is to know in order to use A:M and Photoshop with alpha images.

As a last note, the above processing only applies to compositing in photoshop or another similar paint application. Most video editing application which can read TGA files knows how to interpret the premultiplied flag bit in those files.

Compositing by selecting the inverse "alpha 1" channel and deleting the selection. Note the dark halo around the image

Compositing with a demultiplied image removes the dark halo around the image. Note that the alpha channel is exactly the same but the image colors where the alpha channel fades to black are now non-antialiased. The result is that instead of mixing the background with black which produces the halo, the background is now mixed with more natural image colors.

  
The two alpha merges side by side at 100%.