Sometimes there are images that are either to low quality or with backgrounds so busy that a one shot channel mask just will not work, and that’s when we pull out the big guns..

Creating multiple masks for different parts of the image and then combining 2 or more masks in photoshop into one finished mask.

Imagine that you are working with an image that is giving you trouble creating a mask in one go with channel masking or calculations. You can get part of the mask but there’s an area where the background and foreground is to close in colors to allow you to dodge and burn and enhance it to separate the foreground from the background.

Now you need to pull out all the tricks you know to create the mask in parts, and then combine the different masked out areas into one finished mask.

First take a stab at getting as much of the mask using channel masking or calculations masking and save that mask. Now try LAB and see if the area that doesn’t work for you in RGB will give you that area and save that as another mask. If that doesn’t work, trace the area with the pen tool, go to the paths palette and click the little icon in the bottom that creates a selection from the path, go back and create a new alpha channel and fill the selection with white. Keep pulling out all your masking tricks until all the parts of the object is selected using multiple masks.

Before going further, you will need to make sure that all the masks are white where the subject you want to mask out are, if not, hit control+i to invert those parts.

Now open up the calculations dialog. In the source 1 channel drop down list select the first mask, in the source 2 channel select the second mask. Now for the magic. Select the screen blending mode and your 2 masks will be combined into one. Only the areas above 50% grey will be combined, and anything below will be discarded, and that gives you a combined mask. If you have more masks to combine, use calculations again, this time selecting the combined mask you just create, and the next part of the mask puzzle until all parts have been combined.

In the Masking and removing backgrounds in photoshop tutorial we learned how do grab a mask from the channels palette, and how to enhance it with curves, dodge and burn.

Sometimes you can get a jump start on a perfect mask, using the calculations command and masking with calculations.

We will use the same photo of the flower, just to stay in familiar territory, so if you have not download that photo, click on the image below a couple times and download the large version and load it into photoshop.

redflower channel masking photo

With the flower layer selected, go the to channels palette and from the image menu select the calculations command.

photoshop calculations command

Yeah, scary dialog time I know

All the calculations command allows you to do, is choose 2 channels and apply a blending mode, it’s realy not that scary now is it?.

What you do is select the two channels that are most different from eachother by using the source 1 and source 2 Channel dropdown lists. For this image, that would be the red channel for source 1 and the blue channel for source 2

Now we want to apply a blending mode that creates a target alpha channel (Mask) with the most contrast. This is largely a trial and error kinda deal. You know you are looking for something that makes the background black, and the object you want to mask white, or the other way around. What’s black and what’s white doesn’t matter, since you can always just hit control+i to invert the resulting mask.

To get a feel for what will happen, go through the list of blending modes in the blending modes dropdown list and watch what happens. Also, for each blending mode you select, try checking off the Invert checkboxes and see what happens. This is why I told you it’s largely a trial and error process. Nobody is going to remember what each blending mode will do with invert on/off for all the modes since the effect they will have is dependant on the colors and luminace in the image and no two will be alike.

Many times though, the blending mode that is only available here in the calculations command.. Add is going to work, and that’s the one we will use for this tutorial.

Select the red channel for source 1 and the blue channel for source 2. Set the blending mode to Add, opacity to 100 (Opacity is rarely changed ffrom 100%), Offset to -200 and leave the Scale set at 1. Lastly, put a checkmark in invert for the Blue channel.

Hey, would you look at that, that looks like the start of a super mask.

Make sure the Result dropdown listbox is set to New Channel which is the default unless you changed it, and hit OK.

calculations channelmask 1

Now’s the time for some curves to bump up the white and blacks, so hit control+m.

Set the black input levels to 120 and the white input levels to 225, that’s what I found the best compromise. Feel free to play around and see if you can make it better and then hit OK.

photoshop calculations curves

Now we burn the grey areas like we already learned, and dodge the white areas to further enchance the white and black areas. The process is the same, but notice this time you don’t have the problems with the ratty edges, and the grey background and white flower responds much better to the dodge and burn tools, making the whole process a lot faster.

I’m going to stop with the images showing progressions of the mask, you should be flying through the mask by now anyway

Take care of the rest of the background by painting it over with black and paint the inside of the flower white.

Invert the mask to check if you got everything. Nope, still some stuff there. A quick dodge and burn and that goes away with no effort.

Invert the mask back again, control click on it to load it as a selection and go back to the layers palette. Doubleclick the background layer to turn it into a normal layer, and with the selection still active click on add layer mask.

Add a black background behind the masked out flower and notice that this time we have no background bleed either. Using calculations for masking made our masking job much faster and painless.

Now you are ready to read: Masking in the LAB Color space (Advanced)

First, lets look at some of techniques that many people use, that are usually the wrong way. I say usually, because there will be times when you have to work with low quality images with jpeg artifacts and just unfortunate backgrounds that makes the right way impossible. Once you learn how photos that will be masked in post production are made the right way, you will learn when to apply the different techniques.

Creating a layer mask and brushing on it with white and black to remove the background.

This takes way to much time and the mask will always be sub-par and sloppy.

Using quick selection tools.

Unless the object you want to mask out is on a pure colored background these tools will give you ratty and jaggy edges that will never blend right when you composite them into other backgrounds.

Quick mask

Same problem as with brushing on a layer mask, to time consuming and sub-par sloppy results.

Whenever possible, try to get the highest quality photo to work with. Jpeg compression is the enemy of great masks so if at all possible get the RAW files to work from. If you have any control over the photographer, instruct him/her to take photos against solid colored backgrounds. If photos are taken outside where that’s not always possible, a blue sky works great and it’s useually just a matter of the photographer finding the right spot to make sure the subject is framed against the sky.

So what’s the right way?

Channel masking, also called Found masks because these masks are found within the image itself.

Start out by downloading this photo of a flower that we will be using for this tutorial. It’s a moderately difficult image that won’t deter us by being to difficult while we are learning these techniques. Click on it a few times until you get to the large version and download that one.

redflower channel masking photo

Before we go further, since we will be using brush tools to enhance the mask there’s a few shortcuts you MUST learn if you want to work fast. Left/Right bracket will resize your brush. Shift+left/right brackets will change the hardness of your brush. Channel masking without learning and using these shortcuts will be very frustrating because you will be constantly changing brush size and hardness, it’s your choice but you have been warned.

Load up the image in photoshop and lets go to the channels palette (windows/channels)

What you see now, is how photoshop actually sees your photos, as grayscale channels. At the top we have the composite channel, labelled RGB since we are in the RGB color space. In RGB, you have a red green and a blue channel and in each channel levels of luminance, or light. The brighter an area is, the brighter that color is in that area. So if you see bright areas in the red channel, you know it’s bright red in that area, and if you see dark areas, you know there’s dark red red in that area. The green and blue channel works the same way. Photoshop then stacks these 3 channels on top of each other and we get the full color RGB image.

We might as well learn some shortcuts for selecting the channels while we are at it. Hit control+1 to select the red channel, control+2 for the green, and control+3 for the blue, and finally hit the tilde key to select the RGB composite. Try holding down control and hitting 1,2,3 – this is called channel walking.

Now what are we doing here in the channels view? We want to find the channel with the most contrast between the background that we want to mask out and the subject we want to keep.

Remember how a mask works in photoshop? Whatever is white is revealed, and what ever is black is hidden, and levels of grey in between defines transparency, so if you have an area in a mask that’s 50% grey, it’s 50% transparent.

OK so let’s find a good channel. hit control+1 and you see the red channel. It’s very bright where the flower is, because the flower is red. Hit control+2 and control+3 to check out the green and blue channels. As you can see they are very dark so they will not be useful, our candidate channel is the red one.

Now we want a copy of the red channel, we can’t mess with the original because that will change the colors in the image. Drag the red channel down to the new channel icon to create a copy of it and rename the channel to Flower Mask

As we want to create a mask from this channel we need a way to enhance the black and white areas, for that we need the curves command so hit control+m to bring it up.

If you don’t know the curves command don’t worry, all we need is the black and white input sliders down there in the bottom. Dragging the input sliders maps more and more of the luminance to either black or white. Drag the black input slider so the value says 130. For this particular image we won’t get any closer by messing with the white input but many images will need some white levels remapped, every image will be different. When you adjust the white and black levels keep a close eye on fine edges in your mask, dragging to far into the middle will create jaggy and harsh transitions, this is something you will learn with experience.

OK out of the curves dialog box for now.

redflower1 curves

Now we are getting somewhere, still not perfect though. We need to turn the darker grays around the edges of the flower to black and the outer edges of the flower to white.

We can do that with the dodge and burn tools. First we need to make sure they are setup correctly. Switch to the dodge tool and make sure the range is set to Highlights, then switch to the burn tool and make sure the range is set to shadows.

Switch to the burn tool, make sure the brush is set a hardness of zero, because we want soft gentle edges.

Now slowly start a bit out from the white edge of the flower and CLICK, don’t click and drag as that will smudge the edges to much, just click a couple times and you will see the gray stuff go away slowly. If you see the white edge of the flower go darker just hit O to switch the the dodge tool and hit the white edge a couple times to bring it up, then hit O again to go back to the burn tool. Work your way around the edge of the flower until you have a nice dark gap all the way around.

redflower1 curves dodge and burn

Let’s clean up the mask a bit before we go work on the white areas of the flower itself. Hit B to select your brush tool, make sure the foreground color is set to black, and now since we have a nice gap around the flower we can paint the background black without worrying about hitting the flower.

redflower1 mask blackbackground

If there’s still some grey stuff around the edge take care of that now with the burn tool.

Now go around the outside of the flower with the dodge tool and make it white, remember to click and never drag. Once the outside is white switch to the brush tool and select white for the foreground color and quickly paint the inside white.

redflower1 raggyedges

OK, getting there now. There’s still some rattyness of a few edges that won’t play ball because the image was shot with a large aparture so the edges blurr a bit, that’s how it goes so lets fix that up. Zoom in and select a small brush and make sure it’s set to white and make the hardness 100% now as we want to paint nice clean edges.

Now doubleclick on your mask layer, that brings up the channel options dialog box. Since the flower is red we don’t want red for the channel options, click on the color swatch and select a nice deep blue that’s not to dark and set the Opacity to 90% and hit OK

Nothing happened yet, but click on the eyeball of the RGB composite, important DON’T click the layer, click on the eyeball only. Now you get a rubylith overlay, well bluelith overlay since you changed the overlay color to blue but whatever..

Be aware that Photoshop has an annoying habit of flipping the background and foreground colors when you activate the overlay so if your painting with black all of a sudden instead of white, that’s what happened, just hit x to switch back to white.

Now you can see the original unmasked image behind the overlay making it easy to see where to paint to fix up the edges.

Paint with white to reveal anything that is part of the flower and paint with black to hide the background where the mask is still ratty. TIP TIME.. To easily make straight lines with the brush tool just hold down shift and click once to set the start point and lift the brush to the endpoint, don’t let go of shift and you can keep clicking to create nice straight lines and quickly take care of the ratty edges.

Almost done

redflower1 mask cleanup

If you want to make absolutely sure there’s nothing that needs cleanup hit control+i with the mask selected to invert it

redflower1 mask lastinspection

Oh dear, that needs to be fixed.

There, much better, perfect mask.

redflower1 mask last inspection clean

All we need to do now is invert the mask back, so hit control+i again.

redflower1 mask done

Now to put this bad boy into play. Control click on the mask to make a selection from it.

redflower1 mask selected

Now go back to your layers palette with the selection active. If the photo layer is named background doubleclick on it and just hit OK to the dialog that pops up, that turns it into a normal layer.

Now with the selection still active click on Add Layer Mask icon below the layers palette and marvel at your perfect mask.

Is it perfect though? Hard to see now, so lets add a new layer and fill it with black and drag it below the masked out flower layer.

Hmm, those edges could need a fixup.

redflower1 lastcheck

Select the mask (Not the layer, but the mask beside it) and go to filters/blur/Gaussian blur and enter 1 for the radius value and hit OK.

Now with the mask still selected hit control+m and pull the black input level slider in towards the middle and you will choke the mask, adjust until the green background bleed is gone and you are done.

Told you it was going to be a long one. If you made it down here pat yourself on the back, good job. With a little more practice and you can now mask images better than probably 80% of photoshop users. (Numbers made up, actual percentages may wary)

If you are thinking that this is tedious and time consuming it’s because you are still trying to keep all this stuff in your head and stumbling along. This and similar images takes me about 5-8 minutes tops because I don’t have to thing about what I’m doing anymore, keep at it you’ll get there.

When you are ready to move on and have channel masking down, have a look at my other articles about masking:

Masking with calculations in photoshop
Masking in LAB Color space
Difference masks for Product photography
Combining 2 or more masks in photoshop
You can also stay up to date with the masking and selections category

Make sure nothing in the scene with your subject moves, this goes for changing lights, moving flags or gobos etc.., the scene must be static. Shoot on a stand and use a remote to trigger the camera so you don’t get any movements at all.

This technique works best of you are already shooting on white seamless backgrounds.

Now take the photo of your subject. Carefully remove the subject from the scene and take another photo.

Import the 2 photos into Photoshop and put them on different layers, order doesn’t matter.

Now for the magic. Select whatever layer is the top layer and change the blending mode to Difference and voila, whatever was changed will become black and the rest of the scene will be white or at least very bright. Hit control+shift+alt+e to create a copy on a new layer on top of all the others and hit control+u and pull the saturation down to make the mask black and white. Go to your channels palette and grab the blue channel and make a copy of it. Now hit control+m to bring up curves and drag the white/black input sliders to wards the middle to make the mask all black and white and clean up any gray areas. Control+click the channel to load it as a selection and go the the layers palette and select your original image, click add layer mask and there’s your masked out image.

Masking and removing backgrounds in photoshop
Masking with calculations in photoshop

LAB masking

Whenever I start to mask something out, my first trip is ALWAYS to the LAB color space, especially if the the object I want to mask out is colored in any way.

Try it right now. Open an image of a flower, or a colored car or something (GIS one of you don’t have one handy), then change the color space to LAB.

Go to the channels palette and look at the A and B channels. One of them will be brighter, pick that one.

Now do the useual and copy the A or B channel you picked, enhance the black and white point with curves and you’r allmost guaranteed to get a mask that’s close the perfect in one go, maybe a bit of dodge/burn and curves but LAB is amazing for many times giving you a perfect mask in a couple minutes that would have taken much much longer in RGB.

In case you’r wondering why, it’s because LAB couldn’t give a rats ass about luminance and stores only color info in the A and B channels. That’s why the mask is so easy to create. The A/B channels will be consistantly light in the areas of color. In RGB luminance is store in each channel so if something goes from dark red to bright red the brightness of the channel will vary making it hard to create a channel mask from it. But in LAB, as long as theres red at all, the A/B channel will be the same brightness, making it dead easy to make the channel mask.

If you’r “one of those” that are worried about the roundrip to LAB just duplicate your image, create the mask and copy the alpha channel over to your RGB version. If you’r working in 16bit I seriously doubt you’r going to be able to detect any degradation, do a diff blend if you don’t belive me.

Here’s my take on how to create fake bloom in photoshop.

First duplicate your original layer.

Now go to filters/blur/lens blur. Set the radius high, around 80 or so and adjust the specular highlight brightness so the brightest areas of the photo blows out. Play around with curvature and rotation if you like. Be aware that the lens blur filter is dog ass sloooow so expect to wait a while on large images. Ok the changes.

The reason we use lens blur and not gausian blur like so many other tutorials is that we want to emulate as closely as possible the effect of bloom in camera and gausian blur will just not be realistic.

Now turn of all but this new layer, we are going to use it to create what is called a “found mask”, a mask that is derived from the image with no brushwork. Go the the channels palette and select the blue layer and drag a copy of it to the new channel icon below the channels palette. Name your new layer “bloom mask” and then hit control+m to open up the curves dialog box.

In the curves dialog box grab the black input triangle below the grid and drag it a bit towards the middle, you want to make the black areas of the bloom mask channel black, don’t go to far as that will create harsh transitions. Grab the white input slider and drag that a bit towards the middle also, not to much, just enough that you see some of the brightest areas go all white. Ok out of the curves dialog box.

Go back to your layers palette and delete the blurred layer, we are done with that. Select the original layer and hit control+j to jump a new copy of it. With the new copy selected, go back to the channels palette and control+click on the bloom mask channel to create a selection from it.

Go back to the layers palette and now click on the add layer mask icon below the layers palette and you have created your “found mask” layer.

Set the layers blending mode to screen and watch the bloom come alive. If the bloome areas are to sharp, select the layer mask and hit it with some gausian blur to smooth it out. You can also try selecting the layer and add a hue/saturation adjustment layer and adjust the colors to be a bit more yellow to simulate sunlight bloom better.

If you realy want to hit this effect over the head, select the channel mask on the layer and drag it to the trashcan icon, when asked, select “apply mask” before deleting. Now select the bloom layer and add some motion blur, adjust the angle of the motion blur so it follows the direction of the light, this creates cool “god rays” and can look amazing on sunshine landscape photos, if the rays become to streaky, hit the layer with some gausion blur after you apply the motion blur.

You may also want to try some other blending modes than screen to see if you get effects that you like, the possibilities are endless.

New photoshop users are stumped by this, there’s no way to create rounded shapes, where’s the rounded corners tool? Well, there is none, this is photoshop not Corel Draw.

Open up the channels palette (Windows/Channels)
Create a new Alpha channel (Click page icon at the bottom of the palette)

Now draw any shaped selection you want to have rounded corners, start with a square using the marquee tool (Hit M, draw a box)

Fill the selection with white since the new alpha channel is black, don’t worry about the color for now. Hit control+d to drop the selection.

Now go to filters/Blur/Gausian blur and hit your shape with a good amount of blur. Hit Control+M to bring up the curves dialog box. See those little triangles down there in the bottom of the grid? Drag them towards the middle until the corners are rounded like you want them. The more you pull towards the middle the harder the edges will be. OK out of the curves dialog box when done.

Now hold down control and click on the alpha channel, this will load it as a selection. Go back to the layers palette (Windows/Layers) and create a new layer. The selection is still active so now you can just fill it with whatever color you like.

That’s all there is to creating rounded corners in photoshop.

Here’s a simple way to defringe colored edges, whatever the cause.

Add a new layer on top of your original image. Set this layer to the color blending mode. Zoom into your image so the colored edge is easy to see. Hit I to select your color picker and pick a color right next to where the colored edge you want to remove, don’t click the colored edge itself. Switch to your brush tool, set opacity real low, like 20 and make sure the brush edge is real soft. Adjust the brush size so it’s around the same size as the edge you want to remove.

Now simply paint over the edge and watch it disappear like magic, it will take on the color of the underlying layer eliminating the colored edge.

If the edge continues and the color next to the edge changes, make sure to pick up the color with the color picker again so the color you paint over the edge matches.

There’s a trick to the pen tool you see.

Whenever you change directions you want to drag out a handle by holding down the ALT key and dragging the point you set, this gives you a little handle.

Make sure you now point the handle you just dragged out in the direction of your next pen point. If you need to realign a point just hold down control.

If you forgot to drag out a handle just hold down alt and drag one out.

Also note that when you drag out a handle there’s two points, you can split them up so they become independant by dragging them while holding down ALT giving you even more presicion.

Voila, no more trouble with the pen tool.