Kens Photoshop Blog

This is going to be a long one so get comfy and go grab a cuppa first, we are going to learn how to mask and remove backgrounds in photoshop the right way.

First, lets look at some of techniques that many people use, that are usually the wrong way. I say usually, because there will be times when you have to work with low quality images with jpeg artifacts and just unfortunate backgrounds that makes the right way impossible. Once you learn how photos that will be masked in post production are made the right way, you will learn when to apply the different techniques.

Creating a layer mask and brushing on it with white and black to remove the background.

This takes way to much time and the mask will always be sub-par and sloppy.

Using quick selection tools.

Unless the object you want to mask out is on a pure colored background these tools will give you ratty and jaggy edges that will never blend right when you composite them into other backgrounds.

Quick mask

Same problem as with brushing on a layer mask, to time consuming and sub-par sloppy results.

Whenever possible, try to get the highest quality photo to work with. Jpeg compression is the enemy of great masks so if at all possible get the RAW files to work from. If you have any control over the photographer, instruct him/her to take photos against solid colored backgrounds. If photos are taken outside where that’s not always possible, a blue sky works great and it’s useually just a matter of the photographer finding the right spot to make sure the subject is framed against the sky.

So what’s the right way?

Channel masking, also called Found masks because these masks are found within the image itself.

Start out by downloading this photo of a flower that we will be using for this tutorial. It’s a moderately difficult image that won’t deter us by being to difficult while we are learning these techniques. Click on it a few times until you get to the large version and download that one.

redflower channel masking photo

Before we go further, since we will be using brush tools to enhance the mask there’s a few shortcuts you MUST learn if you want to work fast. Left/Right bracket will resize your brush. Shift+left/right brackets will change the hardness of your brush. Channel masking without learning and using these shortcuts will be very frustrating because you will be constantly changing brush size and hardness, it’s your choice but you have been warned.

Load up the image in photoshop and lets go to the channels palette (windows/channels)

What you see now, is how photoshop actually sees your photos, as grayscale channels. At the top we have the composite channel, labelled RGB since we are in the RGB color space. In RGB, you have a red green and a blue channel and in each channel levels of luminance, or light. The brighter an area is, the brighter that color is in that area. So if you see bright areas in the red channel, you know it’s bright red in that area, and if you see dark areas, you know there’s dark red red in that area. The green and blue channel works the same way. Photoshop then stacks these 3 channels on top of each other and we get the full color RGB image.

We might as well learn some shortcuts for selecting the channels while we are at it. Hit control+1 to select the red channel, control+2 for the green, and control+3 for the blue, and finally hit the tilde key to select the RGB composite. Try holding down control and hitting 1,2,3 – this is called channel walking.

Now what are we doing here in the channels view? We want to find the channel with the most contrast between the background that we want to mask out and the subject we want to keep.

Remember how a mask works in photoshop? Whatever is white is revealed, and what ever is black is hidden, and levels of grey in between defines transparency, so if you have an area in a mask that’s 50% grey, it’s 50% transparent.

OK so let’s find a good channel. hit control+1 and you see the red channel. It’s very bright where the flower is, because the flower is red. Hit control+2 and control+3 to check out the green and blue channels. As you can see they are very dark so they will not be useful, our candidate channel is the red one.

Now we want a copy of the red channel, we can’t mess with the original because that will change the colors in the image. Drag the red channel down to the new channel icon to create a copy of it and rename the channel to Flower Mask

As we want to create a mask from this channel we need a way to enhance the black and white areas, for that we need the curves command so hit control+m to bring it up.

If you don’t know the curves command don’t worry, all we need is the black and white input sliders down there in the bottom. Dragging the input sliders maps more and more of the luminance to either black or white. Drag the black input slider so the value says 130. For this particular image we won’t get any closer by messing with the white input but many images will need some white levels remapped, every image will be different. When you adjust the white and black levels keep a close eye on fine edges in your mask, dragging to far into the middle will create jaggy and harsh transitions, this is something you will learn with experience.

OK out of the curves dialog box for now.

redflower1 curves

Now we are getting somewhere, still not perfect though. We need to turn the darker grays around the edges of the flower to black and the outer edges of the flower to white.

We can do that with the dodge and burn tools. First we need to make sure they are setup correctly. Switch to the dodge tool and make sure the range is set to Highlights, then switch to the burn tool and make sure the range is set to shadows.

Switch to the burn tool, make sure the brush is set a hardness of zero, because we want soft gentle edges.

Now slowly start a bit out from the white edge of the flower and CLICK, don’t click and drag as that will smudge the edges to much, just click a couple times and you will see the gray stuff go away slowly. If you see the white edge of the flower go darker just hit O to switch the the dodge tool and hit the white edge a couple times to bring it up, then hit O again to go back to the burn tool. Work your way around the edge of the flower until you have a nice dark gap all the way around.

redflower1 curves dodge and burn

Let’s clean up the mask a bit before we go work on the white areas of the flower itself. Hit B to select your brush tool, make sure the foreground color is set to black, and now since we have a nice gap around the flower we can paint the background black without worrying about hitting the flower.

redflower1 mask blackbackground

If there’s still some grey stuff around the edge take care of that now with the burn tool.

Now go around the outside of the flower with the dodge tool and make it white, remember to click and never drag. Once the outside is white switch to the brush tool and select white for the foreground color and quickly paint the inside white.

redflower1 raggyedges

OK, getting there now. There’s still some rattyness of a few edges that won’t play ball because the image was shot with a large aparture so the edges blurr a bit, that’s how it goes so lets fix that up. Zoom in and select a small brush and make sure it’s set to white and make the hardness 100% now as we want to paint nice clean edges.

Now doubleclick on your mask layer, that brings up the channel options dialog box. Since the flower is red we don’t want red for the channel options, click on the color swatch and select a nice deep blue that’s not to dark and set the Opacity to 90% and hit OK

Nothing happened yet, but click on the eyeball of the RGB composite, important DON’T click the layer, click on the eyeball only. Now you get a rubylith overlay, well bluelith overlay since you changed the overlay color to blue but whatever..

Be aware that Photoshop has an annoying habit of flipping the background and foreground colors when you activate the overlay so if your painting with black all of a sudden instead of white, that’s what happened, just hit x to switch back to white.

Now you can see the original unmasked image behind the overlay making it easy to see where to paint to fix up the edges.

Paint with white to reveal anything that is part of the flower and paint with black to hide the background where the mask is still ratty. TIP TIME.. To easily make straight lines with the brush tool just hold down shift and click once to set the start point and lift the brush to the endpoint, don’t let go of shift and you can keep clicking to create nice straight lines and quickly take care of the ratty edges.

Almost done

redflower1 mask cleanup

If you want to make absolutely sure there’s nothing that needs cleanup hit control+i with the mask selected to invert it

redflower1 mask lastinspection

Oh dear, that needs to be fixed.

There, much better, perfect mask.

redflower1 mask last inspection clean

All we need to do now is invert the mask back, so hit control+i again.

redflower1 mask done

Now to put this bad boy into play. Control click on the mask to make a selection from it.

redflower1 mask selected

Now go back to your layers palette with the selection active. If the photo layer is named background doubleclick on it and just hit OK to the dialog that pops up, that turns it into a normal layer.

Now with the selection still active click on Add Layer Mask icon below the layers palette and marvel at your perfect mask.

Is it perfect though? Hard to see now, so lets add a new layer and fill it with black and drag it below the masked out flower layer.

Hmm, those edges could need a fixup.

redflower1 lastcheck

Select the mask (Not the layer, but the mask beside it) and go to filters/blur/Gaussian blur and enter 1 for the radius value and hit OK.

Now with the mask still selected hit control+m and pull the black input level slider in towards the middle and you will choke the mask, adjust until the green background bleed is gone and you are done.

Told you it was going to be a long one. If you made it down here pat yourself on the back, good job. With a little more practice and you can now mask images better than probably 80% of photoshop users. (Numbers made up, actual percentages may wary)

If you are thinking that this is tedious and time consuming it’s because you are still trying to keep all this stuff in your head and stumbling along. This and similar images takes me about 5-8 minutes tops because I don’t have to thing about what I’m doing anymore, keep at it you’ll get there.

When you are ready to move on and have channel masking down, have a look at my other articles about masking:

Masking with calculations in photoshop
Masking in LAB Color space
Difference masks for Product photography
Combining 2 or more masks in photoshop
You can also stay up to date with the masking and selections category