Friday, November 16, 2007

CCD and CMOS image sensor processing pipeline

Page 1 of 3

Video Imaging DesignLine

Camera sources today are overwhelmingly based on either Charge-Coupled Device (CCD) or CMOS technology. Both of these technologies convert light into electrical signals, but they differ in how this conversion occurs.

In CCD devices, an array of millions of light-sensitive picture elements, or pixels, spans the surface of the sensor. After exposure to light, the accumulated charge over the entire CCD pixel array is read out at one end of the device and then digitized via an Analog Front End (AFE) chip or CCD processor. On the other hand, CMOS sensors directly digitize the exposure level at each pixel site.

In general, CCDs have the highest quality and lowest noise, but they are not power-efficient. CMOS sensors are easy to manufacture and have low power dissipation, but at reduced quality. Part of the reason for this is because the transistors at each pixel site tend to occlude light from reaching part of the pixel. However, CMOS has started giving CCD a run for its money in the quality arena, and increasing numbers of mid-tier camera sensors are now CMOS-based.

Regardless of their underlying technology, all pixels in the sensor array are sensitive to grayscale intensity -- from total darkness (black) to total brightness (white). The extent to which they're sensitive is known as their "bit depth." Therefore, 8-bit pixels can distinguish between 28, or 256, shades of gray, whereas 12-bit pixel values differentiate between 4096 shades. Layered over the entire pixel array is a color filter that segments each pixel into several color-sensitive "subpixels." This arrangement allows a measure of different color intensities at each pixel site. Thus, the color at each pixel location can be viewed as the sum of its red, green and blue channel light content, superimposed in an additive manner. The higher the bit depth, the more colors that can be generated in the RGB space. For example, 24-bit color (8 bits each of R,G,B) results in 224, or 16.7 million, discrete colors.

In order to properly represent a color image, a sensor needs 3 color samples -- most commonly, Red, Green and Blue -- for every pixel location. However, putting 3 separate sensors in every camera is not a financially tenable solution (although lately such technology is becoming more practical). What's more, as sensor resolutions increase into the 5-10 Megapixel range, it becomes apparent that some form of image compression is necessary to prevent the need to output 3 bytes (or worse yet, 3 12-bit words for higher-resolution sensors) for each pixel location.

Not to worry, because camera manufacturers have developed clever ways of reducing the number of color samples necessary. The most common approach is to use a Color Filter Array (CFA), which measures only a single color at any given pixel location. Then, the results can be interpolated by the image processor to appear as if 3 colors were measured at every location.

The most popular CFA in use today is the Bayer pattern, shown in Figure 1. This scheme, invented by Kodak, takes advantage of the fact that the human eye discerns differences in green-channel intensities more than red or blue changes. Therefore, in the Bayer color filter array, the Green subfilter occurs twice as often as either the Blue or Red subfilter. This results in an output format sometimes known as '4:2:2 RGB', where 4 Green values are sent for every 2 Red and Blue values.

Figure 1: Bayer pattern image sensor arrangement
Connecting to Image Sensors
CMOS sensors ordinarily output a parallel digital stream of pixel components in either YCbCr or RGB format, along with horizontal and vertical synchronization and a pixel clock. Sometimes, they allow for an external clock and sync signals to control the transfer of image frames out from the sensor.

CCDs, on the other hand, usually hook up to an "Analog Front End" (AFE) chip, such as the AD9948, that processes the analog output signal, digitizes it, and generates appropriate timing to scan the CCD array. A processor supplies synchronization signals to the AFE, which needs this control to manage the CCD array. The digitized parallel output stream from the AFE might be in 10-bit, or even 12-bit, resolution per pixel component.

Recently, LVDS (low-voltage differential signaling) has become an important alternative to the parallel data bus approach. LVDS is a low-cost, low pin-count , high-speed serial interconnect that has better noise immunity and lower power consumption than the standard parallel approach. This is important as sensor resolutions and color depths increase, and as portable multimedia applications become more widespread.

Image Pipe
Of course, the picture-taking process doesn't end at the sensor; on the contrary, its journey is just beginning. Let's take a look at what a raw image has to go through before becoming a pretty picture on a display. In digital cameras, this sequence of processing stages is known as the "image processing pipeline," or just "image pipe." Refer to Figure 2 for one possible dataflow. These algorithms are typically performed on a media processor such as those in Analog Devices' Blackfin family.

Figure 2: Example Software Image Pipe Flow

Mechanical Feedback Control
Before the shutter button is even released, the focus and exposure systems work with the mechanical camera components to control lens position based on scene characteristics.

Auto-exposure algorithms measure brightness over discrete scene regions to compensate for overexposed or underexposed areas by manipulating shutter speed and/or aperture size. The net goals here are to maintain relative contrast between different regions in the image and to achieve a target average luminance.

Auto-focus algorithms divide into two categories. Active methods use infrared or ultrasonic emitters/receivers to estimate the distance between the camera and the object being photographed. Passive methods, on the other hand, make focusing decisions based on the received image in the camera.

In both of these subsystems, the media processor manipulates the various lens and shutter motors via PWM output signals. For auto-exposure control, it also adjusts the Automatic Gain Control (AGC) circuit of the sensor.

As we discussed earlier, a sensor's output needs to be gamma-corrected to account for eventual display, as well as to compensate for nonlinearities in the sensor's capture response.

Since sensors usually have a few inactive or defective pixels, a common preprocessing technique is to eliminate these via median filtering, relying on the fact that sharp changes from pixel to pixel are abnormal, since the optical process blurs the image somewhat.

Lens correction (shading / distortion correction)
This set of algorithms accounts for the physical properties of lenses that warp the output image compared to the actual scene the user is viewing. Different lenses can cause different distortions; for instance, wide-angle lenses create a "barreling" or "bulging" effect, while telephoto lenses create a "pincushion" or "pinching" effect.

Lens shading distortion reduces image brightness in the area around the lens. Chromatic aberration causes color fringes around an image. The media processor needs to mathematically transform the image in order to correct for these distortions.

Image stability compensation, or hand-shaking correction is another area of preprocessing. Here, the processor adjusts for the translational motion of the received image, often with the help of external transducers that relate the real-time motion profile of the sensor.

White balance is another important stage of preprocessing. When we look at a scene, regardless of lighting conditions, our eyes tend to normalize everything to the same set of natural colors. For instance, an apple looks deep red to us whether we're indoors under fluorescent lighting, or outside in sunny weather. However, an image sensor's "perception" of color depends largely on lighting conditions, so it needs to map its acquired image to appear "lighting-agnostic" in its final output. This mapping can be done either manually or automatically.

In manual systems, you point your camera at an object you determine to be "white," and the camera will then shift the "color temperature" of all images it takes to accommodate this mapping. Automatic White Balance (AWB), on the other hand, uses inputs from the image sensor and an extra white balance sensor to determine what should be regarded as "true white" in an image. It tweaks the relative gains between the R, G and B channels of the image. Naturally, AWB requires more image processing than manual methods, and it's another target of proprietary vendor algorithms.

De-mosaic / Pixel interpolation / Noise reduction / Edge enhancement
De-mosaicing is perhaps the most crucial and numerically intensive operation in the image pipeline. Each camera manufacturer typically has their own "secret recipe," but in general, the approaches fall into a few main algorithm categories.

Nonadaptive algorithms like bilinear interpolation or bicubic interpolation are among the simplest to implement, and they work well in smooth areas of an image. However, edges and texture-rich regions present a challenge to these straightforward implementations. Adaptive algorithms, those that change behavior based on localized image traits, can provide better results.

One example of an adaptive approach is edge-directed reconstruction. Here, the algorithm analyzes the region surrounding a pixel and determines in which direction to perform interpolation. If it finds an edge nearby, it interpolates along the edge, rather than across it. Another adaptive scheme assumes a constant hue for an entire object, and this prevents abrupt changes in color gradients within individual objects. Many other de-mosaicing approaches exist, some involving frequency-domain analysis, Bayesian probabilistic estimation, and even neural networks.

Color Transformation
In this stage, the interpolated RGB image is transformed to the targeted output color space (if not already in the right space). For compression or display to a television, this will usually involve an RGB®YCbCr matrix transformation, often with another gamma correction stage to accommodate the target display. The YCbCr outputs may also be chroma subsampled at this stage to the standard 4:2:2 format for color bandwidth reduction with little visual impact.

In this phase, the image is perfected via a variety of filtering operations before being sent to the display and/or storage media. For instance, edge enhancement, pixel thresholding for noise reduction, and color-artifact removal are all common at this stage.

Display / Compress / Store
Once the image itself is ready for viewing, the image pipe branches off in two different directions. In the first, the postprocessed image is output to the target display, usually an integrated LCD screen (but sometimes an NTSC/PAL television monitor, in certain camera modes). In the second, the image is sent to the media processor's compression algorithm, where industry-standard compression techniques (JPEG, for instance) are applied before the picture is stored locally in some storage medium (usually a non-volatile Flash memory card).

About the authors
David Katz is a Senior DSP Applications Engineer at Analog Devices, Inc., where he is involved in specifying and supporting Blackfin media processors. He has published dozens of embedded processor articles both domestically and internationally. Previously, he worked at Motorola, Inc., as a senior design engineer in cable modem and automation groups. David holds both a B.S. and M. Eng. in Electrical Engineering from Cornell University. He can be reached at

Rick Gentile joined ADI in 2000 as a Senior DSP Applications Engineer, and he currently leads the Blackfin DSP Applications Group. Prior to joining ADI, Rick was a Member of the Technical Staff at MIT Lincoln Laboratory, where he designed several signal processors used in a wide range of radar sensors. He received a B.S. in 1987 from the University of Massachusetts at Amherst and an M.S. in 1994 from Northeastern University, both in Electrical and Computer Engineering. He can be reached at


Anonymous said...

Hi I'd love to thank you for such a great made forum!
I was sure this is a perfect way to introduce myself!

Monte Phil
if you're ever bored check out my site!
[url=]unicorn Party Supplies[/url].

Anonymous said...

[url=]hip hop jewelry[/url],[url=]hip hop pendants[/url],hip hop watches,[url=]bling bling[/url] ,hip hop,[url=]hip hop chains[/url],hip hop bling,[url=]iced out chains[/url],[url=]wholesale chains[/url]
hip hop jewelry
wholesale hip hop watches
hip hop rings

Anonymous said...

minneapolis dating escort [url=]best online dating[/url] dreamlover mylovewebsite best international dating site singles groups georgia

Anonymous said...

Hello. My wife and I bought our house about 6 months ago. It was a foreclosure and we were able to get a great deal on it. We also took advantage of the 8K tax credit so that definitely helped. We did an extensive remodeling job and now I want to refinance to cut the term to a 20 or 15 year loan. Does anyone know any good sites for mortgage information? Thanks!


Anonymous said...

employment opportunities bs human health science [url=]casodex[/url] diflucan yasmin antique oak dental cabinet
meal preparation raw vegetable diet [url=]luvox[/url] dental student glossary [url=]indian health service properly disposing computers[/url]

Anonymous said...

males breast [url=]hangover helper[/url] thyroid pills with dyes medical diagnostic imaging phoenix
read blood work [url=]lopid[/url] small blood vessel ischemic disease [url=]vanadium diabetes[/url]

Anonymous said...
[url=]university of maryland medical center hospital pharmacy[/url] lescol xl and erectile dysfunction [url=]cetirizine alcohol absorptionproduct informat[/url]
cvs cialis
[url=]carrs pharmacy anchorage alaska[/url] cals pharmacy snowbords [url=]carbohydrate counter for alcohol[/url]
dot drug testing
[url=]online drugstore michigan[/url] wsu pharmacy and current students [url=]disulfiram[/url] erectile dysfunction massage [url=]xanax online pharmacy no prescription[/url]

Anonymous said...
[url=]drug digest org[/url] cms 11450 drugs [url=]cefdinir missed dose[/url]
laws for drugs and cosmetics india
[url=]erectile dysfunction on cocaine[/url] jim crow drug war [url=]cefdinir 300mg for sinus infections[/url]
inclusion body myositis and erectile dysfunction
[url=]miami drug court 2008[/url] cheap cialis softtabs [url=]claritin weight loss[/url] iowa drug conviction data base [url=]cetirizine symtoms[/url]

Anonymous said...
[url=]using drugs with pens[/url] nose drop bp drug [url=]estrasorb and prometrium[/url]
retail pharmacy
[url=]drug listing[/url] itch relief medicine [url=]nystatin and triamcinolone[/url]
catalytic converters and drugs
[url=]walmart four dollar drug list[/url] withdrawal cialis [url=]david mcculloch group health care[/url] major drug busts [url=]freee market health care[/url]

Anonymous said...
[url=]drugs for infections[/url] online medicine rx cialis viagra order [url=]gaba l glutamine[/url]
chloesterol drug
[url=]cialis problems[/url] caltrate drug class [url=]medrol diphenhydramine and catscan[/url]
use of new drug development
[url=]minnesota department of health surveys[/url] retail pharmacy highest salary pharmacy technician [url=]nasal polyps approval treatment rhinocort[/url] spasmodic dysphonia caused by drug use [url=]levitra slides[/url]

Anonymous said...
[url=]drugs ultram[/url] erectile problem [url=]cardizem cd lysergic acid diethylamide flomax[/url]
all natural drug for focusing and anxiety
[url=]cvs pharmacy drug prices[/url] couch levitra [url=]contradictions of cilostazol[/url]
cave man medicine
[url=]online pharmacys without perscription deliver overnight[/url] drug interaction search prozac ciprofloxacin [url=]what is cefdinir used for[/url] montreal pharmacy [url=]role of carvedilol in hypertension treatment[/url]

Anonymous said...

las vegas travel forums travel money for italy
[url=]best sky mileage credit card for european travel[/url] south india travel package [url=]map of antigua[/url]
travel journals traveler
[url=]travel conditions on i81 tennessee[/url] educational travel consultants [url=]turkmenistan cheap cruises[/url]
health travel intepid travel [url=]strattford inn[/url]

Anonymous said...

mercedes benze ml brake rotor removal mercedes 2008 calendar
[url=]auto sacramento show february[/url] volkswagen competencies [url=]land rover colorado[/url]
free code radio volkswagen
[url=]papyrus nascar racing cheat type talladega[/url] corporate office for napa auto parts jacksonville fl [url=]daytona spring bike rally[/url]
auto dealerships northwest houston
[url=]mercedes sl500 radio code[/url] auto shows ohio columbus [url=]aftermarket steel hoods for 442 oldsmobile[/url]

Anonymous said...

advantage auto mall fort myers dundalk racing
[url=]auto brake problems[/url] auto cell phone holders [url=]utilisation ordinateur de bord opel vectra[/url]
horse racing bet
[url=]jd power and associates 2008 automobile review[/url] mercedes benz ml500 [url=]volvo lease early termination pull ahead[/url]
mercedes dealers nc
[url=]aftermarket auto parts 198 nissan sentra[/url] car racing in central pa [url=]irish rover pub sarasota fl[/url]

Anonymous said...

mephisto shoes new york mason shoes [url=]szilvia lauren[/url] couture designer gown wedding camerons clothes the holiday [url=]cloud nine shoes[/url]

Anonymous said...

nusring shoes shoes for heel spur [url=]brand extension gucci[/url] womens ryka athletic shoes designer takes flight with stuffed birds [url=]naked girls taking off each others clothes[/url]

Anonymous said...

pictures of fashion models cyramed form designer [url=]gardiner fashion[/url] lightning bolt clothes thrown off why casual shoes [url=]graphic designers[/url]

Anonymous said...

parvovirus puppy anal pain
[url=]kanalen adult xxx nl[/url] pandora s box adult comics [url=]gay friendly doctors iowa[/url]
vibrating dildo
[url=]xxx adult yahoo groups[/url] french maid free amateur video [url=]your amteur porn[/url]
dildo porn gallerys
[url=]sexy hot blonde naked[/url] automatic dildo machine [url=]gay anal bareback[/url]
strap on dildo story
[url=]anal stenosis[/url] free girl anal movie [url=]free celebrity lesbian clips[/url]

Anonymous said...

adult video engine sifuni
[url=]sexy girl strech[/url] anal stimulation for men [url=]fetish japan love boots[/url]
podcast directory adult
[url=]anal wart[/url] porn dans [url=]adult video pics[/url]
playboy sexy lingerie series
[url=]ridan active adult at leelan station[/url] barbie griffin dildo [url=]mom oral sex[/url]
women dildoing men porn
[url=]adult chat free fuck[/url] free sexy videos online [url=]what is oral communication[/url]

Anonymous said...

fda position on glp for medical devices [url=]atacand[/url] sunburst medical
number of lives saved yearly by smoke detectors [url=]flonase[/url] effect of exercise on blood pressure and heart rate [url= ]medical school in cuba [/url] breast cyst grains
best herbs to lower blood pressure [url=]cialis soft tabs[/url] ready health care
medical thermometer conversions celsius to fahrenheit [url=]leukeran[/url] alternative medicine sacramento [url= ]the heart of the game music [/url] hamilton tooth whitening

Anonymous said...

home remedy herpes [url=]seroquel[/url] diabetes testing device
dr atkins low carb diet [url=]medications without a prescription[/url] natural penis erection [url= ]health gallerys [/url] calicum deposits in the breast
diuretics for congestive heart failure [url=]valtrex[/url] chemicals health and safety standards uk
emergency health insurance blue cross [url=]anti hongos anti ansiedad[/url] journal topics health [url= ]flow of blood through heart animation [/url] brassler dental

Anonymous said...

lauren davis poland spring harley davidson kid clothes [url=]dog walking shoes[/url] orthotic shoes insert latest fashion in spain [url=]redwing shoes[/url]

Anonymous said...

hard nipples on breast is caused from [url=]antivirali[/url] medical office label systems
cause of calcium buildup in breast [url=]septilin[/url] drug lists [url= ]bacteria beaten egg whites [/url] health insurance missouri rates
behavior disorder in children learning [url=]anti allergic asthma[/url] vitamin need
health clinic henry county ky [url=]ditropan[/url] gum frenum [url= ]warrensburg health center [/url] southern oregon dental society

Anonymous said...

street names for the drug heroine [url=]imuran[/url] health plan registries
carbohydrate diet food low [url=]vitamin c[/url] what can i use from home to pass a drug test [url= ]notes on book bury my heart at wounded knee [/url] dog drug prescription
relationship between drugs and crime [url=]mobic[/url] how to avoid medical collections
burn itchy penis head [url=]zyprexa[/url] dental scraps [url= ]gold heart candles [/url] pharmacy course

Anonymous said...
[url=]why drugs laws should be kept as they are[/url] young people abusing prescription drugs
eli lilly south africa cialis news 2008
[url=]does levitra inhibit orgasm[/url] sleep drugs touch
medicare blue part d drug formulary
[url=]longs drug stores lahaina hi[/url] seizure drug lamictal about levitra

Anonymous said...

gucci handbag imitation wholesale shoes distributor [url=]betty boop shoes[/url] childrens high heeled shoes visual merchandising of christian dior [url=]clothesfree[/url]

Anonymous said...

cheap travel fares british berkefeld arctic travel pure
[url=]travel chicago[/url] low claim cost travel insurance [url=]studio six hotel motel[/url]
southwest travel
[url=]train travel transmongolian[/url] gift ideas for people who travel [url=]discount airline to new york city[/url]
travel toothbrush sanitizer with toothbrush sarah watson india travel [url=]palace hotel san moritz[/url]

Anonymous said...

deauville fashions lauren crane nude [url=]josef seibel shoes online[/url] lauren crane nude urnials designer [url=]new look clothes shop[/url]

Anonymous said...

babes talk off their clothes laurens county sc platt book [url=]henschenklein[/url] climbing shoes designer jewelry [url=]ballerina shoes[/url]

Anonymous said...
[url=]drugs for treating sore throat[/url] mesalamine drug interactions [url=]amoxil 500[/url]
pharmacys in lincoln ne
[url=]factors for drug abuse[/url] highs with out failing drug tests [url=]amo ramipril[/url]
drug free bulletin boartd
[url=]common type two diabetes drugs[/url] levitra alternative lavitra [url=]equine dosage cetirizine[/url] celebrex arthritic drug [url=]bextra celecoxib[/url]

Anonymous said...

samsung blackjack master reset history of the lottery ticket in the renissances
[url=]blackjack play best internet casino[/url] casino morgan stanley atlantic city [url=]germany lottery[/url]
gambling horse racing slots casino indian internet
[url=]grand victoria casino gambling boat[/url] online sports betting with paypal [url=]map of the casinos in tunica ms[/url]
ron harris keno immigration lottery [url=]casino clasic[/url]

Anonymous said...

movie jungle fever spike lee [url=]corpses[/url] movie where asks how was the idiot convention new jurrasic park movie
dirty dancing the movie [url=]the brother from another planet[/url] reasons watch simpsons movie times movie making programs
coming home movie [url=]choose me[/url] new lord of the rins movie
charade movie [url=]executive decision[/url] how can i watch game plan the movie on the computer free online movie trailers
bombastic movie [url=]over the hedge[/url] rogue movie bittorrent movie torrents