Fundamentals of embedded video, part 5

Let's walk through the system of Figure 1 to illustrate some fundamental video processing steps present in various combinations in an embedded video application. In the diagram, an interlaced-scan CMOS sensor sends a 4:2:2 YCbCr video stream through the processor's video port, at which point it is deinterlaced and scan-rate converted. It then passes through some computational algorithm(s) and is prepared for output to an LCD panel. This preparation involves chroma resampling, gamma correction, color conversion, scaling, blending with graphics, and packing into the appropriate output format for display on the LCD panel. Note that this system is only provided as an example, and not all of these components are necessary in a given system. Additionally, these steps may occur in a different order than shown here.

Figure 1. Example flow of camera input to LCD output, with processing stages in-between

Deinterlacing
When taking video source data from a camera that outputs interlaced NTSC data, it's often necessary to deinterlace it so that the odd and even lines are interleaved in memory, instead of being located in two separate field buffers. Deinterlacing is needed not only for efficient block-based video processing, but also for displaying interlaced video in progressive format (for instance, on an LCD panel). There are many ways to deinterlace, including line doubling, line averaging, median filtering, and motion compensation.

Scan Rate Conversion
Once the video has been deinterlaced, a scan-rate conversion process may be necessary, in order to insure that the input frame rate matches the output display refresh rate. In order to equalize the two, fields may need to be dropped or duplicated. Of course, as with deinterlacing, some sort of filtering is preferable in order to smooth out high-frequency artifacts caused by creating abrupt frame transitions.

A special case of frame rate conversion that's employed to convert a 24 frame/sec stream (common for 35 mm and 70 mm movie recordings) to the 30 frame/sec required by NTSC video is 3:2 pulldown. For instance, motion pictures recorded at 24 fps would run 25% faster (=30/24) if each film frame is used only once in an NTSC video system. Therefore, 3:2 pulldown was conceived to adapt the 24 fps stream into a 30 fps video sequence. It does so by repeating frames in a certain periodic pattern, shown in Figure 2.

Figure 2. 3:2 pulldown frame repetition pattern
(24 fps progressive movie 30 fps interlaced TV)

Pixel Processing
As we discussed above, there are a lot of video algorithms in common use, and they're stratified into spatial and temporal classifications. One particularly common video operator is the two-dimensional (2D) convolution kernel, which is used for many different forms of image filtering.

2D convolution
Since a video stream is really an image sequence moving at a specified rate, image filters need to operate fast enough to keep up with the succession of input images. Thus, it is imperative that image filter kernels be optimized for execution in the lowest possible number of processor cycles. This can be illustrated by examining a simple image filter set based on two-dimensional convolution.

Convolution is one of the fundamental operations in image processing. In a two-dimensional convolution, the calculation performed for a given pixel is a weighted sum of intensity values from pixels in its immediate neighborhood. Since the neighborhood of a mask is centered on a given pixel, the mask usually has odd dimensions. The mask size is typically small relative to the image. A 3x3 mask is a common choice, because it is computationally reasonable on a per-pixel basis, yet large enough to detect edges in an image. However, it should be noted that 5x5, 7x7 and beyond are also widely used. Camera image pipes, for example, can employ 11x11 (and larger!) kernels for extremely complex filtering operations.

The basic structure of the 3x3 kernel is shown in Figure 3a. As an example, the output of the convolution process for a pixel at row 20, column 10 in an image would be: Out(20,10)=A*(19,9)+B*(19,10)+C*(19,11)+D*(20,9)+E*(20,10)+ F*(20,11)+G*(21,9)+H*(21,10)+I*(21,11)

Figure 3. The 3x3 Convolution mask and how it can be used

It is important to choose coefficients in a way that aids computation. For instance, scale factors that are powers of 2 (including fractions) are preferred because multiplications can then be replaced by simple shift operations.

Figures 3b-e show several useful 3x3 kernels, each of which is explained briefly below.

The Delta Function shown in Figure 3b is among the simplest of image manipulations—it passes the current pixel through without modification.

Figure 3c shows two popular forms of an edge detection mask. The first one detects vertical edges, while the second one detects horizontal edges. High output values correspond to higher degrees of edge presence.

The kernel in Figure 3d is a smoothing filter. It performs an average of the 8 surrounding pixels and places the result at the current pixel location. This has the effect of "smoothing," or low-pass filtering, the image.

The filter in Figure 3e is known as an "unsharp masking" operator. It can be considered as producing an edge-enhanced image by subtracting from the current pixel a smoothed version of itself (constructed by averaging the 8 surrounding pixels).

Dealing with Image Boundaries
What happens when a function like 2D convolution operates on pixels near an image's border regions? To properly perform pixel filtering requires pixel information "outside" these boundaries. There are a couple of remedies for this situation. The simplest is just to ignore these edge regions. That is, consider that a 5x5 convolution kernel needs 2 pixels to the left, top, bottom, and right of the current pixel in order to function properly. Therefore, why not just shave 2 rows off of the image in each direction, so as to guarantee that the kernel will always act on real data? Of course, this isn't always an ideal approach, since it throws out real image data. Also, in cases where filters are strung together to create more complex pixel manipulations, this scheme will continually narrow the input image with every new filter stage that executes.

Other popular ways of handling the image boundary quandary are to duplicate rows and/or columns of pixels, or to wrap around from the left (top) edge back to the previous right (bottom) edge. While these might be easy to implement in practice, they create data that didn't exist before, and therefore corrupt filtering results to some extent.

Perhaps the most straightforward, and least damaging, method for dealing with image boundaries is to consider everything that lies outside of the actual image to be zero-valued, or black. Although this scheme, too, distorts filtering results, it is not as invasive as creating lines of potentially random non-zero-valued pixels.

Chroma Resampling and Color Conversion
Ultimately, the data stream in our example needs to be converted to RGB space. We already discussed how to convert between 4:4:4 YCbCr and RGB spaces, via a 3x3 matrix multiplication. However, up to this point, our pixel values are still in 4:2:2 YCbCr space. Therefore, we need to resample the chroma values to achieve a 4:4:4 format. Then the transformation to RGB will be straightforward, as we've already seen.

Resampling from 4:2:2 to 4:4:4 involves interpolating Cb and Cr values for those Y samples that are missing one of these components. A clear-cut way to resample is to interpolate the missing chroma values from their nearest neighbors by simple averaging. That is, a missing Cb value at a pixel site would be replaced by the average of the nearest two Cb values. Higher-order filtering might be necessary for some applications, but this simplified approach is often sufficient. Another approach is to replicate the chrominance values of neighboring pixels for those values that are missing in the current pixel's representation.

In general, conversions from 4:1:1 space to 4:2:2 or 4:4:4 formats involve only a one-dimensional filter (with tap values and quantities consistent with the level of filtering desired). However, resampling from 4:2:0 format into 4:2:2 or 4:4:4 format involves vertical sampling as well, necessitating a two-dimensional convolution kernel.

Because chroma resampling and YCbCr→RGB conversion are both linear operations, it is possible to combine the steps into a single mathematical operation, thus achieving 4:2:2 YCbCr→RGB conversion efficiently.

Scaling and Cropping
Video scaling allows the generation of an output stream whose resolution is different from that of the input format. Ideally, the fixed scaling requirements (input data resolution, output panel resolution) are known ahead of time, in order to avoid the computational load of arbitrary scaling between input and output streams.

Depending on the application, scaling can be done either upwards or downwards. It is important to understand the content of the image to be scaled (e.g., the presence of text and thin lines). Improper scaling can make text unreadable or cause some horizontal lines to disappear in the scaled image.

The easiest method to adjust an input frame size to an output frame that's smaller is simply to crop the image. For instance, if the input frame size is 720x480 pixels, and the output is a VGA frame (640x480 pixels), you can drop the first 40 and the last 40 pixels on each line. The advantage here is that there are no artifacts associated with dropping pixels or duplicating them. Of course, the disadvantage is that you'd lose 80 pixels (about 11%) of frame content. Sometimes this isn't too much of an issue, because the leftmost and rightmost extremities of the screen (as well as the top and bottom regions) are often obscured from view by the mechanical enclosure of the display.

If cropping isn't an option, there are several ways to downsample (reduce pixel and/or line count) or upsample (increase pixel and/or line count) an image that allow tradeoffs between processing complexity and resultant image quality.

Increasing or decreasing pixels per row
One straightforward method of scaling involves either dropping pixels (for downsampling) or duplicating existing pixels (for upsampling). That is, when scaling down to a lower resolution, some number of pixels on each line (and/or some number of lines per frame) can be thrown away. While this certainly reduces processing load, the results will yield aliasing and visual artifacts.

A small step up in complexity uses linear interpolation to improve the image quality. For example, when scaling down an image, filtering in either the horizontal or vertical direction obtains a new output pixel, which then replaces the pixels used during the interpolation process. As with the previous technique, information is still thrown away, and artifacts and aliasing will be present again.

If image quality is paramount, there are other ways to perform scaling without reducing quality. These methods strive to maintain the high frequency content of the image consistent with the horizontal and vertical scaling, while reducing the effects of aliasing. For example, assume that an image is to be scaled by a factor of Y:X. To accomplish this scaling, the image could be upsampled ("interpolated") by a factor of Y, filtered to eliminate aliasing, and then downsampled ("decimated") by a factor of X. In practice, these two sampling processes can be combined into a single multirate filter.

Increasing or Reducing lines per frame
The guidelines for increasing or reducing the number of pixels per row generally extend to modifying the number of lines per frame of an image. For example, throwing out every other line (or one entire interlaced field) provides a quick method of reducing vertical resolution. However, as we've mentioned above, some sort of vertical filtering is necessary whenever removing or duplicating lines, because these processes introduce artifacts into the image. The same filter strategies apply here: simple vertical averaging, higher-order FIR filters, or multirate filters to scale vertically to an exact ratio.

Display Processing
Alpha Blending
Often it is necessary to combine two image and/or video buffers prior to display. A practical example of this is overlaying of icons like signal strength and battery level indicators onto a cellular phone's graphics display. An example involving two video streams is picture-in-picture functionality.

When combining two streams, you need to decide which stream "wins" in places where content overlaps. This is where alpha blending comes in. It defines a variable alpha (a) that indicates a "transparency factor" between an overlay stream and a background stream as follows:

Output value = α (foreground pixel value) + (1-α) (background pixel value)

As the equation shows, an α value of 0 results in a completely transparent overlay, whereas a value of 1 results in a completely opaque overlay that disregards the background image entirely.

Alpha is sometimes sent as a separate channel along with the pixel-wise luma and chroma information. This results in the notation "4:2:2:4," where the last digit indicates an alpha key that accompanies each 4:2:2 pixel entity. Alpha is coded in the same way as the luma component, but often only a few discrete levels of transparency (perhaps 16) are needed for most applications. Sometimes a video overlay buffer is premultiplied by alpha or premapped via a lookup table, in which case it's referred to as a "shaped" video buffer.

Compositing
The act of compositing involves positioning an overlay buffer inside a larger image buffer. Common examples are a "Picture-in-Picture" mode on a video display, and placement of graphics icons (like battery and signal strength indicators) over the background image or video. In general, the composition function can take several iterations before the output image is complete. In other words, there may be many "layers" of graphics and video that combine to generate a composite image.

Two-dimensional DMA capability is very useful for compositing functions, because it allows the positioning of arbitrarily-sized rectangular buffers inside a larger buffer. One thing to keep in mind is that any image cropping should take place after the composition process, because the positioned overlay might violate any previously cropped boundaries. Of course, an alternative is to ensure that the overlay won't violate the boundaries in the first place, but this is sometimes asking too much!

Chroma Keying
The term "chroma keying" refers to a process by which a particular color (usually blue or green) in one image is replaced by the content in a second image when the two are composited together. This provides a convenient way to combine two video images by purposefully tailoring parts of the first image to be replaced by the appropriate sections of the second image. Chroma keying can be performed in either software or hardware on a media processor.

Output Formatting
Most color LCD displays targeted for consumer applications (TFT-LCDs) have a digital RGB interface. Each pixel in the display actually has 3 subpixels—one each with Red, Green and Blue filters—that the human eye resolves as a single color pixel. For example, a 320x240 pixel display actually has 960x240 pixel components, accounting for the R, G, and B subpixels. Each subpixel has 8 bits of intensity, thus forming the basis of the common 24-bit color LCD display.

The three most common configurations use either 8 bits per channel for RGB (RGB888 format), 6 bits per channel (RGB666 format), or 5 bits per channel for R and B, and 6 bits for G (RGB565 format).

RGB888 provides the greatest color clarity of the three. With a total of 24 bits of resolution, this format provides over 16 million shades of color. It offers the high resolution and precision needed in high performance applications like LCD TVs.

The RGB666 format is popular in portable electronics. Providing over 262,000 shades of color, this format has a total of 18 bits of resolution. However, because the 18-pin (6+6+6) data bus doesn't conform nicely to 16-bit processor data paths, a popular industry compromise is to use 5 bits each of R and B, and 6 bits of G (5+5+6 = a 16-bit data bus) to connect to a RGB666 panel. This scenario works well because green is the most visually important color of the three. The least-significant bits of both Red and Blue are tied at the panel to their respective most-significant bits. This ensures a full dynamic range for each color channel (full intensity down to total black).

We hope that this article series has given you a good understanding of the basics involved in embedded video processing. For a more in-depth discussion on media processing issues, including data flow and media framework analyses, you may wish to refer to "Embedded Media Processing."

This series is adapted from the book "Embedded Media Processing" (Newnes 2005) by David Katz and Rick Gentile. See the book's web site for more information.

Fundamentals of embedded video, part 4

Video Port Features
To handle video streams, processors must have a suitable interface that can maintain a high data transfer rate into and out of the part. Some processors accomplish this through an FPGA and/or FIFO connected to the processor's external memory interface. Typically, this device will negotiate between the constant, relatively slow stream of video (~27 MB/s for NTSC video) into/out of the processor and the sporadic but speedy and bursty nature of the external memory controller (~133 MWords/sec, or 266 MB/s).

However, there are problems with this arrangement. For example, FPGAs and FIFOs are expensive, often costing as much as the video processor itself. Additionally, using the external memory interface for video transfer steals bandwidth from its other prime use in these systems—moving video buffers back and forth between the processor core and external memory.

Therefore, a dedicated video interface is highly preferable for media processing systems. For example, on Blackfin processors, this is the Parallel Peripheral Interface (PPI). The PPI is a multifunction parallel interface that can be configured between 8 and 16 bits in width. It supports bi-directional data flow and includes three synchronization lines and a clock pin for connection to an externally supplied clock. The PPI can gluelessly decode ITU-R BT.656 data and can also interface to ITU-R BT.601 video sources and displays, as well as TFT-LCD panels. It can serve as a conduit for high-speed analog-to-digital converters (ADCs) and digital-to-analog converters (DACs). It can also emulate a host interface for an external processor.

The PPI has some built-in features that can reduce system costs and improve data flow. For instance, in BT.656 mode the PPI can decode an input video stream and automatically ignore everything except active video, effectively reducing an NTSC input's video stream rate from 27 MB/s to 20 MB/s, and markedly reducing the amount of off-chip memory needed to handle the video. Alternately, it can ignore active video regions and only read in ancillary data that's embedded in vertical blanking intervals. These modes are shown pictorially in Figure 1.

Figure 1: Selective Masking of BT.656 regions in PPI.

Likewise, the PPI can "ignore" every other field of an interlaced stream; in other words, it will not forward this data to the DMA controller. While this instantly decimates input bandwidth requirements by 50%, it also eliminates 50% of the source content, so sometimes this tradeoff might not be acceptable. Nevertheless, this can be a useful feature when the input video resolution is much greater than the required output resolution.

On a similar note, the PPI allows "skipping" of odd- or even-numbered elements, again saving DMA bandwidth for the skipped pixel elements. For example, in a 4:2:2 YCbCr stream, this feature allows only luma or chroma elements to be read in, providing convenient partitioning of an algorithm between different processors; one can read in the luma, and the other can read the chroma. Also, it provides a simple way to convert an image or video stream to grayscale (luma-only). Finally, in high-speed converter applications with interleaved I/Q data, this feature allows partitioning between in-phase and quadrature components.

Importantly, the PPI is format-agnostic, in that it is not hardwired to a specific video standard. It allows for programmable row lengths and frame lengths. This aids applications that need, say, CIF or QCIF video instead of standard NTSC/PAL formats. In general, as long as the incoming video has the proper EAV/SAV codes (for BT.656 video) or hardware synchronization signals (for BT.601 video), the PPI can process it.

Packing
Although the BT.656 and BT.601 recommendations allow for 10-bit pixel elements, this is not a very friendly word length for processing. The problem is that most processors are very efficient at handling data in 8-bit, 16-bit or 32-bit chunks, but anything in-between results in data movement inefficiencies. For example, even though a 10-bit pixel value is only 2 bits wider than an 8-bit value, most processors will treat it as a 16-bit entity with the 6 most significant bits (MSBs) set to 0. Not only does this waste bandwidth on the internal data transfer (DMA) buses, but it also wastes a lot of memory – a disadvantage in video applications, where several entire frame buffers are usually stored in external memory.

A related inefficiency associated with data sizes larger than 8 bits is non-optimal packing. Usually, a high-performance media processor will imbue its peripherals with a data packing mechanism that sits between the outside world and the internal data movement buses of the processor, and its goal is to minimize the overall bandwidth burden that the data entering or exiting the peripheral places on these buses. Therefore, an 8-bit video stream clocking into a peripheral at 27 MB/s might be packed onto a 32-bit internal data movement bus, thereby requesting service from this bus at a rate of only 27/4, or 6.75 MHz. Note that the overall data transfer rate remains the same (6.75 MHz * 32 bits = 27 MB/s). In contrast, a 10-bit video stream running at 27 MB/s would only be packed onto the 32-bit internal bus in 2 16-bit chunks, reducing the overall transfer rate to 27/2, or 13.5 MHz. In this case, since only 10 data bits out of every 16 are relevant, 37.5% of the internal bus bandwidth is wasted.

Possible Data Flows
It is instructive to examine some ways in which a video port connects in multimedia systems, to show how the system as a whole is interdependent on each component flow. In Figure 2a, an image source sends data to the PPI, at which point the DMA engine transfers it to L1 memory, where the data is processed to its final form before being sent out through a high-speed serial port. This model works very well for low-resolution video processing and for image compression algorithms like JPEG, where small blocks of video (several lines worth) can be processed and are not needed again subsequently. This flow also can work well for some data converter applications.

In Figure 2b, the video data is not routed to L1 memory, but instead is directed to L3 memory. This configuration supports algorithms such as MPEG-2 and MPEG-4, which require storage of intermediate video frames in memory in order to perform temporal compression. In such a scenario, a bidirectional DMA stream between L1 and L3 memories allows for transfers of pixel macroblocks and other intermediate data.

Figure 2: Possible video port data transfer scenarios.

Video ALUs
Most video applications need to deal with 8-bit data, since individual pixel components (whether RGB or YCbCr) are usually byte quantities. Therefore, 8-bit video ALUs and byte-based address generation can make a huge difference in pixel manipulation. This is a nontrivial point, because embedded processors typically operate on 16-bit or 32-bit boundaries.

Embedded media processors sometimes have instructions that are geared to processing 8-bit video data efficiently. For instance, Table 1 shows a summary of the specific Blackfin instructions that can be used together to handle a variety of video operations.

Table 1: Native Blackfin video instructions.

Let's look at a few examples of how these instructions can be used.

The Quad 8-bit Subtract-Absolute-Accumulate (SAA) instruction is well-suited for block-based video motion estimation. The instruction subtracts four pairs of bytes, takes the absolute value of each difference, and accumulates the results. All this happens within a single cycle. The actual formula is shown below:

Consider the macroblocks shown in Figure 3a. The reference frame of 16 pixels x 16 pixels can be further divided into 4 groups. A very reasonable assumption is that neighboring video frames are correlated with each other. That is, if there is motion, then pieces of each frame will move in relation to macroblocks in previous frames. It takes less information to encode the movement of macroblocks than it does to encode each video frame as a separate entity—MPEG compression uses this technique.

This motion detection of macroblocks decomposes into two basic steps. Given a reference macroblock in one frame, we can search all surrounding macroblocks (target macroblocks) in a subsequent frame to determine the closest match. The offset in location between the reference macroblock (in Frame n) and the best-matching target macroblock (in Frame n+1) is the motion vector.

Figure 3b shows how this can be visualized in a system.

• Circle = some object in a video frame
• Solid square = reference macroblock
• Dashed square = search area for possible macroblocks
• Dotted square = best-matching target macroblock (i.e., the one representing the motion vector of the circle object)Figure 3: Illustration of Subtract-Absolute-Accumulate (SAA) instruction.

  The SAA instruction on a Blackfin processor is fast because it utilizes four 8-bit ALUs in each clock cycle. We can implement the following loop to iterate over each of the four entities shown in Figure 3b.

  /* used in a loop that iterates over an image block */

  SAA (R1:0,R3:2) || R1 = [I0++] || R2 = [I1++]; /* compute absolute difference and accumulate */

  SAA (R1:0,R3:2) (R) || R0 = [I0++] || R3 = [I1++];

  SAA (R1:0,R3:2) || R1 = [I0 ++ M3] || R2 = [I1++M1]; /* after fetch of 4th word of target block, pointer is made to point to the next row */

  SAA (R1:0,R3:2) (R) || R0 = [I0++] || R2 = [I1++];

  Let's now consider another example, the 4-Neighborhood Average computation whose basic kernel is shown in Figure 4a. Normally, four additions and one division (or multiplication or shift) are necessary to compute the average. The BYTEOP2P instruction can accelerate the implementation of this filter.

  The value of the center pixel of Figure 4b is defined as:

  x = Average(xN, xS, xE, xW)

  The BYTEOP2P can perform this kind of average on two pixels (Figures 6.21c,d) in 1 cycle. So, if x1 = Average(x1N, x1S, x1E, x1W), and x2 = Average(x2N, x2S, x2E, x2W), then

  R3 = BYTEOP2P(R1:0, R3:2)

  will compute both pixel averages in a single cycle, assuming the x1 (N, S, E, W) information is stored in registers R1 and R0, and the x2 (N, S, E, W) data is sourced from R3 and R2.

Figure 4: Neighborhood Average Computation.

DMA Considerations
An embedded media processor with two-dimensional DMA (2D DMA) capability offers several system-level benefits. For starters, 2D DMA can facilitate transfers of macroblocks to and from external memory, allowing data manipulation as part of the actual transfer. This eliminates the overhead typically associated with transferring non-contiguous data. It can also allow the system to minimize data bandwidth by selectively transferring, say, only the desired region of an input image, instead of the entire image.

As another example, 2D DMA allows data to be placed into memory in a sequence more natural to processing. For example, as shown in Figure 5, RGB data may enter a processor's L2 memory from a CCD sensor in interleaved RGB444 format, but using 2D DMA, it can be transferred to L3 memory in separate R, G and B planes. Interleaving/deinterleaving of color space components for video and image data saves additional data moves prior to processing.

Figure 5: Deinterleaving data with 2D DMA.

Planar vs. Interleaved Buffer Formats
How do you decide whether to structure your memory buffers as interleaved or planar? The advantage to interleaved data is that it's the natural output format of image sensors, and the natural input format for video encoders. However, planar buffers (that is, separate memory regions for each pixel component) are more effective structures for many video algorithms, since many of them (JPEG and MPEG included) work on luma and chroma channels separately. What's more, accessing planar buffers in L3 is more efficient than striding through interleaved data, because the latency penalty for SDRAM page misses is spread out over a much larger sample size when the buffers are structured in a planar manner.

Double-Buffering
We had previously discussed the need for double-buffering as a means of ensuring that current data is not overwritten by new data until you're ready for this to happen. Managing a video display buffer serves as a perfect example of this scheme. Normally, in systems involving different rates between source video and the final displayed content, it's necessary to have a smooth switchover between the old content and the new video frame. This is accomplished using a double-buffer arrangement. One buffer points to the present video frame, which is sent to the display at a certain refresh rate. The second buffer fills with the newest output frame. When this latter buffer is full, a DMA interrupt signals that it's time to output the new frame to the display. At this point, the first buffer starts filling with processed video for display, while the second buffer outputs the current display frame. The two buffers keep switching back and forth in a "ping-pong" arrangement.

It should be noted that multiple buffers can be used, instead of just two, in order to provide more margin for synchronization, and to reduce the frequency of interrupts and their associated latencies.

So now we've covered some basic issues and features associated with efficient video data movement in embedded applications. In the final part of this series, we extend these ideas into a "walkthrough" of a sample embedded video application.

This series is adapted from the book "Embedded Media Processing" (Newnes 2005) by David Katz and Rick Gentile. See the book's web site for more information.

Fundamentals of embedded video, part 3

Figure 1 shows a typical end-to-end embedded digital video system. In one case, a video source feeds into a media processor (after being digitized by a video decoder, if necessary). There, it might be compressed via a software encode operation before being stored locally or sent over the network.

Figure 1: System Video Flow for Analog/Digital Sources and Displays.

In an opposite flow, a compressed stream is retrieved from a network or mass storage. It is then decompressed via a software decode operation and sent directly to a digital output display (like a TFT-LCD panel), perhaps being first converted to analog form by a video encoder for display on a conventional CRT.

Keep in mind that compression/decompression represent only a subset of possible video processing algorithms that might run on the media processor. Still, for our purposes, they set a convenient template for discussion. Let's examine in more detail the video-specific portions of these data flows.

Analog Video Sources
Embedded processor cores cannot deal with analog video directly. Instead, the video must be digitized first, via a video decoder. This device converts an analog video signal (e.g., NTSC, PAL, CVBS, S-Video) into a digital form (usually of the ITU-R BT.601/656 YCbCr or RGB variety). This is a complex, multi-stage process. It involves extracting timing information from the input, separating luma from chroma, separating chroma into Cr and Cb components, sampling the output data, and arranging it into the appropriate format. A serial interface such as SPI or I2C configures the decoder's operating parameters. Figure 2 shows a block diagram of a representative video decoder.

Figure 2: Block diagram of ADV7183B video decoder.

Digital Video Sources
Camera sources today are overwhelmingly based on either Charge-Coupled Device (CCD) or CMOS technology. Both of these technologies convert light into electrical signals, but they differ in how this conversion occurs.

CMOS sensors ordinarily output a parallel digital stream of pixel components in either YCbCr or RGB format, along with horizontal and vertical synchronization and a pixel clock. Sometimes, they allow for an external clock and sync signals to control the transfer of image frames out from the sensor.

CCDs, on the other hand, usually hook up to an "Analog Front End" (AFE) chip, such as the AD9948, that processes the analog output signal from the CCD array, digitizes it, and generates appropriate timing to scan the CCD array. A processor supplies synchronization signals to the AFE, which needs this control to manage the CCD array. The digitized parallel output stream from the AFE might be in 10-bit, or even 12-bit, resolution per pixel component.

For a more detailed discussion on tradeoffs between CMOS and CCD sensors, as well as an overview of a typical image processing pipeline, please refer to the article CCD and CMOS image sensor processing pipeline.

Analog Video Displays
Video Encoder
A video encoder converts a digital video stream into an analog video signal. It typically accepts a YCbCr or RGB video stream in either ITU-R BT.656 or BT.601 format and converts it to a signal compliant with one of several different output standards (e.g., NTSC, PAL, SECAM). A host processor controls the encoder via a 2- or 3-wire serial interface like SPI or I2C, programming such settings as pixel timing, input/output formats, and luma/chroma filtering. Figure 3 shows a block diagram of a representative encoder.

Figure 3: Block diagram of ADV7179 video encoder.

Video encoders commonly output in one or more of the following analog formats:

CVBS – This acronym stands for Composite Video Baseband Signal (or Composite Video Blanking and Syncs). Composite video connects through the ubiquitous yellow RCA jack shown in Figure 4a. It contains Luma, Chroma, Sync and Color Burst information all on the same wire.

S Video, using the jack shown in Figure 4b, sends the luma and chroma content separately. Separating the brightness information from the color difference signals dramatically improves image quality, which accounts for the popularity of S Video connections on today's home theater equipment.

Component Video – Also known as YPbPr, this is the analog version of YCbCr digital video. Here, the luma and each chroma channel are output separately, each with its own timing. This offers maximum image quality for analog transmission. Component connections are very popular on higher-end home theatre system components like DVD players and A/V receivers (Figure 4c).

Analog RGB has separate channels for Red, Green and Blue signals. Although this offers image quality similar to Component Video, this format is generally used in the computer graphics realm, whereas Component Video is primarily employed in the consumer electronics arena. RGB connectors are usually of the BNC variety, shown in Figure 4d.

Figure 4: Common Analog Video Connectors.

Cathode Ray Tubes (CRTs)
On the display side, RGB is the most popular interface to computer monitors and LCD panels. Most older computer monitors accept analog RGB inputs on 3 separate pins from the PC video card and modulate 3 separate electron gun beams to generate the image. Depending on which beam(s) excite a phosphor point on the screen, that point will glow either red, green, blue, or some combination of these colors. This is different from analog television, where a composite signal (one that includes all color information superimposed on a single input) modulates a single electron beam. Newer computer monitors use DVI, or Digital Visual Interface, to accept RGB information in both digital and analog formats.

The main advantages of CRTs are that they are very inexpensive and can produce more colors than a comparably sized LCD panel. Also, unlike LCDs, they can be viewed from any angle. On the downside, CRTs are very bulky, emit considerable electromagnetic radiation, and can cause eyestrain due to their refresh-induced flicker.

Liquid Crystal Display (LCD) Panels
There are two main categories of LCD technology: passive matrix and active matrix. In the former (whose common family members include STN, or "Super Twisted Nematic," derivatives), a glass substrate imprinted with rows forms a "liquid crystal sandwich" with a substrate imprinted with columns. Pixels are constructed as row-column intersections. Therefore, to activate a given pixel, a timing circuit energizes the pixel's column while grounding its row. The resultant voltage differential untwists the liquid crystal at that pixel location, which causes it to become opaque and block light from coming through.

Straightforward as it is, passive matrix technology does have some shortcomings. For one, screen refresh times are relatively slow (which can result in "ghosting" for fast-moving images). Also, there is a tendency for the voltage at a row-column intersection to "bleed" over into neighboring pixels, partly untwisting the liquid crystals and blocking some light from passing through the surrounding pixel area. To the observer, this blurs the image and reduces contrast. Moreover, the viewing angle is relatively narrow.

Active matrix LCD technology greatly improves upon passive technology in these respects. Basically, each pixel consists of a capacitor and transistor switch. This arrangement gives rise to the more popular term, "Thin-Film Transistor (TFT) Display." To address a particular pixel, its row is enabled, and then a voltage is applied to its column. This has the effect of isolating only the pixel of interest, so others in the vicinity don't turn on. Also, since the current to control a given pixel is reduced, pixels can be switched at a faster rate, which leads to faster refresh rates for TFTs compared to passive displays. What's more, modulating the voltage level applied to the pixel allows many discrete levels of brightness. Today, it is common to have 256 levels, corresponding to 8 bits of intensity.

Connecting to a TFT-LCD panel can be a confusing endeavor due to all of the different components involved. First, there's the panel itself, which houses an array of pixels arranged for strobing by row and column, referenced to the pixel clock frequency.
The backlight is often a CCFL (Cold Cathode Fluorescent Lamp), which excites gas molecules to emit bright light while generating very little heat. Other reasons for the suitability of CCFLs to LCD panel applications are their durability, long life, and straightforward drive requirements. LEDs are also a popular backlight method, mainly for small- to mid-sized panels. They have the advantages of low cost, low operating voltage, long life, and good intensity control. However, for larger panel sizes, LED backlights can draw a lot of power compared to CCFL solutions.

An LCD controller contains most of the circuitry needed to convert an input video signal into the proper format for display on the LCD panel. It usually includes a timing generator that controls the synchronization and pixel clock timing of the individual pixels on the panel. However, in order to meet the LCD panel size and cost requirements, sometimes timing generation circuitry needs to be supplied externally in a "Timing Generator" or "Timing ASIC" chip. In addition to the standard synchronization and data lines, timing signals are needed to drive the individual rows and columns of the LCD panel. Sometimes, spare general-purpose PWM (pulse-width modulation) timers on a media processor can substitute for this separate chip, saving system cost.

Additional features of LCD controller chips are things like on-screen display support, graphics overlay blending, color lookup tables, dithering, and image rotation. The more elaborate chips can be very expensive, often surpassing the cost of the processor to which they're connected.

An LCD driver chip is necessary to generate the proper voltage levels to the LCD panel. It serves as the "translator" between the output of the LCD Controller and the LCD Panel. The rows and columns are usually driven separately, with timing controlled by the timing generator. Liquid crystal must be driven with periodic polarity inversions, because a dc current will stress the crystal structure and ultimately deteriorate it. Therefore, the voltage polarity applied to each pixel varies on either a per-frame, per-line, or per-pixel basis, depending on the implementation.

With the trend toward smaller, cheaper multimedia devices, there has been a push to integrate these various components into the LCD system. Today, there exist integrated TFT-LCD modules that include timing generation and drive circuitry, requiring only a data bus connection, clocking/synchronization lines, and power supplies. The electrical interface on an integrated TFT-LCD display module is straightforward. It typically consists of data lines, synchronization lines, power supply lines, and a clock. Some panels are also available with a composite analog video input, instead of parallel digital inputs.

OLED (Organic Light-Emitting Diode) Displays
The "Organic" in OLED refers to the material that's encased between two electrodes. When charge is applied through this organic substance, it emits light. This display technology is still very new, but it holds promise because it improves upon several deficiencies in LCD displays. For one, it's a self-emissive technology and does not require a backlight. This has huge implications for saving panel power, cost, and weight—an OLED panel can be extremely thin. Additionally, it can support a wider range of colors than comparable LCD panels can, and its display of moving images is also superior to that of LCDs. What's more, it supports a wide viewing angle and provides high contrast. OLEDs have an electrical signaling and data interface similar to that of TFT-LCD panels.

For all its advantages, so far the most restrictive aspect of the OLED display is its limited lifetime. The organic material breaks down after a few thousand hours of use, although this number has now improved in some displays to over 10,000 hours—quite suitable for many portable multimedia applications. It is here that OLEDs have their brightest future—in cellphones, digital cameras, and the like. It is also quite possible that in the near future we'll see televisions or computer monitors based on OLED technology. For the time being, though, as LCD panel technology keeps improving, the OLED mass production timeline keeps getting pushed out incrementally.

Now that we've covered the basics of connecting video streams within a system, it's time to take a look inside the processor, to see how it handles video efficiently. This is the subject of Part 4.

This series is adapted from the book "Embedded Media Processing" (Newnes 2005) by David Katz and Rick Gentile. See the book's web site for more information.

Fundamentals of embedded video, part 2

Color Spaces
There are many different ways of representing color, and each color system is suited for different purposes. The most fundamental representation is RGB color space.

RGB stands for "Red-Green-Blue," and it is a color system commonly employed in camera sensors and computer graphics displays. As the three primary colors that sum to form white light, they can combine in proportion to create most any color in the visible spectrum. RGB is the basis for all other color spaces, and it is the overwhelming choice of color space for computer graphics.

Gamma Correction
"Gamma" is a crucial phenomenon to understand when dealing with color spaces. This term describes the nonlinear nature of luminance perception and display. Note that this is a twofold manifestation: the human eye perceives brightness in a nonlinear manner, and physical output devices (such as CRTs and LCDs) display brightness nonlinearly. It turns out, by way of coincidence, that human perception of luminance sensitivity is almost exactly the inverse of a CRT's output characteristics.

Stated another way, luminance on a display is roughly proportional to the input analog signal voltage raised to the power of gamma. On a CRT or LCD display, this value is ordinarily between 2.2 and 2.5. A camera's precompensation, then, scales the RGB values to the power of (1/gamma).

The upshot of this effect is that video cameras and computer graphics routines, through a process called "gamma correction," prewarp their RGB output stream both to compensate for the target display's nonlinearity and to create a realistic model of how the eye actually views the scene. Figure 1 illustrates this process.

Gamma-corrected RGB coordinates are referred to as R'G'B' space, and the luma value Y' is derived from these coordinates. Strictly speaking, the term "luma" should only refer to this gamma-corrected luminance value, whereas the true "luminance" Y is a color science term formed from a weighted sum of R, G, and B (with no gamma correction applied).

Often when we talk about YCbCr and RGB color spaces in this series, we are referring to gamma-corrected components – in other words, Y'CbCr or R'G'B'. However, because this notation can be distracting and doesn't affect the substance of our discussion, and since it's clear that gamma correction needs to take place at sensor and/or display interfaces to a processor, we will confine ourselves to the YCbCr/RGB nomenclature even in cases where gamma adjustment has been applied. The exception to this convention is when we discuss actual color space conversion equations.

Figure 1: Gamma correction linearizes the intensity produced for a given input amplitude.

While RGB channel format is a natural scheme for representing real-world color, each of the three channels is highly correlated with the other two. You can see this by independently viewing the R, G, and B channels of a given image – you'll be able to perceive the entire image in each channel. Also, RGB is not a preferred choice for image processing because changes to one channel must be performed in the other two channels as well, and each channel has equivalent bandwidth.

To reduce required transmission bandwidths and increase video compression ratios, other color spaces were devised that are highly uncorrelated, thus providing better compression characteristics than RGB does. The most popular ones – YPbPr, YCbCr, and YUV -- all separate a luminance component from two chrominance components. This separation is performed via scaled color difference factors (B'-Y') and (R'-Y'). The Pb/Cb/U term corresponds to the (B'-Y') factor, and the Pr/Cr/V term corresponds to the (R'-Y') parameter. YPbPr is used in component analog video, YUV applies to composite NTSC and PAL systems, and YCbCr relates to component digital video.

Separating luminance and chrominance information saves image processing bandwidth. Also, as we'll see shortly, we can reduce chrominance bandwidth considerably via subsampling, without much loss in visual perception. This is a welcome feature for video-intensive systems.

As an example of how to convert between color spaces, the following equations illustrate translation between 8-bit representations of Y'CbCr and R'G'B' color spaces, where Y', R', G' and B' normally range from 16-235, and Cr and Cb range from 16-240.

Y' = (0.299)R + (0.587)G + (0.114)B

Cb = -(0.168)R - (0.330)G + (0.498)B + 128

Cr = (0.498)R - (0.417)G - (0.081)B + 128

R = Y' + 1.397(Cr - 128)

G = Y' - 0.711(Cr - 128) - 0.343(Cb - 128)

B = Y' + 1.765(Cb - 128)
Chroma subsampling
With many more rods than cones, the human eye is more attuned to brightness and less to color differences. As luck (or really, design) would have it, the YCbCr color system allows us to pay more attention to Y, and less to Cb and Cr. As a result, by subsampling these chroma values, video standards and compression algorithms can achieve large savings in video bandwidth.

Before discussing this further, let's get some nomenclature straight. Before subsampling, let's assume we have a full-bandwidth YCbCr stream. That is, a video source generates a stream of pixel components in the form of Figure 2a. This is called "4:4:4 YCbCr." This notation looks rather odd, but the simple explanation is this: the first number is always '4', corresponding historically to the ratio between the luma sampling frequency and the NTSC color subcarrier frequency. The second number corresponds to the ratio between luma and chroma within a given line (horizontally): if there's no downsampling of chroma with respect to luma, this number is '4.' The third number, if it's the same as the second digit, implies no vertical subsampling of chroma. On the other hand, if it's a 0, there is a 2:1 chroma subsampling between lines. Therefore, 4:4:4 implies that each pixel on every line has its own unique Y, Cr and Cb components.

Now, if we filter a 4:4:4 YCbCr signal by subsampling the chroma by a factor of 2 horizontally, we end up with 4:2:2 YCbCr. '4:2:2' implies that there are 4 luma values for every 2 chroma values on a given video line. Each (Y,Cb) or (Y,Cr) pair represents one pixel value. Another way to say this is that a chroma pair coincides spatially with every other luma value, as shown in Figure 2b. Believe it or not, 4:2:2 YCbCr qualitatively shows little loss in image quality compared with its 4:4:4 YCbCr source, even though it represents a savings of 33% in bandwidth over 4:4:4 YCbCr. As we'll discuss soon, 4:2:2 YCbCr is a foundation for the ITU-R BT.601 video recommendation, and it is the most common format for transferring digital video between subsystem components.

Figure 2: a) 4:4:4 vs. b) 4:2:2 YCbCr pixel sampling.

Note that 4:2:2 is not the only chroma subsampling scheme. Figure 3 shows others in popular use. For instance, we could subsample the chroma of a 4:4:4 YCbCr stream by a factor of 4 horizontally, as shown in Figure 3c, to end up with a 4:1:1 YCbCr stream. Here, the chroma pairs are spatially coincident with every fourth luma value. This chroma filtering scheme results in a 50% bandwidth savings. 4:1:1 YCbCr is a popular format for inputs to video compression algorithms and outputs from video decompression algorithms.

Another format popular in video compression/uncompression is 4:2:0 YCbCr, and it's more complex than the others we've described for a couple of reasons. For one, the Cb and Cr components are each subsampled by 2 horizontally and vertically. This means we have to store multiple video lines in order to generate this subsampled stream. What's more, there are 2 popular formats for 4:2:0 YCbCr. MPEG-2 compression uses a horizontally co-located scheme (Figure 3d, top), whereas MPEG-1 and JPEG algorithms use a form where the chroma are centered between Y samples (Figure 3d, bottom).

Figure 3: (a) YCbCr 4:4:4 stream and its chroma-subsampled derivatives (b) 4:2:2 (c) 4:1:1 (d) 4:2:0.

Digital Video
Before the mid-1990's, nearly all video was in analog form. Only then did forces like the advent of MPEG-2 compression, proliferation of streaming media on the Internet, and the FCC's adoption of a Digital Television (DTV) Standard create a "perfect storm" that brought the benefits of digital representation into the video world. These advantages over analog include better signal-to-noise performance, improved bandwidth utilization (fitting several digital video channels into each existing analog channel), and reduction in storage space through digital compression techniques.

At its root, digitizing video involves both sampling and quantizing the analog video signal. In the 2D context of a video frame, sampling entails dividing the image space, gridlike, into small regions and assigning relative amplitude values based on the intensities of color space components in each region. Note that analog video is already sampled vertically (discrete number of rows) and temporally (discrete number of frames per second).

Quantization is the process that determines these discrete amplitude values assigned during the sampling process. 8-bit video is common in consumer applications, where a value of 0 is darkest (total black) and 255 is brightest (white), for each color channel (R,G,B or YCbCr). However, it should be noted that 10-bit and 12-bit quantization per color channel is rapidly entering mainstream video products, allowing extra precision that can be useful in reducing received image noise by avoiding roundoff error.

The advent of digital video provided an excellent opportunity to standardize, to a large degree, the interfaces to NTSC and PAL systems. When the ITU (International Telecommunication Union) met to define recommendations for digital video standards, it focused on achieving a large degree of commonality between NTSC and PAL formats, such that the two could share the same coding formats.

They defined 2 separate recommendations – ITU-R BT.601 and ITU-R BT.656. Together, these two define a structure that enables different digital video system components to interoperate. Whereas BT.601 defines the parameters for digital video transfer, BT.656 defines the interface itself.

ITU-R BT.601 (formerly CCIR-601)
BT.601 specifies methods for digitally coding video signals, using the YCbCr color space for better use of channel bandwidth. It proposes 4:2:2 YCbCr as a preferred format for broadcast video. Synchronization signals (HSYNC, VSYNC, FIELD) and a clock are also provided to delineate the boundaries of active video regions. Figure 4 shows typical timing relationships between sync signals, clock and data.

(Click to enlarge)
Figure 4: Common Digital Video Format Timing.

Each BT.601 pixel component (Y, Cr, or Cb) is quantized to either 8 or 10 bits, and both NTSC and PAL have 720 pixels of active video per line. However, they differ in their vertical resolution. While 30 frames/sec NTSC has 525 lines (including vertical blanking, or retrace, regions), the 25 frame/sec rate of PAL is accommodated by adding 100 extra lines, or 625 total, to the PAL frame.

BT.601 specifies Y with a nominal range from 16 (total black) to 235 (total white). The color components Cb and Cr span from 16 to 240, but a value of 128 corresponds to no color. Sometimes, due to noise or rounding errors, a value might dip outside the nominal boundaries, but never all the way to 0 or 255.
ITU-R BT.656 (formerly CCIR-656)
Whereas BT.601 outlines how to digitally encode video, BT.656 actually defines the physical interfaces and data streams necessary to implement BT.601. It defines both bit-parallel and bit-serial modes. The bit-parallel mode requires only a 27 MHz clock (for NTSC 30 frames/sec) and 8 or 10 data lines (depending on the pixel resolution). All synchronization signals are embedded in the data stream, so no extra hardware lines are required.

The bit-serial mode requires only a multiplexed 10 bit/pixel serial data stream over a single channel, but it involves complex synchronization, spectral shaping and clock recovery conditioning. Furthermore, the bit clock rate runs close to 300 MHz, so it can be challenging to implement bit-serial BT.656 in many systems. For our purposes, we'll focus our attention on the bit-parallel mode only.

The frame partitioning and data stream characteristics of ITU-R BT.656 are shown in Figures 5 and 6, respectively, for 525/60 (NTSC) and 625/50 (PAL) systems.

Figure 5: ITU-R BT.656 Frame Partitioning.

Figure 6: ITU-R BT.656 Data Stream.

In BT.656, the Horizontal (H), Vertical (V), and Field (F) signals are sent as an embedded part of the video data stream in a series of bytes that form a control word. The Start of Active Video (SAV) and End of Active Video (EAV) signals indicate the beginning and end of data elements to read in on each line. SAV occurs on a 1-to-0 transition of H, and EAV begins on a 0-to-1 transition of H. An entire field of video is comprised of Active Video + Horizontal Blanking (the space between an EAV and SAV code) and Vertical Blanking (the space where V = 1).

A field of video commences on a transition of the F bit. The "odd field" is denoted by a value of F = 0, whereas F = 1 denotes an even field. Progressive video makes no distinction between Field 1 and Field 2, whereas interlaced video requires each field to be handled uniquely, because alternate rows of each field combine to create the actual video image.

The SAV and EAV codes are shown in more detail in Figure 7. Note there is a defined preamble of three bytes (0xFF, 0x00, 0x00 for 8-bit video, or 0x3FF, 0x000, 0x000 for 10-bit video), followed by the XY Status word, which, aside from the F (Field), V (Vertical Blanking) and H (Horizontal Blanking) bits, contains four protection bits for single-bit error detection and correction. Note that F and V are only allowed to change as part of EAV sequences (that is, transitions from H = 0 to H = 1). Also, notice that for 10-bit video, the two additional bits are actually the least-significant bits, not the most-significant bits.

Figure 7: SAV/EAV Preamble codes.

The bit definitions are as follows:

• F = 0 for Field 1

• F = 1 for Field 2

• V = 1 during Vertical Blanking

• V = 0 when not in Vertical Blanking

• H = 0 at SAV

• H = 1 at EAV

• P3 = V XOR H

• P2 = F XOR H

• P1 = F XOR V

• P0 = F XOR V XOR H

The vertical blanking interval (the time during which V=1) can be used to send non-video information, like audio, teletext, closed-captioning, or even data for interactive television applications. BT.656 accommodates this functionality through the use of ancillary data packets. Instead of the "0xFF, 0x00, 0x00" preamble that normally precedes control codes, the ancillary data packets all begin with a "0x00, 0xFF, 0xFF" preamble.

Assuming that ancillary data is not being sent, during horizontal and vertical blanking intervals the (Cb, Y, Cr, Y, Cb, Y, ) stream is (0x80, 0x10, 0x80, 0x10, 0x80, 0x10). Also, note that because the values 0x00 and 0xFF hold special value as control preamble demarcators, they are not allowed as part of the active video stream. In 10-bit systems, the values (0x000 through 0x003) and (0x3FC through 0x3FF) are also reserved, so as not to cause problems in 8-bit implementations.

So that's a wrap on our discussion of digital video concepts. In part 3 we turn our focus to a systems view of video, covering how video streams enter and exit embedded systems.

This series is adapted from the book "Embedded Media Processing" (Newnes 2005) by David Katz and Rick Gentile. See the book's web site for more information.

Fundamentals of embedded video, part 1

Part 1 of this 5-part series explains how video signals are tailored to the human vision system, and reviews the basics of NSTC and PAL video signals.

By David Katz and Rick Gentile, Analog Devices

DSP DesignLine

(09/24/07, 03:00:00 H EDT)

[Part 2 looks at the basics of digital video. For the accompanying intro to audio, see Fundamentals of embedded audio.]

As consumers, we're intimately familiar with video systems in many embodiments. However, from the embedded developer's viewpoint, video represents a tangled web of different resolutions, formats, standards, sources and displays.

In this series, we will strive to untangle some of this intricate web, focusing on the most common circumstances you're likely to face in today's media processing systems. After reviewing the basics of video, we will discuss some common scenarios you may encounter in embedded video design and provide some tips and tricks for dealing with challenging video design issues.

Human Visual Perception
Let's start by discussing a little physiology. As we'll see, understanding how our eyes work has paved an important path in the evolution of video and imaging.

Our eyes contain 2 types of vision cells: rods and cones. Rods are primarily sensitive to light intensity as opposed to color, and they give us night vision capability. Cones, on the other hand, are not tuned to intensity, but instead are sensitive to wavelengths of light between 400nm(violet) and 770nm(red). Thus, the cones provide the foundation for our color perception.

There are 3 types of cones, each with a different pigment that's most sensitive to either red, green or blue energy, although there's a lot of overlap between the three responses. Taken together, the response of our cones peaks in the green region, at around 555 nm. This is why, as we'll see, we can make compromises in LCD displays by assigning the Green channel more bits of resolution than the Red or Blue channels.

The discovery of the Red, Green and Blue cones ties into the development of the trichromatic color theory, which states that almost any color of light can be conveyed by combining proportions of monochromatic Red, Green and Blue wavelengths.

Because our eyes have a lot more rods than cones, they are more sensitive to intensity than color. This allows us to save bandwidth in video and image representations by subsampling the color information.

Our perception of brightness is logarithmic, not linear. In other words, the actual intensity required to produce a 50% gray image (exactly between total black and total white) is only around 18% of the intensity we need to produce total white. This characteristic is extremely important in camera sensor and display technology, as we'll see in our discussion of gamma correction. Also, this effect leads to a reduced sensitivity to quantization distortion at high intensities, a trait that many media encoding algorithms use to their advantage.

Another visual novelty is that our eyes continually adjust to the viewing environment, creating their own reference for white, even in low-lighting or artificial-lighting situations. Because camera sensors don't innately act the same way, this gives rise to a white balance control in which the camera picks its reference point for absolute white.

Perhaps most important for image and video codecs, the eye is less sensitive to high-frequency information than low-frequency information. What's more, although it can detect fine details and color resolution in still images, it cannot do so for rapidly moving images. As a result, transform coding (DCT, FFT, etc.) and low-pass filtering can be used to reduce the total bandwidth needed to represent an image or video sequence.

Our eyes can notice a "flicker" effect at image update rates less than 50-60 times per second, or 50-60 Hz, in bright light. Under dim lighting conditions, this rate drops to about 24 Hz. Additionally, we tend to notice flicker in large uniform regions more so than in localized areas. These traits have important implications for interlaced video, refresh rates and display technologies.

What's a video signal?
At its root, a video signal is basically just a two-dimensional array of intensity and color data that is updated at a regular frame rate, conveying the perception of motion. On conventional cathode-ray tube (CRT) TVs and monitors, an electron beam modulated by the analog video signal shown in Figure 1 illuminates phosphors on the screen in a top-bottom, left-right fashion. Synchronization signals embedded in the analog signal define when the beam is actively "painting" phosphors and when it is inactive, so that the electron beam can retrace from right to left to start on the next row, or from bottom to top to begin the next video field or frame. These synchronization signals are represented in Figure 2.

Figure 1. Composition of Luma signal.

HSYNC is the horizontal synchronization signal. It demarcates the start of active video on each row (left to right) of a video frame. Horizontal Blanking is the interval in which the electron gun retraces from the right side of the screen back over to the next row on the left side.

VSYNC is the vertical synchronization signal. It defines the start (top to bottom) of a new video image. Vertical Blanking is the interval in which the electron gun retraces from the bottom right corner of the screen image back up to the top left corner.

FIELD distinguishes, for interlaced video, which field is currently being displayed. This signal is not applicable for progressive-scan video systems.

Figure 2: Typical timing relationships between HSYNC, VSYNC, FIELD.

The transmission of video information originated as a display of relative luminance from black to white – thus was born the black-and-white television system. The voltage level at a given point in space correlates to the brightness level of the image at that point.

When color TV became available, it had to be backward-compatible with B/W systems, so the color burst information was added on top of the existing luminance signal, as shown in Figure 3. Color information is also called chrominance. We'll talk more about it in our discussion on color spaces (in part 2 of this series).

Figure 3. Analog video signal with color burst.

Broadcast TV – NTSC and PAL
Analog video standards differ in the ways they encode brightness and color information. Two standards dominate the broadcast television realm – NTSC and PAL. NTSC, devised by the National Television System Committee, is prevalent in Asia and North America, whereas PAL ("Phase Alternation Line") dominates Europe and South America. PAL developed as an offshoot of NTSC, improving on its color distortion performance. A third standard, SECAM, is popular in France and parts of eastern Europe, but many of these areas use PAL as well. Our discussions will center on NTSC systems, but the results relate also to PAL-based systems.

Video Resolution
Horizontal resolution indicates the number of pixels on each line of the image, and vertical resolution designates how many horizontal lines are displayed on the screen to create the entire frame. Standard definition (SD) NTSC systems are interlaced-scan, with 480 lines of active pixels, each with 720 active pixels per line (i.e., 720x480 pixels). Frames refresh at a rate of roughly 30 frames/second (actually 29.97 fps), with interlaced fields updating at a rate of 60 fields/second (actually 59.94 fields/sec).

High definition systems (HD) often employ progressive scanning and can have much higher horizontal and vertical resolutions than SD systems. We will focus on SD systems rather than HD systems, but most of our discussion also generalizes to the higher frame and pixel rates of the high-definition systems.

When discussing video, there are two main branches along which resolutions and frame rates have evolved. These are computer graphics formats and broadcast video formats. Table 1 shows some common screen resolutions and frame rates belonging to each category. Even though these two branches emerged from separate domains with different requirements (for instance, computer graphics uses RGB progressive-scan schemes, while broadcast video uses YCbCr interlaced schemes), today they are used almost interchangeably in the embedded world. That is, VGA compares closely with the NTSC "D-1" broadcast format, and QVGA parallels CIF. It should be noted that although D-1 is 720 pixels x 486 rows, it's commonly referred to as being 720x480 pixels (which is really the arrangement of the NTSC "DV" format used for DVDs and other digital video).

Table 1. Graphics vs Broadcast standards.

nterlaced vs. Progressive Scanning
Interlaced scanning originates from early analog television broadcast, where the image needed to be updated rapidly in order to minimize visual flicker, but the technology available did not allow for refreshing the entire screen this quickly. Therefore, each frame was "interlaced," or split into two fields, one consisting of odd-numbered scan lines, and the other composed of even-numbered scan lines, as depicted in Figure 4. The frame refresh rate for NTSC/(PAL) was set at approximately 30/(25) frames/sec. Thus, large areas flicker at 60 (50) Hz, while localized regions flicker at 30 (25) Hz. This was a compromise to conserve bandwidth while accounting for the eye's greater sensitivity to flicker in large uniform regions.

Not only does some flickering persist, but interlacing also causes other artifacts. For one, the scan lines themselves are often visible. Because each NTSC field is a snapshot of activity occurring at 1/60 second intervals, a video frame consists of two temporally different fields. This isn't a problem when you're watching the display, because it presents the video in a temporally appropriate manner. However, converting interlaced fields into progressive frames (a process known as "deinterlacing"), can cause jagged edges when there's motion in an image. Deinterlacing is important because it's often more efficient to process video frames as a series of adjacent lines.

With the advent of digital television, progressive (that is, non-interlaced) scan has become a very popular input and output video format for improved image quality. Here, the entire image updates sequentially from top to bottom, at twice the scan rate of a comparable interlaced system. This eliminates many of the artifacts associated with interlaced scanning. In progressive scanning, the notion of two fields composing a video frame does not apply.

Figure 4. Interlaced Scan vs Progressive Scan illustration.

Now that we've briefly discussed the basis for video signals and some common terminology, we're almost ready to move to the really interesting stuff—digital video. We'll get to that in part 2.

This series is adapted from the book "Embedded Media Processing" (Newnes 2005) by David Katz and Rick Gentile. See the book's web site for more information.

尋求差異化發展 擴展FPGA的汽車應用領域

為了能與市場上的大型競爭對手相抗衡，Actel公司正針對FPGA在汽車領域的各種可能應用開闢新市場，包括可進軍油電混合車(hybrid)、電動汽車或燃料電池供電汽車的控制系統設計。

根據Gartner Dataquest預測，2007年全球汽車半導體市場規模將達到201億美元，而這個數字到2010年更將成長到259億美元。其中，FPGA目前約佔9,300萬美元，而且在今後的三年內還可望成長到3.12億美元。

目前，車用FPGA主要集中在資訊娛樂/售後市場，以及無線通訊/車身電子等應用。賽靈思(Xilinx)和Altera等較大型的 FPGA供應商們已經將其策略重點放在這些較‘值得關注的’應用領域中。為了能與這些大型廠商競爭，Actel等小型FPGA供應商就必須另謀出路。

其中一條出路就是動力傳動系統(powertrain) ──這一設計理念幾十年來變化不大，而今馬上就要面臨轉變。隨著油電混合汽車、電動汽車、燃料電池汽車、E85(編註：指用85%的乙醇潔凈燃料與15% 的汽油混合後的燃料)燃料汽車，以及改良柴油引擎的推出，汽車製造商們急於開發能夠勝任新型動力傳動控制任務的電子系統。

同時，汽車安全應用的市場也敞開了大門，例如防撞系統、盲點檢測和警示系統，以及倒車攝影機等。除了快速的上市時間和滿足後期設計變化所需的靈活性，動力傳動和安全系統的設計師們還要求電子元件具有高可靠性、小功耗，並能適應嚴苛的汽車環境。

大多數傳統的電子零件並不適合這些應用。ASIC的NRE和認證成本太高、設計週期過長、欠缺彈性且風險較大。微控制器雖然靈活，但功耗卻 相對較大，無法滿足具有極端反應時間需求的應用，如高階柴油引擎中的控制單元等設備。(這些閉環控制單元必須能夠在逐週期的基礎上即時進行監控和調整燃料 噴射。

CPLD曾在這些市場中找到部份應用，但其只能提供中等的邏輯密度，通常製造製程比較陳舊，而且性能和I/O功能也相對有限。反之，基於 SRAM的FPGA雖然具有很高的密度和豐富的性能，但卻容易產生韌體錯誤。再者，基於SRAM的FPGA功耗也相對較大，且會導致自發熱，限制了更大規 模的元件在最高溫度達100℃環境中的運作。

上述事實可用來解釋為何Actel公司要擴展其ProASIC3系列產品，並使其發展成為據稱是業界首個通過AEC-Q100一級認證的FPGA。AEC-Q100是汽車電子協會(AEC)針對車用IC所發佈的應力測試認證標準。

這些低功率、單晶片、基於快閃記憶體的元件能上電即行(LAPU)，而且不會產生韌體錯誤。它們功能豐富且具有彈性，能在汽車極端應用要求的高溫環境下運作(125℃的環境溫度和135℃的結溫)。

這些元件採用了英飛凌CMOS+Flash的130nm汽車元件製程，氧化層比較厚，因此能支援較高工作溫度，同時也減少漏電流。相對於 SRAM或快閃記憶體/SRAM混合配置，純快閃記憶體的配置不會產生韌體錯誤，而且功耗特別低、產生的熱量非常少。這些特性也使得該元件能夠運作在極端 溫度環境下。

除了AEC標準外，這些元件還支援一些汽車製造商所要求的‘生產元件批准程式’(PPAP)規定和TS16949品質管理規範等產業標 準。據Actel透露，這些元件已被用於多種設備，範圍從倒車視覺系統(在該系統中，攝影機、FPGA和其它設備必須裝配進一個1英吋的管子中)，到用於 拖曳機和採礦設備的重型柴油引擎等下一代控制器。

ProASIC3的規模在6萬閘到100萬閘之間。100萬閘的A3P1000元件已通過AEC-Q100認證，目前已能供貨，該系列中的其它產品將在今年底推出。A3P060、A3P125和A3P250的工程樣品現已出貨。

“作為一家小型的FPGA供應商，Actel必須專注於自身的差異化發展。”Gartner Dataquest公司分析師Bryan Lewis表示，“對他們來說，這是一個能實現穩定發展的利基市場。”

Gartner Dataquest公司目前只追蹤了FPGA在汽車資訊娛樂以及無線通訊領域中的使用情況，還沒有把FPGA在動力傳動和安全系統中的應用一併考慮在內。但是Lewis指出：“根據Actel發佈的資訊來看，這個局面馬上就會產生改變。”

各種汽車應用領域正為基於FPGA的方案打開大門。

'Personal Robot' wins iRobot's challenge

A personal robot that can water plants, remind owners to take their medication, turn lights on and off, and control appliances has won a contest sponsored by iRobot.

Danh Trinh, 35, of Towson, Md., won iRobot's Create Challenge contest and its $5,000 prize, with his Personal Home Robot, the company announced Tuesday.

iRobot Create is a preassembled programmable robot designed so developers can create new robots without having to build everything from scratch.

It features standard connections for electronics and threaded mounting holes so users can attach their own inventions to the robots and integrate third-party electronics.

Trinh attached sensors and video cameras that enable the robot to move around and perform household tasks. He also designed the robot so it can play music.

"This is an excellent example of the enthusiasm people bring to the idea of building robots," Helen Greiner, co-founder and chairman of iRobot, said in a prepared statement. "Contestants put in amazing efforts creating exciting and imaginative robots for the challenge. We saw everything from robots that serve food and drinks to robots that paint pictures and can be remotely controlled from distant locations."

The Create platform provides access to robot sensors and actuators via an open interface.

Electronic enthusiast Web site Tom's Hardware Guide sponsored the contest and hobbyist Web site Instructables.com hosted it. Judges chose the winning entry based on aesthetics, intelligence, utility, entertainment value, completeness, and originality.

IRobot sells bots that perform dull, dirty, or dangerous tasks for consumer and military use. The company's proprietary technology, iRobot AWARE Robot Intelligence Systems, includes technologies for navigation, mobility, manipulation, and artificial intelligence.

NTUST Goldfish-Catching System

NTUST mobile robot project

導覽機器人

日本Toyota汽車發展出導覽機器人「Robina」，透過輪子可以自由移動。「Robina」有一點二公尺高，圖為二十七日機器人在Toyota的展覽館執勤。（法新社）

步行機器人揭開人類邁步前行之謎

來自德國的研究人員日前表示，一種能適應不同地形的步行機器人(walking robot)將可協助科學家理解人類如何行走的奧秘，甚至在未來改善針對脊髓神經(spinal cord)和其它部位損傷的治療方法。

發明機器人的研究人員表示，過去這款名為RunBot的30公分高機器人只能在平地上行走，遇到斜坡就會跌倒；但自從採用了紅外線眼(infrared eye)之後，RunBot現在可以探測行進路線上的斜坡，並在4~5次嘗試之後調整自己的步伐克服坡度。

在學會爬坡之前，這個機器人總是不斷跌倒，但它每秒可邁出3~4步長，比普通人類每秒1.5~2.5的步長要快；參與RunBot設計的德 國Goettingen大學研究人員Florentin Woergoetter表示：「它會不斷反覆摸索學習，需要經過約4~5次的跌倒才能學會。」

Woergoetter在《Computational Biology》期刊上發表了自己的研究成果，並把RunBot的學習過程與學走路的幼兒進行比較。和人類一樣，RunBot在直立行走時身體會稍稍前傾，而爬坡時步伐會更短一些。

RunBot能走路的關鍵之一在其“大腦”，它的紅外線眼和控制電路相連，引導它需要的時候改變步伐。之前的研究顯示，人體內的動力控制系統是由肌肉與脊髓神經之間交互作用的多個層級所組成，這個部份大多數是自主運作，但某些運動則需要更高層級的控制──即大腦。

Woergoetter表示，上述關係解釋了為何某些下半身癱瘓的病人使用輔具之後就可在跑步機上使用雙腳，也是RunBot研究的核心。他並指出，透過機器人研究進一步了解人體各個不同部份如何在行走時互相合作，對於改善醫療保健有著實質性的作用。

這類機器人研究不僅有助於為殘障者設計出更好的義肢，也能協助臨床治療師與病患一起對抗脊髓損傷等症狀，重新恢復運動能力。Woergoetter表示：「RunBot實際上就是人類直立行走的一個模型，將幫助我們進一步了解箇中奧祕並帶來更好的治療方法。」

Word 內嵌 Visio 檔案，轉換成 PDF 檔案圖形亂掉的解決方案

當您轉換 Word 2002 文件， 包含 Visio 2002 或 Visio 2003 圖形轉換成 PDF 格式是在圖形文字會顯示不正確

解決方案

如果要解決這個問題， 將 PDF [ 列印品質 ] 選項設定成 600 dpi 設定值。 如果要執行這項操作，請依照下列步驟執行。:

1.	如果程式正在執行結束 Visio 2003 或 Visio 2002、 Word 2002 及 Adobe Acrobat。
2.	按一下 [ 開始 ] 按一下 [ 執行 ] 在 [ 開啟 ] 方塊， 鍵入 印表機控制項 ， 然後按一下 [ 確定 ] 。
3.	以滑鼠右鍵按一下 Adobe PDF ， 並按一下 [ 列印喜好設定 ] 。
4.	按一下 [ 版面配置 ] 索引標籤， 及 [ 進階 ] 。
5.	在 [ 進階 PDF 轉換程式進階選項 ] 對話方塊， 再展開 圖形 ， 及 [ 列印品質 ] 。
6.	600dpi ， 請按一下及兩次 [ 確定 ] 。
7.	啟動 Word 2002、 開啟該文件， 及再列印文件， 以 Adobe Acrobat。 若要列印繪圖以 Adobe Acrobat： a. 在 Word 2002， 按一下 [ 檔案 ] 功能表上 [ 列印 ] 。 b. 按一下 [ 在 [ 名稱 ] 方塊， Adobe PDF 然後再按一下 [ 確定 ] c. 在 另存新 PDF 檔 ] 對話方塊， 指定檔案名稱及您要儲存 PDF 檔案， 位置及 [ 儲存 ]。

沒想到問題是在 Visio 與 PDF 無法在列印品質上取得共識啊~
http://support.microsoft.com/kb/892955/zh-tw?spid=2963&sid=480

Using Embedded Linux in a reconfigurable high-res network camera

Using Embedded Linux in a reconfigurable high-res network camera by Andrey Filippov (Dec. 3, 2002)

Background

About a year ago I wrote an article which was published by LinuxDevices.com, and after it was mentioned on Slashdot my company (Elphel Inc.) was flooded with inquiries regarding general purpose network cameras, rather than the "high speed gated intensified" ones I wrote about. Also, the Model 303 network camera I wrote about, being high resolution, was rather slow -- the ETRAX100LX requires nearly 5 seconds for JPEG compression of a 1280x1024 color frame.



The Model 303 High Speed Gated Intensified Camera

My intention to increase camera frame rate was mentioned in the "TODO" section of the previous article, but the way to actually do that turned out to be very different from what I had anticipated. I decided not to use the JPEG-2000 compressor chip from Analog Devices. Nor did I make use of the new ETRAX multi-chip module from Axis Communications, as I wanted more memory (both SDRAM and Flash) than Axis put into the MCM version of its ETRAX controller. Also, in the new camera there was no place for a Quicklogic FPGA that I had intended to use for fixed-pattern noise elimination; this function needs 10 times less resources than image compression, and definitely fits in the same FPGA.

Instead, what I began to contemplate was . . .

An Open Source reconfigurable camera

I first investigated the possibility of using a large enough reprogrammable FPGA to be able to handle basic image acquisition tasks, fixed-pattern noise elimination, and image compression (i.e. baseline JPEG), without slowing down a sensor (~20 MHz pixel rate). An additional goal was to be able to use free FPGA development software, so it would make sense for me to post Verilog sources so that users would be able not only to build the camera software from sources but to do the same with the hardware (FPGA) part.

Incidentally, I'm not sure if it still makes sense to call it "hardware", as you do not even need to open the case to modify it. But there are at least two arguments that it still is hardware: (1) it's easy to fry the thing, by installing the wrong code in the FPGA (I had to hold my finger on the chip while first debugging the download process); and (2) the speed -- namely, a nearly 100x increase in compression performance and the fact that my Athlon-700 based PC is about 2.5 times slower in decoding than the camera FPGA in encoding (both require approximately the same amount of calculations) and the FPGA does not have any heat sink and is just slightly warmer than the environment.

Picking an FPGA

It was not difficult to find a good FPGA candidate. The latest member of Xilinx's low-cost Spartan IIe FPGA family -- a 300K gates XC2S300E chip (see note below, for an update). Plus, they have free ISE Webpack development software available for download that worked fine for the design and was able to make use of 98% of the chip's resources.

Unfortunately, the free version of the 3rd party simulator Xilinx included with their free development software package proved useless for my purposes, as its 500 line limit is not serious for simulating such a design. I do not think this is a real problem for the Linux community, since some of the Open Source simulators can probably be combined with the Xilinx ISE for Linux.

Before starting an actual design, I had to evaluate whether the JPEG compressor and other required circuitry would fit into the selected FPGA (I did not have prior experience with Xilinx devices). So I looked for commercial IPs and found that they really do exist (although they're rather expensive), and thereby determined that the chip should handle the job.

I also located an XAPP610 application note which includes source code for an 8x8 DCT core that is fast enough and uses less than 30% of the chip (I later found out that I had to modify it).

Architecture of the Model 313

I didn't get around to really starting the new design until August, at which time I downloaded the Xilinx development software and designed the schematic and PCB layout for the Model 313 camera, making it exactly the same physical dimensions as the old one.


Block diagram: Model 313 Reconfigurable Network Camera
(click to enlarge)

Together with the new FPGA came some other components . . .

• 16MB SDRAM memory, connected directly to the FPGA so image processing does not reduce CPU bus bandwidth.
  
  
• multi-channel programmable clock: its 20 MHz crystal oscillator output and one of the three PLLs (25MHz) are used to drive ETRAX100LX and Ethernet transceiver respectively, leaving the two other PLLs for FPGA flexible clocking. This Cypress CY22393FC part combines EEPROM memory (so the right frequencies will be applied to the CPU and network transceiver upon power up) and the I²C-compatible interface making it possible to provide an extra degree of flexibility to a reconfigurable FPGA.

The SRAM-based FPGA is configured using the bit-stream file that is generated by the Xilinx development software and stored in the camera flash memory. It is transferred to the chip using just 4 pins of the ETRAX general purpose interface port which is connected to the dedicated JTAG pins of the XC2S300E. It takes just a single line in one of the init scripts ("cat /etc/x313.bit > /dev/fpgaconfjtag") and a fraction of a second to bring it to life.

The FPGA code is designed around a four channel SDRAM controller. It uses internal "Block RAM" embedded memory (there are 16 of 4096 bit blocks in the XC2S300E chip) for ping-pong buffering of each channel. The controller provides interleaved access to the following channels . . .

• Channel 0 -- raw or processed, 8 or 16 bits per pixel data from the sensor to the memory. Data is arranged in horizontal 256 pixel lines (128 for 16-bit data). It is also possible to write partial blocks (last in a scan line).
  
  
• Channel 1 -- used to read from the memory per-pixel calibration data prepared by software in advance. For each pixel, there is an 8-bit value to subtract from the 10-bit sensor data. This data may be prescaled by 1, 2, or 4. The other byte contains sensitivity calibration, so depending on a global prescaling factor each pixel value may be individually adjusted in the +/- 12.5%, +/-25% or +/- 50% range.
  
  
• Channel 2 -- provides data for the JPEG encoder. For the 4:2:0 encoding where two color components (Cb and Cr) have half of brightness resolution in both directions (that matches the Bayer color filters of the sensor) the minimal coding unit (MCU) is a square of 16x16 pixels that are later encoded as 4 8x8 blocks for the intensity (Y) component, and one 8x8 for each of Cb and Cr color ones (total 6 per MCU). If the data is encoded "live", the SDRAM controller provides a "ready" signal for this channel whenever there are at least 16 lines written by the sensor (channel 0).
  
  
• Channel 3 -- provides CPU access to the SDRAM. Normally it is used to read out raw sensor data and write calibration data for the FPN elimination (that is calculated by the CPU from the raw pixel data).

The SDRAM controller runs at 75MHz (16-bit wide data), which is enough for a pixel rate of up to 25MHz and quasi-simultaneous channel operation.

The synchronization module provides the camera with the capability of registering short asynchronous events. The camera is designed to work with both Zoran (2/3, 1/2, and 1/3 in.) and Kodak (1/2 in.) imagers which can work only in continuous "rolling shutter" mode. In that case, an asynchronous event (i.e. a laser pulse) will likely be registered in two consecutive frames (part in the first, and the balance in the second), but since the camera is continuously writing data into a circular buffer it is possible to reconstruct the complete image. The synchronization module can work in 2 ways: using an external electrical signal, or just comparing average the pixel value in each scan line with some predefined threshold. This makes it possible to register short light pulses without any additional electrical connections to the camera.

The JPEG compression itself is performed in a chain of stages, some of them using embedded Block RAMs as buffers and/or data tables (quantization and Huffman). This function uses approximately two-thirds of the resources of the FPGA . . .

• First stage -- the Bayer-to-YCbCr converter receives 16x16 pixel MCU tiles and writes simultaneously into two buffers: one, 16x16 for Y data; and the other, 2x8x8 for Cb and Cr data. In parallel, it calculates average pixel value (DC component) for each of them and subtracts it on the output to bypass the DCT conversion. On the output, data goes out from the buffers in 64-sample (9 bits signed) blocks, 4 for Y component followed by 1 Cb an 1 Cr. The next 3 stages (DCT, Quantizator/Zigzag reorderer, and RLL encoder) are designed to process data in blocks of 64 consecutive samples with arbitrary (>=0) gaps between them.
  
  
• Second stage -- the 8x8 DCT converter is based on a Xilinx reference design described here (PDF download). I had to modify it to make it work in asynchronous mode, so each 64-sample block can start with arbitrary delay (0 or more cycles) after the previous one, and to increase the dynamic range (the test fixture in the reference design had just 6-bit -- not 8-bit -- input data). This stage uses a 2x64x10-bit ping-pong memory buffer between horizontal and vertical stages of the 2-d DCT. Output data comes in the down first, then right order for each 64-sample block.
  
  
• Third stage -- the Quantizator/Zigzag reorderer receives 8-bit signed average block value (directly from Bayer-to-YCbCr converter stage) and combines it with 12 bit signed data output from the DCT. It uses Two Block RAMs - one to store 2 alternative 2-table sets of 64x12-bit quantization data, the other - to reorder the output data in zigzag order (starting from the lowest frequencies and going to the highest) as required by the JPEG standard. This reordering increases the probability of long sequences of zeroes that are encoded on the next RLL stage. Quantizator uses uses multiplication by 12 bits (together with >>12) instead of division by 8 bits. The software that generates the tables makes corrections to the divisor table (written in the JPEG file header) so that for high divisor values they match the FPGA multiplicands). Quantization tables are written by the CPU prior to compression.
  
  
• Fourth stage -- the RLL encoder is the first to break uniform 64-cycle long data packets. It combines the data output from the quantizator with the number of preceding zero-value data samples. This stage also maintains the previous DC value for each component (Y, Cb and Cr) and sends out the difference from the previous instead of the value itself for DC components.
  
  
• Fifth stage -- the Huffman encoder uses 256x16bit FIFO for the input data it receives from RLL stage. Three more Block RAM modules (2x256x20) are required to store 2 sets of Huffman tables (one for Y, and the other - for Cb and Cr components). In each output 20-bit word 4 MSBs specify the number of bits to send, and the 16 LSBs - the data bits to send. The DC Huffman tables are rather short so they are stored in unused parts of the AC tables.
  
  
• Sixth stage -- the bit stuffer receives number of bits to send and the data to send, combines them into continuous bit stream and formats into 16-bit output words. It also inserts 0x00 bytes after each 0xff, as the 0xff is a prefix to the marker in the JFIF data stream.

The output encoded data goes to a 256x16 FIFO and then is transferred to the system memory using CPU DMA channel as 32-bit long words.

Results and plans

The code compiles into 98% of the FPGA's resources. It takes about twenty minutes to compile on my 700 MHz Athlon PC. And it works -- and works really fast! The compressor works at the full sensor rate (15 fps @1280x1024), and I can get 12 fps (some frames are still skipped) of the Quicktime format clips saved on the PC. There are a couple things that need to be cleaned up to fix that frame skipping, and then the camera will provide 15fps at 1280x1024 pixels, 60 fps at 640x480 pixels, and 240 fps at 320x240 pixels over the LAN connection.



Model 313 Reconfigurable Network Camera

There is no video streaming server software in the camera yet. It can only provide Quicktime clips of some predefined length (although it is possible to make that length really big). To view the clips live (before they are completely transferred) all the index information is provided before the actual video data, so each JPEG frame is padded to make them all the same size. To make the size of the padding smaller (and make most of the frames fit) the JPEG compression quality is adjusted after each frame.

Incidentally, on November 18, 2002, Xilinx announced availability of two new members of Spartan IIe series, with 600K and 400K gates. The 600K uses a bigger package, whereas the 400K gates version has the option of matching the pinout of the XC2S300E currently used in the model 313 camera, so it can be used in the camera without any schematic of PCB changes. Using this device, I believe it will be possible to implement the full frame MPEG encoder.

Here is a product description of the resulting camera . . .

About the Elphel Model 313 Reconfigurable Network Camera

There are many network cameras (cameras that can serve images/video without computer) on the market today. Some can provide high frame rate video, but limited to 705x480 pixels or less. There are even some high-resolution (megapixel) network cameras, but they usually need one second or longer to compress a full size image.

The Model 313 can do both. It is a 1.3 megapixel network camera and it can serve full size images really fast -- at 15 frames per second. High resolution may be very useful for security applications: for example, a single camera with a wide angle lens placed in the corner can see the whole room with the same quality as a narrow angle NTSC camera placed on a pan/tilt platform; and it can see it all at the same time, without any need of scanning.

Full resolution high frame rate even makes it possible not to use "digital pan-and-tilt" (sending out just a subwindow of the whole frame), the usual way to overcome the slow operation of high resolution network cameras.

The Model 313 camera is powered by 48VDC through the LAN cable, compliant to the IEEE 802.3af standard. This voltage makes possible to use four times longer cables to the camera than when using 24VDC power, and 16 times longer than 12VDC. Such lower voltages (not IEEE 802.3af compliant) are still used in some powered-over-LAN cameras.

All of the camera's embedded software and FPGA bitstream are stored in the camera flash memory, which can be upgraded through the Internet. Unlike the very dangerous procedure of rewriting flash memory with BIOS in a PC (if it was a wrong file or the power went off during flashing, the motherboard will likely be wasted), the Model 313 camera uses an important feature of the Axis ETRAX100LX 32-bit CPU which has an internal bootloader from the LAN that does not depend on the current flash memory data, so it is always possible to start over again with camera software installation.

Another important feature for developers is that both the embedded software and FPGA hardware algorithms are open source. Four levels of customization of the camera are thereby possible . . .

1. Modification of the user interface using web design tools -- The camera has three file systems that makes it easy and safe to modify preinstalled web pages and be able to restore everything back if something went wrong.
   
   
2. Applications written in C -- It is possible to compile C code on a computer running Linux after installing software from the downloads page (and links from there). The executable file may be transferred to the camera using ftp to RAM disk or a flash memory file system (jffs). That user application may have CGI interface, and can respond to http requests from the web browser.
   
   
3. Adding (or modifying) drivers to the camera operating system -- This will require building the new OS kernel and there are two ways to try it on the camera: boot the camera from the LAN with the new kernel (it will not change anything in the camera flash memory, so just turning it off and back on will restore initial software); or flashing it instead of original one (in that case, after power cycling camera will always boot with the new system).
   
   
4. FPGA modification that gives full control over the power of the reconfigurable computing in the camera -- This level requires different tools: FPGA development software from Xilinx (free for download available), and the camera sources posted on Elphel's website.

---

About the author: Andrey N. Filippov has a passion for applying modern technologies to embedded devices, especially in advanced imaging applications. He has over twenty years of experience in R&D, including high-speed high-resolution, mixed signal design, PDDs and FPGAs, and microprocessor-based embedded system hardware and software design, with a special focus on image acquisition methods for Laser Physics studies and computer automation of scientific experiments. Andrey holds a PhD in Physics from the Moscow Institute for Physics and Technology. This photo of the author was made using a Model 303 High Speed Gated Intensified Camera.

---

NOTE: A version of this article translated into Russian is available here.

---

Related stories:

• Using Embedded Linux in a High Speed Gated Intensified Camera
  
• ETRAX 100LX high-integration RISC system-on-chip
  
• The StarDot NetCam
  
• Embedded Linux powers intelligent online video camera

實現全功能、低成本家庭安全系統設計

今天，家庭用戶對一些易於使用、具有多媒體彩色介面、功能豐富與高性能的電子裝置已習以為常，這些設備的擁有成本正不斷下降，並具備與PC、筆記型電腦、手機、PDA及可攜式遊戲機等裝置互動的能力。使用者對這些產品的體驗，包括對這些家電產品的舒適度及滿意度，使他們對這類家電系統具有更高的期待。然而，相較於用戶擁有的行動設備，目前許多已安裝的住宅安全系統仍無法滿足這些不斷升高的期待。

使用者對個人安全的渴求，加上目前安全系統可被察覺的弱點持續增加，正推動業界廠商們研發用戶可負擔得起的全功能型多媒體家庭安全、監控與控制系統。

當然，家庭用戶希望安全系統能檢測出入侵，並在系統感測器?動時發出聲音或警報。然而，許多人也許更喜歡在開門前看見來訪者，或當身處別的房間、做家事時看見是誰在按門鈴。目前開發的新系統均具備能支援這種可存取、控制及監控家庭系統的能力。

目前系統設計師面臨的挑戰在於必須在低成本、全功能設備中，以能滿足各種潛在用戶需求和預算的價格提供彈性化功能。為使產品更具競爭力，設計團隊還面臨必須以盡可能低的價格，在盡最短時間內開發、測試並使產品上市的挑戰。

對價格敏感產品的架構通常採用低成本、高整合度的零組件來實現。不過，隨著網路通訊、複雜控制和視訊等新功能的增加，這些系統也必須具備高性能。

圖1所示為一款全功能、低成本的安全系統，內含QVGA LCD、乙太網路埠、本地感測器和控制功能介面線、通道門(access door )視訊輸入源、本地及遠端麥克風和揚聲器。該系統還包含一個記憶體擴展槽，可支援用戶定製檔案及系統軟體升級的載入，同時能發送視訊。

該方案基於ADI公司具備嵌入式乙太網路MAC模組的ADSP-536 Blackfin 處理器。這種系統架構具備可擴展與連網特性。Blackfin 處理器系列的程式碼均相容，部份元件為接腳相容，能讓製造商以一個公共模組系統架構開發出具有不同功能、性能和價格的產品。

利用這套系統，用戶可在遠端監控室內狀況，並能播放記錄下的視訊。還可開發出能透過開放標準和專供家用連網的協議，對家庭空調系統監視和控制，以及控制房間照明和家電的產品。

許多家庭用戶一直在等待可負擔起的新一代網路居家安全和監控系統，這些系統所帶來的功能、性能及便利性，與其對個人通訊及娛樂等系統的期待水準相當。隨著新系統的推出，這些用戶的願望將獲得實現。

">
圖：基於ADSP-536 Blackfin處理器的全功能、低成本安全系統。

作者：RC Cofer

John Schippanoski

現場應用工程師

安富利公司