In digital signal processing, anti-aliasing is a key approach to reducing distorting artifacts, which are known by the term ‘aliasing’ when dealing with high resolution signals received by a lower resolution imager (sensor).

In digital photography, aliasing artefacts appear in several forms. Chiefly they are most visible as wavy noise or moiré patterns, strobing or as sparkle-type spots. 

Anti-aliasing therefore refers to the removal of higher frequencies that the sensor cannot properly resolve during live capture. In other words when the sampling is performed, undesirable artifacts can occur if they are not removed beforehand, causing visible noise in the image. The below illustration is a receding checkerboard pattern showing what happens when no anti-aliasing filter is in place.

It clearly shows the visible distortion occurring when anti-aliasing is not deployed. As the checkerboard pattern converges into the vanishing point, the image becomes too difficult to resolve for the sensor.

In contrast, Figure 2, which is anti-aliased, fares considerably better where the same area now blends into greyness. This is what the anti-aliasing filter does when the resolution is incapable of displaying details clearly. Furthermore, towards to the bottom of the image, the edges of the lines are smoother.

Digital cameras usually resolve this problem with the use of an analog anti-aliasing filter (aka Low Pass Filter, LPF, AA or blur filter) in order to remove the out-of-band component of the input image signal prior to sampling with an A/D converter. These filters are optical in nature and are integral in the sensor to block invisible light. They are often made from the use of two layers of birefringent materials such as lithium niobate (LiNbO3), which spreads each optical point into a cluster of four points.  

(Note: ‘Birefringent’ means ‘double refraction.’ Therefore a birefringent material is able to decompose light into two separate rays – one ordinary and the other extraordinary. Cellophane is a cheap example.)

Figure 3 shows a portion of the original image with no filtering applied. When resized or photographed with a digital camera, these converging lines will alias, causing artifacts.

Figure 4 is of the same section of the image with an anti-aliasing filter applied to the image.

Because of this spread, the shortcoming of an anti-aliasing filter is that it curtails part of the image sharpness as well as fill factor and that is part and parcel of its effort in reducing the resolution to a level that the sensor can accept. This loss of image sharpness is able to be somewhat corrected with a sharpening filter to counteract the loss of apparent resolution.

A central part of understanding anti-aliasing is that in signal processing, downsampling (aka sub-sampling) is an integral part of the effort to reduce the sampling rate to make it more acceptable for the imaging sensor to cope. As this is normally done, the data rate or the size of the image data is affected and because of this, it is important to ensure that the criterion proposed by the Shannon-Nyquist Sampling Theorem is maintained. Otherwise the resulting digital signal will be aliased.