Introduction

With the advent of CHDK (Canon Hackers' Development Kit), many inexpensive point&shoot Canon cameras can be turned into versatile photographic instruments, with capabilities that far exceed other cameras in their price range, e.g bracketing, taking raw pictures, shooting indefinite time lapses.

When saving images in the DNG raw format (digital negative), CHDK also offers the possibility to remove bad pixels. However, these bad pixels seem to be particularly volatile. I have documented wide variations of number and position of bad pixels in a variety of situations. Although there is as yet no explanation for why bad pixels should vary in such a striking manner, determining how they vary might provide strategies for improving picture quality by removing them.

Definitions

For clarity the terms, "bad", "dead" and "hot" have to be determined for pixel behaviour:

• Bad pixels have a constant R, G or B value over a range of exposures where they would be expected to show response to changes in light intensity;
• Dead pixels have a constant R, G or B value of 0;
• Hot pixels have a constant non-zero R, G or B value.

Hardware and Software

The camera used in this report is a Canon PowerShot SX120 IS, purchased in August 2010. Comparisons were made to other cameras, such as a Canon PowerShot SX110IS (purchased November 2008) and a Canon PowerShot A470 (purchased March 2008)
The version of CHDK used was as close to the currently compiled version as practicable, but had no discernible influence on the production of hot pixels.
The DNG version employed was 1.1. There have been reports of version 1.3 producing hot pixels which cannot be removed.
Raw DNG images were processed primarily by DCRAW in the current version. (UFRaw has also been used, but does not have the option of removing bad pixels. However, the same hot pixels turned up when this program was used.)

PHP Script for Converting and Comparing badpixel.bin Files

With the files to be examined in a common subdirectory, named and arranged thus:

• badpixel_001.bin
• badpixel_002.bin
• badpixel_003.bin, etc.

#chdir ("badpixel.bin directory");
$bin_files = glob ("badpixel*.bin");
foreach ($bin_files as $bin)
{
  $text_file = basename ($bin, ".bin") . ".txt";
  $data = fopen ($bin, "rb");
  $bad_pixels = "";
  while (!feof ($data))
  {
    if ($ds = fread ($data, 2))
    {
      list (, $x) = unpack ("v", $ds);
      if ($ds = fread ($data, 2))
      {
        list (, $y) = unpack ("v", $ds);
        $bad_pixels .= "$x $y\r\n";
      }
    }
  }
  file_put_contents ($text_file, $bad_pixels);
  fclose ($data);
}
$text_files = glob ("badpixel*.txt");
$i_max = count ($text_files) - 1;
for ($i = 0; $i < $i_max; $i++)
{
  $bad_pixels_o = file ($text_files[$i]);
  $bad_pixels_c = file ($text_files[$i + 1]);
  $same = $new = $removed = 0;
  foreach ($bad_pixels_o as $bp) if (in_array ($bp, $bad_pixels_c)) $same++; else $removed++;
  foreach ($bad_pixels_c as $bp) if (!in_array ($bp, $bad_pixels_o)) $new++;
  echo "{$text_files[$i]} compared to " . $text_files[$i + 1] . ": same: $same, new: $new, removed: $removed<br>\r\n";
}

List of Dead Pixels Produced by CHDK's Create badpixel.bin Function at Different Times

At various times, the badpixel.bin files have been saved separately and are available as follows:

Legend for comparisons: =: unchanged; +: new; -: removed. Numbers of dead pixels retrieved from badpixel.bin

Even this table shows a counterintuitive decrease in the number of dead pixels over time, which on its own requires an explanation.

CHDK's Dead (Bad) Pixels Vary with Exposure Conditions

The rather puzzling decrease in dead pixel numbers documented above can be explained by generating a number of different badpixel.bin files in a single session and comparing the results. CHDK comes in doubly handy here, because the "File Browser" function enables the renaming of the badpixel file after it has been generated, thus protecting it from being overwritten. In this way, any number of badpixel.bin files can be generated under varying conditions.

First, the ISO setting was varied. Subject was a fairly bright but overcast sky with a B_v of 859-986, resulting in a fairly uniformly textured image. It was immediately apparent that different ISO values caused different badpixel.bin files to be generated:

Numbers within brackets indicate the number of dead pixels actually recoverable from the badpixel.bin file; those within brackets the number of dead pixels shown by CHDK. Identical counts represent identical dead pixels. Closer inspection of the gains/losses of dead pixels between ISO steps revealed:

Once established, very few dead pixels were "lost" again when a new badpixel.bin file was generated.

True Effect of Ageing on Bad Pixel Counts

It is now possible to determine the true effect of ageing on the number of dead pixels by comparing a newer set of bad pixels with its corresponding older set:

Attempts at Determining Thresholds between Sets 1, 2, and 3

Although it was possible to determine that all three sets of dead pixels could be produced at ISO 80 and 100, and two sets of dead pixels could be produced at ISO 200 and 400, determining the threshold at which any particular set came into play — while apparently dependent on the B_v value — reliable, exact determination was elusive. Two methods were used when attempting to determine the transition thresholds. In the Shutter Series method, the shutter speed was reduced step by step in M mode, ensuring all the time that the image was well-exposed. For the B_v Manipulation method, first the brightness value was determined by reading out property #34 in P mode, the badpixel file generated and the brightness value determined immediately afterwards, with the two values being averaged. Then another field was chosen with a different brightness value, and the determination continued until the threshold was approached as closely as was practicable. The results can be summarised as follows:

No threshold transition was observed at ISO 800 or 1600. The mean error for determination by B_v manipulation was approx. ±2.

Behaviour of Other Cameras

The SX110 appeared to produce only one set of dead pixels (P mode, 6291 pixels, according to the CHDK count), which were the same at all ISO settings. The A470 produced 2 sets of dead pixels (Auto mode, 2717 dead pixels counted by CHDK up to and including ISO 400; 4180 pixels counted by CHDK for ISO 800 and 1600).

Recommendations

Two items form the basis of a recommendation for using the badpixel.bin file:

• Bad pixels age well, so that age (normally) does not need to be taken into account;
• The threshold where one bad pixel set starts and the next begins is unpredictable; the differences between the set sizes is so large that it is understandable to attempt to use the appropriate set, but the unpredictability of the thresholds means that it is impracticable.

To generate such a dead pixel file, call the Create badpixel.bin function at the highest ISO rating possible for a shot in the low-to-medium light range.

Problems with Hot Pixels Begin

Over time a number of images have come to my attention that show serious problems with hot pixels. Below are excerpts from pictures in an exposure series. The overexposed image, but also the underexposed image, show many more hot pixels than could possibly be contained in the small files from above:

DNG Images, extreme exposures of an exposure bracket
ISO 200, f 6.3, M mode
1/20 s, 2013/03/01 09:35:58	1/1250 s, 2013/03/01 09:35:40

These two pictures are part of an exposure bracket with 5 pictures of intermediate exposure intervening. Once the bracket is set going the camera is not moved or adjusted in any way.

However, the camera's processing of its raw images show that the bad pixels have been successfully compensated for (images offset by the displacement between JPG and DNG images):

JPG Images, extreme exposures of an exposure bracket
Images correspond to those above

A Method for Determining Bad Pixels in Exposure Bracketing Sequences

The real extent of the variability of bad pixel distribution only comes to light when many such exposure bracketing series can be compared with one another.

A set of 36 exposure bracket series, each series consisting of 7 images at an exposure range of 6 EV and taken within a two month period (February-March 2013), was examined for bad pixels. Raw images were processed by DCRAW with no .badpixel file present in the current or lower directories, and wavelet denoising set at 0. A bad pixel was judged to be any pixel in the first of the series (underexposed image, -3 EV) that had exactly the same R, G and B values as the pixel in the same position in the last of the series (overexposed image, +3 EV). The number of bad pixels in an image pair varied widely:

Also quite extraordinary is the reliability at which any one pixel appeared to be bad. The next table shows how many pixels appeared to be bad in how many pairs of images:

Among the unusual phenomena in this table are:

1. A total of about 143,000 pixels at some time or another were deemed to be bad pixels. This represents about 1.2% of all the pixels in the sensor (12 MB, 3652 x 2758 pixels for DNG images);
2. There is a solitary peak of 940 bad pixels which appear in just over a third of the image pairs;
3. This peak appears to have no or very little error associated with it;
4. One particular hot pixel which was already immediately obvious in many of the images shot, whether brackets or not, was the Bright red pixel, which was found to be hot in 35 of 36 pairs;
5. No pixel was bad in all 36 pairs examined.

Sudden Appearance of Hot Pixels in Time-Lapse Series

It has been found by examining time-lapse series carefully, that whole sets of hot pixels turn up spontaneously during the course of the time-lapse. The following excerpt of four shots taking during a sunset series shows ordinary sensor noise in the first two shots, followed by hot pixels quite distinguishable from the noise in the shot immediately following. Image excerpts are from the areas identical to those in the previous images. Interestingly, these appear initially to be almost identical to the hot pixels in those images. After the hot pixels remain stable for about 6 minutes, picture quality deteriorates further as new hot pixels suddenly appear (and remain until the end of the series).

Stepwise appearance of hot pixels
All images taken at ISO 80, f 2.8 in P mode
1/30 s, 2013/02/28 20:22:48	1/30 s, 2013/02/28 20:23:18	1/25 s, 2013/02/28 20:23:48	1/6 s, 2013/02/28 20:30:18

Questions

This poses a number of questions that I hope to find answers to, among others:

• Has anyone noticed this happening on their camera as well? Can you provide further examples?
• What programs (preferably ones that can run as batch files) can be used to determine hot pixels and what kind of results do they produce?
• How does the Canon firmware deal with hot pixels and is it possible to access their lists of hot pixels and the rules used to apply them?

Note on this Work in Progress

This file will be updated as new information is obtained. If you have any information to contribute, or questions about data that might be useful in understanding these issues, please contact me here. A forum for discussing questions around this issue can be found here.

History of this Document

Prior versions of this document can be found in this list:

• 2013, October 15.