Unusual Bad Pixel Behaviour on a Canon PowerShot SX120

Lee Traynor

First published: October 15, 2013, expanded and revised October 20, 2013
Table of Contents

Introduction

With the advent of CHDK (Canon Hackers' Development Kit), many inexpensive point&shoot Canon cameras can be turned into versatile photographic instruments, with capabilities that far exceed other cameras in their price range, e.g bracketing, taking raw pictures, shooting indefinite time lapses.

When saving images in the DNG raw format (digital negative), CHDK also offers the possibility to remove bad pixels. However, these bad pixels seem to be particularly volatile. I have documented wide variations of number and position of bad pixels in a variety of situations. Although there is as yet no explanation for why bad pixels should vary in such a striking manner, determining how they vary might provide strategies for improving picture quality by removing them.

Definitions

For clarity the terms, "bad", "dead" and "hot" have to be determined for pixel behaviour:

Hardware and Software

The camera used in this report is a Canon PowerShot SX120 IS, purchased in August 2010. Comparisons were made to other cameras, such as a Canon PowerShot SX110IS (purchased November 2008) and a Canon PowerShot A470 (purchased March 2008)
The version of CHDK used was as close to the currently compiled version as practicable, but had no discernible influence on the production of hot pixels.
The DNG version employed was 1.1. There have been reports of version 1.3 producing hot pixels which cannot be removed.
Raw DNG images were processed primarily by DCRAW in the current version. (UFRaw has also been used, but does not have the option of removing bad pixels. However, the same hot pixels turned up when this program was used.)

PHP Script for Converting and Comparing badpixel.bin Files

With the files to be examined in a common subdirectory, named and arranged thus:

files can be converted into text files and compared pairwise (i.e. first, badpixel_001.bin with badpixel.002.bin, then badpixel.002.bin with badpixel.003.bin, etc., with the following script:

#chdir ("badpixel.bin directory");
$bin_files = glob ("badpixel*.bin");
foreach ($bin_files as $bin)
{
  $text_file = basename ($bin, ".bin") . ".txt";
  $data = fopen ($bin, "rb");
  $bad_pixels = "";
  while (!feof ($data))
  {
    if ($ds = fread ($data, 2))
    {
      list (, $x) = unpack ("v", $ds);
      if ($ds = fread ($data, 2))
      {
        list (, $y) = unpack ("v", $ds);
        $bad_pixels .= "$x $y\r\n";
      }
    }
  }
  file_put_contents ($text_file, $bad_pixels);
  fclose ($data);
}
$text_files = glob ("badpixel*.txt");
$i_max = count ($text_files) - 1;
for ($i = 0; $i < $i_max; $i++)
{
  $bad_pixels_o = file ($text_files[$i]);
  $bad_pixels_c = file ($text_files[$i + 1]);
  $same = $new = $removed = 0;
  foreach ($bad_pixels_o as $bp) if (in_array ($bp, $bad_pixels_c)) $same++; else $removed++;
  foreach ($bad_pixels_c as $bp) if (!in_array ($bp, $bad_pixels_o)) $new++;
  echo "{$text_files[$i]} compared to " . $text_files[$i + 1] . ": same: $same, new: $new, removed: $removed<br>\r\n";
}

List of Dead Pixels Produced by CHDK's Create badpixel.bin Function at Different Times

At various times, the badpixel.bin files have been saved separately and are available as follows:

Apparent Development of Dead Pixels with Time
Date, Binary FileText FileSize, kB# Dead PixelsCompare to: 110509121213130728
May 9, 2011badpixel_120_110509_L.txt31.88152---
December 13, 2012badpixel_120_121213_L.txt31.78116=8080, +36, -72--
July 28, 2013badpixel_120_130728_L.txt10.52689=2668, +21, -5484=2677, +12, -5439-
October 7, 2013badpixel_120_131007_L.txt6.891764=1752, +12, -6400=1734, +30, -6382=1743, +21, -946

Legend for comparisons: =: unchanged; +: new; -: removed. Numbers of dead pixels retrieved from badpixel.bin

Even this table shows a counterintuitive decrease in the number of dead pixels over time, which on its own requires an explanation.

CHDK's Dead (Bad) Pixels Vary with Exposure Conditions

The rather puzzling decrease in dead pixel numbers documented above can be explained by generating a number of different badpixel.bin files in a single session and comparing the results. CHDK comes in doubly handy here, because the "File Browser" function enables the renaming of the badpixel file after it has been generated, thus protecting it from being overwritten. In this way, any number of badpixel.bin files can be generated under varying conditions.

First, the ISO setting was varied. Subject was a fairly bright but overcast sky with a Bv of 859-986, resulting in a fairly uniformly textured image. It was immediately apparent that different ISO values caused different badpixel.bin files to be generated:

Dependence of Dead Pixel Count on ISO
ISO# Dead Pixels
801764 (1776)
1001764 (1776)
2002710 (2731)
4002710 (2731)
8008152 (8188)
16008152 (8188)

Numbers within brackets indicate the number of dead pixels actually recoverable from the badpixel.bin file; those within brackets the number of dead pixels shown by CHDK. Identical counts represent identical dead pixels. Closer inspection of the gains/losses of dead pixels between ISO steps revealed:

Development of Dead Pixels with ISO
ISODead Pixel Set#Dead PixelsUnchangedNewLostUnchangedNewLost
 cf. Set 1cf. Set 2
80, 10011764------
200, 4002271017599515---
800, 1600381521753639911270254508

Once established, very few dead pixels were "lost" again when a new badpixel.bin file was generated.

True Effect of Ageing on Bad Pixel Counts

It is now possible to determine the true effect of ageing on the number of dead pixels by comparing a newer set of bad pixels with its corresponding older set:

Bad Pixel Numbers and Age at T2 = October 18, 2013
T1Dead Pixel Set#Dead Pixels at T2#Dead Pixels at T2NewLostRate (px/a), New only
October 7, 2013117641764000
July 28, 201322689271021093
May 9, 2011381528152363615
Even the highest rate (ca. 100 pixels / year) can be neglected when compared to the sample size.

Attempts at Determining Thresholds between Sets 1, 2, and 3

Although it was possible to determine that all three sets of dead pixels could be produced at ISO 80 and 100, and two sets of dead pixels could be produced at ISO 200 and 400, determining the threshold at which any particular set came into play — while apparently dependent on the Bv value — reliable, exact determination was elusive. Two methods were used when attempting to determine the transition thresholds. In the Shutter Series method, the shutter speed was reduced step by step in M mode, ensuring all the time that the image was well-exposed. For the Bv Manipulation method, first the brightness value was determined by reading out property #34 in P mode, the badpixel file generated and the brightness value determined immediately afterwards, with the two values being averaged. Then another field was chosen with a different brightness value, and the determination continued until the threshold was approached as closely as was practicable. The results can be summarised as follows:

Threshold Determination for Dead Pixel Sets
Threshold, BvMethod
ISO, TransitionShutter SeriesBv Manipulation
80, 1→2647-662458
100, 1→2-415
80, 2→3343-359258
100, 2→3-218
200, 2→3-339
400, 2→3-244

No threshold transition was observed at ISO 800 or 1600. The mean error for determination by Bv manipulation was approx. 2.

Behaviour of Other Cameras

The SX110 appeared to produce only one set of dead pixels (P mode, 6291 pixels, according to the CHDK count), which were the same at all ISO settings. The A470 produced 2 sets of dead pixels (Auto mode, 2717 dead pixels counted by CHDK up to and including ISO 400; 4180 pixels counted by CHDK for ISO 800 and 1600).

Recommendations

Two items form the basis of a recommendation for using the badpixel.bin file:

Therefore I recommend applying the Precautionary Principle, and using the largest possible dead pixel set that can be generated on an individual camera. In relative terms, the number of dead pixels is typically < 0.1% at a maximum, which should hardly affect picture quality at normal resolutions. In this way, the worst case is covered.

To generate such a dead pixel file, call the Create badpixel.bin function at the highest ISO rating possible for a shot in the low-to-medium light range.

Problems with Hot Pixels Begin

Over time a number of images have come to my attention that show serious problems with hot pixels. Below are excerpts from pictures in an exposure series. The overexposed image, but also the underexposed image, show many more hot pixels than could possibly be contained in the small files from above:

DNG Images, extreme exposures of an exposure bracket
ISO 200, f 6.3, M mode
TotaranuiTotaranui
1/20 s, 2013/03/01 09:35:581/1250 s, 2013/03/01 09:35:40

These two pictures are part of an exposure bracket with 5 pictures of intermediate exposure intervening. Once the bracket is set going the camera is not moved or adjusted in any way.

However, the camera's processing of its raw images show that the bad pixels have been successfully compensated for (images offset by the displacement between JPG and DNG images):

JPG Images, extreme exposures of an exposure bracket
Images correspond to those above
TotaranuiTotaranui

A Method for Determining Bad Pixels in Exposure Bracketing Sequences

The real extent of the variability of bad pixel distribution only comes to light when many such exposure bracketing series can be compared with one another.

A set of 36 exposure bracket series, each series consisting of 7 images at an exposure range of 6 EV and taken within a two month period (February-March 2013), was examined for bad pixels. Raw images were processed by DCRAW with no .badpixel file present in the current or lower directories, and wavelet denoising set at 0. A bad pixel was judged to be any pixel in the first of the series (underexposed image, -3 EV) that had exactly the same R, G and B values as the pixel in the same position in the last of the series (overexposed image, +3 EV). The number of bad pixels in an image pair varied widely:

Image Pair #No of bad pixels
11
21
31
41
51
610
71
81
91
103
114
121
131
141
151
161
171946
182371
191791
201381
211197
221188
231013
241021
251057
261133
2777050
281211
291172
30598
313586
326219
3313594
345894
351089
3632006

Also quite extraordinary is the reliability at which any one pixel appeared to be bad. The next table shows how many pixels appeared to be bad in how many pairs of images:

No of PixelsAppear to be bad in this number of image pairsComment
1412381
6432
2243
1534
545
146
47
18
09
110
011
012
94013Solitary peak
614
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
135Bright red pixel (BRP)
036

Among the unusual phenomena in this table are:

  1. A total of about 143,000 pixels at some time or another were deemed to be bad pixels. This represents about 1.2% of all the pixels in the sensor (12 MB, 3652 x 2758 pixels for DNG images);
  2. There is a solitary peak of 940 bad pixels which appear in just over a third of the image pairs;
  3. This peak appears to have no or very little error associated with it;
  4. One particular hot pixel which was already immediately obvious in many of the images shot, whether brackets or not, was the Bright red pixel, which was found to be hot in 35 of 36 pairs;
  5. No pixel was bad in all 36 pairs examined.

Sudden Appearance of Hot Pixels in Time-Lapse Series

It has been found by examining time-lapse series carefully, that whole sets of hot pixels turn up spontaneously during the course of the time-lapse. The following excerpt of four shots taking during a sunset series shows ordinary sensor noise in the first two shots, followed by hot pixels quite distinguishable from the noise in the shot immediately following. Image excerpts are from the areas identical to those in the previous images. Interestingly, these appear initially to be almost identical to the hot pixels in those images. After the hot pixels remain stable for about 6 minutes, picture quality deteriorates further as new hot pixels suddenly appear (and remain until the end of the series).

Stepwise appearance of hot pixels
All images taken at ISO 80, f 2.8 in P mode
1/30 s, 2013/02/28 20:22:481/30 s, 2013/02/28 20:23:181/25 s, 2013/02/28 20:23:481/6 s, 2013/02/28 20:30:18

Questions

This poses a number of questions that I hope to find answers to, among others:

And, of course, we would all like to know how bad pixels originate, and where they go when they cease to exist.

Note on this Work in Progress

This file will be updated as new information is obtained. If you have any information to contribute, or questions about data that might be useful in understanding these issues, please contact me here. A forum for discussing questions around this issue can be found here.

History of this Document

Prior versions of this document can be found in this list: