Table of Contents | |
---|---|
With the advent of CHDK (Canon Hackers' Development Kit), many inexpensive point&shoot Canon cameras can be turned into versatile photographic instruments, with capabilities that far exceed other cameras in their price range, e.g bracketing, taking raw pictures, shooting indefinite time lapses.
When saving images in the DNG raw format (digital negative), CHDK also offers the possibility to remove bad pixels. However, these bad pixels seem to be particularly volatile. I have documented wide variations of number and position of bad pixels in a variety of situations. Although there is as yet no explanation for why bad pixels should vary in such a striking manner, determining how they vary might provide strategies for improving picture quality by removing them.
For clarity the terms, "bad", "dead" and "hot" have to be determined for pixel behaviour:
The camera used in this report is a Canon PowerShot SX120 IS, purchased in August 2010. Comparisons were made to other cameras, such as a Canon PowerShot SX110IS (purchased November 2008) and a Canon PowerShot A470 (purchased March 2008)
The version of CHDK used was as close to the currently compiled version as practicable, but had no discernible influence on the production of hot pixels.
The DNG version employed was 1.1. There have been reports of version 1.3 producing hot pixels which cannot be removed.
Raw DNG images were processed primarily by DCRAW in the current version. (UFRaw has also been used, but does not have the option of removing bad pixels. However, the same hot pixels turned up when this program was used.)
With the files to be examined in a common subdirectory, named and arranged thus:
#chdir ("badpixel.bin directory");
$bin_files = glob ("badpixel*.bin");
foreach ($bin_files as $bin)
{
$text_file = basename ($bin, ".bin") . ".txt";
$data = fopen ($bin, "rb");
$bad_pixels = "";
while (!feof ($data))
{
if ($ds = fread ($data, 2))
{
list (, $x) = unpack ("v", $ds);
if ($ds = fread ($data, 2))
{
list (, $y) = unpack ("v", $ds);
$bad_pixels .= "$x $y\r\n";
}
}
}
file_put_contents ($text_file, $bad_pixels);
fclose ($data);
}
$text_files = glob ("badpixel*.txt");
$i_max = count ($text_files) - 1;
for ($i = 0; $i < $i_max; $i++)
{
$bad_pixels_o = file ($text_files[$i]);
$bad_pixels_c = file ($text_files[$i + 1]);
$same = $new = $removed = 0;
foreach ($bad_pixels_o as $bp) if (in_array ($bp, $bad_pixels_c)) $same++; else $removed++;
foreach ($bad_pixels_c as $bp) if (!in_array ($bp, $bad_pixels_o)) $new++;
echo "{$text_files[$i]} compared to " . $text_files[$i + 1] . ": same: $same, new: $new, removed: $removed<br>\r\n";
}
At various times, the badpixel.bin files have been saved separately and are available as follows:
Apparent Development of Dead Pixels with Time | ||||||
---|---|---|---|---|---|---|
Date, Binary File | Text File | Size, kB | # Dead Pixels | Compare to: 110509 | 121213 | 130728 |
May 9, 2011 | badpixel_120_110509_L.txt | 31.8 | 8152 | - | - | - |
December 13, 2012 | badpixel_120_121213_L.txt | 31.7 | 8116 | =8080, +36, -72 | - | - |
July 28, 2013 | badpixel_120_130728_L.txt | 10.5 | 2689 | =2668, +21, -5484 | =2677, +12, -5439 | - |
October 7, 2013 | badpixel_120_131007_L.txt | 6.89 | 1764 | =1752, +12, -6400 | =1734, +30, -6382 | =1743, +21, -946 |
Legend for comparisons: =: unchanged; +: new; -: removed. Numbers of dead pixels retrieved from badpixel.bin
Even this table shows a counterintuitive decrease in the number of dead pixels over time, which on its own requires an explanation.
The rather puzzling decrease in dead pixel numbers documented above can be explained by generating a number of different badpixel.bin files in a single session and comparing the results. CHDK comes in doubly handy here, because the "File Browser" function enables the renaming of the badpixel file after it has been generated, thus protecting it from being overwritten. In this way, any number of badpixel.bin files can be generated under varying conditions.
First, the ISO setting was varied. Subject was a fairly bright but overcast sky with a Bv of 859-986, resulting in a fairly uniformly textured image. It was immediately apparent that different ISO values caused different badpixel.bin files to be generated:
Dependence of Dead Pixel Count on ISO | |
---|---|
ISO | # Dead Pixels |
80 | 1764 (1776) |
100 | 1764 (1776) |
200 | 2710 (2731) |
400 | 2710 (2731) |
800 | 8152 (8188) |
1600 | 8152 (8188) |
Numbers within brackets indicate the number of dead pixels actually recoverable from the badpixel.bin file; those within brackets the number of dead pixels shown by CHDK. Identical counts represent identical dead pixels. Closer inspection of the gains/losses of dead pixels between ISO steps revealed:
Development of Dead Pixels with ISO | ||||||||
---|---|---|---|---|---|---|---|---|
ISO | Dead Pixel Set | #Dead Pixels | Unchanged | New | Lost | Unchanged | New | Lost |
cf. Set 1 | cf. Set 2 | |||||||
80, 100 | 1 | 1764 | - | - | - | - | - | - |
200, 400 | 2 | 2710 | 1759 | 951 | 5 | - | - | - |
800, 1600 | 3 | 8152 | 1753 | 6399 | 11 | 2702 | 5450 | 8 |
Once established, very few dead pixels were "lost" again when a new badpixel.bin file was generated.
It is now possible to determine the true effect of ageing on the number of dead pixels by comparing a newer set of bad pixels with its corresponding older set:
Bad Pixel Numbers and Age at T2 = October 18, 2013 | ||||||
---|---|---|---|---|---|---|
T1 | Dead Pixel Set | #Dead Pixels at T2 | #Dead Pixels at T2 | New | Lost | Rate (px/a), New only |
October 7, 2013 | 1 | 1764 | 1764 | 0 | 0 | 0 |
July 28, 2013 | 2 | 2689 | 2710 | 21 | 0 | 93 |
May 9, 2011 | 3 | 8152 | 8152 | 36 | 36 | 15 |
Although it was possible to determine that all three sets of dead pixels could be produced at ISO 80 and 100, and two sets of dead pixels could be produced at ISO 200 and 400, determining the threshold at which any particular set came into play while apparently dependent on the Bv value reliable, exact determination was elusive. Two methods were used when attempting to determine the transition thresholds. In the Shutter Series method, the shutter speed was reduced step by step in M mode, ensuring all the time that the image was well-exposed. For the Bv Manipulation method, first the brightness value was determined by reading out property #34 in P mode, the badpixel file generated and the brightness value determined immediately afterwards, with the two values being averaged. Then another field was chosen with a different brightness value, and the determination continued until the threshold was approached as closely as was practicable. The results can be summarised as follows:
Threshold Determination for Dead Pixel Sets | |||
---|---|---|---|
Threshold, Bv | Method | ||
ISO, Transition | Shutter Series | Bv Manipulation | |
80, 1→2 | 647-662 | 458 | |
100, 1→2 | - | 415 | |
80, 2→3 | 343-359 | 258 | |
100, 2→3 | - | 218 | |
200, 2→3 | - | 339 | |
400, 2→3 | - | 244 |
No threshold transition was observed at ISO 800 or 1600. The mean error for determination by Bv manipulation was approx. ±2.
The SX110 appeared to produce only one set of dead pixels (P mode, 6291 pixels, according to the CHDK count), which were the same at all ISO settings. The A470 produced 2 sets of dead pixels (Auto mode, 2717 dead pixels counted by CHDK up to and including ISO 400; 4180 pixels counted by CHDK for ISO 800 and 1600).
Two items form the basis of a recommendation for using the badpixel.bin file:
To generate such a dead pixel file, call the Create badpixel.bin function at the highest ISO rating possible for a shot in the low-to-medium light range.
Over time a number of images have come to my attention that show serious problems with hot pixels. Below are excerpts from pictures in an exposure series. The overexposed image, but also the underexposed image, show many more hot pixels than could possibly be contained in the small files from above:
DNG Images, extreme exposures of an exposure bracket | |
---|---|
ISO 200, f 6.3, M mode | |
![]() | ![]() |
1/20 s, 2013/03/01 09:35:58 | 1/1250 s, 2013/03/01 09:35:40 |
These two pictures are part of an exposure bracket with 5 pictures of intermediate exposure intervening. Once the bracket is set going the camera is not moved or adjusted in any way.
However, the camera's processing of its raw images show that the bad pixels have been successfully compensated for (images offset by the displacement between JPG and DNG images):
JPG Images, extreme exposures of an exposure bracket | |
---|---|
Images correspond to those above | |
![]() | ![]() |
The real extent of the variability of bad pixel distribution only comes to light when many such exposure bracketing series can be compared with one another.
A set of 36 exposure bracket series, each series consisting of 7 images at an exposure range of 6 EV and taken within a two month period (February-March 2013), was examined for bad pixels. Raw images were processed by DCRAW with no .badpixel file present in the current or lower directories, and wavelet denoising set at 0. A bad pixel was judged to be any pixel in the first of the series (underexposed image, -3 EV) that had exactly the same R, G and B values as the pixel in the same position in the last of the series (overexposed image, +3 EV). The number of bad pixels in an image pair varied widely:
Image Pair # | No of bad pixels |
---|---|
1 | 1 |
2 | 1 |
3 | 1 |
4 | 1 |
5 | 1 |
6 | 10 |
7 | 1 |
8 | 1 |
9 | 1 |
10 | 3 |
11 | 4 |
12 | 1 |
13 | 1 |
14 | 1 |
15 | 1 |
16 | 1 |
17 | 1946 |
18 | 2371 |
19 | 1791 |
20 | 1381 |
21 | 1197 |
22 | 1188 |
23 | 1013 |
24 | 1021 |
25 | 1057 |
26 | 1133 |
27 | 77050 |
28 | 1211 |
29 | 1172 |
30 | 598 |
31 | 3586 |
32 | 6219 |
33 | 13594 |
34 | 5894 |
35 | 1089 |
36 | 32006 |
Also quite extraordinary is the reliability at which any one pixel appeared to be bad. The next table shows how many pixels appeared to be bad in how many pairs of images:
No of Pixels | Appear to be bad in this number of image pairs | Comment |
---|---|---|
141238 | 1 | |
643 | 2 | |
224 | 3 | |
153 | 4 | |
54 | 5 | |
14 | 6 | |
4 | 7 | |
1 | 8 | |
0 | 9 | |
1 | 10 | |
0 | 11 | |
0 | 12 | |
940 | 13 | Solitary peak |
6 | 14 | |
0 | 15 | |
0 | 16 | |
0 | 17 | |
0 | 18 | |
0 | 19 | |
0 | 20 | |
0 | 21 | |
0 | 22 | |
0 | 23 | |
0 | 24 | |
0 | 25 | |
0 | 26 | |
0 | 27 | |
0 | 28 | |
0 | 29 | |
0 | 30 | |
0 | 31 | |
0 | 32 | |
0 | 33 | |
0 | 34 | |
1 | 35 | Bright red pixel (BRP) |
0 | 36 | |
Among the unusual phenomena in this table are:
It has been found by examining time-lapse series carefully, that whole sets of hot pixels turn up spontaneously during the course of the time-lapse. The following excerpt of four shots taking during a sunset series shows ordinary sensor noise in the first two shots, followed by hot pixels quite distinguishable from the noise in the shot immediately following. Image excerpts are from the areas identical to those in the previous images. Interestingly, these appear initially to be almost identical to the hot pixels in those images. After the hot pixels remain stable for about 6 minutes, picture quality deteriorates further as new hot pixels suddenly appear (and remain until the end of the series).
Stepwise appearance of hot pixels | |||
---|---|---|---|
All images taken at ISO 80, f 2.8 in P mode | |||
![]() | ![]() | ![]() | ![]() |
1/30 s, 2013/02/28 20:22:48 | 1/30 s, 2013/02/28 20:23:18 | 1/25 s, 2013/02/28 20:23:48 | 1/6 s, 2013/02/28 20:30:18 |
This poses a number of questions that I hope to find answers to, among others:
This file will be updated as new information is obtained. If you have any information to contribute, or questions about data that might be useful in understanding these issues, please contact me here. A forum for discussing questions around this issue can be found here.
Prior versions of this document can be found in this list: