Occasionally we go “deep” into areas of astronomical image processing to discuss some unique applications or methods, of interest to the astroimagers out there. This is one I came across recently which addresses a problem inherent to the ccd sensor, namely “column defects”. The ccd or charge-coupled device was a revolution for astronomy. Pixels represented by metal-oxide semiconductor capacitors allow conversion of incoming photons from a distant galaxy for example into electric charges which are then read out by the sensor as digital signal. The result of this is that the more photons are accumulated, the more information is gathered i.e. the more you will see. One of the issues imagers face with these sensors is that they may have aberrations in how some groups of pixels operate due to the manufacturing process such that artifacts can be present in your image.
Those of us who are using ccd sensors for imaging are familiar with the so-called “column defects” which we think of typically as a single white single pixel vertical line, maybe one or two, either full length or partial, that appear in our standard raw images. In the past I would ignore them, expecting routine calibration to remove them. Calibration is the process where you remove noise and other artifacts from the image. Unfortunately, the more image processing I did, the more often I would get to the end only to see an ugly black or white line cutting right across a busy field of bright nebulosity! Ouch! It took me awhile to realize that in your average ccd sensor there can be quite a number of these defects which often are next to impossible to see because they often lie very close to the dimmest noise detectable in your image, i.e. the “noise floor”. Most of us imagers are familiar with “hot” pixels which are single pixels or clusters of pixels that have a much higher dark current than their neighbors and appear as white speckles. Black pixels have a lower response than their neighbors and are not as common. These types of defects are fairly easily removed with standard “cosmetic correction” methods that many processing programs offer. Much more potentially damaging are the column abnormalities. In a similar manner to individual pixels you can have a whole column of pixels that are either “hot” or “black”. These are next to impossible to remove cleanly. Even the ones you can obviously see are not easily suppressed.
Of course there is a happy ending to all of this as I recently stumbled across a script in the processing program Pixinsight which addresses this very problem! As I looked into this and tried it out I was floored honestly by how many of these types of defects there really are. The bottom line is that the process takes a few simple steps with absolutely pristine results! The strategy is a combination of multiscale and statistical methods rather than simply painting the columns with adjacent pixels which can lead to artifacts. Here is basically what you do and how it works for all of you astroimagers out there:
First off lets take a look at this “Master Dark” frame which is a record of the camera’s “dark current” Most of this is thermal energy generated within the sensor which is independent of any light falling on it. However you can see that there are 2 white vertical lines and 1 dark line in addition to the speckled background. These are the column defects. What I have learned is that there are a LOT more defects in there that you cannot even see!
In the past I would expect basic calibration to take care of these. In other words, to simplify for those not familiar with this, you subtract the dark frame, and hence the speckling (noise), from the light frame, and the “flat” frame from the light frame , which removes other aberrations in your optical set-up such as dust on your optics, internal reflections etc.
This is an application of “basic” calibration (dark and flat subtraction) to the image of the Iris nebula on the left to arrive at the result on the right. Overall it looks pretty good. Just about all of the bright speckling is gone although it might be difficult to see in this formatted size.
Unfortunately if we look very carefully at the background and magnify the image we find these unsightly vertical lines. This obvious black column is going to be a major problem because after you combine all of your images things like this will be enhanced which we do not want.
So how do we accomplish removal of these? To do this you need the most current version of the Pixinsight application which at the time of this post is version 1.8.8-5
Here is the “path to victory”!:
- Calibrate your images as you would normally with darks, flats, etc
- Do an image integration without registering your images. This is very important because this is how the linear defects are best seen by the script. The integrated result looks weird, like you didn’t focus properly but this is correct!
- Make sure your integrated image is already open in Pixinsight before opening the script
- Go to “script”> “utilities” > “linear defect detection” as shown here:
The script will open and look like this:
You do NOT have to change anything here. Just enter a location for the output text file which will list the x, y coordinates and length of each column defect. This file will be used for the next step.
5. Click on “run” and the script will generate the text file as well as 3 other image files: line model, line detection and partial line detection. The only purpose of these 3 other image files ,for me anyway, was to graphically demonstrate the scope of the defect problem!
These are the myriads of column abnormalities discovered! Crazy! You would think maybe one or 2 but not all of this! You can discard these images as they won’t be needed for the next steps.
6. Next you will open the “Linear Pattern Subtraction” script found right underneath the Linear Defect Detection also under “script> utilities”
7. The script dialog looks like this:
Once again you typically will not need to change any of these default settings. When you first open the script you might see the “target is active image” box checked. Un-check this so you can add individual files. Type in an output directory folder where the corrected files will appear with the suffix “_lps” added. Click on “add files” to add your calibrated files and also navigate to the previously generated defect text file and click on it so it will appear in the “Defects file” box.
8. Click on “Run”!
On the right is the original image file with the black column defect and on the left is the corrected file after linear pattern subtraction. Amazing! The other background “noise” is removed in later stages of processing.
So this was a “quick start” demo and I can say that it has worked consistently as described with no additional changes. For a more detailed explanation and great tutorial you can visit the official Pixinsight page here: https://pixinsight.com/tutorials/LDD-LPS/
Thanks for reading!