[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[task #16447] Median Absolute Deviation based clipping
From: |
Mohammad Akhlaghi |
Subject: |
[task #16447] Median Absolute Deviation based clipping |
Date: |
Tue, 3 Oct 2023 20:54:22 -0400 (EDT) |
URL:
<https://savannah.gnu.org/task/?16447>
Summary: Median Absolute Deviation based clipping
Group: GNU Astronomy Utilities
Submitter: makhlaghi
Submitted: Wed 04 Oct 2023 02:54:20 AM CEST
Should Start On: Wed 04 Oct 2023 12:00:00 AM CEST
Should be Finished on: Wed 04 Oct 2023 12:00:00 AM CEST
Category: Libraries
Priority: 5 - Normal
Item Group: Enhancement
Status: None
Privacy: Public
Percent Complete: 0%
Assigned to: makhlaghi
Open/Closed: Open
Discussion Lock: Any
Effort: 0.00
_______________________________________________________
Follow-up Comments:
-------------------------------------------------------
Date: Wed 04 Oct 2023 02:54:20 AM CEST By: Mohammad Akhlaghi <makhlaghi>
Currently, the most commonly used way to reject outliers is through
[https://www.gnu.org/software/gnuastro/manual/html_node/Sigma-clipping.html
sigma-clipping]. However, sigma-clipping is itself based on the standard
deviation (which is strongly affected by outliers!).
To demonstrate the problem, in the P.S. I am attaching a small script to
generate 9 noisy images, where one of them contains a bright circle in the
middle. Afterwards the script stacks the images by sigma-clipping and opens
DS9 to show the sigma-clipped result. The footprint of the circle is present
in the sigma-clipped mean (very strong) and sigma-clipped median (weaker, but
still present).
As mentioned above, the cause of this is that the standard deviation is very
easily/strongly affected by outliers. You can see this through the few numbers
of this command (which contain the strong outlier of 15). The third command
doesn't contain the '15':
$ printf "2\n1\n4\n2\n3\n5\n15\n"
2
1
4
2
3
5
15
$ printf "2\n1\n4\n2\n3\n5\n15\n" \
| aststatistics --sigclip-mean --sigclip-std
4.571429e+00 4.435479e+00
$ printf "2\n1\n4\n2\n3\n5\n" \
| aststatistics --sigclip-mean --sigclip-std
2.833333e+00 1.343710e+00
As you see here, sigma-clipping failed to clip/exclude the '15' because the
standard deviation itself is so strongly shifted due to it!
In this task, I have started work on generalizing the clipping algorithm to
also do the clipping based on
[https://en.wikipedia.org/wiki/Median_absolute_deviation Median Absolute
Deviation] (MAD).
P.S.
# Constants
profn=4
sigma=10
width=201
number=9
radius=30
profsum=1e5
# Initialize internal parameters.
imglist=""
# For the final profile, put a flat circle in the center of the image.
nn=$number-no-noise.fits
center=$(echo $width | awk '{print int($1/2)+1}')
echo "1 $center $center 5 $radius $profn 0 1 $profsum 1" \
| astmkprof --mode=img --mergedsize=$width,$width --oversample=1 \
--output=$nn --mcolissum
astarithmetic $nn $sigma mknoise-sigma -o$number.fits
imglist="$imglist $number.fits"
# Build pure noise.
numnoise=$(echo $number | awk '{print $1-1}')
for i in $(seq 1 $numnoise); do
imglist="$imglist $i.fits"
astarithmetic $width $width 2 makenew $sigma mknoise-sigma -o$i.fits
done
# Stack the images.
astarithmetic $imglist $number 3 0.2 sigclip-std -g1 -o stack-std.fits
astarithmetic $imglist $number 3 0.2 sigclip-number -g1 -o stack-num.fits
astarithmetic $imglist $number 3 0.2 sigclip-mean -g1 -o stack-mean.fits
astarithmetic $imglist $number 3 0.2 sigclip-median -g1 -o stack-median.fits
# Show the images and the stack.
#$imglist
#astscript-fits-view $imglist
#astscript-fits-view $number.fits stack-std.fits stack-median.fits
astscript-fits-view stack-num.fits stack-mean.fits stack-std.fits
stack-median.fits
_______________________________________________________
File Attachments:
-------------------------------------------------------
Date: Wed 04 Oct 2023 02:54:20 AM CEST Name: inputs.png Size: 485KiB By:
makhlaghi
<http://savannah.gnu.org/task/download.php?file_id=55196>
-------------------------------------------------------
Date: Wed 04 Oct 2023 02:54:20 AM CEST Name: sigma-clipping.png Size: 345KiB
By: makhlaghi
<http://savannah.gnu.org/task/download.php?file_id=55197>
_______________________________________________________
Reply to this item at:
<https://savannah.gnu.org/task/?16447>
_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/
- [task #16447] Median Absolute Deviation based clipping,
Mohammad Akhlaghi <=