Re: Possibly wrong statistics calculation

[Mohammad Akhlaghi]

Hi,

Thank you very much for reporting this.

This is an interesting situation. From the command below, we see that 
the image has 223112 pixels.

astfits 019_00.fits -h0 | awk '$1=="NAXIS1"{x=$3} $1=="NAXIS2"{y=$3} 
END{print x*y}'

But when I look at the "Number of elements" reported by the 
aststatistics program is one less: 223111

Then I noticed that in your calculation, the maximum is 65535 (the 
maximum value for unsigned 16-bit integers) while aststatistics says it 
is 65184.

This explains the problem: in Gnuastro, for unsigned types, the maximum 
value is defined as a blank value. In a single exposure this is natural: 
this value can only occur when the pixel is saturated and in such cases, 
indeed, it shouldn't be included in the statistics. This is explained in 
this part of the Gnuastro book:

https://www.gnu.org/software/gnuastro/manual/html_node/Blank-pixels.html

But if in your scenario, that single saturated pixel is meaningful, 
indeed it is a bug.

So I just defined bug #58360 for this [1]. Currently I am too busy to 
implement it immediately, but hopefully we'll implement a fix soon. I 
have added you in the "Carbon copy" of that bug so you are informed when 
it is fixed ;-).

If there is any other thoughts on that bug, it would be great if we 
discuss it on Savannah so it is recorded under there. You can just 
register on Savannah and post bugs there.

Thanks again for reporting this,
Cheers,
Mohammad

[1] https://savannah.gnu.org/bugs/?58360

On 5/12/20 9:27 PM, rmorales wrote:
> Summary:
> Possibly wrong statistics calculation
>
> gnuastro version:
> 0.11
>
> test image set:
> https://cloud.iaa.csic.es/public.php?service=files&t=9344cdb85c640c7dd74446d8c266591d 
>
>
> image pixel sequence as csv file:
> https://cloud.iaa.csic.es/public.php?service=files&t=56b7ade952ede68f496b313c52887aa5 
>
>
> command:
> aststatistics 019_00.fits --hdu=0
>
> result :
>    Number of elements:                      223111
>    Minimum:                                 0
>    Maximum:                                 65184
>    Median:                                  1295
>    Mean:                                    1340.290403
>    Standard deviation:                      524.5495077
>
> question:
>
> Using my own statistics some values are different: average, stdDev and max:
> average = 1340.57812668077
> variance = 486126.1640894976
> stdDev = 697.2274837450813
> mode = 0
> median = 1295
> min = 0
> max = 65535
>
> Writing all pixels in to a calc sheet (see download link at beginning of 
> this message) :
> Mean    1340.57812668077
> Mode    0
> Median    1295
> Variance    486126.164089498
> Standard Deviation    697.227483745081
> Minimum    0
> Maximum    65535
>
> suggestion:
> Review how the pixel values are obtained from FITS file applying the 
> linear transformation with BSCALE and BZERO keywords.