Gnuastro 0.18.71 released [alpha]

[Mohammad Akhlaghi]

Dear all,

The last few months (since the release of Gnuastro 0.18) have been
very productive in Gnuastro (16866 lines added and 4864 lines removed,
in 71 commits) with some pretty exciting, and long-awaited new
features now available (for the full list, see [1] below).

 - Warp: align images with the WCS (usually RA/Dec), or another image
   (with any rotation/projection/distortion), while accounting for all
   WCSLIB-recognized distortions and projections (for the full list of
   projection and distortion models see [1] below). In other words, we
   are finally free from the chains of the TPV and can directly align
   images with SIP or other distortions in creating aligned coadds!
   Tip: if you get unreasonable results, upgrade to WCSLIB 7.12.

 - ConvertType: draw vector graphics markers based on input catalogs
   in the output PDFs (very good for producing high-quality figures in
   your papers).

 - Statistics: least squares fitting (currently in 1D).

In fact, Pedram (author of the newly added Warp feature above) has
recorded a video highlighing the first two points in Warp and
ConvertType, please watch it to get a hands-on feeling of the power of
these new capabilities using SDSS and J-PLUS images:

  https://peertube.stream/w/uq7SBDYZS1HRtJwCkbcDsz

If you have any questions or ideas, feel free to get in touch with
Pedram and the rest of the users and developers in our Matrix chat:

  https://openastronomy.element.io/#/room/#gnuastro:openastronomy.org

As always, complete tutorials and documentation for all these features
is also available. Currently the documentation is only in 'info'
format, installed with Gnuastro (the webpage will be updated with
version 0.19). A PDF copy of the latest documentation is available
here: https://akhlaghi.org/gnuastro.pdf

We are therefore almost ready for the next official release of
Gnuastro; hence the reason for this alpha release. It would be great
if you could try building and using this release to help us fix any
important bugs, crashes, compiler warnings and etc. Your feedback will
allow Gnuastro 0.19 to be smoothly usable for all of us :-).

Here is the compressed source and the GPG detached signature for
this release. To uncompress Lzip tarballs, see [2]. To check the
validity of the tarballs using the GPG detached signature (*.sig)
see [3]:

  https://alpha.gnu.org/gnu/gnuastro/gnuastro-0.18.71-c982.tar.lz
  https://alpha.gnu.org/gnu/gnuastro/gnuastro-0.18.71-c982.tar.lz.sig

Here are the SHA1 and SHA256 checksums (other ways to check if the
tarball you download is what we distributed). Just note that the
SHA256 checksum is base64 encoded, instead of the hexadecimal encoding
that most checksum tools default to (see [4] on how to generate it).
For the list of software used to bootstrap this tarball, see [5].

67df5849a8717da55b73982eaa72082fc9769296  gnuastro-0.18.71-c982.tar.lz
Qw8bU2sZDVR28q8ZaNppFKJUQ0Dgqa106nzUIq25y0c  gnuastro-0.18.71-c982.tar.lz

For their direct contribution to this version's source code, I am very
grateful to Pedram Ashofteh-Ardakani (6 commits), Raul Infante-Sainz
(6), Sepideh Eskandarlou (5), Jash Shah (3), Marjan Akbari (2), Elham
Saremi (1) and Faezeh Bijarchian (1). I am also grateful to (in
alphabetical order) Marjan Akbari, Faezeh Bijarchian, Sepideh
Eskandarlou, Giulia Golini, Raul Infante-Sainz, Teet Kuutma and
Richard Stallman for their good suggestions or reported bugs that have
been implemented in this release.

If any of Gnuastro's programs or libraries are useful in your work,
please cite _and_ acknowledge them. For citation and acknowledgment
guidelines, run the relevant programs with a `--cite' option (it can
be different for different programs, so run it for all the programs
you use). Citations _and_ acknowledgments are vital for the continued
work on Gnuastro, so please don't forget to support us by doing so.

Best wishes,
Mohammad

--
Staff Researcher
Centro de Estudios de Física del Cosmos de Aragón (CEFCA),
Plaza San Juan 1, Planta 2, Teruel 44001, Spain





* Noteworthy changes in release 0.18.71

** New features

  Book:
  - Two new sections added to the "General program usage tutorial" for
    describing how to prepare a FITS image for high quality publication (in
    PDF), and drawing vector graphics marks from a catalog over it (for
    example to show your selected galaxies in the field of view). The use
    the newly added features of ConvertType.

  Arithmetic:
  - Added new type operators and physical constants. All are also available
    in Table's column arithmetic also).
    - e: Base of the natural logarithm (no units).
    - pi: Fraction of Circle cirumference to diameter (no units).
    - c: speed of light in vaccume (in units of m/s).
    - G: Gravitational constant (in units of m^3/kg/s^2).
    - h: Plank's constant (in units of J/Hz).
    - au: Astronomical Units (in units of meters).
    - ly: Light years (in units of meters).
    - avogadro: Avogadro's constant (in units of 1/mol).
    - fine-structure: Fine structure constant (no units).
    - counts-to-sb: convert counts to surface brightness (mag/arcsec^2).
    - sb-to-counts: convert surface brightness (mag/arcsec^2) to counts.
    - mag-to-sb: convert magnitudes to surface brightness over a 
certain area.
    - sb-to-mag: convert surface brightness to magnitudes over a 
certain area.
  - New operators that are specific to Arithmetic:
    - collapse-median: collapse input dataset by calculating the median
      along the given dimension.
    - collapse-sigclip-std: Collapse with sigma-clipped standard deviation.
    - collapse-sigclip-mean: Collapse with sigma-clipped mean.
    - collapse-sigclip-median: Collapse with sigma-clipped median.
    - collapse-sigclip-number: Collapse with number remaining after 
sigma-clip.

  ConvertType:
  - It is now possible to draw vector graphics marks from a catalog over
    the output PDF images. The following options have been added to
    ConvertType for doing this. See the "General program usage tutorial"
    for a fully working example.
    --marks: name of table containing mark information.
    --markshdu: HDU of table if file given to '--marks' is FITS.
    --markcoords: name or number of two columns containing coordinates.
    --mode: if the coordinates are in 'img' (image) or 'wcs' (RA/Dec).
    --markshape: name or number of column containing the shape of each 
mark.
    --markrotate: name or number of column containing rotation of each 
mark.
    --marksize: name or number of column containing the size of the mark.
    --sizeinpix: interpret the values in the size column as pixels.
    --sizeinarcsec: interpret the values in the size column as arc-seconds.
    --sizeinarcmin: interpret the values in the size column as arc-minutes.
    --marklinewidth: name or number of column containing mark's line width.
    --markcolor: name or number of column containing mark's color.
    --listcolors: List all the 140 available colors, and show the colors on
                  24-bit (true color) terminal.
    --marktext: name or number of column containing text under each mark.
    --marktextprecision: number of decimals to print as text when the text
                         column (given to '--marktext') is floating point.
    --markfont: name or number of column containing the font to use for the
                the mark text (given to '--marktext').
    --markfontsize: name or number of column containing the size of the
                    font to use for the mark text (given to '--marktext').
    --showfonts: build a demo PDF with one page per font to show the
                 various available fonts on the system.
    --listfonts: List the names of the available fonts on the terminal.
  - It is now possible to select the color of the border of images produced
    in vector graphics outputs (EPS and PDF) with '--bordercolor'.

  Fits:
  --pixelareaarcsec2: print the image pixel area in units of arcsec^2 to
    standard output. Among other things, this is useful in creating a
    surface brightness image using the new 'counts-to-sb' operator of
    Arithmetic.

  Statistics:
  - Linear and Polynomial least squares fitting are now available and very
    easy to call on the command-line. They are wrappers over the respective
    least squares fitting functions of the GNU Scientific Library. The
    interface is pretty simple, like the example below:
       aststatistics table.fits -cX,Y,Yerr --fit=linear-weighted
    It is also possible to estimate values and errors of the fitted model
    on a new X axis. A complete example has been added to the newly added
    "Least squares fitting" section of the book (under the Statistics
    program documentation). Please see that tutorial to easily use this
    feature. The following new options have been added to the Statistics
    program for this purpose:
    --fit: the model to use. Currently the following models are supported:
           linear
           linear-weighted
           linear-no-constant
           linear-no-constant-weighted
           polynomial
           polynomial-weighted
           polynomial-robust
    --fitweight: nature of the "weight" column (default: standard dev).
    --fitmaxpower: maximum power of X in polynomial models.
    --fitrobust: weight function to use in the "robust" polynomial model.
    --fitestimate: File name, or number to estimate the fit on.
    --fitestimatehdu: HDU containing table in file given to 
'--fitestimate'.
    --fitestimatecol: Column containing X axis values for '--fitestimate'.

  Warp:
  - Can correct distortions (with any standard recognized by WCSLIB) and
    simultaneously align the image to the coordinate system. When no named
    linear operation (like '--rotate', '--scale' or etc) is requested, Warp
    will go into this mode. It is highly customizable through the following
    options. See the "Invoking Warp" section of the book for more. This
    feature has been written by Pedram Ashofteh Ardakani.
    --center: RA, DEC of the center of the central pixel of output.
    --width: Width of output in degrees or pixels (see '--widthinpix').
    --widthinpix: interpret values of '--width' as pixels.
    --cdelt: Pixel scale of output ('CDELTi' keywords in FITS).
    --ctype: Coordinates and projection algorithm. Default: RA/Dec and
      Gnomonic or 'TAN').
    --edgesampling: extra sampling of pixel polygon to account for strong
      non-linear projection or distortion effects, when necessary.
    --gridfile: warp the input to the exact WCS and pixel grid of the file
      given to this option. This is very useful when matching images from
      differetn surveys. Using this option, you can also insert distortions
      in an image (for example on a mock image, to make it match an
      observed exposure with dithering+distortion).
    --gridhdu: HDU containing image to be matched in '--gridfile'.
  - List of WCS projections available in WCSLIB 7.12 (and therefore in
    Gnuastro's Warp):
       AZP: Zenithal/azimuthal perspective.
       SZP: Slant zenithal perspective.
       TAN: Gnomonic (tangential).
       STG: Stereographic.
       SIN: Orthographic/synthesis.
       ARC: Zenithal/azimuthal equidistant.
       ZPN: Zenithal/azimuthal polynomial.
       ZEA: Zenithal/azimuthal equal area.
       AIR: Airy.
       CYP: Cylindrical perspective.
       CEA: Cylindrical equal area.
       CAR: Plate carree.
       MER: Mercator.
       SFL: Sanson-Flamsteed.
       PAR: Parabolic.
       MOL: Mollweide.
       AIT: Hammer-Aitoff.
       COP: Conic perspective.
       COE: Conic equal area.
       COD: Conic equidistant.
       COO: Conic orthomorphic.
       BON: Bonne.
       PCO: Polyconic.
       TSC: Tangential spherical cube.
       CSC: COBE spherical cube.
       QSC: Quadrilateralized spherical cube.
       HPX: HEALPix.
       XPH: HEALPix polar, aka "butterfly".
  - List of WCS distortions available in WCSLIB 7.12 (and therefore in
    Gnuastro's Warp):
       TPD: Template Polynomial Distortion.
       SIP: Simple Imaging Polynomial.
       TPV: Polynomial distortion for Gnomonic (TAN) projection.
       DSS: Digitized Sky Survey.
       WAT: Chebyshev or Legendre polynomials for TNX and ZPX projections.

  astscript-fits-view:
  --ds9colorbarmulti: show a separate color-bar for each image in DS9. By
    default this script will show a single color-bar for all the images to
    help save space on the monitor when there are many images.

  astscript-psf-stamp:
  - sub-pixel warping is applied to ensure that your coordinate is at the
    center of the central pixel of the output image. This results in a
    _major_ improvement when estimating the center of the PSF.
  --nocentering: disable sub-pixel warping when creating the PSF stamp. As
    described above, the sub-pixel warping is critical for the central part
    of the PSF, but for the outer parts it is statistically negligible. So
    to avoid slowing down you pipeline, you can disable sub-pixel warping
    with this option.
  --snthresh: if given, the value to this option is assumed to be a
    signal-to-noise ratio (S/N) threshold and all pixels below that S/N
    will be masked. This is useful because we are often forced to stack
    stars of differing magnitudes. While the fainter ones are good for the
    inner parts of the star, they degrade the stack's outer parts. With
    this option, the fainter stars won't harm the outer parts.

  GNU Make extensions (in a Makefile)
    It is now possible to use custom Gnuastro functions in GNU Make, using
    its extension facilities with Dynamic libraries. GNU Make is a very
    powerful workflow manager that is also used for data analysis (not just
    for compilation). With the Gnaustro Make functions, (astronomical) data
    analysis becomes even more easier and faster. In the following, you can
    see the first set of such functions (they all begin with 'ast-'). For
    more, see the newly added chapter in the Gnuastro manual.
    - ast-version-is: will return '1' if the running Gnuastro has the given
      version (argument of this function). This can be used to ensure
      reproducibility in combination with Make's conditional features, see
      the minimal working example in the manual.
    - ast-text-contains: will return space-separated words within a larger
      list that contain a certain string. The to-contain string can be
      anywhere within the words of the larger list.
    - ast-text-not-contains: will return space-separated words within a
      larger list that DO NOT contain a certain string. The to-not-contain
      string can be anywhere within the words of the larger list.
    - ast-fits-with-keyvalue: takes a keyword name, a list of keyword
      values, a HDU and a list of FITS files. It will return only those
      FITS files that have the requested value(s) in the requested keyword
      of the requested HDU.
    - ast-fits-unique-keyvalues: takes a keyword name, a HDU and a list of
      FITS files. It will return all the unique values given to that
      keyword within the FITS files.

  Library:
  - GAL_CONFIG_HAVE_PYTHON: non-zero if Python3+Numpy features included.
  - GAL_CONFIG_HAVE_GNUMAKE_H: non-zero if GNU Make extensions can be made.
  - gal_box_border_rotate_around_center: width of box after rotation.
  - gal_color_id_to_name: return the name of a color from its ID.
  - gal_color_in_rgb: return the fraction of red-green-blue in a color.
  - gal_color_name_to_id: return the ID of a color from its name.
  - gal_dimension_collapse_median: collapse input along dim. using median.
  - gal_dimension_collapse_sigclip_mean: collapse with sig-clipped mean.
  - gal_dimension_collapse_sigclip_std: collapse with sig-clipped STD.
  - gal_dimension_collapse_sigclip_median: collapse with sig-clipped 
median.
  - gal_dimension_collapse_sigclip_number: collapse with num. after 
sig-clip.
  - gal_eps_shape_id_to_name: return the name of a shape from its ID.
  - gal_eps_shape_name_to_id: return the ID of a shape from its name.
  - gal_fit_name_to_id: Convert string name to ID of fitting model.
  - gal_fit_name_from_id: Convert ID of fitting model to string name.
  - gal_fit_name_robust_to_id: Convert name of robust weights to ID.
  - gal_fit_name_robust_from_id: Convert ID of robust weights to name.
  - gal_fit_1d_linear: linear fit of input columns.
  - gal_fit_1d_linear_no_constant: linear fit with no constant.
  - gal_fit_1d_linear_estimate: estimate a linear fit on a new X column.
  - gal_fit_1d_polynomial: polynomial fit of input columns.
  - gal_fit_1d_polynomial_robust: robust polynomial fit of input columns.
  - gal_fit_1d_polynomial_estimate: estimate a polynomial fit on new X 
column.
  - gal_fits_unique_keyvalues: extract all unique values to a certain
    keyword in many files.
  - gal_fits_with_keyvalue: select FITS image with a certain key value.
  - gal_list_data_select_by_name: select a dataset from a list by its name.
  - gal_list_str_cat: Concatenate (append) list to a space-separated 
string.
  - gal_list_str_extract: Extract space-separated tokens to a list.
  - gal_python_type_from_numpy: Convert Numpy's type id. to Gnuastro's.
  - gal_python_type_to_numpy: Convert Gnuastro's type id. to Numpy's.
  - gal_txt_contains_string: Check a certain string within in a larger one.
  - gal_units_counts_to_sb: SB from counts, zeropoint and area.
  - gal_units_mag_to_sb: surface brightness (SB) from magnitude and area.
  - gal_units_sb_to_counts: counts from SB, zeropoint and area.
  - gal_units_sb_to_mag: magnitude from SB and area.
  - gal_warp_wcsalign_init: initialize the WCS aligning structure.
  - gal_warp_wcsalign_onpix: Per-pixel filling of output.
  - gal_warp_wcsalign_onthread: function to give to pthreads.
  - gal_warp_wcsalign: high-level function to align input by its WCS.
  - gal_warp_wcsalign_free: free the contents of the WCS aligning 
structure.
  - gal_wcs_free: free a WCS structure that is created or read by Gnuastro.

** Removed features

  Statistics:
  --refcol has been removed because it breaks the modularity principle
    (given that it is the job of Gnuastro's Table program to limit rows
    from a larger table based on many different criteria). The output of
    Table can be directly piped to Statistics to achieve the same (and much
    more feature-rich effect).

  Warp:
  --align: has been removed. This is because aligning an image (while
    correcting for any possible distortion) is now the default behavior of
    Warp (when no linear operations have been requested).

** Changed features

  Book:
  - The "General program usage tutorial" section is now the first section
    of the Tutorial chapter, since it introduces the tools at a more basic
    level. The "Sufi simulates a detection" (which was previously first)
    has been moved to the fourth section.

  Warp:
  - The short format of the '--centeroncorner' option has been removed. The
    '-c' is now the short format for the new '--center' option to Warp.

  astscript-psf-scale-factor:
  --widthinpix: new name for the old '--stampwidth' option. This was done
    to have the same name to a similar option in Crop and help in
    remembering.

  astscript-psf-stamp:
  --widthinpix: new name for the old '--stampwidth' option. This was done
    to have the same name to a similar option in Crop and help in
    remembering.

  Library
  - gal_eps_write: two new arguments have been added to draw marks, and to
        set the border color.
  - gal_pdf_write: similar to 'gal_eps_write'.
  - gal_fits_hdu_open: new argument to optionally exit program if HDU
        couldn't be opened.

** Bugs fixed
  bug #62861: '--printkeynames' of Fits program gets caught in an infinite
              loop on FITS files that have empty keywords before
              'END'. Found by Pedram Ashofteh Ardakani.
  bug #62892: Installed scripts don't account for differing LANG and
              LC_NUMERIC. Found by Teet Kuutma and fixed by Raul
              Infante-Sainz.
  bug #62937: psf-scale-factor not being quiet, when requested. Found and
              fixed by Sepideh Eskandarlou.
  bug #62943: Couldn't read the value of width with '--snthresh' is called.
              Found and fixed by Sepideh Eskandarlou.
  bug #62944: No warning when the option value isn't immediately after the
              equal sign in long format. Found by Faezeh Bijarchian.
  bug #63013: Sigma clip segmentation fault when input has an integer type
              with values close to saturation-level.
  bug #63022: psf-scale-factor not saving the result in the output file,
              found and fixed by Raul Infante-Sainz.





[2] Lzip has better compression ratio and archival features compared
to the `.gz' or `.xz' formats. Therefore Gnuastro's alpha/test
releases are only in this format, but for historical reasons we also
include `.gz' tarballs in the official releases. If you don't have
Lzip (you can check with `lzip --version' command), download and
install it from its webpage:

  https://www.nongnu.org/lzip/lzip.html

If Lzip is present and you use GNU Tar, then the single command below
should uncompress and un-pack the tarball:

  $ tar xf gnuastro-0.18.71-c982.tar.lz

If the command above doesn't work, you have to un-compress and un-pack
it with two separate commands (or use a pipe to feed the output of the
first into the second: `lzip -cd gnuastro-0.18.71-c982.tar.lz | tar -xf -'):

  $ lzip -d gnuastro-0.18.71-c982.tar.lz
  $ tar xf gnuastro-0.18.71-c982.tar





[3] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify gnuastro-0.18.71-c982.tar.lz.sig

The signature should match the fingerprint of the following key:

  pub   rsa4096 2018-12-08 [SC]
        52B0 4484 D806 C90D CB52  7249 71E8 9901 2D17 4B66
  uid           [ unknown] Mohammad Akhlaghi <mohammad@akhlaghi.org>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --recv-keys 71E899012D174B66

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify gnuastro-0.18.71-c982.tar.lz.sig





[4] To get the base64 SHA256 checksum, you can use the command below
(assuming you already have the 'openssl' command-line program). Just
note that an extra '=' is printed as the last character that you
should ignore.

  cat gnuastro-0.18.71-c982.tar.lz \
      | openssl dgst -binary -sha256 | openssl base64 -A





[5] This tarball was bootstrapped (created) with the tools below. Note
that you don't need these to build Gnuastro from the tarball, these
are the tools that were used to make the tarball itself. They are only
mentioned here to be able to reproduce/recreate this tarball later.

  Texinfo 6.8
  Autoconf 2.71
  Automake 1.16.4
  Help2man 1.49.2
  ImageMagick 7.1.0-50
  Gnulib v0.1-5527-gcd1fdabe8b
  Autoconf archives v2022.09.03-4-gda89908