[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[gnuastro-commits] master 1f9d941 090/113: Imported recent work in maste
|
From: |
Mohammad Akhlaghi |
|
Subject: |
[gnuastro-commits] master 1f9d941 090/113: Imported recent work in master, conflicts fixed |
|
Date: |
Fri, 16 Apr 2021 10:33:56 -0400 (EDT) |
branch: master
commit 1f9d9417cac9c4d8e0488ec29d5951670a0c01b0
Merge: 22d0fba 596b88f
Author: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Commit: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Imported recent work in master, conflicts fixed
There weren't any conflicts, but there was building crash that happened due
to the new format of `gal_table_read' in MakeProfile's reading of 3D input
catalogs. It is fixed with this merge.
---
NEWS | 23 +-
bin/arithmetic/arithmetic.c | 2 +-
bin/arithmetic/operands.c | 2 +-
bin/arithmetic/ui.c | 1 +
bin/buildprog/ui.c | 1 +
bin/convertt/ui.c | 38 +-
bin/convolve/ui.c | 5 +-
bin/cosmiccal/ui.c | 1 +
bin/crop/ui.c | 6 +-
bin/fits/ui.c | 1 +
bin/gnuastro.conf | 1 +
bin/match/main.h | 1 +
bin/match/match.c | 10 +-
bin/match/ui.c | 68 +-
bin/mkcatalog/ui.c | 13 +-
bin/mknoise/ui.c | 3 +-
bin/mkprof/ui.c | 30 +-
bin/noisechisel/ui.c | 5 +-
bin/segment/ui.c | 11 +-
bin/statistics/ui.c | 102 +-
bin/table/ui.c | 17 +-
bin/warp/ui.c | 3 +-
doc/gnuastro.texi | 2682 ++++++++++++++++++++----------------
lib/array.c | 22 +-
lib/gnuastro-internal/commonopts.h | 15 +-
lib/gnuastro-internal/options.h | 10 +-
lib/gnuastro/array.h | 12 +-
lib/gnuastro/table.h | 10 +-
lib/gnuastro/txt.h | 17 +-
lib/options.c | 44 +
lib/statistics.c | 25 +-
lib/table.c | 19 +-
lib/txt.c | 364 +++--
tests/Makefile.am | 6 +-
tests/statistics/from-stdin.sh | 56 +
35 files changed, 2159 insertions(+), 1467 deletions(-)
diff --git a/NEWS b/NEWS
index e84f45c..1d59fe6 100644
--- a/NEWS
+++ b/NEWS
@@ -8,6 +8,9 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
- Input files and parameters written as keywords in the first extension
of output when it is FITS. This is only relevant for some programs,
(for example not the Fits or Table programs).
+ - Standard input (for example from pipes) is now available to feed input
+ to all programs that accept plain text input (ConvertType, Match,
+ MakeProfiles, Statistics, Table).
Arithmetic:
--onedasimage: write output as an image if it has one dimension, not table.
@@ -51,7 +54,14 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
across the image.
Statistics:
- - Sky estimation: new outlier estimation algorithm similar to NoiseChisel.
+ - If an input table has only one column, Statistics won't complain and
+ abort when no `--column' (`-c') is given: there is only one column to
+ use anyway, so it will be used. In the absence of which column to
+ use, it will still complain and abort if the input has more than one
+ column.
+ - Sky estimation: new outlier estimation algorithm similar to
+ NoiseChisel.
+ - Input can be given using the standard input (for example a pipe).
Library:
- gal_blank_flag_apply: Set all flagged/masked elements to blank.
@@ -61,6 +71,7 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
- gal_fits_key_write_version_in_ptr: old gal_fits_key_write_version.
- gal_fits_key_write_config: write key list and version as config header.
- gal_statistics_outlier_positive: Find the first positive outlier.
+ - gal_txt_stdin_read: Read lines in standard input into a list of strings.
** Removed features
@@ -99,6 +110,16 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
- gal_fits_key_write: filename and hdu instead of FITS pointer.
- gal_fits_key_write_version: filename and hdu instead of FITS pointer.
- gal_fits_key_write_filename: write at the top or end of the input list.
+ - gal_array_read: list of strings (from standard input) acceptable.
+ - gal_array_read_to_type: list of strings (from stin) acceptable.
+ - gal_array_read_one_ch: list of strings (from stdin) acceptable.
+ - gal_array_read_one_ch_to_type: list of strings (from stdin) acceptable.
+ - gal_table_info: list of strings (from stdin) acceptable.
+ - gal_table_read: list of strings (from stdin) acceptable.
+ - gal_txt_table_info: list of strings (from stdin) acceptable.
+ - gal_txt_image_info: list of strings (from stdin) acceptable.
+ - gal_txt_table_read: list of strings (from stdin) acceptable.
+ - gal_txt_image_read: list of strings (from stdin) acceptable.
** Bugs fixed
bug #54493: Warp crashes when type isn't set.
diff --git a/bin/arithmetic/arithmetic.c b/bin/arithmetic/arithmetic.c
index 8afcb7b..d747e85 100644
--- a/bin/arithmetic/arithmetic.c
+++ b/bin/arithmetic/arithmetic.c
@@ -1139,7 +1139,7 @@ reversepolish(struct arithmeticparams *p)
filename=p->operands->filename;
if( gal_fits_name_is_fits(filename) )
{
- p->operands->data=gal_array_read_one_ch(filename, hdu,
+ p->operands->data=gal_array_read_one_ch(filename, hdu, NULL,
p->cp.minmapsize);
p->refdata.wcs=gal_wcs_read(filename, hdu, 0, 0, &p->refdata.nwcs);
if(!p->cp.quiet) printf(" - %s (hdu %s) is read.\n", filename, hdu);
diff --git a/bin/arithmetic/operands.c b/bin/arithmetic/operands.c
index 400a25d..2a00a29 100644
--- a/bin/arithmetic/operands.c
+++ b/bin/arithmetic/operands.c
@@ -335,7 +335,7 @@ operands_pop(struct arithmeticparams *p, char *operator)
filename=operands->filename;
/* Read the dataset. */
- data=gal_array_read_one_ch(filename, hdu, p->cp.minmapsize);
+ data=gal_array_read_one_ch(filename, hdu, NULL, p->cp.minmapsize);
/* Arithmetic changes the contents of a dataset, so the existing name
(in the FITS `EXTNAME' keyword) should not be passed on beyond
diff --git a/bin/arithmetic/ui.c b/bin/arithmetic/ui.c
index 9ca9e6e..61075cc 100644
--- a/bin/arithmetic/ui.c
+++ b/bin/arithmetic/ui.c
@@ -140,6 +140,7 @@ ui_initialize_options(struct arithmeticparams *p,
case GAL_OPTIONS_KEY_TYPE:
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
diff --git a/bin/buildprog/ui.c b/bin/buildprog/ui.c
index 46aef6e..8667aa5 100644
--- a/bin/buildprog/ui.c
+++ b/bin/buildprog/ui.c
@@ -125,6 +125,7 @@ ui_initialize_options(struct buildprogparams *p,
case GAL_OPTIONS_KEY_NUMTHREADS:
case GAL_OPTIONS_KEY_MINMAPSIZE:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
cp->coptions[i].mandatory=GAL_OPTIONS_NOT_MANDATORY;
break;
diff --git a/bin/convertt/ui.c b/bin/convertt/ui.c
index 0df8390..47b2194 100644
--- a/bin/convertt/ui.c
+++ b/bin/convertt/ui.c
@@ -271,14 +271,11 @@ ui_read_check_only_options(struct converttparams *p)
static void
ui_check_options_and_arguments(struct converttparams *p)
{
- /* Check if there was any inputs. */
- if(p->inputnames==NULL)
- error(EXIT_FAILURE, 0, "no input files given");
-
- /* Reverse the `inputnames' linked list, note that the `hdu' linked list
- was reversed during option parsing.*/
+ /* Reverse the `inputnames' linked list if it was given (recall that we
+ also accept input from the standard input). Note that the `hdu' linked
+ list was reversed during option parsing, so we don't need to do it
+ here any more. */
gal_list_str_reverse(&p->inputnames);
-
}
@@ -396,11 +393,27 @@ ui_make_channels_ll(struct converttparams *p)
{
char *hdu=NULL;
gal_data_t *data;
- gal_list_str_t *name;
size_t dsize=0, dirnum;
+ gal_list_str_t *name, *lines;
- /* Go through the input files and add the channel(s). */
+ /* Initialize the counting of channels. */
p->numch=0;
+
+ /* If any standard input is provided, we want to process that first. Note
+ that since other input arguments are also allowed (as other channels),
+ we'll need to process the standard input independently first, then go
+ onto the possible list of other files.*/
+ lines=gal_txt_stdin_read(p->cp.stdintimeout);
+ if(lines)
+ {
+ data=gal_txt_image_read(NULL, lines, p->cp.minmapsize);
+ gal_list_data_add(&p->chll, data);
+ gal_list_str_free(lines, 1);
+ ++p->numch;
+ }
+
+
+ /* Go through the input files and add the channel(s). */
for(name=p->inputnames; name!=NULL; name=name->next)
{
/* Check if p->numch has not exceeded 4. */
@@ -492,12 +505,15 @@ ui_make_channels_ll(struct converttparams *p)
/* Text: */
else
{
- data=gal_txt_image_read(name->v, p->cp.minmapsize);
+ data=gal_txt_image_read(name->v, NULL, p->cp.minmapsize);
gal_list_data_add(&p->chll, data);
++p->numch;
}
}
+ /* If there weren't any channels, abort with an error. */
+ if(p->numch==0)
+ error(EXIT_FAILURE, 0, gal_options_stdin_error(p->cp.stdintimeout, 0));
/* Reverse the list of channels into the input order. */
gal_list_data_reverse(&p->chll);
@@ -616,7 +632,7 @@ void
ui_add_dot_use_automatic_output(struct converttparams *p)
{
gal_list_str_t *stll;
- char *tmp, *firstname="output.txt", *suffix=p->cp.output;
+ char *tmp, *firstname="converttype.txt", *suffix=p->cp.output;
/* Find the first non-blank file name in the input(s). */
for(stll=p->inputnames; stll!=NULL; stll=stll->next)
diff --git a/bin/convolve/ui.c b/bin/convolve/ui.c
index 356de22..b67aec2 100644
--- a/bin/convolve/ui.c
+++ b/bin/convolve/ui.c
@@ -132,6 +132,7 @@ ui_initialize_options(struct convolveparams *p,
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
case GAL_OPTIONS_KEY_INTERPNUMNGB:
case GAL_OPTIONS_KEY_INTERPONLYBLANK:
cp->coptions[i].flags=OPTION_HIDDEN;
@@ -296,7 +297,7 @@ ui_read_kernel(struct convolveparams *p)
/* Read the image into file. */
p->kernel = gal_array_read_one_ch_to_type(p->kernelname, p->khdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Convert all the NaN pixels to zero if the kernel contains blank
@@ -343,7 +344,7 @@ ui_preparations(struct convolveparams *p)
/* Read the input image as a float64 array. */
- p->input=gal_array_read_one_ch_to_type(p->filename, cp->hdu,
+ p->input=gal_array_read_one_ch_to_type(p->filename, cp->hdu, NULL,
GAL_TYPE_FLOAT32, cp->minmapsize);
p->input->wcs=gal_wcs_read(p->filename, cp->hdu, 0, 0, &p->input->nwcs);
diff --git a/bin/cosmiccal/ui.c b/bin/cosmiccal/ui.c
index f792d5d..6b60557 100644
--- a/bin/cosmiccal/ui.c
+++ b/bin/cosmiccal/ui.c
@@ -138,6 +138,7 @@ ui_initialize_options(struct cosmiccalparams *p,
case GAL_OPTIONS_KEY_MINMAPSIZE:
case GAL_OPTIONS_KEY_IGNORECASE:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
}
diff --git a/bin/crop/ui.c b/bin/crop/ui.c
index 8b69c17..de00c22 100644
--- a/bin/crop/ui.c
+++ b/bin/crop/ui.c
@@ -145,6 +145,10 @@ ui_initialize_options(struct cropparams *p,
case GAL_OPTIONS_KEY_IGNORECASE:
cp->coptions[i].group=UI_GROUP_CENTER_CATALOG;
break;
+
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
+ cp->coptions[i].group=OPTION_HIDDEN;
+ break;
}
/* Select by group. */
@@ -671,7 +675,7 @@ ui_read_cols(struct cropparams *p)
/* Read the desired columns from the file. */
- cols=gal_table_read(p->catname, p->cathdu, colstrs, p->cp.searchin,
+ cols=gal_table_read(p->catname, p->cathdu, NULL, colstrs, p->cp.searchin,
p->cp.ignorecase, p->cp.minmapsize, NULL);
if(cols==NULL)
error(EXIT_FAILURE, 0, "%s: is empty! No usable information "
diff --git a/bin/fits/ui.c b/bin/fits/ui.c
index 6482d17..7b5ef33 100644
--- a/bin/fits/ui.c
+++ b/bin/fits/ui.c
@@ -123,6 +123,7 @@ ui_initialize_options(struct fitsparams *p,
case GAL_OPTIONS_KEY_LOG:
case GAL_OPTIONS_KEY_MINMAPSIZE:
case GAL_OPTIONS_KEY_NUMTHREADS:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
diff --git a/bin/gnuastro.conf b/bin/gnuastro.conf
index c9a6d42..3652e6a 100644
--- a/bin/gnuastro.conf
+++ b/bin/gnuastro.conf
@@ -21,6 +21,7 @@
hdu 1
ignorecase 1
searchin name
+ stdintimeout 100000
# Tessellation
tilesize 30,30
diff --git a/bin/match/main.h b/bin/match/main.h
index 87664a1..49ddf93 100644
--- a/bin/match/main.h
+++ b/bin/match/main.h
@@ -69,6 +69,7 @@ struct matchparams
char *logname; /* Name of log file. */
char *out1name; /* Name of first matched output. */
char *out2name; /* Name of second matched output. */
+ gal_list_str_t *stdinlines; /* Lines given by Standard input. */
/* Output: */
time_t rawtime; /* Starting time of the program. */
diff --git a/bin/match/match.c b/bin/match/match.c
index 011de6c..74d3ef4 100644
--- a/bin/match/match.c
+++ b/bin/match/match.c
@@ -66,8 +66,9 @@ match_catalog_read_write_all(struct matchparams *p, size_t
*permutation,
__func__, "numcolmatch");
/* Read the full table. */
- cat=gal_table_read(filename, hdu, cols, p->cp.searchin, p->cp.ignorecase,
- p->cp.minmapsize, *numcolmatch);
+ cat=gal_table_read(filename, hdu, filename ? NULL : p->stdinlines, cols,
+ p->cp.searchin, p->cp.ignorecase, p->cp.minmapsize,
+ *numcolmatch);
origsize=cat->size;
@@ -316,7 +317,10 @@ match(struct matchparams *p)
the output. */
if(gal_fits_name_is_fits(p->out1name))
{
- gal_fits_key_write_filename("input1", p->input1name, &p->cp.okeys, 1);
+ gal_fits_key_write_filename("input1", ( p->input1name
+ ? p->input1name
+ : "Standard input" ),
+ &p->cp.okeys, 1);
gal_fits_key_write_filename("input2", p->input2name, &p->cp.okeys, 1);
gal_fits_key_write_config(&p->cp.okeys, "Match configuration",
"MATCH-CONFIG", p->out1name, "0");
diff --git a/bin/match/ui.c b/bin/match/ui.c
index e7461ba..74de816 100644
--- a/bin/match/ui.c
+++ b/bin/match/ui.c
@@ -119,6 +119,7 @@ ui_initialize_options(struct matchparams *p,
break;
case GAL_OPTIONS_KEY_TYPE:
case GAL_OPTIONS_KEY_NUMTHREADS:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
}
@@ -169,7 +170,7 @@ parse_opt(int key, char *arg, struct argp_state *state)
{
if(p->input2name)
argp_error(state, "only two arguments (input files) should be "
- "given");
+ "given, not any more");
else
p->input2name=arg;
}
@@ -227,21 +228,34 @@ ui_read_check_only_options(struct matchparams *p)
static void
ui_check_options_and_arguments(struct matchparams *p)
{
- /* Make sure two input file names were given and if they a FITS file,
- that a HDU is also given for each. */
+ /* The first input might come from the standard input, in that case, the
+ second input's name is stored in `input1name' and `input2name' will be
+ left to NULL. So we'll first correct for this. */
if(p->input1name)
{
- if( gal_fits_name_is_fits(p->input1name) && p->cp.hdu==NULL )
- error(EXIT_FAILURE, 0, "no HDU for first input. When the input is "
- "a FITS file, a HDU must also be specified, you can use the "
- "`--hdu' (`-h') option and give it the HDU number (starting "
- "from zero), extension name, or anything acceptable by "
- "CFITSIO");
+ if(p->input2name==NULL)
+ {
+ p->input2name=p->input1name;
+ p->input1name=NULL;
+ }
}
else
- error(EXIT_FAILURE, 0, "no input file is specified: two inputs are "
- "necessary");
-
+ error(EXIT_FAILURE, 0, "no inputs! two inputs are necessary. The first "
+ "can be from the standard input (e.g., a pipe)");
+
+
+ /* If the first input is a FITS file, make sure its HDU is also given. */
+ if( p->input1name
+ && gal_fits_name_is_fits(p->input1name)
+ && p->cp.hdu==NULL )
+ error(EXIT_FAILURE, 0, "no HDU for first input. When the input is "
+ "a FITS file, a HDU must also be specified, you can use the "
+ "`--hdu' (`-h') option and give it the HDU number (starting "
+ "from zero), extension name, or anything acceptable by "
+ "CFITSIO");
+
+ /* If the second input is a FITS file, make sure its HDU is also
+ given. */
if(p->input2name)
{
if( gal_fits_name_is_fits(p->input2name) && p->hdu2==NULL )
@@ -282,7 +296,7 @@ ui_set_mode(struct matchparams *p)
{
/* Check if we are in image or catalog mode. We will base the mode on the
first input, then check with the second. */
- if( gal_fits_name_is_fits(p->input1name) )
+ if( p->input1name && gal_fits_name_is_fits(p->input1name) )
p->mode = ( (gal_fits_hdu_format(p->input1name, p->cp.hdu) == IMAGE_HDU)
? MATCH_MODE_WCS
: MATCH_MODE_CATALOG );
@@ -533,9 +547,16 @@ ui_read_columns_to_double(struct matchparams *p, char
*filename, char *hdu,
"Please give more specific values to `--ccol1' (column "
"numberes are the only identifiers guaranteed to be unique).";
- /* Read the columns. */
- tout=gal_table_read(filename, hdu, cols, cp->searchin, cp->ignorecase,
- cp->minmapsize, NULL);
+ /* Read the columns. Note that the first input's name can be NULL (it the
+ user intended to use the standrad input). Also note that this function
+ is called more than one time, so if the Standard input is already read
+ once, we don't want to write a blank list over it (the Standard input
+ will be empty after being read). */
+ if(p->stdinlines==NULL)
+ p->stdinlines=gal_options_check_stdin(filename, p->cp.stdintimeout);
+ tout=gal_table_read(filename, hdu, filename ? NULL : p->stdinlines,
+ cols, cp->searchin, cp->ignorecase, cp->minmapsize,
+ NULL);
/* A small sanity check. */
if(gal_list_data_number(tout)!=numcols)
@@ -689,6 +710,7 @@ ui_preparations_out_name(struct matchparams *p)
{
/* To temporarily keep the original value. */
uint8_t keepinputdir_orig;
+ char *refname = p->input1name ? p->input1name : p->input2name;
/* Set the output file(s) name(s). */
if(p->logasoutput)
@@ -698,10 +720,10 @@ ui_preparations_out_name(struct matchparams *p)
else
{
if(p->cp.tableformat==GAL_TABLE_FORMAT_TXT)
- p->logname=gal_checkset_automatic_output(&p->cp, p->input1name,
+ p->logname=gal_checkset_automatic_output(&p->cp, refname,
"_matched.txt");
else
- p->logname=gal_checkset_automatic_output(&p->cp, p->input1name,
+ p->logname=gal_checkset_automatic_output(&p->cp, refname,
"_matched.fits");
}
@@ -721,7 +743,7 @@ ui_preparations_out_name(struct matchparams *p)
gal_checkset_allocate_copy(p->cp.output, &p->out1name);
else
p->out1name = gal_checkset_automatic_output(&p->cp,
- p->input1name, ( p->cp.tableformat==GAL_TABLE_FORMAT_TXT
+ refname, ( p->cp.tableformat==GAL_TABLE_FORMAT_TXT
? "_matched.txt" : "_matched.fits") );
gal_checkset_writable_remove(p->out1name, 0, p->cp.dontdelete);
}
@@ -758,8 +780,7 @@ ui_preparations_out_name(struct matchparams *p)
{
if(p->cp.tableformat==GAL_TABLE_FORMAT_TXT)
{
- p->out1name=gal_checkset_automatic_output(&p->cp,
- p->input1name,
+ p->out1name=gal_checkset_automatic_output(&p->cp, refname,
"_matched_1.txt");
p->out2name=gal_checkset_automatic_output(&p->cp,
p->input2name,
@@ -767,8 +788,7 @@ ui_preparations_out_name(struct matchparams *p)
}
else
{
- p->out1name=gal_checkset_automatic_output(&p->cp,
- p->input1name,
+ p->out1name=gal_checkset_automatic_output(&p->cp, refname,
"_matched.fits");
gal_checkset_allocate_copy(p->out1name, &p->out2name);
}
@@ -891,7 +911,6 @@ ui_read_check_inputs_setup(int argc, char *argv[], struct
matchparams *p)
keywords to write in output later. */
if(gal_fits_name_is_fits(p->out1name))
gal_options_as_fits_keywords(&p->cp);
-
}
@@ -926,6 +945,7 @@ ui_free_report(struct matchparams *p, struct timeval *t1)
free(p->cp.output);
gal_data_free(p->ccol1);
gal_data_free(p->ccol2);
+ gal_list_str_free(p->stdinlines, 1);
/* Print the final message.
if(!p->cp.quiet)
diff --git a/bin/mkcatalog/ui.c b/bin/mkcatalog/ui.c
index e038270..f9fdcad 100644
--- a/bin/mkcatalog/ui.c
+++ b/bin/mkcatalog/ui.c
@@ -142,6 +142,7 @@ ui_initialize_options(struct mkcatalogparams *p,
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
case GAL_OPTIONS_KEY_WORKOVERCH:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
case GAL_OPTIONS_KEY_INTERPNUMNGB:
case GAL_OPTIONS_KEY_INTERPONLYBLANK:
cp->coptions[i].flags=OPTION_HIDDEN;
@@ -540,7 +541,7 @@ ui_read_labels(struct mkcatalogparams *p)
gal_data_t *tmp, *keys=gal_data_array_calloc(2);
/* Read it into memory. */
- p->objects = gal_array_read_one_ch(p->objectsfile, p->cp.hdu,
+ p->objects = gal_array_read_one_ch(p->objectsfile, p->cp.hdu, NULL,
p->cp.minmapsize);
@@ -609,7 +610,7 @@ ui_read_labels(struct mkcatalogparams *p)
/* Read the clumps image. */
p->clumps = gal_array_read_one_ch(p->usedclumpsfile, p->clumpshdu,
- p->cp.minmapsize);
+ NULL, p->cp.minmapsize);
/* Check its size. */
if( gal_dimension_is_different(p->objects, p->clumps) )
@@ -927,7 +928,7 @@ ui_preparations_read_inputs(struct mkcatalogparams *p)
/* Read the values dataset. */
p->values=gal_array_read_one_ch_to_type(p->usedvaluesfile, p->valueshdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Make sure it has the correct size. */
@@ -974,7 +975,7 @@ ui_preparations_read_inputs(struct mkcatalogparams *p)
/* Read the Sky dataset. */
p->sky=gal_array_read_one_ch_to_type(p->usedskyfile, p->skyhdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Check its size and prepare tile structure. */
@@ -1003,7 +1004,7 @@ ui_preparations_read_inputs(struct mkcatalogparams *p)
/* Read the Sky standard deviation image into memory. */
p->std=gal_array_read_one_ch_to_type(p->usedstdfile, p->stdhdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Check its size and prepare tile structure. */
@@ -1037,7 +1038,7 @@ ui_preparations_read_inputs(struct mkcatalogparams *p)
/* Read the mask image. */
p->upmask = gal_array_read_one_ch(p->upmaskfile, p->upmaskhdu,
- p->cp.minmapsize);
+ NULL, p->cp.minmapsize);
/* Check its size. */
if( gal_dimension_is_different(p->objects, p->upmask) )
diff --git a/bin/mknoise/ui.c b/bin/mknoise/ui.c
index 471328d..5221db0 100644
--- a/bin/mknoise/ui.c
+++ b/bin/mknoise/ui.c
@@ -131,6 +131,7 @@ ui_initialize_options(struct mknoiseparams *p,
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
}
@@ -282,7 +283,7 @@ void
ui_preparations(struct mknoiseparams *p)
{
/* Read the input image as a double type */
- p->input=gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu,
+ p->input=gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu, NULL,
GAL_TYPE_FLOAT64, p->cp.minmapsize);
p->input->wcs=gal_wcs_read(p->inputname, p->cp.hdu, 0, 0, &p->input->nwcs);
diff --git a/bin/mkprof/ui.c b/bin/mkprof/ui.c
index 5475d63..2ffaab6 100644
--- a/bin/mkprof/ui.c
+++ b/bin/mkprof/ui.c
@@ -562,10 +562,6 @@ ui_check_options_and_arguments(struct mkprofparams *p)
error(EXIT_FAILURE, 0, "no `hdu' specified for the input FITS "
"table '%s', to ", p->catname);
}
- else
- error(EXIT_FAILURE, 0, "no input catalog provided. To build "
- "profiles, you need to give a catalog/table containing "
- "the information of the profiles");
}
@@ -587,7 +583,10 @@ ui_check_options_and_arguments(struct mkprofparams *p)
{
gal_checkset_allocate_copy(p->cp.output, &p->outdir);
gal_checkset_check_dir_write_add_slash(&p->outdir);
- tmpname=gal_checkset_automatic_output(&p->cp, p->catname, ".fits");
+ tmpname=gal_checkset_automatic_output(&p->cp,
+ ( p->catname
+ ? p->catname
+ : "makeprofiles" ), ".fits");
p->mergedimgname=gal_checkset_malloc_cat(p->outdir, tmpname);
free(tmpname);
}
@@ -629,8 +628,8 @@ ui_read_cols_2d(struct mkprofparams *p)
int checkblank;
size_t i, counter=0;
char *colname=NULL, **strarr;
- gal_list_str_t *colstrs=NULL, *ccol;
gal_data_t *cols, *tmp, *corrtype=NULL;
+ gal_list_str_t *ccol, *lines, *colstrs=NULL;
/* The coordinate columns are a linked list of strings. */
ccol=p->ccol;
@@ -654,8 +653,17 @@ ui_read_cols_2d(struct mkprofparams *p)
gal_list_str_reverse(&colstrs);
/* Read the desired columns from the file. */
- cols=gal_table_read(p->catname, p->cp.hdu, colstrs, p->cp.searchin,
+ lines=gal_options_check_stdin(p->catname, p->cp.stdintimeout);
+ cols=gal_table_read(p->catname, p->cp.hdu, lines, colstrs, p->cp.searchin,
p->cp.ignorecase, p->cp.minmapsize, NULL);
+ gal_list_str_free(lines, 1);
+
+ /* The name of the input catalog is only for informative steps from now
+ on (we won't be dealing with the actual file any more). So if the
+ standard input was used (therefore `catname==NULL', set it to
+ `stdin'). */
+ if(p->catname==NULL)
+ gal_checkset_allocate_copy("standard-input", &p->catname);
/* Set the number of objects. */
p->num=cols->size;
@@ -830,8 +838,8 @@ ui_read_cols_3d(struct mkprofparams *p)
int checkblank;
size_t i, counter=0;
char *colname=NULL, **strarr;
- gal_list_str_t *colstrs=NULL, *ccol;
gal_data_t *cols, *tmp, *corrtype=NULL;
+ gal_list_str_t *lines, *ccol, *colstrs=NULL;
/* The 3D-specific columns are not mandatory in `args.h', so we need to
check here if they are given or not before starting to read them. */
@@ -865,8 +873,10 @@ ui_read_cols_3d(struct mkprofparams *p)
gal_list_str_reverse(&colstrs);
/* Read the desired columns from the file. */
- cols=gal_table_read(p->catname, p->cp.hdu, colstrs, p->cp.searchin,
+ lines=gal_options_check_stdin(p->catname, p->cp.stdintimeout);
+ cols=gal_table_read(p->catname, p->cp.hdu, lines, colstrs, p->cp.searchin,
p->cp.ignorecase, p->cp.minmapsize, NULL);
+ gal_list_str_free(lines, 1);
/* Set the number of objects. */
p->num=cols->size;
@@ -1343,7 +1353,7 @@ ui_prepare_canvas(struct mkprofparams *p)
{
/* Read the image. */
p->out=gal_array_read_one_ch_to_type(p->backname, p->backhdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
p->out->wcs=gal_wcs_read(p->backname, p->backhdu, 0, 0,
&p->out->nwcs);
diff --git a/bin/noisechisel/ui.c b/bin/noisechisel/ui.c
index a6cef92..04b7358 100644
--- a/bin/noisechisel/ui.c
+++ b/bin/noisechisel/ui.c
@@ -126,6 +126,7 @@ ui_initialize_options(struct noisechiselparams *p,
case GAL_OPTIONS_KEY_TYPE:
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
@@ -566,7 +567,7 @@ ui_preparations_read_input(struct noisechiselparams *p)
/* Read the input as a single precision floating point dataset. */
p->input = gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
p->input->wcs = gal_wcs_read(p->inputname, p->cp.hdu, 0, 0,
&p->input->nwcs);
@@ -663,7 +664,7 @@ ui_preparations(struct noisechiselparams *p)
{
/* Read the input convolved image. */
p->conv = gal_array_read_one_ch_to_type(p->convolvedname, p->chdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Make sure the convolved image is the same size as the input. */
diff --git a/bin/segment/ui.c b/bin/segment/ui.c
index 8048deb..ef39735 100644
--- a/bin/segment/ui.c
+++ b/bin/segment/ui.c
@@ -131,6 +131,7 @@ ui_initialize_options(struct segmentparams *p,
case GAL_OPTIONS_KEY_TYPE:
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
@@ -404,7 +405,7 @@ ui_prepare_inputs(struct segmentparams *p)
/* Read the input as a single precision floating point dataset. */
p->input = gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
p->input->wcs = gal_wcs_read(p->inputname, p->cp.hdu, 0, 0,
&p->input->nwcs);
@@ -428,7 +429,7 @@ ui_prepare_inputs(struct segmentparams *p)
{
/* Read the input convolved image. */
p->conv = gal_array_read_one_ch_to_type(p->convolvedname, p->chdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
p->conv->wcs=gal_wcs_copy(p->input->wcs);
@@ -447,7 +448,7 @@ ui_prepare_inputs(struct segmentparams *p)
{
/* Read the dataset into memory. */
p->olabel = gal_array_read_one_ch(p->useddetectionname, p->dhdu,
- p->cp.minmapsize);
+ NULL, p->cp.minmapsize);
if( gal_dimension_is_different(p->input, p->olabel) )
error(EXIT_FAILURE, 0, "`%s' (hdu: %s) and `%s' (hdu: %s) have a"
"different dimension/size", p->useddetectionname, p->dhdu,
@@ -736,7 +737,7 @@ ui_read_std_and_sky(struct segmentparams *p)
/* Read the STD image. */
p->std=gal_array_read_one_ch_to_type(p->usedstdname, p->stdhdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Make sure it has the correct size. */
@@ -794,7 +795,7 @@ ui_read_std_and_sky(struct segmentparams *p)
/* Read the Sky dataset. */
sky=gal_array_read_one_ch_to_type(p->skyname, p->skyhdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Check its size. */
diff --git a/bin/statistics/ui.c b/bin/statistics/ui.c
index 4503fb2..3cedec1 100644
--- a/bin/statistics/ui.c
+++ b/bin/statistics/ui.c
@@ -27,6 +27,7 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include <error.h>
#include <stdio.h>
+#include <gnuastro/txt.h>
#include <gnuastro/wcs.h>
#include <gnuastro/fits.h>
#include <gnuastro/tile.h>
@@ -508,8 +509,6 @@ ui_read_check_only_options(struct statisticsparams *p)
static void
ui_check_options_and_arguments(struct statisticsparams *p)
{
- char *name=NULL;
-
if(p->inputname)
{
/* If input is FITS. */
@@ -523,39 +522,15 @@ ui_check_options_and_arguments(struct statisticsparams *p)
"(starting from zero), extension name, or anything "
"acceptable by CFITSIO");
- /* If its a table, make sure a column is also specified. */
+ /* If its an image, make sure column isn't given (in case the
+ user confuses an image with a table). */
p->hdu_type=gal_fits_hdu_format(p->inputname, p->cp.hdu);
- if(p->hdu_type==IMAGE_HDU)
- {
- if(p->column)
- error(EXIT_FAILURE, 0, "%s (hdu: %s): is a FITS image "
- "extension. The `--column' option is only applicable "
- "to tables.", p->inputname, p->cp.hdu);
- }
- else if(p->column==NULL)
- if( asprintf(&name, "%s (hdu: %s)", p->inputname, p->cp.hdu)<0 )
- error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
- }
-
- /* If its not FITS, it must be a table. */
- else
- {
- if(p->column==NULL) name=p->inputname;
+ if(p->hdu_type==IMAGE_HDU && p->column)
+ error(EXIT_FAILURE, 0, "%s (hdu: %s): is a FITS image "
+ "extension. The `--column' option is only applicable "
+ "to tables.", p->inputname, p->cp.hdu);
}
-
- /* If a column was necessary, but not given, print an error. */
- if(name)
- error(EXIT_FAILURE, 0, "%s is a table but no column is "
- "specified. Please use the `--column' (`-c') option to "
- "specify a column.\n\nYou can either give it the column number "
- "(couting from 1), or a match/search in its meta-data (e.g., "
- "column names). For more information, please run the "
- "following command (press the `SPACE' key to go down and "
- "`q' to return to the command-line):\n\n"
- " $ info gnuastro \"Selecting table columns\"\n", name);
}
- else
- error(EXIT_FAILURE, 0, "no input file is specified");
}
@@ -721,20 +696,68 @@ ui_make_sorted_if_necessary(struct statisticsparams *p)
void
ui_read_columns(struct statisticsparams *p)
{
- int toomanycols=0;
- size_t size, counter=0;
- gal_data_t *cols, *tmp;
+ int toomanycols=0, tformat;
gal_list_str_t *column=NULL;
+ gal_data_t *cols, *tmp, *cinfo;
+ size_t size, ncols, nrows, counter=0;
+ gal_list_str_t *lines=gal_options_check_stdin(p->inputname,
+ p->cp.stdintimeout);
- /* Define the columns that we want, note that they should be added to the
- list in reverse. */
+ /* If a reference column is also given, add it to the list of columns to
+ read. */
if(p->refcol)
gal_list_str_add(&column, p->refcol, 0);
+
+ /* If no column is specified, Statistics will abort and an error will be
+ printed when the table has more than one column. If there is only one
+ column, there is no need to specify any, so Statistics will use it. */
+ if(p->column==NULL)
+ {
+ /* Get the basic table information. */
+ cinfo=gal_table_info(p->inputname, p->cp.hdu, lines, &ncols, &nrows,
+ &tformat);
+ gal_data_array_free(cinfo, ncols, 1);
+
+ /* See how many columns it has and take the proper action. */
+ switch(ncols)
+ {
+ case 0:
+ error(EXIT_FAILURE, 0, "%s contains no usable information",
+ ( p->inputname
+ ? gal_checkset_dataset_name(p->inputname, p->cp.hdu)
+ : "Standard input" ));
+ case 1:
+ gal_checkset_allocate_copy("1", &p->column);
+ break;
+ default:
+ error(EXIT_FAILURE, 0, "%s is a table containing more than one "
+ "column. However, the specific column to work on isn't "
+ "specified.\n\n"
+ "Please use the `--column' (`-c') option to specify a "
+ "column. You can either give it the column number "
+ "(couting from 1), or a match/search in its meta-data (e.g., "
+ "column names).\n\n"
+ "For more information, please run the following command "
+ "(press the `SPACE' key to go down and `q' to return to the "
+ "command-line):\n\n"
+ " $ info gnuastro \"Selecting table columns\"\n",
+ ( p->inputname
+ ? gal_checkset_dataset_name(p->inputname, p->cp.hdu)
+ : "Standard input" ));
+ }
+
+ }
gal_list_str_add(&column, p->column, 0);
/* Read the desired column(s). */
- cols=gal_table_read(p->inputname, p->cp.hdu, column, p->cp.searchin,
+ cols=gal_table_read(p->inputname, p->cp.hdu, lines, column, p->cp.searchin,
p->cp.ignorecase, p->cp.minmapsize, NULL);
+ gal_list_str_free(lines, 1);
+
+ /* If the input was from standard input, we can actually write this into
+ it (for future reporting). */
+ if(p->inputname==NULL)
+ gal_checkset_allocate_copy("Standard input", &p->inputname);
/* Put the columns into the proper gal_data_t. */
size=cols->size;
@@ -803,7 +826,8 @@ ui_preparations(struct statisticsparams *p)
if(p->isfits && p->hdu_type==IMAGE_HDU)
{
p->inputformat=INPUT_FORMAT_IMAGE;
- p->input=gal_array_read_one_ch(p->inputname, cp->hdu, cp->minmapsize);
+ p->input=gal_array_read_one_ch(p->inputname, cp->hdu, NULL,
+ cp->minmapsize);
p->input->wcs=gal_wcs_read(p->inputname, cp->hdu, 0, 0,
&p->input->nwcs);
}
diff --git a/bin/table/ui.c b/bin/table/ui.c
index 6a77393..b94cc96 100644
--- a/bin/table/ui.c
+++ b/bin/table/ui.c
@@ -242,8 +242,6 @@ ui_check_options_and_arguments(struct tableparams *p)
"zero), extension name, or anything acceptable by CFITSIO");
}
- else
- error(EXIT_FAILURE, 0, "no input file is specified");
}
@@ -274,11 +272,15 @@ ui_print_info_exit(struct tableparams *p)
char *tmp;
int tableformat;
gal_data_t *allcols;
+ gal_list_str_t *lines;
size_t i, numcols, numrows;
/* Read the table information for the number of columns and rows. */
- allcols=gal_table_info(p->filename, p->cp.hdu, &numcols,
+ lines=gal_options_check_stdin(p->filename, p->cp.stdintimeout);
+ allcols=gal_table_info(p->filename, p->cp.hdu, lines, &numcols,
&numrows, &tableformat);
+ if(p->filename==NULL) p->filename="Standard-input";
+ gal_list_str_free(lines, 1);
/* If there was no actual data in the file, then inform the user */
if(allcols==NULL)
@@ -360,6 +362,7 @@ ui_columns_prepare(struct tableparams *p)
static void
ui_preparations(struct tableparams *p)
{
+ gal_list_str_t *lines;
struct gal_options_common_params *cp=&p->cp;
/* If there were no columns specified or the user has asked for
@@ -371,8 +374,12 @@ ui_preparations(struct tableparams *p)
ui_columns_prepare(p);
/* Read in the table columns. */
- p->table=gal_table_read(p->filename, cp->hdu, p->columns, cp->searchin,
- cp->ignorecase, cp->minmapsize, NULL);
+ lines=gal_options_check_stdin(p->filename, p->cp.stdintimeout);
+ p->table=gal_table_read(p->filename, cp->hdu, lines, p->columns,
+ cp->searchin, cp->ignorecase, cp->minmapsize,
+ NULL);
+ if(p->filename==NULL) p->filename="Standard-input";
+ gal_list_str_free(lines, 1);
/* If there was no actual data in the file, then inform the user and
abort. */
diff --git a/bin/warp/ui.c b/bin/warp/ui.c
index 2804859..66c49e1 100644
--- a/bin/warp/ui.c
+++ b/bin/warp/ui.c
@@ -128,6 +128,7 @@ ui_initialize_options(struct warpparams *p,
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
}
@@ -340,7 +341,7 @@ ui_check_options_and_arguments(struct warpparams *p)
/* Read the input image as double type and its WCS structure. */
p->input=gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu,
- GAL_TYPE_FLOAT64,
+ NULL, GAL_TYPE_FLOAT64,
p->cp.minmapsize);
p->input->wcs=gal_wcs_read(p->inputname, p->cp.hdu, p->hstartwcs,
p->hendwcs, &p->input->nwcs);
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index 704b9c0..ed25a44 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -291,18 +291,19 @@ Common program behavior
* Command-line:: How to use the command-line.
* Configuration files:: Values for unspecified variables.
+* Getting help:: Getting more information on the go.
* Multi-threaded operations:: How threads are managed in Gnuastro.
* Numeric data types:: Different types and how to specify them.
* Tables:: Recognized table formats.
* Tessellation:: Tile the dataset into non-overlapping bins.
-* Getting help:: Getting more information on the go.
* Automatic output:: About automatic output names.
* Output FITS files:: Common properties when outputs are FITS.
Command-line
-* Arguments and options:: Basics of options and arguments.
-* Common options:: Common options to all Gnuastro programs.
+* Arguments and options:: Different ways to specify inputs and
configuration.
+* Common options:: Options that are shared between all programs.
+* Standard input:: Using output of another program as input.
Arguments and options
@@ -322,6 +323,14 @@ Configuration files
* Current directory and User wide:: Local and user configuration files.
* System wide:: System wide configuration files.
+Getting help
+
+* --usage:: View option names and value formats.
+* --help:: List all options with description.
+* Man pages:: Man pages generated from --help.
+* Info:: View complete book in terminal.
+* help-gnuastro mailing list:: Contacting experienced users.
+
Multi-threaded operations
* A note on threads:: Caution and suggestion on using threads.
@@ -337,14 +346,6 @@ Recognized table formats
* Gnuastro text table format:: Reading plain text tables
-Getting help
-
-* --usage:: View option names and value formats.
-* --help:: List all options with description.
-* Man pages:: Man pages generated from --help.
-* Info:: View complete book in terminal.
-* help-gnuastro mailing list:: Contacting experienced users.
-
Data containers
* Fits:: View and manipulate extensions and keywords.
@@ -7130,28 +7131,60 @@ If your problem was not listed above, please file a bug
report
@chapter Common program behavior
All the programs in Gnuastro share a set of common behavior mainly to do
-with user interaction to facilitate their usage. The most basic is how you
-can configure each program to do what you want: define the input, change
-parameter/option values, or identify the output. All Gnuastro programs can
-also read your desired configuration from pre-defined or user-specified
-files so you don't have to specify all the (sometimes numerous) parameters
-on the command-line each time you run a program. These files define the
-``default'' program behavior in each directory, for each user, or on each
-system. In other cases, some programs can greatly benefit from the many
-threads available in modern CPUs, so here we'll also discuss how you can
-get the most out of your hardware. Among some other issues, we will also
-discuss how you can get immediate and distraction-free (without taking your
-hands off the keyboard!) help, or access to this whole book, on the
-command-line.
+with user interaction to facilitate their usage and development. This
+includes how to feed input datasets into the programs, how to configure
+them, specifying the outputs, numerical data types, treating columns of
+information in tables and etc. This chapter is devoted to describing this
+common behavior in all programs. Because the behaviors discussed here are
+common to several programs, they are not repeated in each program's
+description.
+
+In @ref{Command-line}, a very general description of running the programs
+on the command-line is discussed, like difference between arguments and
+options, as well as options that are common/shared between all
+programs. None of Gnuastro's programs keep any internal configuration value
+(values for their different operational steps), they read their
+configuration primarily from the command-line, then from specific files in
+directory, user, or system-wide settings. Using these configuration files
+can greatly help reproducible and robust usage of Gnuastro, see
+@ref{Configuration files} for more.
+
+It is not possible to always have the different options and configurations
+of each program on the top of your head. It is very natural to forget the
+options of a program, their current default values, or how it should be run
+and what it did. Gnuastro's programs have multiple ways to help you refresh
+your memory in multiple levels (just an option name, a short description,
+or fast access to the relevant section of the manual. See @ref{Getting
+help} for more for more on benefiting from this very convenient feature.
+
+Many of the programs use the multi-threaded character of modern CPUs, in
+@ref{Multi-threaded operations} we'll discuss how you can configure this
+behavior, along with some tips on making best use of them. In @ref{Numeric
+data types}, we'll review the various types to store numbers in your
+datasets: setting the proper type for the usage context@footnote{For
+example if the values in your dataset can only be integers between 0 or
+65000, store them in a unsigned 16-bit type, not 64-bit floating point type
+(which is the default in most systems). It takes four times less space and
+is much faster to process.} can grealy improve the file size and also speed
+of reading, writing or processing them.
+
+We'll then look into the recognized table formats in @ref{Tables} and how
+large datasets are broken into tiles, or mesh grid in
+@ref{Tessellation}. Finally, we'll take a look at the behavior regarding
+output files: @ref{Automatic output} discribes how the programs set a
+default name for their output when you don't give one explicitly (using
+@option{--output}). When the output is a FITS file, all the programs also
+store some very useful information in the header that is discussed in
+@ref{Output FITS files}.
@menu
* Command-line:: How to use the command-line.
* Configuration files:: Values for unspecified variables.
+* Getting help:: Getting more information on the go.
* Multi-threaded operations:: How threads are managed in Gnuastro.
* Numeric data types:: Different types and how to specify them.
* Tables:: Recognized table formats.
* Tessellation:: Tile the dataset into non-overlapping bins.
-* Getting help:: Getting more information on the go.
* Automatic output:: About automatic output names.
* Output FITS files:: Common properties when outputs are FITS.
@end menu
@@ -7159,42 +7192,40 @@ command-line.
@node Command-line, Configuration files, Common program behavior, Common
program behavior
@section Command-line
-All the programs in Gnuastro are customized through the standard GNU style
-command-line options. Thus, we'll start by defining this general style that
-is very common in many command-line tools on Unix-like operating
-systems. Finally, the options that are common to all the programs in
-Gnuastro are discussed.
-
-@cindex Metacharacters
-@cindex Token separation
-@cindex Command-line token separation
-@cindex Separating tokens on the command-line
-The command-line text that you type is passed onto the shell (or program
-managing the command-line) as a string of characters. See the ``Invoking
-ProgramName'' sections in this manual for some examples of commands with
-each program, for example @ref{Invoking asttable}. That string is then
-broken up into separate @emph{tokens} or @emph{words} by any
-@emph{metacharacters} (like space, tab, @command{|}, @command{>} or
-@command{;}) that might exist in the text. To learn more, please see the
-GNU Bash manual, for the complete list of meta-characters and other GNU
-Bash definitions (GNU Bash is the most common shell program). Its ``Shell
-Operation'' section has a short summary of the steps the shell takes before
-passing the commands to the program you called.
+Gnuastro's programs are customized through the standard Unix-like
+command-line environment and GNU style command-line options. Both are very
+common in many Unix-like operating system programs. In @ref{Arguments and
+options} we'll start with the difference between arguments and options and
+elaborate on the GNU style of options. Afterwards, in @ref{Common options},
+we'll go into the detailed list of all the options that are common to all
+the programs in Gnuastro.
@menu
-* Arguments and options:: Basics of options and arguments.
-* Common options:: Common options to all Gnuastro programs.
+* Arguments and options:: Different ways to specify inputs and
configuration.
+* Common options:: Options that are shared between all programs.
+* Standard input:: Using output of another program as input.
@end menu
@node Arguments and options, Common options, Command-line, Command-line
@subsection Arguments and options
+@cindex Shell
@cindex Options to programs
@cindex Command-line options
@cindex Arguments to programs
@cindex Command-line arguments
-On the command-line, the first thing you usually enter is the name of the
-program you want to run. After that, you can specify two types of input:
+When you type a command on the command-line, it is passed onto the shell (a
+generic name for the program that manages the command-line) as a string of
+characters. As an example, see the ``Invoking ProgramName'' sections in
+this manual for some examples of commands with each program, like
+@ref{Invoking asttable}, @ref{Invoking astfits}, or @ref{Invoking
+aststatistics}.
+
+The shell then brokes up your string into separate @emph{tokens} or
+@emph{words} using any @emph{metacharacters} (like white-space, tab,
+@command{|}, @command{>} or @command{;}) that are in the string. On the
+command-line, the first thing you usually enter is the name of the program
+you want to run. After that, you can specify two types of tokens:
@emph{arguments} and @emph{options}. In the GNU-style, arguments are those
tokens that are not preceded by any hyphens (@command{-}, see
@ref{Arguments}). Here is one example:
@@ -7203,29 +7234,32 @@ tokens that are not preceded by any hyphens
(@command{-}, see
$ astcrop --center=53.162551,-27.789676 -w10/3600 --mode=wcs udf.fits
@end example
-In this example, the argument is @file{udf.fits}. Arguments are most
-commonly the input file names containing your data. Options start with one
-or two hyphens, followed by an identifier for the option (the option's
-name) and its value (see @ref{Options}). Through options you tell the
-program how to interpret the data. In this example, we are running
-@ref{Crop} to crop a region of width 10 arc-seconds centered at the given
-RA and Dec from the input Hubble Ultra-Deep Field (UDF) FITS image. So
-options come with an identifier (the option name which is separate from
-their value).
+In the example above, we are running @ref{Crop} to crop a region of width
+10 arc-seconds centered at the given RA and Dec from the input Hubble
+Ultra-Deep Field (UDF) FITS image. Here, the argument is
+@file{udf.fits}. Arguments are most commonly the input file names
+containing your data. Options start with one or two hyphens, followed by an
+identifier for the option (the option's name, for example,
+@option{--center}, @option{-w}, @option{--mode} in the example above) and
+its value (anything after the option name, or the optional @key{=}
+character). Through options you can configure how the program runs
+(interprets the data you provided).
@vindex --help
@vindex --usage
@cindex Mandatory arguments
-Arguments can be both mandatory and optional and unlike options they don't
-have any identifiers (or help from you). Hence, their order might also
-matter (for example in @command{cp} which is used for copying one file to
-another location). The outputs of @option{--usage} and @option{--help}
+Arguments can be mandatory and optional and unlike options, they don't have
+any identifiers. Hence, when there multiple arguments, their order might
+also matter (for example in @command{cp} which is used for copying one file
+to another location). The outputs of @option{--usage} and @option{--help}
shows which arguments are optional and which are mandatory, see
-@ref{--usage}. As their name suggests, @emph{options} on the command-line
-can be considered to be optional and most of the time, you don't have to
-worry about what order you specify them in. When the order does matter, or
-the option can be invoked multiple times, it is explicitly mentioned in the
-``Invoking ProgramName'' section of each program.
+@ref{--usage}.
+
+As their name suggests, @emph{options} can be considered to be optional and
+most of the time, you don't have to worry about what order you specify them
+in. When the order does matter, or the option can be invoked multiple
+times, it is explicitly mentioned in the ``Invoking ProgramName'' section
+of each program (this is a very important aspect of an option).
@cindex Metacharacters on the command-line
In case your arguments or option values contain any of the shell's
@@ -7241,17 +7275,16 @@ For example, let's say you want to specify the header
data unit (HDU) of
your FITS file using a complex expression like `@command{3; images(exposure
> 100)}'. If you simply add these after the @option{--hdu} (@option{-h})
option, the programs in Gnuastro will read the value to the HDU option as
-`@command{3}' and run. Then, Bash will attempt to run a separate command
-`@command{images(exposure > 100)}' and complain about a syntax error. This
-is because the semicolon (@command{;}) is an `end of command' character in
-the shell. To solve this problem you can simply put double quotes around
-the whole string you want to pass to @option{--hdu} as seen below:
+`@command{3}' and run. Then, the shell will attempt to run a separate
+command `@command{images(exposure > 100)}' and complain about a syntax
+error. This is because the semicolon (@command{;}) is an `end of command'
+character in the shell. To solve this problem you can simply put double
+quotes around the whole string you want to pass to @option{--hdu} as seen
+below:
@example
-$ astcrop --hdu="3; images(exposure > 100)" FITSimage.fits
+$ astcrop --hdu="3; images(exposure > 100)" image.fits
@end example
-Alternatively you can put a `@command{\}' before every meta-character in
-this string, but try doing that, and probably you will agree that the
-double quotes are much more easier, elegant and readable.
+
@@ -7465,7 +7498,7 @@ others 1. You can assume by default that counting starts
from 1, if it
starts from 0 for a special option, it will be explicitly mentioned.
@end cartouche
-@node Common options, , Arguments and options, Command-line
+@node Common options, Standard input, Arguments and options, Command-line
@subsection Common options
@cindex Options common to all programs
@@ -7493,6 +7526,32 @@ programs.
@vtable @option
+@cindex Timeout
+@cindex Standard input
+@item --stdintimeout
+Number of micro-seconds to wait for writing/typing in the @emph{first line}
+of standard input from the command-line (see @ref{Standard input}). This is
+only relevant for programs that also accept input from the standard input,
+@emph{and} you want to manually write/type the contents on the
+terminal. When the standard input is already connected to a pipe (output of
+another program), there won't be any waiting (hence no timeout, thus making
+this option redundant).
+
+If the first line-break (for example with the @key{ENTER} key) is not
+provided before the timeout, the program will abort with an error that no
+input was given. Note that this time interval is @emph{only} for the first
+line that you type. Once the first line is given, the program will assume
+that more data will come and accept rest of your inputs without any time
+limit. You need to specify the ending of the standard input, for example by
+pressing @key{CTRL-D} after a new line.
+
+Note that any input you write/type into a program on the command-line with
+Standard input will be discarded (lost) once the program is finished. It is
+only recoverable manually from your command-line (where you actually typed)
+as long as the terminal is open. So only use this feature when you are sure
+that you don't need the dataset (or have a copy of it somewhere else).
+
+
@cindex HDU
@cindex Header data unit
@item -h STR/INT
@@ -7956,8 +8015,103 @@ others, this option will be ignored.
+@node Standard input, , Common options, Command-line
+@subsection Standard input
-@node Configuration files, Multi-threaded operations, Command-line, Common
program behavior
+@cindex Standard input
+@cindex Stream: standard input
+The most common way to feed the primary/first input dataset into a program
+is to give its filename as an argument (discussed in @ref{Arguments}). When
+you want to run a series of programs in sequence, this means that each will
+have to keep the output of each program in a separate file and re-type that
+file's name in the next command. This can be very slow and frustrating
+(mis-typing a file's name).
+
+@cindex Standard output stream
+@cindex Stream: standard output
+To solve the problem, the founders of Unix defined pipes to directly feed
+the output of one program (its ``Standard output'' stream) into the
+``standard input'' of a next program. This removes the need to make
+temporary files between separate processes and became one of the best
+demonstrations of the Unix-way, or Unix philosopy.
+
+Every program has three streams identifying where it reads/writes non-file
+inputs/outputs: @emph{Standard input}, @emph{Standard output}, and
+@emph{Standard error}. When a program is called alone, all three are
+directed to the terminal that you are using. If it needs an input, it will
+prompt you for one and you can type it in. Or, it prints its results in the
+terminal for you to see.
+
+For example, say you have a FITS table/catalog containing the B and V band
+magnitudes (@code{MAG_B} and @code{MAG_V} columns) of a selection of
+galaxies along with many other columns. If you want to see only these two
+columns in your terminal, can use Gnuastro's @ref{Table} program like
+below:
+
+@example
+$ asttable cat.fits -cMAG_B,MAG_V
+@end example
+
+Through the Unix pipe mechanism, when the shell confronts the pipe
+character (@key{|}), it connects the standard output of the program before
+the pipe, to the standard input of the program after it. So it is literally
+a ``pipe'': everything that you would see printed by the first program on
+the command (without any pipe), is now passed to the second program (and
+not seen by you).
+
+@cindex AWK
+@cindex GNU AWK
+To continue the previous example, let's say you want to see the B-V
+color. To do this, you can pipe Table's output to AWK (a wonderful tool for
+processing things like plain text tables):
+
+@example
+$ asttable cat.fits -cMAG_B,MAG_V | awk '@{print $1-$2@}'
+@end example
+
+But understanding the distribution by visually seeing all the numbers under
+each other is not too useful! You can therefore feed this single column
+information into @ref{Statistics} to give you a general feeling of the
+distribution with the same command:
+
+@example
+$ asttable cat.fits -cMAG_B,MAG_V | awk '@{print $1-$2@}' | aststatistics
+@end example
+
+Gnuastro's programs that accept input from standard input, only look into
+the Standard input stream if there is no first argument. In other words,
+arguments take precedence over Standard input. When no argument is
+provided, the programs check if the standard input stream is already full
+or not (output from another program is waiting to be used). If data is
+present in the standard input stream, it is used.
+
+When the standard input is empty, the program will wait
+@option{--stdintimeout} micro-seconds for you to manually enter the first
+line (ending with a new-line character, or the @key{ENTER} key, see
+@ref{Input output options}). If it detects the first line in this time,
+there is no more time limit, and you can manually write/type all the lines
+for as long as it takes. To inform the program that Standard input has
+finished, press @key{CTRL-D} after a new line. If the program doesn't catch
+the first line before the time-out finishes, it will abort with an error
+saying that no input was provided.
+
+@cartouche
+@noindent
+@strong{Manual input in Standard input is discarded: } Be careful that when
+you manually fill the Standard input, the data will be discarded once the
+program finishes and reproducing the result will be impossible. Therefore
+this form of providing input is only good for temporary tests.
+@end cartouche
+
+@cartouche
+@noindent
+@strong{Standard input currently only for plain text: } Currently Standard
+input only works for plain text inputs like the example above. We will
+later allow FITS files into the programs through standard input also.
+@end cartouche
+
+
+@node Configuration files, Getting help, Command-line, Common program behavior
@section Configuration files
@cindex @file{etc}
@@ -8209,1192 +8363,1200 @@ source by your self as in @ref{Quick start}.
-@node Multi-threaded operations, Numeric data types, Configuration files,
Common program behavior
-@section Multi-threaded operations
+@node Getting help, Multi-threaded operations, Configuration files, Common
program behavior
+@section Getting help
-@pindex nproc
-@cindex pthread
-@cindex CPU threads
-@cindex GNU Coreutils
-@cindex Using CPU threads
-@cindex CPU, using all threads
-@cindex Multi-threaded programs
-@cindex Using multiple CPU cores
-@cindex Simultaneous multithreading
-Some of the programs benefit significantly when you use all the threads
-your computer's CPU has to offer to your operating system. The number of
-threads available can be larger than the number of physical (hardware)
-cores in the CPU (also known as Simultaneous multithreading). For example,
-in Intel's CPUs (those that implement its Hyper-threading technology) the
-number of threads is usually double the number of physical cores in your
-CPU. On a GNU/Linux system, the number of threads available can be found
-with the command @command{$ nproc} command (part of GNU Coreutils).
+@cindex Help
+@cindex Book formats
+@cindex Remembering options
+@cindex Convenient book formats
+Probably the first time you read this book, it is either in the PDF
+or HTML formats. These two formats are very convenient for when you
+are not actually working, but when you are only reading. Later on,
+when you start to use the programs and you are deep in the middle of
+your work, some of the details will inevitably be forgotten. Going to
+find the PDF file (printed or digital) or the HTML webpage is a major
+distraction.
-@vindex --numthreads
-@cindex Number of threads available
-@cindex Available number of threads
-@cindex Internally stored option value
-Gnuastro's programs can find the number of threads available to your system
-internally at run-time (when you execute the program). However, if a value
-is given to the @option{--numthreads} option, the given number will be
-used, see @ref{Operating mode options} and @ref{Configuration files} for ways
to
-use this option. Thus @option{--numthreads} is the only common option in
-Gnuastro's programs with a value that doesn't have to be specified anywhere
-on the command-line or in the configuration files.
+@cindex Online help
+@cindex Command-line help
+GNU software have a very unique set of tools for aiding your memory on
+the command-line, where you are working, depending how much of it you
+need to remember. In the past, such command-line help was known as
+``online'' help, because they were literally provided to you `on'
+the command `line'. However, nowadays the word ``online'' refers to
+something on the internet, so that term will not be used. With this
+type of help, you can resume your exciting research without taking
+your hands off the keyboard.
-@menu
-* A note on threads:: Caution and suggestion on using threads.
-* How to run simultaneous operations:: How to run things simultaneously.
-@end menu
+@cindex Installed help methods
+Another major advantage of such command-line based help routines is
+that they are installed with the software in your computer, therefore
+they are always in sync with the executable you are actually
+running. Three of them are actually part of the executable. You don't
+have to worry about the version of the book or program. If you rely
+on external help (a PDF in your personal print or digital archive or
+HTML from the official webpage) you have to check to see if their
+versions fit with your installed program.
-@node A note on threads, How to run simultaneous operations, Multi-threaded
operations, Multi-threaded operations
-@subsection A note on threads
+If you only need to remember the short or long names of the options,
+@option{--usage} is advised. If it is what the options do, then
+@option{--help} is a great tool. Man pages are also provided for those
+who are use to this older system of documentation. This full book is
+also available to you on the command-line in Info format. If none of
+these seems to resolve the problems, there is a mailing list which
+enables you to get in touch with experienced Gnuastro users. In the
+subsections below each of these methods are reviewed.
-@cindex Using multiple threads
-@cindex Best use of CPU threads
-@cindex Efficient use of CPU threads
-Spinning off threads is not necessarily the most efficient way to run an
-application. Creating a new thread isn't a cheap operation for the
-operating system. It is most useful when the input data are fixed and you
-want the same operation to be done on parts of it. For example one input
-image to Crop and multiple crops from various parts of it. In this fashion,
-the image is loaded into memory once, all the crops are divided between the
-number of threads internally and each thread cuts out those parts which are
-assigned to it from the same image. On the other hand, if you have multiple
-images and you want to crop the same region(s) out of all of them, it is
-much more efficient to set @option{--numthreads=1} (so no threads spin off)
-and run Crop multiple times simultaneously, see @ref{How to run
-simultaneous operations}.
-@cindex Wall-clock time
-You can check the boost in speed by first running a program on one of the
-data sets with the maximum number of threads and another time (with
-everything else the same) and only using one thread. You will notice that
-the wall-clock time (reported by most programs at their end) in the former
-is longer than the latter divided by number of physical CPU cores (not
-threads) available to your operating system. Asymptotically these two times
-can be equal (most of the time they aren't). So limiting the programs to
-use only one thread and running them independently on the number of
-available threads will be more efficient.
+@menu
+* --usage:: View option names and value formats.
+* --help:: List all options with description.
+* Man pages:: Man pages generated from --help.
+* Info:: View complete book in terminal.
+* help-gnuastro mailing list:: Contacting experienced users.
+@end menu
-@cindex System Cache
-@cindex Cache, system
-Note that the operating system keeps a cache of recently processed
-data, so usually, the second time you process an identical data set
-(independent of the number of threads used), you will get faster
-results. In order to make an unbiased comparison, you have to first
-clean the system's cache with the following command between the two
-runs.
+@node --usage, --help, Getting help, Getting help
+@subsection @option{--usage}
+@vindex --usage
+@cindex Usage pattern
+@cindex Mandatory arguments
+@cindex Optional and mandatory tokens
+If you give this option, the program will not run. It will only print a
+very concise message showing the options and arguments. Everything within
+square brackets (@option{[]}) is optional. For example here are the first
+and last two lines of Crop's @option{--usage} is shown:
@example
-$ sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
+$ astcrop --usage
+Usage: astcrop [-Do?IPqSVW] [-d INT] [-h INT] [-r INT] [-w INT]
+ [-x INT] [-y INT] [-c INT] [-p STR] [-N INT] [--deccol=INT]
+ ....
+ [--setusrconf] [--usage] [--version] [--wcsmode]
+ [ASCIIcatalog] FITSimage(s).fits
@end example
-@cartouche
-@noindent
-@strong{SUMMARY: Should I use multiple threads?} Depends:
-@itemize
-@item
-If you only have @strong{one} data set (image in most cases!), then
-yes, the more threads you use (with a maximum of the number of threads
-available to your OS) the faster you will get your results.
-
-@item
-If you want to run the same operation on @strong{multiple} data sets, it is
-best to set the number of threads to 1 and use Make, or GNU Parallel, as
-explained in @ref{How to run simultaneous operations}.
-@end itemize
-@end cartouche
+There are no explanations on the options, just their short and long
+names shown separately. After the program name, the short format of
+all the options that don't require a value (on/off options) is
+displayed. Those that do require a value then follow in separate
+brackets, each displaying the format of the input they want, see
+@ref{Options}. Since all options are optional, they are shown in
+square brackets, but arguments can also be optional. For example in
+this example, a catalog name is optional and is only required in some
+modes. This is a standard method of displaying optional arguments for
+all GNU software.
+@node --help, Man pages, --usage, Getting help
+@subsection @option{--help}
+@vindex --help
+If the command-line includes this option, the program will not be
+run. It will print a complete list of all available options along with
+a short explanation. The options are also grouped by their
+context. Within each context, the options are sorted
+alphabetically. Since the options are shown in detail afterwards, the
+first line of the @option{--help} output shows the arguments and if
+they are optional or not, similar to @ref{--usage}.
+In the @option{--help} output of all programs in Gnuastro, the
+options for each program are classified based on context. The first
+two contexts are always options to do with the input and output
+respectively. For example input image extensions or supplementary
+input files for the inputs. The last class of options is also fixed in
+all of Gnuastro, it shows operating mode options. Most of these
+options are already explained in @ref{Operating mode options}.
+@cindex Long outputs
+@cindex Redirection of output
+@cindex Command-line, long outputs
+The help message will sometimes be longer than the vertical size of
+your terminal. If you are using a graphical user interface terminal
+emulator, you can scroll the terminal with your mouse, but we promised
+no mice distractions! So here are some suggestions:
-@node How to run simultaneous operations, , A note on threads, Multi-threaded
operations
-@subsection How to run simultaneous operations
+@itemize
+@item
+@cindex Scroll command-line
+@cindex Command-line scroll
+@cindex @key{Shift + PageUP} and @key{Shift + PageDown}
+@key{Shift + PageUP} to scroll up and @key{Shift + PageDown} to scroll
+down. For most help output this should be enough. The problem is that
+it is limited by the number of lines that your terminal keeps in
+memory and that you can't scroll by lines, only by whole screens.
-There are two@footnote{A third way would be to open multiple terminal
-emulator windows in your GUI, type the commands separately on each and
-press @key{Enter} once on each terminal, but this is far too frustrating,
-tedious and prone to errors. It's therefore not a realistic solution when
-tens, hundreds or thousands of operations (your research targets,
-multiplied by the operations you do on each) are to be done.} approaches to
-simultaneously execute a program: using GNU Parallel or Make (GNU Make is
-the most common implementation). The first is very useful when you only
-want to do one job multiple times and want to get back to your work without
-actually keeping the command you ran. The second is usually for more
-important operations, with lots of dependencies between the different
-products (for example a full scientific research).
+@item
+@cindex Pipe
+@cindex @command{less}
+Pipe to @command{less}. A pipe is a form of shell re-direction. The
+@command{less} tool in Unix-like systems was made exactly for such
+outputs of any length. You can pipe (@command{|}) the output of any
+program that is longer than the screen to it and then you can scroll
+through (up and down) with its many tools. For example:
+@example
+$ astnoisechisel --help | less
+@end example
+@noindent
+Once you have gone through the text, you can quit @command{less} by
+pressing the @key{q} key.
-@table @asis
-@item GNU Parallel
-@cindex GNU Parallel
-When you only want to run multiple instances of a command on different
-threads and get on with the rest of your work, the best method is to
-use GNU parallel. Surprisingly GNU Parallel is one of the few GNU
-packages that has no Info documentation but only a Man page, see
-@ref{Info}. So to see the documentation after installing it please run
+@item
+@cindex Save output to file
+@cindex Redirection of output
+Redirect to a file. This is a less convenient way, because you will
+then have to open the file in a text editor! You can do this with the
+shell redirection tool (@command{>}):
+@example
+$ astnoisechisel --help > filename.txt
+@end example
+@end itemize
+
+@cindex GNU Grep
+@cindex Searching text
+@cindex Command-line searching text
+In case you have a special keyword you are looking for in the help, you
+don't have to go through the full list. GNU Grep is made for this job. For
+example if you only want the list of options whose @option{--help} output
+contains the word ``axis'' in Crop, you can run the following command:
@example
-$ man parallel
+$ astcrop --help | grep axis
@end example
-@noindent
-As an example, let's assume we want to crop a region fixed on the
-pixels (500, 600) with the default width from all the FITS images in
-the @file{./data} directory ending with @file{sci.fits} to the current
-directory. To do this, you can run:
+@cindex @code{ARGP_HELP_FMT}
+@cindex Argp argument parser
+@cindex Customize @option{--help} output
+@cindex @option{--help} output customization
+If the output of this option does not fit nicely within the confines
+of your terminal, GNU does enable you to customize its output through
+the environment variable @code{ARGP_HELP_FMT}, you can set various
+parameters which specify the formatting of the help messages. For
+example if your terminals are wider than 70 spaces (say 100) and you
+feel there is too much empty space between the long options and the
+short explanation, you can change these formats by giving values to
+this environment variable before running the program with the
+@option{--help} output. You can define this environment variable in
+this manner:
@example
-$ parallel astcrop --numthreads=1 --xc=500 --yc=600 ::: \
- ./data/*sci.fits
+$ export ARGP_HELP_FMT=rmargin=100,opt-doc-col=20
@end example
+@cindex @file{.bashrc}
+This will affect all GNU programs using GNU C library's @file{argp.h}
+facilities as long as the environment variable is in memory. You can
+see the full list of these formatting parameters in the ``Argp User
+Customization'' part of the GNU C library manual. If you are more
+comfortable to read the @option{--help} outputs of all GNU software in
+your customized format, you can add your customization (similar to
+the line above, without the @command{$} sign) to your @file{~/.bashrc}
+file. This is a standard option for all GNU software.
+
+@node Man pages, Info, --help, Getting help
+@subsection Man pages
+@cindex Man pages
+Man pages were the Unix method of providing command-line documentation
+to a program. With GNU Info, see @ref{Info} the usage of this method
+of documentation is highly discouraged. This is because Info provides
+a much more easier to navigate and read environment.
+However, some operating systems require a man page for packages that
+are installed and some people are still used to this method of command
+line help. So the programs in Gnuastro also have Man pages which are
+automatically generated from the outputs of @option{--version} and
+@option{--help} using the GNU help2man program. So if you run
+@example
+$ man programname
+@end example
@noindent
-GNU Parallel can help in many more conditions, this is one of the
-simplest, see the man page for lots of other examples. For absolute
-beginners: the backslash (@command{\}) is only a line breaker to fit
-nicely in the page. If you type the whole command in one line, you
-should remove it.
+You will be provided with a man page listing the options in the
+standard manner.
-@item Make
-@cindex Make
-Make is a program for building ``targets'' (e.g., files) using ``recipes''
-(a set of operations) when their known ``prerequisites'' (other files) have
-been updated. It elegantly allows you to define dependency structures for
-building your final output and updating it efficiently when the inputs
-change. It is the most common infra-structure to build software
-today.
-Scientific research methodology is very similar to software development:
-you start by testing a hypothesis on a small sample of objects/targets with
-a simple set of steps. As you are able to get promising results, you
-improve the method and use it on a larger, more general, sample. In the
-process, you will confront many issues that have to be corrected (bugs in
-software development jargon). Make a wonderful tool to manage this style of
-development. It has been used to make reproducible papers, for example see
-@url{https://gitlab.com/makhlaghi/NoiseChisel-paper, the reproduction
-pipeline} of the paper introducing @ref{NoiseChisel} (one of Gnuastro's
-programs).
-@cindex GNU Make
-GNU Make@footnote{@url{https://www.gnu.org/software/make/}} is the most
-common implementation which (similar to nearly all GNU programs, comes with
-a wonderful
-manual@footnote{@url{https://www.gnu.org/software/make/manual/}}). Make is
-very basic and simple, and thus the manual is short (the most important
-parts are in the first roughly 100 pages) and easy to read/understand.
-Make comes with a @option{--jobs} (@option{-j}) option which allows you to
-specify the maximum number of jobs that can be done simultaneously. For
-example if you have 8 threads available to your operating system. You can
-run:
+
+@node Info, help-gnuastro mailing list, Man pages, Getting help
+@subsection Info
+
+@cindex GNU Info
+@cindex Command-line, viewing full book
+Info is the standard documentation format for all GNU software. It is
+a very useful command-line document viewing format, fully equipped
+with links between the various pages and menus and search
+capabilities. As explained before, the best thing about it is that it
+is available for you the moment you need to refresh your memory on any
+command-line tool in the middle of your work without having to take
+your hands off the keyboard. This complete book is available in Info
+format and can be accessed from anywhere on the command-line.
+
+To open the Info format of any installed programs or library on your
+system which has an Info format book, you can simply run the command
+below (change @command{executablename} to the executable name of the
+program or library):
@example
-$ make -j8
+$ info executablename
@end example
-With this command, Make will process your @file{Makefile} and create all
-the targets (can be thousands of FITS images for example) simultaneously on
-8 threads, while fully respecting their dependencies (only building a
-file/target when its prerequisites are successfully built). Make is thus
-strongly recommended for managing scientific research where robustness,
-archiving, reproducibility and speed@footnote{Besides its multi-threaded
-capabilities, Make will only re-build those targets that depend on a change
-you have made, not the whole work. For example, if you have set the
-prerequisites properly, you can easily test the changing of a parameter on
-your paper's results without having to re-do everything (which is much
-faster). This allows you to be much more productive in easily checking
-various ideas/assumptions of the different stages of your research and thus
-produce a more robust result for your exciting science.} are important.
+@noindent
+@cindex Learning GNU Info
+@cindex GNU software documentation
+In case you are not already familiar with it, run @command{$ info
+info}. It does a fantastic job in explaining all its capabilities its
+self. It is very short and you will become sufficiently fluent in
+about half an hour. Since all GNU software documentation is also
+provided in Info, your whole GNU/Linux life will significantly
+improve.
-@end table
+@cindex GNU Emacs
+@cindex GNU C library
+Once you've become an efficient navigator in Info, you can go to any
+part of this book or any other GNU software or library manual, no
+matter how long it is, in a matter of seconds. It also blends nicely
+with GNU Emacs (a text editor) and you can search manuals while you
+are writing your document or programs without taking your hands off
+the keyboard, this is most useful for libraries like the GNU C
+library. To be able to access all the Info manuals installed in your
+GNU/Linux within Emacs, type @key{Ctrl-H + i}.
+To see this whole book from the beginning in Info, you can run
+@example
+$ info gnuastro
+@end example
+@noindent
+If you run Info with the particular program executable name, for
+example @file{astcrop} or @file{astnoisechisel}:
+@example
+$ info astprogramname
+@end example
-@node Numeric data types, Tables, Multi-threaded operations, Common program
behavior
-@section Numeric data types
+@noindent
+you will be taken to the section titled ``Invoking ProgramName'' which
+explains the inputs and outputs along with the command-line options for
+that program. Finally, if you run Info with the official program name, for
+example Crop or NoiseChisel:
-@cindex Bit
-@cindex Type
-At the lowest level, the computer stores everything in terms of @code{1} or
-@code{0}. For example, each program in Gnuastro, or each astronomical image
-you take with the telescope is actually a string of millions of these zeros
-and ones. The space required to keep a zero or one is the smallest unit of
-storage, and is known as a @emph{bit}. However, understanding and
-manipulating this string of bits is extremely hard for most
-people. Therefore, we define packages of these bits along with a standard
-on how to interpret the bits in each package as a @emph{type}.
+@example
+$ info ProgramName
+@end example
-@cindex Byte
-@cindex Signed integer
-@cindex Unsigned integer
-@cindex Integer, Signed
-The most basic standard for reading the bits is integer numbers
-(@mymath{..., -2, -1, 0, 1, 2, ...}, more bits will give larger
-limits). The common integer types are 8, 16, 32, and 64 bits wide. For each
-width, there are two standards for reading the bits: signed and unsigned
-integers. In the former, negative numbers are allowed and in the latter,
-they aren't. The @code{unsigned} types thus have larger positive limits
-(one extra bit), but no negative value. When the context of your work
-doesn't involve negative numbers (for example counting, where negative is
-not defined), it is best to use the @code{unsigned} types. For full
-numerical range of all integer types, see below.
+@noindent
+you will be taken to the top section which introduces the
+program. Note that in all cases, Info is not case sensitive.
-Another standard of converting a given number of bits to numbers is the
-floating point standard, this standard can approximately store any real
-number with a given precision. There are two common floating point types:
-32-bit and 64-bit, for single and double precision floating point numbers
-respectively. The former is sufficient for data with less than 8
-significant decimal digits (most astronomical data), while the latter is
-good for less than 16 significant decimal digits. The representation of
-real numbers as bits is much more complex than integers. If you are
-interested, you can start with the
-@url{https://en.wikipedia.org/wiki/Floating_point, Wikipedia article}.
-With the conversion operators in Gnuastro's Arithmetic, you can change the
-types of data to each other, which is necessary in some contexts. For
-example the program/library, that you intend to feed the data into, only
-accepts floating point values, but you have an integer image. Another
-situation that conversion can be helpful is when you know that your data
-only has values that fit within @code{int8} or @code{uint16}. However it is
-currently formatted in the @code{float64} type. Operations involving
-floating point or larger integer types are significantly slower than
-integer or smaller-width types respectively. In the latter case, it also
-requires much more (by 8 or 4 times in the example above) storage space. So
-when you confront such situations and want to store/archive/transfter the
-data, it is best convert them to the most efficient type.
-The short and long names for the recognized numeric data types in Gnuastro
-are listed below. Both short and long names can be used when you want to
-specify a type. For example, as a value to the common option
-@option{--type} (see @ref{Input output options}), or in the information
-comment lines of @ref{Gnuastro text table format}. The ranges listed below
-are inclusive.
+@node help-gnuastro mailing list, , Info, Getting help
+@subsection help-gnuastro mailing list
-@table @code
-@item u8
-@itemx uint8
-8-bit unsigned integers, range:@*
-@mymath{[0\rm{\ to\ }2^8-1]} or @mymath{[0\rm{\ to\ }255]}.
+@cindex help-gnuastro mailing list
+@cindex Mailing list: help-gnuastro
+Gnuastro maintains the help-gnuastro mailing list for users to ask any
+questions related to Gnuastro. The experienced Gnuastro users and some
+of its developers are subscribed to this mailing list and your email
+will be sent to them immediately. However, when contacting this
+mailing list please have in mind that they are possibly very busy and
+might not be able to answer immediately.
-@item i8
-@itemx int8
-8-bit signed integers, range:@*
-@mymath{[-2^7\rm{\ to\ }2^7-1]} or @mymath{[-128\rm{\ to\ }127]}.
+@cindex Mailing list archives
+@cindex @code{help-gnuastro@@gnu.org}
+To ask a question from this mailing list, send a mail to
+@code{help-gnuastro@@gnu.org}. Anyone can view the mailing list
+archives at @url{http://lists.gnu.org/archive/html/help-gnuastro/}. It
+is best that before sending a mail, you search the archives to see if
+anyone has asked a question similar to yours. If you want to make a
+suggestion or report a bug, please don't send a mail to this mailing
+list. We have other mailing lists and tools for those purposes, see
+@ref{Report a bug} or @ref{Suggest new feature}.
-@item u16
-@itemx uint16
-16-bit unsigned integers, range:@*
-@mymath{[0\rm{\ to\ }2^{16}-1]} or @mymath{[0\rm{\ to\ }65535]}.
-@item i16
-@itemx int16
-16-bit signed integers, range:@* @mymath{[-2^{15}\rm{\ to\ }2^{15}-1]} or
-@mymath{[-32768\rm{\ to\ }32767]}.
-@item u32
-@itemx uint32
-32-bit unsigned integers, range:@* @mymath{[0\rm{\ to\ }2^{32}-1]} or
-@mymath{[0\rm{\ to\ }4294967295]}.
-@item i32
-@itemx int32
-32-bit signed integers, range:@* @mymath{[-2^{31}\rm{\ to\ }2^{31}-1]} or
-@mymath{[-2147483648\rm{\ to\ }2147483647]}.
-@item u64
-@itemx uint64
-64-bit unsigned integers, range@* @mymath{[0\rm{\ to\ }2^{64}-1]} or
-@mymath{[0\rm{\ to\ }18446744073709551615]}.
+@node Multi-threaded operations, Numeric data types, Getting help, Common
program behavior
+@section Multi-threaded operations
-@item i64
-@itemx int64
-64-bit signed integers, range:@* @mymath{[-2^{63}\rm{\ to\ }2^{63}-1]} or
-@mymath{[-9223372036854775808\rm{\ to\ }9223372036854775807]}.
+@pindex nproc
+@cindex pthread
+@cindex CPU threads
+@cindex GNU Coreutils
+@cindex Using CPU threads
+@cindex CPU, using all threads
+@cindex Multi-threaded programs
+@cindex Using multiple CPU cores
+@cindex Simultaneous multithreading
+Some of the programs benefit significantly when you use all the threads
+your computer's CPU has to offer to your operating system. The number of
+threads available can be larger than the number of physical (hardware)
+cores in the CPU (also known as Simultaneous multithreading). For example,
+in Intel's CPUs (those that implement its Hyper-threading technology) the
+number of threads is usually double the number of physical cores in your
+CPU. On a GNU/Linux system, the number of threads available can be found
+with the command @command{$ nproc} command (part of GNU Coreutils).
-@item f32
-@itemx float32
-32-bit (single-precision) floating point types. The maximum (minimum is its
-negative) possible value is
-@mymath{3.402823\times10^{38}}. Single-precision floating points can
-accurately represent a floating point number up to @mymath{\sim7.2}
-significant decimals. Given the heavy noise in astronomical data, this is
-usually more than sufficient for storing results.
+@vindex --numthreads
+@cindex Number of threads available
+@cindex Available number of threads
+@cindex Internally stored option value
+Gnuastro's programs can find the number of threads available to your system
+internally at run-time (when you execute the program). However, if a value
+is given to the @option{--numthreads} option, the given number will be
+used, see @ref{Operating mode options} and @ref{Configuration files} for ways
to
+use this option. Thus @option{--numthreads} is the only common option in
+Gnuastro's programs with a value that doesn't have to be specified anywhere
+on the command-line or in the configuration files.
-@item f64
-@itemx float64
-64-bit (double-precision) floating point types. The maximum (minimum is its
-negative) possible value is @mymath{\sim10^{308}}. Double-precision
-floating points can accurately represent a floating point number
-@mymath{\sim15.9} significant decimals. This is usually good for processing
-(mixing) the data internally, for example a sum of single precision data
-(and later storing the result as @code{float32}).
-@end table
+@menu
+* A note on threads:: Caution and suggestion on using threads.
+* How to run simultaneous operations:: How to run things simultaneously.
+@end menu
-@cartouche
-@noindent
-@strong{Some file formats don't recognize all types.} Some file formats
-don't recognize all the types, for example the FITS standard (see
-@ref{Fits}) does not define @code{uint64} in binary tables or images. When
-a type is not acceptable for output into a given file format, the
-respective Gnuastro program or library will let you know and abort. On the
-command-line, you can use the @ref{Arithmetic} program to convert the
-numerical type of a dataset, in the libraries, you can call
-@code{gal_data_copy_to_new_type}.
-@end cartouche
+@node A note on threads, How to run simultaneous operations, Multi-threaded
operations, Multi-threaded operations
+@subsection A note on threads
+
+@cindex Using multiple threads
+@cindex Best use of CPU threads
+@cindex Efficient use of CPU threads
+Spinning off threads is not necessarily the most efficient way to run an
+application. Creating a new thread isn't a cheap operation for the
+operating system. It is most useful when the input data are fixed and you
+want the same operation to be done on parts of it. For example one input
+image to Crop and multiple crops from various parts of it. In this fashion,
+the image is loaded into memory once, all the crops are divided between the
+number of threads internally and each thread cuts out those parts which are
+assigned to it from the same image. On the other hand, if you have multiple
+images and you want to crop the same region(s) out of all of them, it is
+much more efficient to set @option{--numthreads=1} (so no threads spin off)
+and run Crop multiple times simultaneously, see @ref{How to run
+simultaneous operations}.
+@cindex Wall-clock time
+You can check the boost in speed by first running a program on one of the
+data sets with the maximum number of threads and another time (with
+everything else the same) and only using one thread. You will notice that
+the wall-clock time (reported by most programs at their end) in the former
+is longer than the latter divided by number of physical CPU cores (not
+threads) available to your operating system. Asymptotically these two times
+can be equal (most of the time they aren't). So limiting the programs to
+use only one thread and running them independently on the number of
+available threads will be more efficient.
+@cindex System Cache
+@cindex Cache, system
+Note that the operating system keeps a cache of recently processed
+data, so usually, the second time you process an identical data set
+(independent of the number of threads used), you will get faster
+results. In order to make an unbiased comparison, you have to first
+clean the system's cache with the following command between the two
+runs.
-@node Tables, Tessellation, Numeric data types, Common program behavior
-@section Tables
+@example
+$ sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
+@end example
-``A table is a collection of related data held in a structured format
-within a database. It consists of columns, and rows.'' (from
-Wikipedia). Each column in the table contains the values of one property
-and each row is a collection of properties (columns) for one target
-object. For example, let's assume you have just ran MakeCatalog (see
-@ref{MakeCatalog}) on an image to measure some properties for the labeled
-regions (which might be detected galaxies for example) in the image. For
-each labeled region (detected galaxy), there will be a @emph{row} which
-groups its measured properties as @emph{columns}, one column for each
-property. One such property can be the object's magnitude, which is the sum
-of pixels with that label, or its center can be defined as the
-light-weighted average value of those pixels. Many such properties can be
-derived from the raw pixel values and their position, see @ref{Invoking
-astmkcatalog} for a long list.
+@cartouche
+@noindent
+@strong{SUMMARY: Should I use multiple threads?} Depends:
+@itemize
+@item
+If you only have @strong{one} data set (image in most cases!), then
+yes, the more threads you use (with a maximum of the number of threads
+available to your OS) the faster you will get your results.
-As a summary, for each labeled region (or, galaxy) we have one @emph{row}
-and for each measured property we have one @emph{column}. This high-level
-structure is usually the first step for higher-level analysis, for example
-finding the stellar mass or photometric redshift from magnitudes in
-multiple colors. Thus, tables are not just outputs of programs, in fact it
-is much more common for tables to be inputs of programs. For example, to
-make a mock galaxy image, you need to feed in the properties of each galaxy
-into @ref{MakeProfiles} for it do the inverse of the process above and make
-a simulated image from a catalog, see @ref{Sufi simulates a detection}. In
-other cases, you can feed a table into @ref{Crop} and it will crop out
-regions centered on the positions within the table, see @ref{Hubble
-visually checks and classifies his catalog}. So to end this relatively long
-introduction, tables play a very important role in astronomy, or generally
-all branches of data analysis.
+@item
+If you want to run the same operation on @strong{multiple} data sets, it is
+best to set the number of threads to 1 and use Make, or GNU Parallel, as
+explained in @ref{How to run simultaneous operations}.
+@end itemize
+@end cartouche
-In @ref{Recognized table formats} the currently recognized table formats in
-Gnuastro are discussed. You can use any of these tables as input or ask for
-them to be built as output. The most common type of table format is a
-simple plain text file with each row on one line and columns separated by
-white space characters, this format is easy to read/write by eye/hand. To
-give it the full functionality of more specific table types like the FITS
-tables, Gnuastro has a special convention which you can use to give each
-column a name, type, unit, and comments, while still being readable by
-other plain text table readers. This convention is described in
-@ref{Gnuastro text table format}.
-When tables are input to a program, the program reading it needs to know
-which column(s) it should use for its desired purposes. Gnuastro's programs
-all follow a similar convention, on the way you can select columns in a
-table. They are thoroughly discussed in @ref{Selecting table columns}.
-@menu
-* Recognized table formats:: Table formats that are recognized in Gnuastro.
-* Gnuastro text table format:: Gnuastro's convention plain text tables.
-* Selecting table columns:: Identify/select certain columns from a table
-@end menu
-@node Recognized table formats, Gnuastro text table format, Tables, Tables
-@subsection Recognized table formats
+@node How to run simultaneous operations, , A note on threads, Multi-threaded
operations
+@subsection How to run simultaneous operations
-The list of table formats that Gnuastro can currently read from and write
-to are described below. Each has their own advantage and disadvantages, so a
-short review of the format is also provided to help you make the best
-choice based on how you want to define your input tables or later use your
-output tables.
+There are two@footnote{A third way would be to open multiple terminal
+emulator windows in your GUI, type the commands separately on each and
+press @key{Enter} once on each terminal, but this is far too frustrating,
+tedious and prone to errors. It's therefore not a realistic solution when
+tens, hundreds or thousands of operations (your research targets,
+multiplied by the operations you do on each) are to be done.} approaches to
+simultaneously execute a program: using GNU Parallel or Make (GNU Make is
+the most common implementation). The first is very useful when you only
+want to do one job multiple times and want to get back to your work without
+actually keeping the command you ran. The second is usually for more
+important operations, with lots of dependencies between the different
+products (for example a full scientific research).
@table @asis
-@item Plain text table
-This is the most basic and simplest way to create, view, or edit the table
-by hand on a text editor. The other formats described below are less
-eye-friendly and have a more formal structure (for easier computer
-readability). It is fully described in @ref{Gnuastro text table format}.
+@item GNU Parallel
+@cindex GNU Parallel
+When you only want to run multiple instances of a command on different
+threads and get on with the rest of your work, the best method is to
+use GNU parallel. Surprisingly GNU Parallel is one of the few GNU
+packages that has no Info documentation but only a Man page, see
+@ref{Info}. So to see the documentation after installing it please run
-@cindex FITS Tables
-@cindex Tables FITS
-@cindex ASCII table, FITS
-@item FITS ASCII tables
-The FITS ASCII table extension is fully in ASCII encoding and thus easily
-readable on any text editor (assuming it is the only extension in the FITS
-file). If the FITS file also contains binary extensions (for example an
-image or binary table extensions), then there will be many hard to print
-characters. The FITS ASCII format doesn't have new line characters to
-separate rows. In the FITS ASCII table standard, each row is defined as a
-fixed number of characters (value to the @code{NAXIS1} keyword), so to
-visually inspect it properly, you would have to adjust your text editor's
-width to this value. All columns start at given character positions and
-have a fixed width (number of characters).
-
-Numbers in a FITS ASCII table are printed into ASCII format, they are not
-in binary (that the CPU uses). Hence, they can take a larger space in
-memory, loose their precision, and take longer to read into memory. If you
-are dealing with integer type columns (see @ref{Numeric data types}),
-another issue with FITS ASCII tables is that the type information for the
-column will be lost (there is only one integer type in FITS ASCII
-tables). One problem with the binary format on the other hand is that it
-isn't portable (different CPUs/compilers) have different standards for
-translating the zeros and ones. But since ASCII characters are defined on a
-byte and are well recognized, they are better for portability on those
-various systems. Gnuastro's plain text table format described below is much
-more portable and easier to read/write/interpret by humans manually.
-
-Generally, as the name implies, this format is useful for when your table
-mainly contains ASCII columns (for example file names, or
-descriptions). They can be useful when you need to include columns with
-structured ASCII information along with other extensions in one FITS
-file. In such cases, you can also consider header keywords (see
-@ref{Fits}).
-
-@cindex Binary table, FITS
-@item FITS binary tables
-The FITS binary table is the FITS standard's solution to the issues
-discussed with keeping numbers in ASCII format as described under the FITS
-ASCII table title above. Only columns defined as a string type (a string of
-ASCII characters) are readable in a text editor. The portability problem
-with binary formats discussed above is mostly solved thanks to the
-portability of CFITSIO (see @ref{CFITSIO}) and the very long history of the
-FITS format which has been widely used since the 1970s.
-
-In the case of most numbers, storing them in binary format is more memory
-efficient than ASCII format. For example, to store @code{-25.72034} in
-ASCII format, you need 9 bytes/characters. But if you keep this same number
-(to the approximate precision possible) as a 4-byte (32-bit) floating point
-number, you can keep/transmit it with less than half the amount of
-memory. When catalogs contain thousands/millions of rows in tens/hundreds
-of columns, this can lead to significant improvements in memory/band-width
-usage. Moreover, since the CPU does its operations in the binary formats,
-reading the table in and writing it out is also much faster than an ASCII
-table.
-
-When you are dealing with integer numbers, the compression ratio can be
-even better, for example if you know all of the values in a column are
-positive and less than @code{255}, you can use the @code{unsigned char}
-type which only takes one byte! If they are between @code{-128} and
-@code{127}, then you can use the (signed) @code{char} type. So if you are
-thoughtful about the limits of your integer columns, you can greatly reduce
-the size of your file and also the speed at which it is read/written. This
-can be very useful when sharing your results with collaborators or
-publishing them. To decrease the file size even more you can name your
-output as ending in @file{.fits.gz} so it is also compressed after
-creation. Just note that compression/decompressing is CPU intensive and can
-slow down the writing/reading of the file.
+@example
+$ man parallel
+@end example
+@noindent
+As an example, let's assume we want to crop a region fixed on the
+pixels (500, 600) with the default width from all the FITS images in
+the @file{./data} directory ending with @file{sci.fits} to the current
+directory. To do this, you can run:
-Fortunately the FITS Binary table format also accepts ASCII strings as
-column types (along with the various numerical types). So your dataset can
-also contain non-numerical columns.
+@example
+$ parallel astcrop --numthreads=1 --xc=500 --yc=600 ::: \
+ ./data/*sci.fits
+@end example
-@end table
+@noindent
+GNU Parallel can help in many more conditions, this is one of the
+simplest, see the man page for lots of other examples. For absolute
+beginners: the backslash (@command{\}) is only a line breaker to fit
+nicely in the page. If you type the whole command in one line, you
+should remove it.
-@menu
-* Gnuastro text table format:: Reading plain text tables
-@end menu
+@item Make
+@cindex Make
+Make is a program for building ``targets'' (e.g., files) using ``recipes''
+(a set of operations) when their known ``prerequisites'' (other files) have
+been updated. It elegantly allows you to define dependency structures for
+building your final output and updating it efficiently when the inputs
+change. It is the most common infra-structure to build software
+today.
-@node Gnuastro text table format, Selecting table columns, Recognized table
formats, Tables
-@subsection Gnuastro text table format
+Scientific research methodology is very similar to software development:
+you start by testing a hypothesis on a small sample of objects/targets with
+a simple set of steps. As you are able to get promising results, you
+improve the method and use it on a larger, more general, sample. In the
+process, you will confront many issues that have to be corrected (bugs in
+software development jargon). Make a wonderful tool to manage this style of
+development. It has been used to make reproducible papers, for example see
+@url{https://gitlab.com/makhlaghi/NoiseChisel-paper, the reproduction
+pipeline} of the paper introducing @ref{NoiseChisel} (one of Gnuastro's
+programs).
-Plain text files are the most generic, portable, and easiest way to
-(manually) create, (visually) inspect, or (manually) edit a table. In this
-format, the ending of a row is defined by the new-line character (a line on
-a text editor). So when you view it on a text editor, every row will occupy
-one line. The delimiters (or characters separating the columns) are white
-space characters (space, horizontal tab, vertical tab) and a comma
-(@key{,}). The only further requirement is that all rows/lines must have
-the same number of columns.
+@cindex GNU Make
+GNU Make@footnote{@url{https://www.gnu.org/software/make/}} is the most
+common implementation which (similar to nearly all GNU programs, comes with
+a wonderful
+manual@footnote{@url{https://www.gnu.org/software/make/manual/}}). Make is
+very basic and simple, and thus the manual is short (the most important
+parts are in the first roughly 100 pages) and easy to read/understand.
-The columns don't have to be exactly under each other and the rows can be
-arbitrarily long with different lengths. For example the following contents
-in a file would be interpreted as a table with 4 columns and 2 rows, with
-each element interpreted as a @code{double} type (see @ref{Numeric data
-types}).
+Make comes with a @option{--jobs} (@option{-j}) option which allows you to
+specify the maximum number of jobs that can be done simultaneously. For
+example if you have 8 threads available to your operating system. You can
+run:
@example
-1 2.234948 128 39.8923e8
-2 , 4.454 792 72.98348e7
+$ make -j8
@end example
-However, the example above has no other information about the columns (it
-is just raw data, with no meta-data). To use this table, you have to
-remember what the numbers in each column represent. Also, when you want to
-select columns, you have to count their position within the table. This can
-become frustrating and prone to bad errors (getting the columns wrong)
-especially as the number of columns increase. It is also bad for sending to
-a colleague, because they will find it hard to remember/use the columns
-properly.
+With this command, Make will process your @file{Makefile} and create all
+the targets (can be thousands of FITS images for example) simultaneously on
+8 threads, while fully respecting their dependencies (only building a
+file/target when its prerequisites are successfully built). Make is thus
+strongly recommended for managing scientific research where robustness,
+archiving, reproducibility and speed@footnote{Besides its multi-threaded
+capabilities, Make will only re-build those targets that depend on a change
+you have made, not the whole work. For example, if you have set the
+prerequisites properly, you can easily test the changing of a parameter on
+your paper's results without having to re-do everything (which is much
+faster). This allows you to be much more productive in easily checking
+various ideas/assumptions of the different stages of your research and thus
+produce a more robust result for your exciting science.} are important.
-To solve these problems in Gnuastro's programs/libraries you aren't limited
-to using the column's number, see @ref{Selecting table columns}. If the
-columns have names, units, or comments you can also select your columns
-based on searches/matches in these fields, for example see @ref{Table}.
-Also, in this manner, you can't guide the program reading the table on how
-to read the numbers. As an example, the first and third columns above can
-be read as integer types: the first column might be an ID and the third can
-be the number of pixels an object occupies in an image. So there is no need
-to read these to columns as a @code{double} type (which takes more memory,
-and is slower).
+@end table
-In the bare-minimum example above, you also can't use strings of
-characters, for example the names of filters, or some other identifier that
-includes non-numerical characters. In the absence of any information, only
-numbers can be read robustly. Assuming we read columns with non-numerical
-characters as string, there would still be the problem that the strings
-might contain space (or any delimiter) character for some rows. So, each
-`word' in the string will be interpreted as a column and the program will
-abort with an error that the rows don't have the same number of columns.
-To correct for these limitations, Gnuastro defines the following convention
-for storing the table meta-data along with the raw data in one plain text
-file. The format is primarily designed for ease of reading/writing by
-eye/fingers, but is also structured enough to be read by a program.
-When the first non-white character in a line is @key{#}, or there are no
-non-white characters in it, then the line will not be considered as a row
-of data in the table (this is a pretty standard convention in many
-programs, and higher level languages). In the former case, the line is
-interpreted as a @emph{comment}. If the comment line starts with `@code{#
-Column N:}', then it is assumed to contain information about column
-@code{N} (a number, counting from 1). Comment lines that don't start with
-this pattern are ignored and you can use them to include any further
-information you want to store with the table in the text file. A column
-information comment is assumed to have the following format:
-@example
-# Column N: NAME [UNIT, TYPE, BLANK] COMMENT
-@end example
-@cindex NaN
-@noindent
-Any sequence of characters between `@key{:}' and `@key{[}' will be
-interpreted as the column name (so it can contain anything except the
-`@key{[}' character). Anything between the `@key{]}' and the end of the
-line is defined as a comment. Within the brackets, anything before the
-first `@key{,}' is the units (physical units, for example km/s, or erg/s),
-anything before the second `@key{,}' is the short type identifier (see
-below, and @ref{Numeric data types}). Finally (still within the brackets),
-any non-white characters after the second `@key{,}' are interpreted as the
-blank value for that column (see @ref{Blank pixels}). Note that blank
-values will be stored in the same type as the column, not as a
-string@footnote{For floating point types, the @code{nan}, or @code{inf}
-strings (both not case-sensitive) refer to IEEE NaN (not a number) and
-infinity values respectively and will be stored as a floating point, so
-they are acceptable.}.
+@node Numeric data types, Tables, Multi-threaded operations, Common program
behavior
+@section Numeric data types
-When a formatting problem occurs (for example you have specified the wrong
-type code, see below), or the the column was already given meta-data in a
-previous comment, or the column number is larger than the actual number of
-columns in the table (the non-commented or empty lines), then the comment
-information line will be ignored.
+@cindex Bit
+@cindex Type
+At the lowest level, the computer stores everything in terms of @code{1} or
+@code{0}. For example, each program in Gnuastro, or each astronomical image
+you take with the telescope is actually a string of millions of these zeros
+and ones. The space required to keep a zero or one is the smallest unit of
+storage, and is known as a @emph{bit}. However, understanding and
+manipulating this string of bits is extremely hard for most
+people. Therefore, we define packages of these bits along with a standard
+on how to interpret the bits in each package as a @emph{type}.
-When a comment information line can be used, the leading and trailing white
-space characters will be stripped from all of the elements. For example in
-this line:
+@cindex Byte
+@cindex Signed integer
+@cindex Unsigned integer
+@cindex Integer, Signed
+The most basic standard for reading the bits is integer numbers
+(@mymath{..., -2, -1, 0, 1, 2, ...}, more bits will give larger
+limits). The common integer types are 8, 16, 32, and 64 bits wide. For each
+width, there are two standards for reading the bits: signed and unsigned
+integers. In the former, negative numbers are allowed and in the latter,
+they aren't. The @code{unsigned} types thus have larger positive limits
+(one extra bit), but no negative value. When the context of your work
+doesn't involve negative numbers (for example counting, where negative is
+not defined), it is best to use the @code{unsigned} types. For full
+numerical range of all integer types, see below.
-@example
-# Column 5: column name [km/s, f32,-99] Redshift as speed
-@end example
+Another standard of converting a given number of bits to numbers is the
+floating point standard, this standard can approximately store any real
+number with a given precision. There are two common floating point types:
+32-bit and 64-bit, for single and double precision floating point numbers
+respectively. The former is sufficient for data with less than 8
+significant decimal digits (most astronomical data), while the latter is
+good for less than 16 significant decimal digits. The representation of
+real numbers as bits is much more complex than integers. If you are
+interested, you can start with the
+@url{https://en.wikipedia.org/wiki/Floating_point, Wikipedia article}.
-The @code{NAME} field will be `@code{column name}' and the @code{TYPE}
-field will be `@code{f32}'. Note how all the white space characters before
-and after strings are not used, but those in the middle remained. Also,
-white space characters aren't mandatory. Hence, in the example above, the
-@code{BLANK} field will be given the value of `@code{-99}'.
+With the conversion operators in Gnuastro's Arithmetic, you can change the
+types of data to each other, which is necessary in some contexts. For
+example the program/library, that you intend to feed the data into, only
+accepts floating point values, but you have an integer image. Another
+situation that conversion can be helpful is when you know that your data
+only has values that fit within @code{int8} or @code{uint16}. However it is
+currently formatted in the @code{float64} type. Operations involving
+floating point or larger integer types are significantly slower than
+integer or smaller-width types respectively. In the latter case, it also
+requires much more (by 8 or 4 times in the example above) storage space. So
+when you confront such situations and want to store/archive/transfter the
+data, it is best convert them to the most efficient type.
-Except for the column number (@code{N}), the rest of the fields are
-optional. Also, the column information comments don't have to be in
-order. In other words, the information for column @mymath{N+m}
-(@mymath{m>0}) can be given in a line before column @mymath{N}. Also, you
-don't have to specify information for all columns. Those columns that don't
-have this information will be interpreted with the default settings (like
-the case above: values are double precision floating point, and the column
-has no name, unit, or comment). So these lines are all acceptable for any
-table (the first one, with nothing but the column number is redundant):
+The short and long names for the recognized numeric data types in Gnuastro
+are listed below. Both short and long names can be used when you want to
+specify a type. For example, as a value to the common option
+@option{--type} (see @ref{Input output options}), or in the information
+comment lines of @ref{Gnuastro text table format}. The ranges listed below
+are inclusive.
-@example
-# Column 5:
-# Column 1: ID [,i] The Clump ID.
-# Column 3: mag_f160w [AB mag, f] Magnitude from the F160W filter
-@end example
+@table @code
+@item u8
+@itemx uint8
+8-bit unsigned integers, range:@*
+@mymath{[0\rm{\ to\ }2^8-1]} or @mymath{[0\rm{\ to\ }255]}.
-@noindent
-The data type of the column should be specified with one of the following
-values:
+@item i8
+@itemx int8
+8-bit signed integers, range:@*
+@mymath{[-2^7\rm{\ to\ }2^7-1]} or @mymath{[-128\rm{\ to\ }127]}.
-@itemize
-@item
-For a numeric column, you can use any of the numeric types (and their
-recognized identifiers) described in @ref{Numeric data types}.
-@item
-`@code{strN}': for strings. The @code{N} value identifies the length of the
-string (how many characters it has). The start of the string on each row is
-the first non-delimiter character of the column that has the string
-type. The next @code{N} characters will be interpreted as a string and all
-leading and trailing white space will be removed.
+@item u16
+@itemx uint16
+16-bit unsigned integers, range:@*
+@mymath{[0\rm{\ to\ }2^{16}-1]} or @mymath{[0\rm{\ to\ }65535]}.
-If the next column's characters, are closer than @code{N} characters to the
-start of the string column in that line/row, they will be considered part
-of the string column. If there is a new-line character before the ending of
-the space given to the string column (in other words, the string column is
-the last column), then reading of the string will stop, even if the
-@code{N} characters are not complete yet. See @file{tests/table/table.txt}
-for one example. Therefore, the only time you have to pay attention to the
-positioning and spaces given to the string column is when it is not the
-last column in the table.
+@item i16
+@itemx int16
+16-bit signed integers, range:@* @mymath{[-2^{15}\rm{\ to\ }2^{15}-1]} or
+@mymath{[-32768\rm{\ to\ }32767]}.
-The only limitation in this format is that trailing and leading white space
-characters will be removed from the columns that are read. In most cases,
-this is the desired behavior, but if trailing and leading white-spaces are
-critically important to your analysis, define your own starting and ending
-characters and remove them after the table has been read. For example in
-the sample table below, the two `@key{|}' characters (which are arbitrary)
-will remain in the value of the second column and you can remove them
-manually later. If only one of the leading or trailing white spaces is
-important for your work, you can only use one of the `@key{|}'s.
+@item u32
+@itemx uint32
+32-bit unsigned integers, range:@* @mymath{[0\rm{\ to\ }2^{32}-1]} or
+@mymath{[0\rm{\ to\ }4294967295]}.
-@example
-# Column 1: ID [label, uc]
-# Column 2: Notes [no unit, str50]
-1 leading and trailing white space is ignored here 2.3442e10
-2 | but they will be preserved here | 8.2964e11
-@end example
+@item i32
+@itemx int32
+32-bit signed integers, range:@* @mymath{[-2^{31}\rm{\ to\ }2^{31}-1]} or
+@mymath{[-2147483648\rm{\ to\ }2147483647]}.
-@end itemize
+@item u64
+@itemx uint64
+64-bit unsigned integers, range@* @mymath{[0\rm{\ to\ }2^{64}-1]} or
+@mymath{[0\rm{\ to\ }18446744073709551615]}.
-Note that the FITS binary table standard does not define the @code{unsigned
-int} and @code{unsigned long} types, so if you want to convert your tables
-to FITS binary tables, use other types. Also, note that in the FITS ASCII
-table, there is only one integer type (@code{long}). So if you convert a
-Gnuastro plain text table to a FITS ASCII table with the @ref{Table}
-program, the type information for integers will be lost. Conversely if
-integer types are important for you, you have to manually set them when
-reading a FITS ASCII table (for example with the Table program when
-reading/converting into a file, or with the @file{gnuastro/table.h} library
-functions when reading into memory).
+@item i64
+@itemx int64
+64-bit signed integers, range:@* @mymath{[-2^{63}\rm{\ to\ }2^{63}-1]} or
+@mymath{[-9223372036854775808\rm{\ to\ }9223372036854775807]}.
+@item f32
+@itemx float32
+32-bit (single-precision) floating point types. The maximum (minimum is its
+negative) possible value is
+@mymath{3.402823\times10^{38}}. Single-precision floating points can
+accurately represent a floating point number up to @mymath{\sim7.2}
+significant decimals. Given the heavy noise in astronomical data, this is
+usually more than sufficient for storing results.
-@node Selecting table columns, , Gnuastro text table format, Tables
-@subsection Selecting table columns
+@item f64
+@itemx float64
+64-bit (double-precision) floating point types. The maximum (minimum is its
+negative) possible value is @mymath{\sim10^{308}}. Double-precision
+floating points can accurately represent a floating point number
+@mymath{\sim15.9} significant decimals. This is usually good for processing
+(mixing) the data internally, for example a sum of single precision data
+(and later storing the result as @code{float32}).
+@end table
-At the lowest level, the only defining aspect of a column in a table is its
-number, or position. But selecting columns purely by number is not very
-convenient and, especially when the tables are large it can be very
-frustrating and prone to errors. Hence, table file formats (for example see
-@ref{Recognized table formats}) have ways to store additional information
-about the columns (meta-data). Some of the most common pieces of
-information about each column are its @emph{name}, the @emph{units} of data
-in the it, and a @emph{comment} for longer/informal description of the
-column's data.
+@cartouche
+@noindent
+@strong{Some file formats don't recognize all types.} Some file formats
+don't recognize all the types, for example the FITS standard (see
+@ref{Fits}) does not define @code{uint64} in binary tables or images. When
+a type is not acceptable for output into a given file format, the
+respective Gnuastro program or library will let you know and abort. On the
+command-line, you can use the @ref{Arithmetic} program to convert the
+numerical type of a dataset, in the libraries, you can call
+@code{gal_data_copy_to_new_type}.
+@end cartouche
-To facilitate research with Gnuastro, you can select columns by matching,
-or searching in these three fields, besides the low-level column number. To
-view the full list of information on the columns in the table, you can use
-the Table program (see @ref{Table}) with the command below (replace
-@file{table-file} with the filename of your table, if its FITS, you might
-also need to specify the HDU/extension which contains the table):
-@example
-$ asttable --information table-file
-@end example
-Gnuastro's programs need the columns for different purposes, for example in
-Crop, you specify the columns containing the central coordinates of the
-crop centers with the @option{--coordcol} option (see @ref{Crop
-options}). On the other hand, in MakeProfiles, to specify the column
-containing the profile position angles, you must use the @option{--pcol}
-option (see @ref{MakeProfiles catalog}). Thus, there can be no unified
-common option name to select columns for all programs (different columns
-have different purposes). However, when the program expects a column for a
-specific context, the option names end in the @option{col} suffix like the
-examples above. These options accept values in integer (column number), or
-string (metadata match/search) format.
+@node Tables, Tessellation, Numeric data types, Common program behavior
+@section Tables
-If the value can be parsed as a positive integer, it will be seen as the
-low-level column number. Note that column counting starts from 1, so if you
-ask for column 0, the respective program will abort with an error. When the
-value can't be interpreted as an a integer number, it will be seen as a
-string of characters which will be used to match/search in the table's
-meta-data. The meta-data field which the value will be compared with can be
-selected through the @option{--searchin} option, see @ref{Input output
-options}. @option{--searchin} can take three values: @code{name},
-@code{unit}, @code{comment}. The matching will be done following this
-convention:
+``A table is a collection of related data held in a structured format
+within a database. It consists of columns, and rows.'' (from
+Wikipedia). Each column in the table contains the values of one property
+and each row is a collection of properties (columns) for one target
+object. For example, let's assume you have just ran MakeCatalog (see
+@ref{MakeCatalog}) on an image to measure some properties for the labeled
+regions (which might be detected galaxies for example) in the image. For
+each labeled region (detected galaxy), there will be a @emph{row} which
+groups its measured properties as @emph{columns}, one column for each
+property. One such property can be the object's magnitude, which is the sum
+of pixels with that label, or its center can be defined as the
+light-weighted average value of those pixels. Many such properties can be
+derived from the raw pixel values and their position, see @ref{Invoking
+astmkcatalog} for a long list.
-@itemize
-@item
-If the value is enclosed in two slashes (for example @command{-x/RA_/}, or
-@option{--coordcol=/RA_/}, see @ref{Crop options}), then it is assumed to
-be a regular expression with the same convention as GNU AWK. GNU AWK has a
-very well written
-@url{https://www.gnu.org/software/gawk/manual/html_node/Regexp.html,
-chapter} describing regular expressions, so we we will not continue
-discussing them here. Regular expressions are a very powerful tool in
-matching text and useful in many contexts. We thus strongly encourage
-reviewing this chapter for greatly improving the quality of your work in
-many cases, not just for searching column meta-data in Gnuastro.
+As a summary, for each labeled region (or, galaxy) we have one @emph{row}
+and for each measured property we have one @emph{column}. This high-level
+structure is usually the first step for higher-level analysis, for example
+finding the stellar mass or photometric redshift from magnitudes in
+multiple colors. Thus, tables are not just outputs of programs, in fact it
+is much more common for tables to be inputs of programs. For example, to
+make a mock galaxy image, you need to feed in the properties of each galaxy
+into @ref{MakeProfiles} for it do the inverse of the process above and make
+a simulated image from a catalog, see @ref{Sufi simulates a detection}. In
+other cases, you can feed a table into @ref{Crop} and it will crop out
+regions centered on the positions within the table, see @ref{Hubble
+visually checks and classifies his catalog}. So to end this relatively long
+introduction, tables play a very important role in astronomy, or generally
+all branches of data analysis.
-@item
-When the string isn't enclosed between `@key{/}'s, any column that exactly
-matches the given value in the given field will be selected.
-@end itemize
+In @ref{Recognized table formats} the currently recognized table formats in
+Gnuastro are discussed. You can use any of these tables as input or ask for
+them to be built as output. The most common type of table format is a
+simple plain text file with each row on one line and columns separated by
+white space characters, this format is easy to read/write by eye/hand. To
+give it the full functionality of more specific table types like the FITS
+tables, Gnuastro has a special convention which you can use to give each
+column a name, type, unit, and comments, while still being readable by
+other plain text table readers. This convention is described in
+@ref{Gnuastro text table format}.
-Note that in both cases, you can ignore the case of alphabetic characters
-with the @option{--ignorecase} option, see @ref{Input output options}. Also, in
-both cases, multiple columns may be selected with one call to this
-function. In this case, the order of the selected columns (with one call)
-will be the same order as they appear in the table.
+When tables are input to a program, the program reading it needs to know
+which column(s) it should use for its desired purposes. Gnuastro's programs
+all follow a similar convention, on the way you can select columns in a
+table. They are thoroughly discussed in @ref{Selecting table columns}.
+@menu
+* Recognized table formats:: Table formats that are recognized in Gnuastro.
+* Gnuastro text table format:: Gnuastro's convention plain text tables.
+* Selecting table columns:: Identify/select certain columns from a table
+@end menu
+@node Recognized table formats, Gnuastro text table format, Tables, Tables
+@subsection Recognized table formats
+The list of table formats that Gnuastro can currently read from and write
+to are described below. Each has their own advantage and disadvantages, so a
+short review of the format is also provided to help you make the best
+choice based on how you want to define your input tables or later use your
+output tables.
-@node Tessellation, Getting help, Tables, Common program behavior
-@section Tessellation
+@table @asis
-It is sometimes necessary to classify the elements in a dataset (for
-example pixels in an image) into a grid of individual, non-overlapping
-tiles. For example when background sky gradients are present in an image,
-you can define a tile grid over the image. When the tile sizes are set
-properly, the background's variation over each tile will be negligible,
-allowing you to measure (and subtract) it. In other cases (for example
-spatial domain convolution in Gnuastro, see @ref{Convolve}), it might
-simply be for speed of processing: each tile can be processed independently
-on a separate CPU thread. In the arts and mathematics, this process is
-formally known as @url{https://en.wikipedia.org/wiki/Tessellation,
-tessellation}.
+@item Plain text table
+This is the most basic and simplest way to create, view, or edit the table
+by hand on a text editor. The other formats described below are less
+eye-friendly and have a more formal structure (for easier computer
+readability). It is fully described in @ref{Gnuastro text table format}.
-The size of the regular tiles (in units of data-elements, or pixels in an
-image) can be defined with the @option{--tilesize} option. It takes
-multiple numbers (separated by a comma) which will be the length along the
-respective dimension (in FORTRAN/FITS dimension order). Divisions are also
-acceptable, but must result in an integer. For example
-@option{--tilesize=30,40} can be used for an image (a 2D dataset). The
-regular tile size along the first FITS axis (horizontal when viewed in SAO
-ds9) will be 30 pixels and along the second it will be 40 pixels. Ideally,
-@option{--tilesize} should be selected such that all tiles in the image
-have exactly the same size. In other words, that the dataset length in each
-dimension is divisible by the tile size in that dimension.
+@cindex FITS Tables
+@cindex Tables FITS
+@cindex ASCII table, FITS
+@item FITS ASCII tables
+The FITS ASCII table extension is fully in ASCII encoding and thus easily
+readable on any text editor (assuming it is the only extension in the FITS
+file). If the FITS file also contains binary extensions (for example an
+image or binary table extensions), then there will be many hard to print
+characters. The FITS ASCII format doesn't have new line characters to
+separate rows. In the FITS ASCII table standard, each row is defined as a
+fixed number of characters (value to the @code{NAXIS1} keyword), so to
+visually inspect it properly, you would have to adjust your text editor's
+width to this value. All columns start at given character positions and
+have a fixed width (number of characters).
-However, this is not always possible: the dataset can be any size and every
-pixel in it is valuable. In such cases, Gnuastro will look at the
-significance of the remainder length, if it is not significant (for example
-one or two pixels), then it will just increase the size of the first tile
-in the respective dimension and allow the rest of the tiles to have the
-required size. When the remainder is significant (for example one pixel
-less than the size along that dimension), the remainder will be added to
-one regular tile's size and the large tile will be cut in half and put in
-the two ends of the grid/tessellation. In this way, all the tiles in the
-central regions of the dataset will have the regular tile sizes and the
-tiles on the edge will be slightly larger/smaller depending on the
-remainder significance. The fraction which defines the remainder
-significance along all dimensions can be set through
-@option{--remainderfrac}.
+Numbers in a FITS ASCII table are printed into ASCII format, they are not
+in binary (that the CPU uses). Hence, they can take a larger space in
+memory, loose their precision, and take longer to read into memory. If you
+are dealing with integer type columns (see @ref{Numeric data types}),
+another issue with FITS ASCII tables is that the type information for the
+column will be lost (there is only one integer type in FITS ASCII
+tables). One problem with the binary format on the other hand is that it
+isn't portable (different CPUs/compilers) have different standards for
+translating the zeros and ones. But since ASCII characters are defined on a
+byte and are well recognized, they are better for portability on those
+various systems. Gnuastro's plain text table format described below is much
+more portable and easier to read/write/interpret by humans manually.
-The best tile size is directly related to the spatial properties of the
-property you want to study (for example, gradient on the image). In
-practice we assume that the gradient is not present over each tile. So if
-there is a strong gradient (for example in long wavelength ground based
-images) or the image is of a crowded area where there isn't too much blank
-area, you have to choose a smaller tile size. A larger mesh will give more
-pixels and and so the scatter in the results will be less (better
-statistics).
+Generally, as the name implies, this format is useful for when your table
+mainly contains ASCII columns (for example file names, or
+descriptions). They can be useful when you need to include columns with
+structured ASCII information along with other extensions in one FITS
+file. In such cases, you can also consider header keywords (see
+@ref{Fits}).
-@cindex CCD
-@cindex Amplifier
-@cindex Bias current
-@cindex Subaru Telescope
-@cindex Hyper Suprime-Cam
-@cindex Hubble Space Telescope (HST)
-For raw image processing, a single tessellation/grid is not sufficient. Raw
-images are the unprocessed outputs of the camera detectors. Modern
-detectors usually have multiple readout channels each with its own
-amplifier. For example the Hubble Space Telescope Advanced Camera for
-Surveys (ACS) has four amplifiers over its full detector area dividing the
-square field of view to four smaller squares. Ground based image detectors
-are not exempt, for example each CCD of Subaru Telescope's Hyper
-Suprime-Cam camera (which has 104 CCDs) has four amplifiers, but they have
-the same height of the CCD and divide the width by four parts.
+@cindex Binary table, FITS
+@item FITS binary tables
+The FITS binary table is the FITS standard's solution to the issues
+discussed with keeping numbers in ASCII format as described under the FITS
+ASCII table title above. Only columns defined as a string type (a string of
+ASCII characters) are readable in a text editor. The portability problem
+with binary formats discussed above is mostly solved thanks to the
+portability of CFITSIO (see @ref{CFITSIO}) and the very long history of the
+FITS format which has been widely used since the 1970s.
-@cindex Channel
-The bias current on each amplifier is different, and initial bias
-subtraction is not perfect. So even after subtracting the measured bias
-current, you can usually still identify the boundaries of different
-amplifiers by eye. See Figure 11(a) in Akhlaghi and Ichikawa (2015) for an
-example. This results in the final reduced data to have non-uniform
-amplifier-shaped regions with higher or lower background flux values. Such
-systematic biases will then propagate to all subsequent measurements we do
-on the data (for example photometry and subsequent stellar mass and star
-formation rate measurements in the case of galaxies).
+In the case of most numbers, storing them in binary format is more memory
+efficient than ASCII format. For example, to store @code{-25.72034} in
+ASCII format, you need 9 bytes/characters. But if you keep this same number
+(to the approximate precision possible) as a 4-byte (32-bit) floating point
+number, you can keep/transmit it with less than half the amount of
+memory. When catalogs contain thousands/millions of rows in tens/hundreds
+of columns, this can lead to significant improvements in memory/band-width
+usage. Moreover, since the CPU does its operations in the binary formats,
+reading the table in and writing it out is also much faster than an ASCII
+table.
-Therefore an accurate analysis requires a two layer tessellation: the top
-layer contains larger tiles, each covering one amplifier channel. For
-clarity we'll call these larger tiles ``channels''. The number of channels
-along each dimension is defined through the @option{--numchannels}. Each
-channel is then covered by its own individual smaller tessellation (with
-tile sizes determined by the @option{--tilesize} option). This will allow
-independent analysis of two adjacent pixels from different channels if
-necessary. If the image is processed or the detector only has one
-amplifier, you can set the number of channels in both dimension to 1.
+When you are dealing with integer numbers, the compression ratio can be
+even better, for example if you know all of the values in a column are
+positive and less than @code{255}, you can use the @code{unsigned char}
+type which only takes one byte! If they are between @code{-128} and
+@code{127}, then you can use the (signed) @code{char} type. So if you are
+thoughtful about the limits of your integer columns, you can greatly reduce
+the size of your file and also the speed at which it is read/written. This
+can be very useful when sharing your results with collaborators or
+publishing them. To decrease the file size even more you can name your
+output as ending in @file{.fits.gz} so it is also compressed after
+creation. Just note that compression/decompressing is CPU intensive and can
+slow down the writing/reading of the file.
-The final tessellation can be inspected on the image with the
-@option{--checktiles} option that is available to all programs which use
-tessellation for localized operations. When this option is called, a FITS
-file with a @file{_tiled.fits} suffix will be created along with the
-outputs, see @ref{Automatic output}. Each pixel in this image has the
-number of the tile that covers it. If the number of channels in any
-dimension are larger than unity, you will notice that the tile IDs are
-defined such that the first channels is covered first, then the second and
-so on. For the full list of processing-related common options (including
-tessellation options), please see @ref{Processing options}.
+Fortunately the FITS Binary table format also accepts ASCII strings as
+column types (along with the various numerical types). So your dataset can
+also contain non-numerical columns.
+@end table
+@menu
+* Gnuastro text table format:: Reading plain text tables
+@end menu
+@node Gnuastro text table format, Selecting table columns, Recognized table
formats, Tables
+@subsection Gnuastro text table format
-@node Getting help, Automatic output, Tessellation, Common program behavior
-@section Getting help
+Plain text files are the most generic, portable, and easiest way to
+(manually) create, (visually) inspect, or (manually) edit a table. In this
+format, the ending of a row is defined by the new-line character (a line on
+a text editor). So when you view it on a text editor, every row will occupy
+one line. The delimiters (or characters separating the columns) are white
+space characters (space, horizontal tab, vertical tab) and a comma
+(@key{,}). The only further requirement is that all rows/lines must have
+the same number of columns.
-@cindex Help
-@cindex Book formats
-@cindex Remembering options
-@cindex Convenient book formats
-Probably the first time you read this book, it is either in the PDF
-or HTML formats. These two formats are very convenient for when you
-are not actually working, but when you are only reading. Later on,
-when you start to use the programs and you are deep in the middle of
-your work, some of the details will inevitably be forgotten. Going to
-find the PDF file (printed or digital) or the HTML webpage is a major
-distraction.
+The columns don't have to be exactly under each other and the rows can be
+arbitrarily long with different lengths. For example the following contents
+in a file would be interpreted as a table with 4 columns and 2 rows, with
+each element interpreted as a @code{double} type (see @ref{Numeric data
+types}).
-@cindex Online help
-@cindex Command-line help
-GNU software have a very unique set of tools for aiding your memory on
-the command-line, where you are working, depending how much of it you
-need to remember. In the past, such command-line help was known as
-``online'' help, because they were literally provided to you `on'
-the command `line'. However, nowadays the word ``online'' refers to
-something on the internet, so that term will not be used. With this
-type of help, you can resume your exciting research without taking
-your hands off the keyboard.
+@example
+1 2.234948 128 39.8923e8
+2 , 4.454 792 72.98348e7
+@end example
-@cindex Installed help methods
-Another major advantage of such command-line based help routines is
-that they are installed with the software in your computer, therefore
-they are always in sync with the executable you are actually
-running. Three of them are actually part of the executable. You don't
-have to worry about the version of the book or program. If you rely
-on external help (a PDF in your personal print or digital archive or
-HTML from the official webpage) you have to check to see if their
-versions fit with your installed program.
+However, the example above has no other information about the columns (it
+is just raw data, with no meta-data). To use this table, you have to
+remember what the numbers in each column represent. Also, when you want to
+select columns, you have to count their position within the table. This can
+become frustrating and prone to bad errors (getting the columns wrong)
+especially as the number of columns increase. It is also bad for sending to
+a colleague, because they will find it hard to remember/use the columns
+properly.
-If you only need to remember the short or long names of the options,
-@option{--usage} is advised. If it is what the options do, then
-@option{--help} is a great tool. Man pages are also provided for those
-who are use to this older system of documentation. This full book is
-also available to you on the command-line in Info format. If none of
-these seems to resolve the problems, there is a mailing list which
-enables you to get in touch with experienced Gnuastro users. In the
-subsections below each of these methods are reviewed.
+To solve these problems in Gnuastro's programs/libraries you aren't limited
+to using the column's number, see @ref{Selecting table columns}. If the
+columns have names, units, or comments you can also select your columns
+based on searches/matches in these fields, for example see @ref{Table}.
+Also, in this manner, you can't guide the program reading the table on how
+to read the numbers. As an example, the first and third columns above can
+be read as integer types: the first column might be an ID and the third can
+be the number of pixels an object occupies in an image. So there is no need
+to read these to columns as a @code{double} type (which takes more memory,
+and is slower).
+In the bare-minimum example above, you also can't use strings of
+characters, for example the names of filters, or some other identifier that
+includes non-numerical characters. In the absence of any information, only
+numbers can be read robustly. Assuming we read columns with non-numerical
+characters as string, there would still be the problem that the strings
+might contain space (or any delimiter) character for some rows. So, each
+`word' in the string will be interpreted as a column and the program will
+abort with an error that the rows don't have the same number of columns.
-@menu
-* --usage:: View option names and value formats.
-* --help:: List all options with description.
-* Man pages:: Man pages generated from --help.
-* Info:: View complete book in terminal.
-* help-gnuastro mailing list:: Contacting experienced users.
-@end menu
+To correct for these limitations, Gnuastro defines the following convention
+for storing the table meta-data along with the raw data in one plain text
+file. The format is primarily designed for ease of reading/writing by
+eye/fingers, but is also structured enough to be read by a program.
-@node --usage, --help, Getting help, Getting help
-@subsection @option{--usage}
-@vindex --usage
-@cindex Usage pattern
-@cindex Mandatory arguments
-@cindex Optional and mandatory tokens
-If you give this option, the program will not run. It will only print a
-very concise message showing the options and arguments. Everything within
-square brackets (@option{[]}) is optional. For example here are the first
-and last two lines of Crop's @option{--usage} is shown:
+When the first non-white character in a line is @key{#}, or there are no
+non-white characters in it, then the line will not be considered as a row
+of data in the table (this is a pretty standard convention in many
+programs, and higher level languages). In the former case, the line is
+interpreted as a @emph{comment}. If the comment line starts with `@code{#
+Column N:}', then it is assumed to contain information about column
+@code{N} (a number, counting from 1). Comment lines that don't start with
+this pattern are ignored and you can use them to include any further
+information you want to store with the table in the text file. A column
+information comment is assumed to have the following format:
@example
-$ astcrop --usage
-Usage: astcrop [-Do?IPqSVW] [-d INT] [-h INT] [-r INT] [-w INT]
- [-x INT] [-y INT] [-c INT] [-p STR] [-N INT] [--deccol=INT]
- ....
- [--setusrconf] [--usage] [--version] [--wcsmode]
- [ASCIIcatalog] FITSimage(s).fits
+# Column N: NAME [UNIT, TYPE, BLANK] COMMENT
@end example
-There are no explanations on the options, just their short and long
-names shown separately. After the program name, the short format of
-all the options that don't require a value (on/off options) is
-displayed. Those that do require a value then follow in separate
-brackets, each displaying the format of the input they want, see
-@ref{Options}. Since all options are optional, they are shown in
-square brackets, but arguments can also be optional. For example in
-this example, a catalog name is optional and is only required in some
-modes. This is a standard method of displaying optional arguments for
-all GNU software.
+@cindex NaN
+@noindent
+Any sequence of characters between `@key{:}' and `@key{[}' will be
+interpreted as the column name (so it can contain anything except the
+`@key{[}' character). Anything between the `@key{]}' and the end of the
+line is defined as a comment. Within the brackets, anything before the
+first `@key{,}' is the units (physical units, for example km/s, or erg/s),
+anything before the second `@key{,}' is the short type identifier (see
+below, and @ref{Numeric data types}). Finally (still within the brackets),
+any non-white characters after the second `@key{,}' are interpreted as the
+blank value for that column (see @ref{Blank pixels}). Note that blank
+values will be stored in the same type as the column, not as a
+string@footnote{For floating point types, the @code{nan}, or @code{inf}
+strings (both not case-sensitive) refer to IEEE NaN (not a number) and
+infinity values respectively and will be stored as a floating point, so
+they are acceptable.}.
-@node --help, Man pages, --usage, Getting help
-@subsection @option{--help}
+When a formatting problem occurs (for example you have specified the wrong
+type code, see below), or the the column was already given meta-data in a
+previous comment, or the column number is larger than the actual number of
+columns in the table (the non-commented or empty lines), then the comment
+information line will be ignored.
-@vindex --help
-If the command-line includes this option, the program will not be
-run. It will print a complete list of all available options along with
-a short explanation. The options are also grouped by their
-context. Within each context, the options are sorted
-alphabetically. Since the options are shown in detail afterwards, the
-first line of the @option{--help} output shows the arguments and if
-they are optional or not, similar to @ref{--usage}.
+When a comment information line can be used, the leading and trailing white
+space characters will be stripped from all of the elements. For example in
+this line:
-In the @option{--help} output of all programs in Gnuastro, the
-options for each program are classified based on context. The first
-two contexts are always options to do with the input and output
-respectively. For example input image extensions or supplementary
-input files for the inputs. The last class of options is also fixed in
-all of Gnuastro, it shows operating mode options. Most of these
-options are already explained in @ref{Operating mode options}.
+@example
+# Column 5: column name [km/s, f32,-99] Redshift as speed
+@end example
-@cindex Long outputs
-@cindex Redirection of output
-@cindex Command-line, long outputs
-The help message will sometimes be longer than the vertical size of
-your terminal. If you are using a graphical user interface terminal
-emulator, you can scroll the terminal with your mouse, but we promised
-no mice distractions! So here are some suggestions:
+The @code{NAME} field will be `@code{column name}' and the @code{TYPE}
+field will be `@code{f32}'. Note how all the white space characters before
+and after strings are not used, but those in the middle remained. Also,
+white space characters aren't mandatory. Hence, in the example above, the
+@code{BLANK} field will be given the value of `@code{-99}'.
-@itemize
-@item
-@cindex Scroll command-line
-@cindex Command-line scroll
-@cindex @key{Shift + PageUP} and @key{Shift + PageDown}
-@key{Shift + PageUP} to scroll up and @key{Shift + PageDown} to scroll
-down. For most help output this should be enough. The problem is that
-it is limited by the number of lines that your terminal keeps in
-memory and that you can't scroll by lines, only by whole screens.
+Except for the column number (@code{N}), the rest of the fields are
+optional. Also, the column information comments don't have to be in
+order. In other words, the information for column @mymath{N+m}
+(@mymath{m>0}) can be given in a line before column @mymath{N}. Also, you
+don't have to specify information for all columns. Those columns that don't
+have this information will be interpreted with the default settings (like
+the case above: values are double precision floating point, and the column
+has no name, unit, or comment). So these lines are all acceptable for any
+table (the first one, with nothing but the column number is redundant):
-@item
-@cindex Pipe
-@cindex @command{less}
-Pipe to @command{less}. A pipe is a form of shell re-direction. The
-@command{less} tool in Unix-like systems was made exactly for such
-outputs of any length. You can pipe (@command{|}) the output of any
-program that is longer than the screen to it and then you can scroll
-through (up and down) with its many tools. For example:
@example
-$ astnoisechisel --help | less
+# Column 5:
+# Column 1: ID [,i] The Clump ID.
+# Column 3: mag_f160w [AB mag, f] Magnitude from the F160W filter
@end example
-@noindent
-Once you have gone through the text, you can quit @command{less} by
-pressing the @key{q} key.
+@noindent
+The data type of the column should be specified with one of the following
+values:
+@itemize
@item
-@cindex Save output to file
-@cindex Redirection of output
-Redirect to a file. This is a less convenient way, because you will
-then have to open the file in a text editor! You can do this with the
-shell redirection tool (@command{>}):
+For a numeric column, you can use any of the numeric types (and their
+recognized identifiers) described in @ref{Numeric data types}.
+@item
+`@code{strN}': for strings. The @code{N} value identifies the length of the
+string (how many characters it has). The start of the string on each row is
+the first non-delimiter character of the column that has the string
+type. The next @code{N} characters will be interpreted as a string and all
+leading and trailing white space will be removed.
+
+If the next column's characters, are closer than @code{N} characters to the
+start of the string column in that line/row, they will be considered part
+of the string column. If there is a new-line character before the ending of
+the space given to the string column (in other words, the string column is
+the last column), then reading of the string will stop, even if the
+@code{N} characters are not complete yet. See @file{tests/table/table.txt}
+for one example. Therefore, the only time you have to pay attention to the
+positioning and spaces given to the string column is when it is not the
+last column in the table.
+
+The only limitation in this format is that trailing and leading white space
+characters will be removed from the columns that are read. In most cases,
+this is the desired behavior, but if trailing and leading white-spaces are
+critically important to your analysis, define your own starting and ending
+characters and remove them after the table has been read. For example in
+the sample table below, the two `@key{|}' characters (which are arbitrary)
+will remain in the value of the second column and you can remove them
+manually later. If only one of the leading or trailing white spaces is
+important for your work, you can only use one of the `@key{|}'s.
+
@example
-$ astnoisechisel --help > filename.txt
+# Column 1: ID [label, uc]
+# Column 2: Notes [no unit, str50]
+1 leading and trailing white space is ignored here 2.3442e10
+2 | but they will be preserved here | 8.2964e11
@end example
+
@end itemize
-@cindex GNU Grep
-@cindex Searching text
-@cindex Command-line searching text
-In case you have a special keyword you are looking for in the help, you
-don't have to go through the full list. GNU Grep is made for this job. For
-example if you only want the list of options whose @option{--help} output
-contains the word ``axis'' in Crop, you can run the following command:
+Note that the FITS binary table standard does not define the @code{unsigned
+int} and @code{unsigned long} types, so if you want to convert your tables
+to FITS binary tables, use other types. Also, note that in the FITS ASCII
+table, there is only one integer type (@code{long}). So if you convert a
+Gnuastro plain text table to a FITS ASCII table with the @ref{Table}
+program, the type information for integers will be lost. Conversely if
+integer types are important for you, you have to manually set them when
+reading a FITS ASCII table (for example with the Table program when
+reading/converting into a file, or with the @file{gnuastro/table.h} library
+functions when reading into memory).
-@example
-$ astcrop --help | grep axis
-@end example
-@cindex @code{ARGP_HELP_FMT}
-@cindex Argp argument parser
-@cindex Customize @option{--help} output
-@cindex @option{--help} output customization
-If the output of this option does not fit nicely within the confines
-of your terminal, GNU does enable you to customize its output through
-the environment variable @code{ARGP_HELP_FMT}, you can set various
-parameters which specify the formatting of the help messages. For
-example if your terminals are wider than 70 spaces (say 100) and you
-feel there is too much empty space between the long options and the
-short explanation, you can change these formats by giving values to
-this environment variable before running the program with the
-@option{--help} output. You can define this environment variable in
-this manner:
-@example
-$ export ARGP_HELP_FMT=rmargin=100,opt-doc-col=20
-@end example
-@cindex @file{.bashrc}
-This will affect all GNU programs using GNU C library's @file{argp.h}
-facilities as long as the environment variable is in memory. You can
-see the full list of these formatting parameters in the ``Argp User
-Customization'' part of the GNU C library manual. If you are more
-comfortable to read the @option{--help} outputs of all GNU software in
-your customized format, you can add your customization (similar to
-the line above, without the @command{$} sign) to your @file{~/.bashrc}
-file. This is a standard option for all GNU software.
+@node Selecting table columns, , Gnuastro text table format, Tables
+@subsection Selecting table columns
+
+At the lowest level, the only defining aspect of a column in a table is its
+number, or position. But selecting columns purely by number is not very
+convenient and, especially when the tables are large it can be very
+frustrating and prone to errors. Hence, table file formats (for example see
+@ref{Recognized table formats}) have ways to store additional information
+about the columns (meta-data). Some of the most common pieces of
+information about each column are its @emph{name}, the @emph{units} of data
+in the it, and a @emph{comment} for longer/informal description of the
+column's data.
-@node Man pages, Info, --help, Getting help
-@subsection Man pages
-@cindex Man pages
-Man pages were the Unix method of providing command-line documentation
-to a program. With GNU Info, see @ref{Info} the usage of this method
-of documentation is highly discouraged. This is because Info provides
-a much more easier to navigate and read environment.
+To facilitate research with Gnuastro, you can select columns by matching,
+or searching in these three fields, besides the low-level column number. To
+view the full list of information on the columns in the table, you can use
+the Table program (see @ref{Table}) with the command below (replace
+@file{table-file} with the filename of your table, if its FITS, you might
+also need to specify the HDU/extension which contains the table):
-However, some operating systems require a man page for packages that
-are installed and some people are still used to this method of command
-line help. So the programs in Gnuastro also have Man pages which are
-automatically generated from the outputs of @option{--version} and
-@option{--help} using the GNU help2man program. So if you run
@example
-$ man programname
+$ asttable --information table-file
@end example
-@noindent
-You will be provided with a man page listing the options in the
-standard manner.
-
-
-
+Gnuastro's programs need the columns for different purposes, for example in
+Crop, you specify the columns containing the central coordinates of the
+crop centers with the @option{--coordcol} option (see @ref{Crop
+options}). On the other hand, in MakeProfiles, to specify the column
+containing the profile position angles, you must use the @option{--pcol}
+option (see @ref{MakeProfiles catalog}). Thus, there can be no unified
+common option name to select columns for all programs (different columns
+have different purposes). However, when the program expects a column for a
+specific context, the option names end in the @option{col} suffix like the
+examples above. These options accept values in integer (column number), or
+string (metadata match/search) format.
-@node Info, help-gnuastro mailing list, Man pages, Getting help
-@subsection Info
+If the value can be parsed as a positive integer, it will be seen as the
+low-level column number. Note that column counting starts from 1, so if you
+ask for column 0, the respective program will abort with an error. When the
+value can't be interpreted as an a integer number, it will be seen as a
+string of characters which will be used to match/search in the table's
+meta-data. The meta-data field which the value will be compared with can be
+selected through the @option{--searchin} option, see @ref{Input output
+options}. @option{--searchin} can take three values: @code{name},
+@code{unit}, @code{comment}. The matching will be done following this
+convention:
-@cindex GNU Info
-@cindex Command-line, viewing full book
-Info is the standard documentation format for all GNU software. It is
-a very useful command-line document viewing format, fully equipped
-with links between the various pages and menus and search
-capabilities. As explained before, the best thing about it is that it
-is available for you the moment you need to refresh your memory on any
-command-line tool in the middle of your work without having to take
-your hands off the keyboard. This complete book is available in Info
-format and can be accessed from anywhere on the command-line.
+@itemize
+@item
+If the value is enclosed in two slashes (for example @command{-x/RA_/}, or
+@option{--coordcol=/RA_/}, see @ref{Crop options}), then it is assumed to
+be a regular expression with the same convention as GNU AWK. GNU AWK has a
+very well written
+@url{https://www.gnu.org/software/gawk/manual/html_node/Regexp.html,
+chapter} describing regular expressions, so we we will not continue
+discussing them here. Regular expressions are a very powerful tool in
+matching text and useful in many contexts. We thus strongly encourage
+reviewing this chapter for greatly improving the quality of your work in
+many cases, not just for searching column meta-data in Gnuastro.
-To open the Info format of any installed programs or library on your
-system which has an Info format book, you can simply run the command
-below (change @command{executablename} to the executable name of the
-program or library):
+@item
+When the string isn't enclosed between `@key{/}'s, any column that exactly
+matches the given value in the given field will be selected.
+@end itemize
-@example
-$ info executablename
-@end example
+Note that in both cases, you can ignore the case of alphabetic characters
+with the @option{--ignorecase} option, see @ref{Input output options}. Also, in
+both cases, multiple columns may be selected with one call to this
+function. In this case, the order of the selected columns (with one call)
+will be the same order as they appear in the table.
-@noindent
-@cindex Learning GNU Info
-@cindex GNU software documentation
-In case you are not already familiar with it, run @command{$ info
-info}. It does a fantastic job in explaining all its capabilities its
-self. It is very short and you will become sufficiently fluent in
-about half an hour. Since all GNU software documentation is also
-provided in Info, your whole GNU/Linux life will significantly
-improve.
-@cindex GNU Emacs
-@cindex GNU C library
-Once you've become an efficient navigator in Info, you can go to any
-part of this book or any other GNU software or library manual, no
-matter how long it is, in a matter of seconds. It also blends nicely
-with GNU Emacs (a text editor) and you can search manuals while you
-are writing your document or programs without taking your hands off
-the keyboard, this is most useful for libraries like the GNU C
-library. To be able to access all the Info manuals installed in your
-GNU/Linux within Emacs, type @key{Ctrl-H + i}.
-To see this whole book from the beginning in Info, you can run
-@example
-$ info gnuastro
-@end example
-@noindent
-If you run Info with the particular program executable name, for
-example @file{astcrop} or @file{astnoisechisel}:
+@node Tessellation, Automatic output, Tables, Common program behavior
+@section Tessellation
-@example
-$ info astprogramname
-@end example
+It is sometimes necessary to classify the elements in a dataset (for
+example pixels in an image) into a grid of individual, non-overlapping
+tiles. For example when background sky gradients are present in an image,
+you can define a tile grid over the image. When the tile sizes are set
+properly, the background's variation over each tile will be negligible,
+allowing you to measure (and subtract) it. In other cases (for example
+spatial domain convolution in Gnuastro, see @ref{Convolve}), it might
+simply be for speed of processing: each tile can be processed independently
+on a separate CPU thread. In the arts and mathematics, this process is
+formally known as @url{https://en.wikipedia.org/wiki/Tessellation,
+tessellation}.
-@noindent
-you will be taken to the section titled ``Invoking ProgramName'' which
-explains the inputs and outputs along with the command-line options for
-that program. Finally, if you run Info with the official program name, for
-example Crop or NoiseChisel:
+The size of the regular tiles (in units of data-elements, or pixels in an
+image) can be defined with the @option{--tilesize} option. It takes
+multiple numbers (separated by a comma) which will be the length along the
+respective dimension (in FORTRAN/FITS dimension order). Divisions are also
+acceptable, but must result in an integer. For example
+@option{--tilesize=30,40} can be used for an image (a 2D dataset). The
+regular tile size along the first FITS axis (horizontal when viewed in SAO
+ds9) will be 30 pixels and along the second it will be 40 pixels. Ideally,
+@option{--tilesize} should be selected such that all tiles in the image
+have exactly the same size. In other words, that the dataset length in each
+dimension is divisible by the tile size in that dimension.
-@example
-$ info ProgramName
-@end example
+However, this is not always possible: the dataset can be any size and every
+pixel in it is valuable. In such cases, Gnuastro will look at the
+significance of the remainder length, if it is not significant (for example
+one or two pixels), then it will just increase the size of the first tile
+in the respective dimension and allow the rest of the tiles to have the
+required size. When the remainder is significant (for example one pixel
+less than the size along that dimension), the remainder will be added to
+one regular tile's size and the large tile will be cut in half and put in
+the two ends of the grid/tessellation. In this way, all the tiles in the
+central regions of the dataset will have the regular tile sizes and the
+tiles on the edge will be slightly larger/smaller depending on the
+remainder significance. The fraction which defines the remainder
+significance along all dimensions can be set through
+@option{--remainderfrac}.
-@noindent
-you will be taken to the top section which introduces the
-program. Note that in all cases, Info is not case sensitive.
+The best tile size is directly related to the spatial properties of the
+property you want to study (for example, gradient on the image). In
+practice we assume that the gradient is not present over each tile. So if
+there is a strong gradient (for example in long wavelength ground based
+images) or the image is of a crowded area where there isn't too much blank
+area, you have to choose a smaller tile size. A larger mesh will give more
+pixels and and so the scatter in the results will be less (better
+statistics).
+@cindex CCD
+@cindex Amplifier
+@cindex Bias current
+@cindex Subaru Telescope
+@cindex Hyper Suprime-Cam
+@cindex Hubble Space Telescope (HST)
+For raw image processing, a single tessellation/grid is not sufficient. Raw
+images are the unprocessed outputs of the camera detectors. Modern
+detectors usually have multiple readout channels each with its own
+amplifier. For example the Hubble Space Telescope Advanced Camera for
+Surveys (ACS) has four amplifiers over its full detector area dividing the
+square field of view to four smaller squares. Ground based image detectors
+are not exempt, for example each CCD of Subaru Telescope's Hyper
+Suprime-Cam camera (which has 104 CCDs) has four amplifiers, but they have
+the same height of the CCD and divide the width by four parts.
+@cindex Channel
+The bias current on each amplifier is different, and initial bias
+subtraction is not perfect. So even after subtracting the measured bias
+current, you can usually still identify the boundaries of different
+amplifiers by eye. See Figure 11(a) in Akhlaghi and Ichikawa (2015) for an
+example. This results in the final reduced data to have non-uniform
+amplifier-shaped regions with higher or lower background flux values. Such
+systematic biases will then propagate to all subsequent measurements we do
+on the data (for example photometry and subsequent stellar mass and star
+formation rate measurements in the case of galaxies).
-@node help-gnuastro mailing list, , Info, Getting help
-@subsection help-gnuastro mailing list
+Therefore an accurate analysis requires a two layer tessellation: the top
+layer contains larger tiles, each covering one amplifier channel. For
+clarity we'll call these larger tiles ``channels''. The number of channels
+along each dimension is defined through the @option{--numchannels}. Each
+channel is then covered by its own individual smaller tessellation (with
+tile sizes determined by the @option{--tilesize} option). This will allow
+independent analysis of two adjacent pixels from different channels if
+necessary. If the image is processed or the detector only has one
+amplifier, you can set the number of channels in both dimension to 1.
-@cindex help-gnuastro mailing list
-@cindex Mailing list: help-gnuastro
-Gnuastro maintains the help-gnuastro mailing list for users to ask any
-questions related to Gnuastro. The experienced Gnuastro users and some
-of its developers are subscribed to this mailing list and your email
-will be sent to them immediately. However, when contacting this
-mailing list please have in mind that they are possibly very busy and
-might not be able to answer immediately.
+The final tessellation can be inspected on the image with the
+@option{--checktiles} option that is available to all programs which use
+tessellation for localized operations. When this option is called, a FITS
+file with a @file{_tiled.fits} suffix will be created along with the
+outputs, see @ref{Automatic output}. Each pixel in this image has the
+number of the tile that covers it. If the number of channels in any
+dimension are larger than unity, you will notice that the tile IDs are
+defined such that the first channels is covered first, then the second and
+so on. For the full list of processing-related common options (including
+tessellation options), please see @ref{Processing options}.
-@cindex Mailing list archives
-@cindex @code{help-gnuastro@@gnu.org}
-To ask a question from this mailing list, send a mail to
-@code{help-gnuastro@@gnu.org}. Anyone can view the mailing list
-archives at @url{http://lists.gnu.org/archive/html/help-gnuastro/}. It
-is best that before sending a mail, you search the archives to see if
-anyone has asked a question similar to yours. If you want to make a
-suggestion or report a bug, please don't send a mail to this mailing
-list. We have other mailing lists and tools for those purposes, see
-@ref{Report a bug} or @ref{Suggest new feature}.
-@node Automatic output, Output FITS files, Getting help, Common program
behavior
+@node Automatic output, Output FITS files, Tessellation, Common program
behavior
@section Automatic output
+@cindex Standard input
@cindex Automatic output file names
@cindex Output file names, automatic
@cindex Setting output file names automatically
-All the programs in Gnuastro are designed such that specifying an
-output file or directory (based on the program context) is optional.
-The outputs of most programs are automatically found based on the
-input and what the program does. For example when you are converting a
-FITS image named @file{FITSimage.fits} to a JPEG image, the JPEG image
-will be saved in @file{FITSimage.jpg}.
+All the programs in Gnuastro are designed such that specifying an output
+file or directory (based on the program context) is optional. When no
+output name is explicitly given (with @option{--output}, see @ref{Input
+output options}), the programs will automatically set an output name based
+on the input name(s) and what the program does. For example when you are
+using ConvertType to save FITS image named @file{dataset.fits} to a JPEG
+image and don't specify a name for it, the JPEG output file will be name
+@file{dataset.jpg}. When the input is from the standard input (for example
+a pipe, see @ref{Standard input}), and @option{--output} isn't given, the
+output name will be the program's name (for example
+@file{converttype.jpg}).
@vindex --keepinputdir
-Another very important part of the automatic output generation is that
-all the directory information of the input file name is stripped off
-of it. This feature can be disabled with the @option{--keepinputdir}
-option, see @ref{Common options}. It is the default because
-astronomical data are usually very large and organized specially with
-special file names. In some cases, the user might not have write
-permissions in those directories. In fact, even if the data is stored
-on your own computer, it is advised to only grant write permissions to
-the super user or root. This way, you won't accidentally delete or
-modify your valuable data!
+Another very important part of the automatic output generation is that all
+the directory information of the input file name is stripped off of
+it. This feature can be disabled with the @option{--keepinputdir} option,
+see @ref{Input output options}. It is the default because astronomical data
+are usually very large and organized specially with special file names. In
+some cases, the user might not have write permissions in those
+directories@footnote{In fact, even if the data is stored on your own
+computer, it is advised to only grant write permissions to the super user
+or root. This way, you won't accidentally delete or modify your valuable
+data!}.
Let's assume that we are working on a report and want to process the
FITS images from two projects (ABC and DEF), which are stored in the
@@ -10131,24 +10293,23 @@ continue with the rest of actions.
@cindex Image format conversion
@cindex Converting image formats
@pindex @r{ConvertType (}astconvertt@r{)}
-The formats of astronomical data were defined mainly for archiving and
-processing. In other situations, the data might be useful in other
-formats. For example, when you are writing a paper or report or if you
-are making slides for a talk, you can't use a FITS image. Other image
-formats should be used. In other cases you might want your pixel
-values in a table format as plain text for input to other programs
-that don't recognize FITS, or to include as a table in your
-report. ConvertType is created for such situations. The various types
-will increase with future updates and based on need.
-
-The conversion is not only one way (from FITS to other formats), but
-two ways (except the EPS and PDF formats). So you can convert a JPEG
-image or text file into a FITS image. Basically, other than EPS, you
-can use any of the recognized formats as different color channel
-inputs to get any of the recognized outputs. So before explaining the
-options and arguments, first a short description of the recognized
-files types will be given followed a short introduction to digital
-color.
+The FITS format used in astronomy was defined mainly for archiving,
+transmission, and processing. In other situations, the data might be useful
+in other formats. For example, when you are writing a paper or report or if
+you are making slides for a talk, you can't use a FITS image. Other image
+formats should be used. In other cases you might want your pixel values in
+a table format as plain text for input to other programs that don't
+recognize FITS, or to include as a table in your report. ConvertType is
+created for such situations. The various types will increase with future
+updates and based on need.
+
+The conversion is not only one way (from FITS to other formats), but two
+ways (except the EPS and PDF formats). So you can convert a JPEG image or
+text file into a FITS image. Basically, other than EPS, you can use any of
+the recognized formats as different color channel inputs to get any of the
+recognized outputs. So before explaining the options and arguments, first a
+short description of the recognized files types will be given followed a
+short introduction to digital color.
@menu
* Recognized file formats:: Recognized file formats
@@ -10417,9 +10578,9 @@ are approximately the same file size.
@node Invoking astconvertt, , Color, ConvertType
@subsection Invoking ConvertType
-ConvertType will convert any recognized input file type to any
-specified output type. The executable name is @file{astconvertt} with
-the following general template
+ConvertType will convert any recognized input file type to any specified
+output type. The executable name is @file{astconvertt} with the following
+general template
@example
$ astconvertt [OPTION...] InputFile [InputFile2] ... [InputFile4]
@@ -10440,31 +10601,39 @@ $ astconvertt f1.txt f2.txt f3.fits -o.jpg
## Use two images and one blank for an RGB EPS output:
$ astconvertt M31_r.fits M31_g.fits blank -oeps
+
+## Directly pass input from output of another program through Standard
+## input (not a file).
+$ cat 2darray.txt | astconvertt -oimg.fits
@end example
@noindent
-The file type of the output will be specified with the (possibly
-complete) file name given to the @option{--output} option, which can
-either be given on the command-line or in any of the configuration
-files (see @ref{Configuration files}). Note that if the output suffix
-is not recognized, it will default to plain text format, see
-@ref{Recognized file formats}.
-
-The order of multiple input files is important. After reading the
-input file(s) the number of color channels in all the inputs will be
-used to define which color space is being used for the outputs and how
-each color channel is interpreted. Note that one file might have more
-than one color channel (for example in the JPEG format). If there is
-one color channel the output is gray-scale, if three input color
-channels are given they are respectively considered to be the red,
-green and blue color channels and if there are four color channels
-they are respectively considered to be cyan, magenta, yellow and
-black.
+The output's file format will be interpretted from the value given to the
+@option{--output} option. It can either be given on the command-line or in
+any of the configuration files (see @ref{Configuration files}). Note that
+if the output suffix is not recognized, it will default to plain text
+format, see @ref{Recognized file formats}.
+
+@cindex Standard input
+At most four input files (one for each color channel for formats that allow
+it) are allowed in ConvertType. The first input dataset can either be a
+file or come from Standard input (see @ref{Standard input}). The order of
+multiple input files is important. After reading the input file(s) the
+number of color channels in all the inputs will be used to define which
+color space to use for the outputs and how each color channel is
+interpreted.
+
+Some formats can allow more than one color channel (for example in the JPEG
+format, see @ref{Recognized file formats}). If there is one input dataset
+(color channel) the output will be gray-scale, if three input datasets
+(color channels) are given, they are respectively considered to be the red,
+green and blue color channels. Finally, if there are four color channels
+they will be be cyan, magenta, yellow and black (CMYK colors).
The value to @option{--output} (or @option{-o}) can be either a full file
name or just the suffix of the desired output format. In the former case,
-that same name will be used for the output. In the latter case, the name of
-the output file will be set based on the automatic output guidelines, see
+it will used for the output. In the latter case, the name of the output
+file will be set based on the automatic output guidelines, see
@ref{Automatic output}. Note that the suffix name can optionally start a
@file{.} (dot), so for example @option{--output=.jpg} and
@option{--output=jpg} are equivalent. See @ref{Recognized file formats}
@@ -10746,19 +10915,24 @@ $ asttable bintab.fits | awk '$10>10e5 @{print@}'
## Sort the output columns by the third column, save output:
$ asttable bintab.fits | 'sort -nk3 > output.txt
+## Subtract the first column from the second in `cat.txt' and keep the
+## third and fourth columns. Feed the columns to Table to write as a
+## FITS table.
+$ awk '!/^#/@{print $2-$1, $3, $4@}' cat.txt | asttable -ocat.fits
+
## Convert a plain text table to a binary FITS table:
$ asttable plaintext.txt --output=table.fits --tabletype=fits-binary
@end example
-@cindex GNU AWK
-In the absence of selected columns, all the input file's columns will be
-output. If the specified output is a FITS file, the type of FITS table
-(binary or ASCII) will be determined from the @option{--tabletype}
-option. If the output is not a FITS file, it will be printed as a plain
-text table (with space characters between the columns). When the columns
-are accompanied by meta-data (like column name, units, or comments), this
-information will also printed in the plain text file before the table, as
-described in @ref{Gnuastro text table format}.
+Table's input dataset can be given either as a file or from Standard input
+(see @ref{Standard input}). In the absence of selected columns, all the
+input's columns will be written to the output. If the specified output is a
+FITS file, the type of FITS table (binary or ASCII) will be determined from
+the @option{--tabletype} option. If the output is not a FITS file, it will
+be printed as a plain text table (with space characters between the
+columns). When the columns are accompanied by meta-data (like column name,
+units, or comments), this information will also printed in the plain text
+file before the table, as described in @ref{Gnuastro text table format}.
For the full list of options common to all Gnuastro programs please see
@ref{Common options}. Options can also be stored in directory, user or
@@ -15089,18 +15263,25 @@ $ aststatistics cat.fits -cMAG_F160W -g26 -l27
--sigmaclip=3,0.2
## Print the median value of all records in column MAG_F160W that
## have a value larger than 3 in column PHOTO_Z:
$ aststatistics tab.txt -rPHOTO_Z -g3 -cMAG_F160W --median
+
+## Calculate the median of the third column in the input table, but only
+## for rows where the mean of the first and second columns is >5.
+$ awk '($1+$2)/2 > 5 @{print $3@}' table.txt | aststatistics --median
@end example
@noindent
-An input image or table is necessary when processing is to be done. If any
-output file is to be created, the value to the @option{--output} option, is
-used as the base name for the generated files. Without @option{--output},
-the input name will be used to generate an output name, see @ref{Automatic
-output}. The options described below are particular to Statistics, but for
-general operations, it shares a large collection of options with the other
-Gnuastro programs, see @ref{Common options} for the full list. Options can
-also be given in configuration files, for more, please see
-@ref{Configuration files}.
+@cindex Standard input
+Statistics can take its input dataset either from a file (image or table)
+or the Standard input (see @ref{Standard input}). If any output file is to
+be created, the value to the @option{--output} option, is used as the base
+name for the generated files. Without @option{--output}, the input name
+will be used to generate an output name, see @ref{Automatic output}. The
+options described below are particular to Statistics, but for general
+operations, it shares a large collection of options with the other Gnuastro
+programs, see @ref{Common options} for the full list. For more on reading
+from standard input, please see the description of @code{--stdintimeout}
+option in @ref{Input output options}. Options can also be given in
+configuration files, for more, please see @ref{Configuration files}.
The input dataset may have blank values (see @ref{Blank pixels}), in this
case, all blank pixels are ignored during the calculation. Initially, the
@@ -15120,9 +15301,9 @@ for those.
@item -c STR/INT
@itemx --column=STR/INT
-The input column selector when the input file is a table. See
-@ref{Selecting table columns} for a full description of how to use this
-option. For more on how tables are read in Gnuastro, please see
+The column to use when the input file is a table with more than one
+column. See @ref{Selecting table columns} for a full description of how to
+use this option. For more on how tables are read in Gnuastro, please see
@ref{Tables}.
@item -r STR/INT
@@ -19265,12 +19446,15 @@ $ astmatch --ccol1=2,3,4 --ccol2=2,3,4
-a0.5/3600,0.5/3600,5e-10 \
@end example
Two inputs are necessary for Match to start processing. The inputs can be
-plain text tables or FITS tables, see @ref{Tables}. Match follows the same
-basic behavior of all Gnuastro programs as fully described in @ref{Common
-program behavior}. If the first input is a FITS file, the common
-@option{--hdu} option (see @ref{Input output options}) should be used to
-identify the extension. When the second input is FITS, the extension must
-be specified with @option{--hdu2}.
+plain text tables or FITS tables, see @ref{Tables}. If only one argument is
+provided, Match will assume look for the first input in Stanard input (see
+@ref{Standard input}).
+
+Match follows the same basic behavior of all Gnuastro programs as fully
+described in @ref{Common program behavior}. If the first input is a FITS
+file, the common @option{--hdu} option (see @ref{Input output options})
+should be used to identify the extension. When the second input is FITS,
+the extension must be specified with @option{--hdu2}.
When @option{--quiet} is not called, Match will print the number of matches
found in standard output (on the command-line). When matches are found, by
@@ -20149,10 +20333,11 @@ The parameters of the mock profiles can either be
given through a catalog
(which stores the parameters of many mock profiles, see @ref{MakeProfiles
catalog}), or the @option{--kernel} option (see @ref{MakeProfiles output
dataset}). The catalog can be in the FITS ASCII, FITS binary format, or
-plain text formats (see @ref{Tables}). The columns related to each
-parameter can be determined both by number, or by match/search criteria
-using the column names, units, or comments. with the options ending in
-@option{col}, see below.
+plain text formats (see @ref{Tables}). A plain text catalog can also be
+provided using the Standard input (see @ref{Standard input}). The columns
+related to each parameter can be determined both by number, or by
+match/search criteria using the column names, units, or comments. with the
+options ending in @option{col}, see below.
Without any file given to the @option{--background} option, MakeProfiles
will make a zero-valued image and build the profiles on that (its size and
@@ -20238,13 +20423,14 @@ Gnuastro's programs to make a complete simulated
image of a mock galaxy.
@node MakeProfiles catalog, MakeProfiles profile settings, Invoking astmkprof,
Invoking astmkprof
@subsubsection MakeProfiles catalog
The catalog containing information about each profile can be in the FITS
-ASCII, FITS binary, or plain text formats (see @ref{Tables}). Its columns
-can be ordered in any desired manner. You can specify which columns belong
-to which parameters using the set of options discussed below. For example
-through the @option{--rcol} and @option{--tcol} options, you can specify
-the column that contains the radial parameter for each profile and its
-truncation respectively. See @ref{Selecting table columns} for a thorough
-discussion on the values to these options.
+ASCII, FITS binary, or plain text formats (see @ref{Tables}). The latter
+can also be provided using standard input (see @ref{Standard input}). Its
+columns can be ordered in any desired manner. You can specify which columns
+belong to which parameters using the set of options discussed below. For
+example through the @option{--rcol} and @option{--tcol} options, you can
+specify the column that contains the radial parameter for each profile and
+its truncation respectively. See @ref{Selecting table columns} for a
+thorough discussion on the values to these options.
The value for the profile center in the catalog (the @option{--ccol}
option) can be a floating point number so the profile center can be on any
@@ -25074,42 +25260,51 @@ types for reading arrays which may contain multiple
extensions (for example
FITS or TIFF) formats.
@end deftypefun
-@deftypefun gal_data_t gal_array_read (char @code{*filename}, char
@code{*extension}, size_t @code{minmapsize})
+@deftypefun gal_data_t gal_array_read (char @code{*filename}, char
@code{*extension}, gal_list_str_t @code{*lines}, size_t @code{minmapsize})
Read the array within the given extension (@code{extension}) of
-@code{filename}. If the array is larger than @code{minmapsize} bytes, then
-it won't be read into RAM, but a file on the HDD/SSD (no difference for the
-programmer). @code{extension} will be ignored for files that don't support
-them (for example JPEG or text). For FITS files, @code{extension} can be a
-number or a string (name of the extension), but for TIFF files, it has to
-be number. In both cases, counting starts from zero.
+@code{filename}, or the @code{lines} list (see below). If the array is
+larger than @code{minmapsize} bytes, then it won't be read into RAM, but a
+file on the HDD/SSD (no difference for the programmer). @code{extension}
+will be ignored for files that don't support them (for example JPEG or
+text). For FITS files, @code{extension} can be a number or a string (name
+of the extension), but for TIFF files, it has to be number. In both cases,
+counting starts from zero.
For multi-channel formats (like RGB images in JPEG or TIFF), this function
will return a @ref{List of gal_data_t}: one data structure per
channel. Thus if you just want a single array (and want to check if the
user hasn't given a multi-channel input), you can check the @code{next}
pointer of the returned @code{gal_data_t}.
+
+@code{lines} is a list of strings with each node representing one line
+(including the new-line character), see @ref{List of strings}. It will
+mostly be the output of @code{gal_txt_stdin_read}, which is used to read
+the program's input as separate lines from the standard input (see
+@ref{Text files}). Note that @code{filename} and @code{lines} are mutually
+exclusive and one of them must be @code{NULL}.
@end deftypefun
-@deftypefun void gal_array_read_to_type (char @code{*filename}, char
@code{*extension}, uint8_t @code{type}, size_t @code{minmapsize})
+@deftypefun void gal_array_read_to_type (char @code{*filename}, char
@code{*extension}, gal_list_str_t @code{*lines}, uint8_t @code{type}, size_t
@code{minmapsize})
Similar to @code{gal_array_read}, but the output data structure(s) will
have a numeric data type of @code{type}, see @ref{Numeric data types}.
@end deftypefun
-@deftypefun void gal_array_read_one_ch (char @code{*filename}, char
@code{*extension}, size_t @code{minmapsize})
+@deftypefun void gal_array_read_one_ch (char @code{*filename}, char
@code{*extension}, gal_list_str_t @code{*lines}, size_t @code{minmapsize})
@cindex Channel
@cindex Color channel
Read the dataset within @code{filename} (extension/hdu/dir
-@code{extension}) and make sure it is only a single channel. Formats like
-JPEG or TIFF support multiple channels per input, but it may happen that
-your program only works on a single dataset. This function can be a
-convenient way to make sure that the data that comes into your program is
-only one channel. It is just a very simple wrapper around
-@code{gal_array_read} that checks if there was more than one dataset and
-aborts with an informative error if there is more than one channel in the
-dataset.
+@code{extension}) and make sure it is only a single channel. This is just a
+simple wrapper around @code{gal_array_read} that checks if there was more
+than one dataset and aborts with an informative error if there is more than
+one channel in the dataset.
+
+Formats like JPEG or TIFF support multiple channels per input, but it may
+happen that your program only works on a single dataset. This function can
+be a convenient way to make sure that the data that comes into your program
+is only one channel.
@end deftypefun
-@deftypefun void gal_array_read_one_ch_to_type (char @code{*filename}, char
@code{*extension}, uint8_t @code{type}, size_t @code{minmapsize})
+@deftypefun void gal_array_read_one_ch_to_type (char @code{*filename}, char
@code{*extension}, gal_list_str_t @code{*lines}, uint8_t @code{type}, size_t
@code{minmapsize})
Similar to @code{gal_array_read_one_ch}, but the output data structure will
has a numeric data type of @code{type}, see @ref{Numeric data types}.
@end deftypefun
@@ -25199,19 +25394,25 @@ string to match, or regular expression to search, be
in the @emph{name},
should be used for the @code{searchin} variables of the functions.
@end deffn
-@deftypefun {gal_data_t *} gal_table_info (char @code{*filename}, char
@code{*hdu}, size_t @code{*numcols}, size_t @code{*numrows}, int
@code{*tableformat})
-Store the information of each column in a table (either as a text file or
-as a FITS table) into an array of data structures with @code{numcols}
-structures (one data structure for each column). The number of rows is
-stored in the space that @code{numrows} points to. The format of the table
-(e.g., ascii text file, or FITS binary or ASCII table) will be put in
-@code{tableformat} (macros defined above). If the filename is not a FITS
-file, then @code{hdu} will not be used (can be @code{NULL}).
+@deftypefun {gal_data_t *} gal_table_info (char @code{*filename}, char
@code{*hdu}, gal_list_str_t @code{*lines}, size_t @code{*numcols}, size_t
@code{*numrows}, int @code{*tableformat})
+Store the information of each column in a table into an array of data
+structures with @code{numcols} datasets (one data structure for each
+column). The number of rows is stored in @code{numrows}. The format of the
+table (e.g., ascii text file, or FITS binary or ASCII table) will be put in
+@code{tableformat} (macros defined above). If the @code{filename} is not a
+FITS file, then @code{hdu} will not be used (can be @code{NULL}).
-Note that other than the character strings (column name, units and
-comments), nothing in the data structure(s) will be allocated by this
-function for the actual data (e.g., the `array' or `dsize' elements). This
-function is just for column information (meta-data), not column contents.
+The input must be either a file (specified by @code{filename}) or a list of
+strings (@code{lines}). @code{lines} is a list of strings with each node
+representing one line (including the new-line character), see @ref{List of
+strings}. It will mostly be the output of @code{gal_txt_stdin_read}, which
+is used to read the program's input as separate lines from the standard
+input (see @ref{Text files}). Note that @code{filename} and @code{lines}
+are mutually exclusive and one of them must be @code{NULL}.
+
+In the output datasets, only the meta-data strings (column name, units and
+comments), will be allocated and set. This function is just for column
+information (meta-data), not column contents.
@end deftypefun
@deftypefun void gal_table_print_info (gal_data_t @code{*allcols}, size_t
@code{numcols}, size_t @code{numrows})
@@ -25223,10 +25424,19 @@ $ asttable --info table.fits
@end example
@end deftypefun
-@deftypefun {gal_data_t *} gal_table_read (char @code{*filename}, char
@code{*hdu}, gal_list_str_t @code{*cols}, int @code{searchin}, int
@code{ignorecase}, size_t @code{minmapsize}, size_t @code{*colmatch})
-Read the specified columns in a text file (named @code{filename}) into a
-linked list of data structures. If the file is FITS, then @code{hdu} will
-also be used, otherwise, @code{hdu} is ignored.
+@deftypefun {gal_data_t *} gal_table_read (char @code{*filename}, char
@code{*hdu}, gal_list_str_t @code{*lines}, gal_list_str_t @code{*cols}, int
@code{searchin}, int @code{ignorecase}, size_t @code{minmapsize}, size_t
@code{*colmatch})
+
+Read the specified columns in a file (named @code{filename}), or list of
+strings (@code{lines}) into a linked list of data structures. If the file
+is FITS, then @code{hdu} will also be used, otherwise, @code{hdu} is
+ignored.
+
+@code{lines} is a list of strings with each node representing one line
+(including the new-line character), see @ref{List of strings}. It will
+mostly be the output of @code{gal_txt_stdin_read}, which is used to read
+the program's input as separate lines from the standard input (see
+@ref{Text files}). Note that @code{filename} and @code{lines} are mutually
+exclusive and one of them must be @code{NULL}.
@cindex AWK
@cindex GNU AWK
@@ -25995,11 +26205,19 @@ data line. The returned values are the macros that
start with
@code{GAL_TXT_LINESTAT}.
@end deftypefun
-@deftypefun {gal_data_t *} gal_txt_table_info (char @code{*filename}, size_t
@code{*numcols}, size_t @code{*numrows})
-Store the information of each column in @code{filename} into an array of
-data structures with @code{numcols} elements (one data structure for each
-column) see @ref{Arrays of datasets}. The total number of rows in the table
-is also put into the memory that @code{numrows} points to.
+@deftypefun {gal_data_t *} gal_txt_table_info (char @code{*filename},
gal_list_str_t @code{*lines}, size_t @code{*numcols}, size_t @code{*numrows})
+Store the information of each column in a text file @code{filename}, or
+list of strings (@code{lines}) into an array of data structures with
+@code{numcols} elements (one data structure for each column) see
+@ref{Arrays of datasets}. The total number of rows in the table is also put
+into the memory that @code{numrows} points to.
+
+@code{lines} is a list of strings with each node representing one line
+(including the new-line character), see @ref{List of strings}. It will
+mostly be the output of @code{gal_txt_stdin_read}, which is used to read
+the program's input as separate lines from the standard input (see
+below). Note that @code{filename} and @code{lines} are mutually exclusive
+and one of them must be @code{NULL}.
This function is just for column information. Therefore it only stores
meta-data like column name, units and comments. No actual data (contents of
@@ -26011,13 +26229,20 @@ variety of table formats based on the filename (see
@ref{Table input
output}).
@end deftypefun
-@deftypefun {gal_data_t *} gal_txt_table_read (char @code{*filename}, size_t
@code{numrows}, gal_data_t @code{*colinfo}, gal_list_sizet_t @code{*indexll},
size_t @code{minmapsize})
-Read the columns given in the list @code{indexll} from a plain text table
-into a linked list of data structures, see @ref{List of size_t} and
-@ref{List of gal_data_t}. If the necessary space for each column is larger
-than @code{minmapsize}, don't keep it in the RAM, but in a file on the
-HDD/SSD, see the description under the same name in @ref{Generic data
-container}.
+@deftypefun {gal_data_t *} gal_txt_table_read (char @code{*filename},
gal_list_str_t @code{*lines}, size_t @code{numrows}, gal_data_t
@code{*colinfo}, gal_list_sizet_t @code{*indexll}, size_t @code{minmapsize})
+Read the columns given in the list @code{indexll} from a plain text file
+(@code{filename}) or list of strings (@code{lines}), into a linked list of
+data structures (see @ref{List of size_t} and @ref{List of gal_data_t}). If
+the necessary space for each column is larger than @code{minmapsize}, don't
+keep it in the RAM, but in a file on the HDD/SSD, see the description under
+the same name in @ref{Generic data container}.
+
+@code{lines} is a list of strings with each node representing one line
+(including the new-line character), see @ref{List of strings}. It will
+mostly be the output of @code{gal_txt_stdin_read}, which is used to read
+the program's input as separate lines from the standard input (see
+below). Note that @code{filename} and @code{lines} are mutually exclusive
+and one of them must be @code{NULL}.
Note that this is a low-level function, so the output data list is the
inverse of the input indexs linked list. It is recommended to use
@@ -26025,11 +26250,44 @@ inverse of the input indexs linked list. It is
recommended to use
@ref{Table input output}.
@end deftypefun
-@deftypefun {gal_data_t *} gal_txt_image_read (char @code{*filename}, size_t
@code{minmapsize})
-Read the 2D plain text dataset in @code{filename} into a dataset and return
-the dataset. If the necessary space for the image is larger than
-@code{minmapsize}, don't keep it in the RAM, but in a file on the HDD/SSD,
-see the description under the same name in @ref{Generic data container}.
+@deftypefun {gal_data_t *} gal_txt_image_read (char @code{*filename},
gal_list_str_t @code{*lines}, size_t @code{minmapsize})
+Read the 2D plain text dataset in file (@code{filename}) or list of strings
+(@code{lines}) into a dataset and return the dataset. If the necessary
+space for the image is larger than @code{minmapsize}, don't keep it in the
+RAM, but in a file on the HDD/SSD, see the description under the same name
+in @ref{Generic data container}.
+
+@code{lines} is a list of strings with each node representing one line
+(including the new-line character), see @ref{List of strings}. It will
+mostly be the output of @code{gal_txt_stdin_read}, which is used to read
+the program's input as separate lines from the standard input (see
+below). Note that @code{filename} and @code{lines} are mutually exclusive
+and one of them must be @code{NULL}.
+@end deftypefun
+
+@deftypefun {gal_list_str_t *} gal_txt_stdin_read (long
@code{timeout_microsec})
+@cindex Standard input
+Read the complete standard input and return a list of strings with each
+line (including the new-line character) as one node of that list. If the
+standard input is already filled (for example connected to another
+program's output with a pipe), then this function will parse the whole
+stream.
+
+If Standard input is not pre-configured and the @emph{first line} is
+typed/written in the terminal before @code{timeout_microsec} micro-seconds,
+it will continue parsing until reaches an end-of-file character
+(@key{CTRL-D} after a new-line on the keyboard) with no time limit. If
+nothing is entered before @code{timeout_microsec} micro-seconds, it will
+return @code{NULL}.
+
+All the functions that can read plain text tables will accept a filename as
+well as a list of strings (intended to be the output of this function for
+using Standard input). The reason for keeping the standard input is that
+once something is read from the standard input, it is hard to put it
+back. We often need to read a text file several times: once to count how
+many columns it has and which ones are requested, and another time to read
+the desired columns. So it easier to keep it all in allocated memory and
+pass it on from the start for each round.
@end deftypefun
@deftypefun void gal_txt_write (gal_data_t @code{*cols}, gal_list_str_t
@code{*comment}, char @code{*filename}, uint8_t @code{colinfoinstdout})
@@ -28019,10 +28277,18 @@ shifted to accommodate this request.
@deftypefun {gal_data_t *} gal_statistics_histogram (gal_data_t @code{*input},
gal_data_t @code{*bins}, int @code{normalize}, int @code{maxone})
Make a histogram of all the elements in the given dataset with bin values
that are defined in the @code{inbins} structure (see
-@code{gal_statistics_regular_bins}). @code{inbins} is not mandatory, if you
-pass a @code{NULL} pointer, the bins structure will be built within this
-function based on the @code{numbins} input. As a result, when you have
-already defined the bins, @code{numbins} is not used.
+@code{gal_statistics_regular_bins}, they currently have to be equally
+spaced). @code{inbins} is not mandatory, if you pass a @code{NULL} pointer,
+the bins structure will be built within this function based on the
+@code{numbins} input. As a result, when you have already defined the bins,
+@code{numbins} is not used.
+
+Let's write the center of the @mymath{i}th element of the bin array as
+@mymath{b_i}, and the fixed half-bin width as @mymath{h}. Then element
+@mymath{j} of the input array (@mymath{in_j}) will be counted in
+@mymath{b_i} if @mymath{(b_i-h) \le in_j < (b_i+h)}. However, if
+@mymath{in_j} is somewhere in the last bin, the condition changes to
+@mymath{(b_i-h) \le in_j \le (b_i+h)}.
@end deftypefun
diff --git a/lib/array.c b/lib/array.c
index 6c63afc..c6db2f5 100644
--- a/lib/array.c
+++ b/lib/array.c
@@ -86,7 +86,8 @@ gal_array_name_recognized_multiext(char *name)
/* Read (all the possibly existing) color channels within each
extension/dir of the given file. */
gal_data_t *
-gal_array_read(char *filename, char *extension, size_t minmapsize)
+gal_array_read(char *filename, char *extension, gal_list_str_t *lines,
+ size_t minmapsize)
{
size_t ext;
@@ -107,7 +108,7 @@ gal_array_read(char *filename, char *extension, size_t
minmapsize)
/* Default: plain text. */
else
- return gal_txt_image_read(filename, minmapsize);
+ return gal_txt_image_read(filename, lines, minmapsize);
/* Control should not get to here, but just to avoid compiler warnings,
we'll return a NULL. */
@@ -123,11 +124,13 @@ gal_array_read(char *filename, char *extension, size_t
minmapsize)
/* Read the contents of the given file/extension to a specific type. */
gal_data_t *
-gal_array_read_to_type(char *filename, char *extension, uint8_t type,
+gal_array_read_to_type(char *filename, char *extension,
+ gal_list_str_t *lines, uint8_t type,
size_t minmapsize)
{
gal_data_t *out=NULL;
- gal_data_t *next, *in=gal_array_read(filename, extension, minmapsize);
+ gal_data_t *next, *in=gal_array_read(filename, extension, lines,
+ minmapsize);
/* Go over all the channels. */
while(in)
@@ -149,11 +152,12 @@ gal_array_read_to_type(char *filename, char *extension,
uint8_t type,
/* Read the input array and make sure it is only one channel. */
gal_data_t *
-gal_array_read_one_ch(char *filename, char *extension, size_t minmapsize)
+gal_array_read_one_ch(char *filename, char *extension, gal_list_str_t *lines,
+ size_t minmapsize)
{
char *fname;
gal_data_t *out;
- out=gal_array_read(filename, extension, minmapsize);
+ out=gal_array_read(filename, extension, lines, minmapsize);
if(out->next)
{
@@ -183,10 +187,12 @@ gal_array_read_one_ch(char *filename, char *extension,
size_t minmapsize)
/* Read a single-channel dataset into a specific type. */
gal_data_t *
-gal_array_read_one_ch_to_type(char *filename, char *extension, uint8_t type,
+gal_array_read_one_ch_to_type(char *filename, char *extension,
+ gal_list_str_t *lines, uint8_t type,
size_t minmapsize)
{
- gal_data_t *out=gal_array_read_one_ch(filename, extension, minmapsize);
+ gal_data_t *out=gal_array_read_one_ch(filename, extension, lines,
+ minmapsize);
return gal_data_copy_to_new_type_free(out, type);
}
diff --git a/lib/gnuastro-internal/commonopts.h
b/lib/gnuastro-internal/commonopts.h
index 300f498..50f3397 100644
--- a/lib/gnuastro-internal/commonopts.h
+++ b/lib/gnuastro-internal/commonopts.h
@@ -60,7 +60,7 @@ struct argp_option gal_commonopts_options[] =
GAL_OPTIONS_KEY_SEARCHIN,
"STR",
0,
- "Select column(s) in: `name', `unit', `comment'.",
+ "Select column(s): `name', `unit', `comment'.",
GAL_OPTIONS_GROUP_INPUT,
&cp->searchin,
GAL_TYPE_STRING,
@@ -82,6 +82,19 @@ struct argp_option gal_commonopts_options[] =
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
+ {
+ "stdintimeout",
+ GAL_OPTIONS_KEY_STDINTIMEOUT,
+ "INT",
+ 0,
+ "Micro-seconds to wait for standard input.",
+ GAL_OPTIONS_GROUP_INPUT,
+ &cp->stdintimeout,
+ GAL_TYPE_LONG,
+ GAL_OPTIONS_RANGE_GE_0,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
diff --git a/lib/gnuastro-internal/options.h b/lib/gnuastro-internal/options.h
index ef3ba41..3f6ea09 100644
--- a/lib/gnuastro-internal/options.h
+++ b/lib/gnuastro-internal/options.h
@@ -108,7 +108,8 @@ enum options_common_keys
GAL_OPTIONS_KEY_REMAINDERFRAC = 'F',
/* Only long option (integers for keywords). */
- GAL_OPTIONS_KEY_MINMAPSIZE = 500,
+ GAL_OPTIONS_KEY_STDINTIMEOUT = 500,
+ GAL_OPTIONS_KEY_MINMAPSIZE,
GAL_OPTIONS_KEY_LOG,
GAL_OPTIONS_KEY_CITE,
GAL_OPTIONS_KEY_CONFIG,
@@ -184,6 +185,7 @@ struct gal_options_common_params
char *hdu; /* Image extension. */
uint8_t searchin; /* Column meta-data to match/search. */
uint8_t ignorecase; /* Ignore case when matching col info. */
+ long stdintimeout; /* Timeout (micro-seconds) for stdin. */
/* Output. */
char *output; /* Directory containg output. */
@@ -302,6 +304,12 @@ gal_options_set_from_key(int key, char *arg, struct
argp_option *options,
error_t
gal_options_common_argp_parse(int key, char *arg, struct argp_state *state);
+char *
+gal_options_stdin_error(long stdintimeout, int precedence);
+
+gal_list_str_t *
+gal_options_check_stdin(char *inputname, long stdintimeout);
+
diff --git a/lib/gnuastro/array.h b/lib/gnuastro/array.h
index faff5f2..9b92fc1 100644
--- a/lib/gnuastro/array.h
+++ b/lib/gnuastro/array.h
@@ -59,17 +59,21 @@ int
gal_array_name_recognized_multiext(char *name);
gal_data_t *
-gal_array_read(char *filename, char *extension, size_t minmapsize);
+gal_array_read(char *filename, char *extension, gal_list_str_t *lines,
+ size_t minmapsize);
gal_data_t *
-gal_array_read_to_type(char *filename, char *extension, uint8_t type,
+gal_array_read_to_type(char *filename, char *extension,
+ gal_list_str_t *lines, uint8_t type,
size_t minmapsize);
gal_data_t *
-gal_array_read_one_ch(char *filename, char *extension, size_t minmapsize);
+gal_array_read_one_ch(char *filename, char *extension, gal_list_str_t *lines,
+ size_t minmapsize);
gal_data_t *
-gal_array_read_one_ch_to_type(char *filename, char *extension, uint8_t type,
+gal_array_read_one_ch_to_type(char *filename, char *extension,
+ gal_list_str_t *lines, uint8_t type,
size_t minmapsize);
diff --git a/lib/gnuastro/table.h b/lib/gnuastro/table.h
index b92a340..68b2012 100644
--- a/lib/gnuastro/table.h
+++ b/lib/gnuastro/table.h
@@ -125,8 +125,8 @@ enum gal_table_where_to_search
/*************** Information about a table ***************/
/************************************************************************/
gal_data_t *
-gal_table_info(char *filename, char *hdu, size_t *numcols,
- size_t *numrows, int *tableformat);
+gal_table_info(char *filename, char *hdu, gal_list_str_t *lines,
+ size_t *numcols, size_t *numrows, int *tableformat);
void
gal_table_print_info(gal_data_t *allcols, size_t numcols, size_t numrows);
@@ -137,9 +137,9 @@ gal_table_print_info(gal_data_t *allcols, size_t numcols,
size_t numrows);
/*************** Read a table ***************/
/************************************************************************/
gal_data_t *
-gal_table_read(char *filename, char *hdu, gal_list_str_t *cols,
- int searchin, int ignorecase, size_t minmapsize,
- size_t *colmatch);
+gal_table_read(char *filename, char *hdu, gal_list_str_t *lines,
+ gal_list_str_t *cols, int searchin, int ignorecase,
+ size_t minmapsize, size_t *colmatch);
diff --git a/lib/gnuastro/txt.h b/lib/gnuastro/txt.h
index e26e7f5..4429b00 100644
--- a/lib/gnuastro/txt.h
+++ b/lib/gnuastro/txt.h
@@ -79,14 +79,23 @@ int
gal_txt_line_stat(char *line);
gal_data_t *
-gal_txt_table_info(char *filename, size_t *numcols, size_t *numrows);
+gal_txt_table_info(char *filename, gal_list_str_t *lines, size_t *numcols,
+ size_t *numrows);
gal_data_t *
-gal_txt_table_read(char *filename, size_t numrows, gal_data_t *colinfo,
- gal_list_sizet_t *indexll, size_t minmapsize);
+gal_txt_image_info(char *filename, gal_list_str_t *lines, size_t *numimg,
+ size_t *dsize);
gal_data_t *
-gal_txt_image_read(char *filename, size_t minmapsize);
+gal_txt_table_read(char *filename, gal_list_str_t *lines, size_t numrows,
+ gal_data_t *colinfo, gal_list_sizet_t *indexll,
+ size_t minmapsize);
+
+gal_data_t *
+gal_txt_image_read(char *filename, gal_list_str_t *lines, size_t minmapsize);
+
+gal_list_str_t *
+gal_txt_stdin_read(long timeout_microsec);
void
gal_txt_write(gal_data_t *input, gal_list_str_t *comment, char *filename,
diff --git a/lib/options.c b/lib/options.c
index dcb4a3d..1f4ceaa 100644
--- a/lib/options.c
+++ b/lib/options.c
@@ -1358,6 +1358,50 @@ gal_options_common_argp_parse(int key, char *arg, struct
argp_state *state)
+char *
+gal_options_stdin_error(long stdintimeout, int precedence)
+{
+ char *out;
+
+ if( asprintf(&out, "no input!\n\n"
+ "The (first) input dataset can be read from a file "
+ "(specified as an argument), or the standard input.%s "
+ "Standard input can come from a pipe (output of another "
+ "program) or typed on the command-line before %ld "
+ "micro-seconds (configurable with the `--stdintimeout' "
+ "option).", ( precedence
+ ? "If both are provided, a file takes precedence"
+ : " " ),
+ stdintimeout )<0 )
+ error(EXIT_FAILURE, 0, "%s: `asprintf' allocation error", __func__);
+
+ return out;
+}
+
+
+
+
+
+/* Make the notice that is printed before program terminates, when no input
+ is given and Standard input is also available. */
+gal_list_str_t *
+gal_options_check_stdin(char *inputname, long stdintimeout)
+{
+ gal_list_str_t *lines=inputname ? NULL : gal_txt_stdin_read(stdintimeout);
+
+ /* See if atleast one of the two inputs is given. */
+ if(inputname==NULL && lines==NULL)
+ error( EXIT_FAILURE, 0, gal_options_stdin_error(stdintimeout,1));
+
+ /* Return the output. */
+ return lines;
+}
+
+
+
+
+
+
diff --git a/lib/statistics.c b/lib/statistics.c
index 981bbec..11a04d1 100644
--- a/lib/statistics.c
+++ b/lib/statistics.c
@@ -1550,7 +1550,9 @@ gal_statistics_regular_bins(gal_data_t *input, gal_data_t
*inrange,
{
tmp=gal_data_copy_to_new_type_free(gal_statistics_maximum(input),
GAL_TYPE_FLOAT64);
- max=*((double *)(tmp->array))+1e-6;
+ max=*((double *)(tmp->array));
+
+ /* Clean up. */
gal_data_free(tmp);
}
else max=ra[1];
@@ -1568,7 +1570,9 @@ gal_statistics_regular_bins(gal_data_t *input, gal_data_t
*inrange,
gal_data_free(tmp);
tmp=gal_data_copy_to_new_type_free(gal_statistics_maximum(input),
GAL_TYPE_FLOAT64);
- max=*((double *)(tmp->array)) + 1e-6;
+ max=*((double *)(tmp->array));
+
+ /* Clean up. */
gal_data_free(tmp);
}
@@ -1605,7 +1609,7 @@ gal_statistics_regular_bins(gal_data_t *input, gal_data_t
*inrange,
printf("onebinstart: %.10f\n", onebinstart);
printf("binwidth: %g\n", binwidth);
for(i=0;i<numbins;++i)
- printf("%zu: %.4f\t(%f, %f)\n", i, b[i], b[i]-hbw, b[i]+hbw);
+ printf("%zu: %.4g\t(%g, %g)\n", i, b[i], b[i]-hbw, b[i]+hbw);
*/
/* Set the status of the bins to regular and return. */
@@ -1626,7 +1630,13 @@ gal_statistics_regular_bins(gal_data_t *input,
gal_data_t *inrange,
#define HISTOGRAM_TYPESET(IT) { \
IT *a=input->array, *af=a+input->size; \
- do if( *a>=min && *a<max) ++h[ (size_t)( (*a-min)/binwidth ) ]; \
+ do \
+ if(*a>=min) \
+ { \
+ h_i=(*a-min)/binwidth; \
+ if(h_i == last_i) { if( *a<=max ) ++h[ h_i ]; } \
+ else { if( *a< max ) ++h[ h_i ]; } \
+ } \
while(++a<af); \
}
@@ -1634,9 +1644,9 @@ gal_data_t *
gal_statistics_histogram(gal_data_t *input, gal_data_t *bins, int normalize,
int maxone)
{
- size_t *h;
float *f, *ff;
gal_data_t *hist;
+ size_t *h, h_i, last_i;
double *d, min, max, ref=NAN, binwidth;
@@ -1668,8 +1678,9 @@ gal_statistics_histogram(gal_data_t *input, gal_data_t
*bins, int normalize,
/* Set the minimum and maximum range of the histogram from the bins. */
d=bins->array;
binwidth=d[1]-d[0];
- max = d[bins->size - 1] + binwidth/2;
- min = d[0] - binwidth/2;
+ last_i=bins->size-1;
+ min = d[ 0 ] - binwidth/2;
+ max = d[ last_i ] + binwidth/2;
/* Go through all the elements and find out which bin they belong to. */
diff --git a/lib/table.c b/lib/table.c
index 84bfdd5..69246ae 100644
--- a/lib/table.c
+++ b/lib/table.c
@@ -60,16 +60,16 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
comments), nothing in the data structure(s) will be allocated by this
function for the actual data (e.g., the `array' or `dsize' elements). */
gal_data_t *
-gal_table_info(char *filename, char *hdu, size_t *numcols, size_t *numrows,
- int *tableformat)
+gal_table_info(char *filename, char *hdu, gal_list_str_t *lines,
+ size_t *numcols, size_t *numrows, int *tableformat)
{
/* Get the table format and size (number of columns and rows). */
- if(gal_fits_name_is_fits(filename))
+ if(filename && gal_fits_name_is_fits(filename))
return gal_fits_tab_info(filename, hdu, numcols, numrows, tableformat);
else
{
*tableformat=GAL_TABLE_FORMAT_TXT;
- return gal_txt_table_info(filename, numcols, numrows);
+ return gal_txt_table_info(filename, lines, numcols, numrows);
}
/* Abort with an error if we get to this point. */
@@ -403,9 +403,9 @@ make_list_of_indexs(gal_list_str_t *cols, gal_data_t
*allcols,
So the first requested column is the first popped data structure and so
on. */
gal_data_t *
-gal_table_read(char *filename, char *hdu, gal_list_str_t *cols,
- int searchin, int ignorecase, size_t minmapsize,
- size_t *colmatch)
+gal_table_read(char *filename, char *hdu, gal_list_str_t *lines,
+ gal_list_str_t *cols, int searchin, int ignorecase,
+ size_t minmapsize, size_t *colmatch)
{
int tableformat;
gal_list_sizet_t *indexll;
@@ -413,7 +413,8 @@ gal_table_read(char *filename, char *hdu, gal_list_str_t
*cols,
gal_data_t *allcols, *out=NULL;
/* First get the information of all the columns. */
- allcols=gal_table_info(filename, hdu, &numcols, &numrows, &tableformat);
+ allcols=gal_table_info(filename, hdu, lines, &numcols, &numrows,
+ &tableformat);
/* If there was no actual data in the file, then return NULL. */
if(allcols==NULL) return NULL;
@@ -435,7 +436,7 @@ gal_table_read(char *filename, char *hdu, gal_list_str_t
*cols,
switch(tableformat)
{
case GAL_TABLE_FORMAT_TXT:
- out=gal_txt_table_read(filename, numrows, allcols, indexll,
+ out=gal_txt_table_read(filename, lines, numrows, allcols, indexll,
minmapsize);
break;
diff --git a/lib/txt.c b/lib/txt.c
index 6b16337..a9fe6b9 100644
--- a/lib/txt.c
+++ b/lib/txt.c
@@ -27,6 +27,7 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include <stdio.h>
#include <errno.h>
#include <error.h>
+#include <unistd.h>
#include <string.h>
#include <stdlib.h>
@@ -290,11 +291,21 @@ txt_info_from_comment(char *line, gal_data_t **datall,
char *comm_start)
given text file. If the file is a text table with string columns, the
contents of the string column will be counted as one token.*/
static size_t
-txt_info_from_first_row(char *line, gal_data_t **datall, int format)
+txt_info_from_first_row(char *in_line, gal_data_t **datall, int format,
+ int inplace)
{
gal_data_t *col, *prev, *tmp;
size_t n=0, maxcnum=0, numtokens;
- char *token, *end=line+strlen(line);
+ char *line, *token, *end, *aline=NULL;
+
+ /* Make a copy of the input line if necessary. */
+ if(inplace) line=in_line;
+ else
+ {
+ gal_checkset_allocate_copy(in_line, &line);
+ aline=line; /* We are going to change `line' during this function. */
+ }
+ end=line+strlen(line);
/* Remove the line termination character(s) from the end of the line. In
Unix, the line terminator is just the new-line character, however, in
@@ -307,8 +318,8 @@ txt_info_from_first_row(char *line, gal_data_t **datall,
int format)
the available space on the line, we don't want to have the line's
new-line character. Its better for it to actually be shorter than the
space. */
- if( *(end-2)==13 ) *(end-2)='\0';
- else *(end-1)='\0';
+ if( end>line+2 && *(end-2)==13 ) *(end-2)='\0';
+ else if( *(end-1)=='\n' ) *(end-1)='\0';
/* Get the maximum number of columns read from the comment
information. */
@@ -438,6 +449,7 @@ txt_info_from_first_row(char *line, gal_data_t **datall,
int format)
}
/* Return the total number of columns/second-img-dimension. */
+ if(inplace==0) free(aline);
return numtokens;
}
@@ -517,18 +529,58 @@ txt_infoll_to_array(gal_data_t *datall, size_t *numdata)
+static void
+txt_get_info_line(char *line, gal_data_t **datall, char *comm_start,
+ int *firstlinedone, int format, size_t *dsize, int inplace)
+{
+ size_t numtokens;
+
+ switch( gal_txt_line_stat(line) )
+ {
+ /* Line is a comment, see if it has formatted information. */
+ case GAL_TXT_LINESTAT_COMMENT:
+ txt_info_from_comment(line, datall, comm_start);
+ break;
+
+ /* Line is actual data, use it to fill in the gaps. */
+ case GAL_TXT_LINESTAT_DATAROW:
+ ++dsize[0];
+ if(*firstlinedone==0)
+ {
+ *firstlinedone=1;
+ numtokens=txt_info_from_first_row(line, datall, format, inplace);
+ if(format==TXT_FORMAT_IMAGE) dsize[1]=numtokens;
+ }
+ break;
+
+ /* We also have the case of GAL_TXT_LINESTAT_BLANK, but we don't
+ need to do anything about it. */
+ }
+}
+
+
+
+
+
/* Return the information about a text file table. If there were no
readable rows, it will return NULL.*/
static gal_data_t *
-txt_get_info(char *filename, int format, size_t *numdata, size_t *dsize)
+txt_get_info(char *filename, gal_list_str_t *lines, int format,
+ size_t *numdata, size_t *dsize)
{
FILE *fp;
- size_t numtokens;
- int firstlinedone=0;
- gal_data_t *datall=NULL, *dataarr;
+ gal_list_str_t *tmp;
+ gal_data_t *datall=NULL;
+ int test, firstlinedone=0;
char *line, *format_err="empty", *comm_start;
size_t linelen=10; /* `linelen' will be increased by `getline'. */
+ /* `filename' and `lines' cannot both be non-NULL. */
+ test = (filename!=NULL) + (lines!=NULL);
+ if( test!=1 )
+ error(EXIT_FAILURE, 0, "%s: one of the `filename' and `lines' "
+ "arguments must be NULL, but they are both %s", __func__,
+ test==2 ? "non-NULL" : "NULL");
/* Set the constant strings */
switch(format)
@@ -540,75 +592,59 @@ txt_get_info(char *filename, int format, size_t *numdata,
size_t *dsize)
__func__, format);
}
+ /* Initialize the first `dsize' element. */
+ dsize[0]=0;
- /* Open the file. */
- errno=0;
- fp=fopen(filename, "r");
- if(fp==NULL)
- error(EXIT_FAILURE, errno, "%s: couldn't open to read as a plain "
- "text %s (from Gnuastro's `%s')", filename, format_err, __func__);
+ /* Parse the file or go over the lines. */
+ if(filename)
+ {
+ /* Open the file. */
+ errno=0;
+ fp=fopen(filename, "r");
+ if(fp==NULL)
+ error(EXIT_FAILURE, errno, "%s: couldn't open to read as a plain "
+ "text %s (from Gnuastro's `%s')", filename, format_err,
+ __func__);
- /* Allocate the space necessary to keep each line as we parse it. Note
- that `getline' is going to later `realloc' this space to fit the line
- length. */
- errno=0;
- line=malloc(linelen*sizeof *line);
- if(line==NULL)
- error(EXIT_FAILURE, errno, "%s: allocating %zu bytes for line",
- __func__, linelen*sizeof *line);
+ /* Allocate the space necessary to keep each line as we parse
+ it. Note that `getline' is going to later `realloc' this space to
+ fit the line length. */
+ errno=0;
+ line=malloc(linelen*sizeof *line);
+ if(line==NULL)
+ error(EXIT_FAILURE, errno, "%s: allocating %zu bytes for line",
+ __func__, linelen*sizeof *line);
- /* Read the comments of the line for possible information about the
- lines, but also confirm/complete the info by parsing the first
- uncommented line. */
- dsize[0]=0;
- while( getline(&line, &linelen, fp) != -1 )
- switch( gal_txt_line_stat(line) )
- {
- /* Line is a comment, see if it has formatted information. */
- case GAL_TXT_LINESTAT_COMMENT:
- txt_info_from_comment(line, &datall, comm_start);
- break;
-
- /* Line is actual data, use it to fill in the gaps. */
- case GAL_TXT_LINESTAT_DATAROW:
- ++dsize[0];
- if(firstlinedone==0)
- {
- firstlinedone=1;
- numtokens=txt_info_from_first_row(line, &datall, format);
- if(format==TXT_FORMAT_IMAGE) dsize[1]=numtokens;
- }
- break;
-
- /* We also have the case of GAL_TXT_LINESTAT_BLANK. */
- }
+ /* Read the comments of the line for possible information about the
+ lines, but also confirm/complete the info by parsing the first
+ uncommented line. */
+ while( getline(&line, &linelen, fp) != -1 )
+ txt_get_info_line(line, &datall, comm_start, &firstlinedone, format,
+ dsize, 1);
+
+ /* Clean up and close the file. */
+ free(line);
+ errno=0;
+ if(fclose(fp))
+ error(EXIT_FAILURE, errno, "%s: couldn't close file after reading "
+ "plain text %s information in %s", filename, format_err,
+ __func__);
+ }
+ else
+ {
+ for(tmp=lines; tmp!=NULL; tmp=tmp->next)
+ txt_get_info_line(tmp->v, &datall, comm_start, &firstlinedone,
+ format, dsize, 0);
+ }
/* The final dataset linked list can have any order (depending on how the
user gave column information in tables for example). So here, we will
convert the list into a nicely sorted array, note that this function
frees list as part of the process. */
- dataarr=txt_infoll_to_array(datall, numdata);
-
-
- /* Clean up. Note that even if there were no usable columns, there might
- have been meta-data comments, so we need to free `colsll' in any
- case. If the list is indeed empty, then `gal_data_free_ll' won't do
- anything. */
- free(line);
-
-
- /* Close the file. */
- errno=0;
- if(fclose(fp))
- error(EXIT_FAILURE, errno, "%s: couldn't close file after reading plain "
- "text %s information in %s", filename, format_err, __func__);
-
-
- /* Return the array of column information. */
- return dataarr;
+ return txt_infoll_to_array(datall, numdata);
}
@@ -617,9 +653,10 @@ txt_get_info(char *filename, int format, size_t *numdata,
size_t *dsize)
/* Get the information of each column in a text file */
gal_data_t *
-gal_txt_table_info(char *filename, size_t *numcols, size_t *numrows)
+gal_txt_table_info(char *filename, gal_list_str_t *lines, size_t *numcols,
+ size_t *numrows)
{
- return txt_get_info(filename, TXT_FORMAT_TABLE, numcols, numrows);
+ return txt_get_info(filename, lines, TXT_FORMAT_TABLE, numcols, numrows);
}
@@ -628,9 +665,10 @@ gal_txt_table_info(char *filename, size_t *numcols, size_t
*numrows)
/* Get the information of a 2D array in a text file. */
gal_data_t *
-gal_txt_image_info(char *filename, size_t *numimg, size_t *dsize)
+gal_txt_image_info(char *filename, gal_list_str_t *lines, size_t *numimg,
+ size_t *dsize)
{
- return txt_get_info(filename, TXT_FORMAT_IMAGE, numimg, dsize);
+ return txt_get_info(filename, lines, TXT_FORMAT_IMAGE, numimg, dsize);
}
@@ -768,17 +806,27 @@ txt_read_token(gal_data_t *data, gal_data_t *info, char
*token,
static void
-txt_fill(char *line, char **tokens, size_t maxcolnum, gal_data_t *info,
- gal_data_t *out, size_t rowind, char *filename, size_t lineno)
+txt_fill(char *in_line, char **tokens, size_t maxcolnum, gal_data_t *info,
+ gal_data_t *out, size_t rowind, char *filename, size_t lineno,
+ int inplace)
{
size_t i, n=0;
gal_data_t *data;
int notenoughcols=0;
- char *end=line+strlen(line);
+ char *end, *line, *aline=NULL;
+
+ /* Make a copy of the input line if necessary. */
+ if(inplace) line=in_line;
+ else
+ {
+ gal_checkset_allocate_copy(in_line, &line);
+ aline=line; /* We are going to change `line' during this function. */
+ }
+ end=line+strlen(line);
/* See explanations in `txt_info_from_first_row'. */
- if( *(end-2)==13 ) *(end-2)='\0';
- else *(end-1)='\0';
+ if( end>line+2 && *(end-2)==13 ) *(end-2)='\0';
+ else if( *(end-1)=='\n' ) *(end-1)='\0';
/* Start parsing the line. Note that `n' and `maxcolnum' start from
one. */
@@ -848,6 +896,9 @@ txt_fill(char *line, char **tokens, size_t maxcolnum,
gal_data_t *info,
error(EXIT_FAILURE, 0, "%s: currently only 1 and 2 dimensional "
"datasets acceptable", __func__);
}
+
+ /* Clean up. */
+ if(inplace==0) free(aline);
}
@@ -855,23 +906,26 @@ txt_fill(char *line, char **tokens, size_t maxcolnum,
gal_data_t *info,
static gal_data_t *
-gal_txt_read(char *filename, size_t *dsize, gal_data_t *info,
- gal_list_sizet_t *indexll, size_t minmapsize, int format)
+txt_read(char *filename, gal_list_str_t *lines, size_t *dsize,
+ gal_data_t *info, gal_list_sizet_t *indexll, size_t minmapsize,
+ int format)
{
FILE *fp;
+ int test;
char *line;
char **tokens;
+ gal_list_str_t *tmp;
gal_data_t *out=NULL;
gal_list_sizet_t *ind;
size_t one=1, maxcolnum=0, rowind=0, lineno=0, ndim;
size_t linelen=10; /* `linelen' will be increased by `getline'. */
- /* Open the file. */
- errno=0;
- fp=fopen(filename, "r");
- if(fp==NULL)
- error(EXIT_FAILURE, errno, "%s: couldn't open to read as a text table "
- "in %s", filename, __func__);
+ /* `filename' and `lines' cannot both be non-NULL. */
+ test = (filename!=NULL) + (lines!=NULL);
+ if( test!=1 )
+ error(EXIT_FAILURE, 0, "%s: one of the `filename' and `lines' "
+ "arguments must be NULL, but they are both %s", __func__,
+ test==2 ? "non-NULL" : "NULL");
/* Allocate the space necessary to keep a copy of each line as we parse
it. Note that `getline' is going to later `realloc' this space to fit
@@ -947,24 +1001,42 @@ gal_txt_read(char *filename, size_t *dsize, gal_data_t
*info,
error(EXIT_FAILURE, errno, "%s: allocating %zu bytes for `tokens'",
__func__, (maxcolnum+1)*sizeof *tokens);
- /* Read the data columns. */
- while( getline(&line, &linelen, fp) != -1 )
+ if(filename)
{
- ++lineno;
- if( gal_txt_line_stat(line) == GAL_TXT_LINESTAT_DATAROW )
- txt_fill(line, tokens, maxcolnum, info, out, rowind++,
- filename, lineno);
+ /* Open the file. */
+ errno=0;
+ fp=fopen(filename, "r");
+ if(fp==NULL)
+ error(EXIT_FAILURE, errno, "%s: couldn't open to read as a text "
+ "table in %s", filename, __func__);
+
+ /* Read the data columns. */
+ while( getline(&line, &linelen, fp) != -1 )
+ {
+ ++lineno;
+ if( gal_txt_line_stat(line) == GAL_TXT_LINESTAT_DATAROW )
+ txt_fill(line, tokens, maxcolnum, info, out, rowind++,
+ filename, lineno, 1);
+ }
+
+ /* Clean up and close the file. */
+ errno=0;
+ if(fclose(fp))
+ error(EXIT_FAILURE, errno, "%s: couldn't close file after reading "
+ "ASCII table information in %s", filename, __func__);
+ free(line);
}
+ else
+ for(tmp=lines; tmp!=NULL; tmp=tmp->next)
+ {
+ ++lineno;
+ if( gal_txt_line_stat(tmp->v) == GAL_TXT_LINESTAT_DATAROW )
+ txt_fill(tmp->v, tokens, maxcolnum, info, out, rowind++,
+ filename, lineno, 0);
+ }
- /* Clean up and close the file. */
- errno=0;
- if(fclose(fp))
- error(EXIT_FAILURE, errno, "%s: couldn't close file after reading ASCII "
- "table information in %s", filename, __func__);
+ /* Clean up and return. */
free(tokens);
- free(line);
-
- /* Return the array of column information. */
return out;
}
@@ -973,11 +1045,12 @@ gal_txt_read(char *filename, size_t *dsize, gal_data_t
*info,
gal_data_t *
-gal_txt_table_read(char *filename, size_t numrows, gal_data_t *colinfo,
- gal_list_sizet_t *indexll, size_t minmapsize)
+gal_txt_table_read(char *filename, gal_list_str_t *lines, size_t numrows,
+ gal_data_t *colinfo, gal_list_sizet_t *indexll,
+ size_t minmapsize)
{
- return gal_txt_read(filename, &numrows, colinfo, indexll, minmapsize,
- TXT_FORMAT_TABLE);
+ return txt_read(filename, lines, &numrows, colinfo, indexll, minmapsize,
+ TXT_FORMAT_TABLE);
}
@@ -985,18 +1058,18 @@ gal_txt_table_read(char *filename, size_t numrows,
gal_data_t *colinfo,
gal_data_t *
-gal_txt_image_read(char *filename, size_t minmapsize)
+gal_txt_image_read(char *filename, gal_list_str_t *lines, size_t minmapsize)
{
size_t numimg, dsize[2];
gal_data_t *img, *imginfo;
gal_list_sizet_t *indexll=NULL;
/* Get the image information. */
- imginfo=gal_txt_image_info(filename, &numimg, dsize);
+ imginfo=gal_txt_image_info(filename, lines, &numimg, dsize);
/* Read the table. */
- img=gal_txt_read(filename, dsize, imginfo, indexll, minmapsize,
- TXT_FORMAT_IMAGE);
+ img=txt_read(filename, lines, dsize, imginfo, indexll, minmapsize,
+ TXT_FORMAT_IMAGE);
/* Clean up and return. */
gal_data_free(imginfo);
@@ -1006,6 +1079,87 @@ gal_txt_image_read(char *filename, size_t minmapsize)
+/* See if there is anything in the standard input already. This function is
+ modeled on the solution provided in:
+
+ https://stackoverflow.com/questions/3711830/set-a-timeout-for-reading-stdin
*/
+static int
+txt_stdin_has_contents(long timeout_microsec)
+{
+ fd_set fds;
+ struct timeval tv;
+
+ /* Set the timeout time. */
+ tv.tv_sec = 0;
+ tv.tv_usec = timeout_microsec;
+
+ /* Initialize `fd_set'. */
+ FD_ZERO(&fds);
+
+ /* Set standard input (STDIN_FILENO is 0) as the FD that must be read. */
+ FD_SET(STDIN_FILENO, &fds);
+
+ /* `select' takes the last file descriptor value + 1 in the fdset to
+ check, the fdset for reads, writes, and errors. We are only passing
+ in reads. the last parameter is the timeout. select will return if
+ an FD is ready or the timeout has occurred. */
+ select(STDIN_FILENO+1, &fds, NULL, NULL, &tv);
+
+ // return 0 if STDIN is not ready to be read.
+ return FD_ISSET(STDIN_FILENO, &fds);
+}
+
+
+
+
+/* Read each line of the standard input into a linked list of strings. */
+gal_list_str_t *
+gal_txt_stdin_read(long timeout_microsec)
+{
+ char *line;
+ gal_list_str_t *out=NULL;
+ size_t lineno=0, linelen=10;/* `linelen' will be increased by `getline'. */
+
+ /* If there is nothing */
+ if( txt_stdin_has_contents(timeout_microsec) )
+ {
+ /* Allocate the space necessary to keep a copy of each line as we
+ parse it. Note that `getline' is going to later `realloc' this
+ space to fit the line length. */
+ errno=0;
+ line=malloc(linelen*sizeof *line);
+ if(line==NULL)
+ error(EXIT_FAILURE, errno, "%s: allocating %zu bytes for `line'",
+ __func__, linelen*sizeof *line);
+
+ /* Read the whole standard input. We are using getline because it can
+ deal with a `NULL' in the input, while also handing allocation
+ issues while reading (allocating by line, not by a fixed buffer
+ size). */
+ while( getline(&line, &linelen, stdin) != -1 )
+ {
+ /* To help in reporting (when necessary), keep a count of how
+ many lines we have. */
+ ++lineno;
+
+ /* Add the line to the output list. */
+ gal_list_str_add(&out, line, 1);
+ }
+
+ /* Reverse the list (to be the same order as input). */
+ gal_list_str_reverse(&out);
+
+ /* Clean up. */
+ free(line);
+ }
+
+ /* Return the result. */
+ return out;
+}
+
+
+
+
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 81768a5..35b37d7 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -175,10 +175,12 @@ if COND_SEGMENT
segment/segment-3d.sh: noisechisel/noisechisel-3d.sh.log
endif
if COND_STATISTICS
- MAYBE_STATISTICS_TESTS = statistics/basicstats.sh statistics/estimate_sky.sh
+ MAYBE_STATISTICS_TESTS = statistics/basicstats.sh statistics/estimate_sky.sh
\
+ statistics/from-stdin.sh
- statistics/basicstats.sh: mknoise/addnoise.sh.log
+ statistics/basicstats.sh: mknoise/addnoise.sh.log
statistics/estimate_sky.sh: mknoise/addnoise.sh.log
+ statistics/from-stdin.sh: prepconf.sh.log
endif
if COND_TABLE
MAYBE_TABLE_TESTS = table/txt-to-fits-binary.sh \
diff --git a/tests/statistics/from-stdin.sh b/tests/statistics/from-stdin.sh
new file mode 100755
index 0000000..7105fdb
--- /dev/null
+++ b/tests/statistics/from-stdin.sh
@@ -0,0 +1,56 @@
+# Measure basic table statistics using standard input.
+#
+# See the Tests subsection of the manual for a complete explanation
+# (in the Installing gnuastro section).
+#
+# Original author:
+# Mohammad Akhlaghi <mohammad@akhlaghi.org>
+# Contributing author(s):
+#
+# Copying and distribution of this file, with or without modification,
+# are permitted in any medium without royalty provided the copyright
+# notice and this notice are preserved. This file is offered as-is,
+# without any warranty.
+
+
+
+
+
+# Preliminaries
+# =============
+#
+# Set the variables (The executable is in the build tree). Do the
+# basic checks to see if the executable is made or if the defaults
+# file exists (basicchecks.sh is in the source tree).
+prog=statistics
+execname=../bin/$prog/ast$prog
+in=$topsrc/tests/$prog/stdin-input.txt
+
+
+
+
+
+# Skip?
+# =====
+#
+# If the dependencies of the test don't exist, then skip it. There are two
+# types of dependencies:
+#
+# - The executable was not made (for example due to a configure option),
+#
+# - The input data was not made (for example the test that created the
+# data file failed).
+if [ ! -f $execname ]; then echo "$execname not created."; exit 77; fi
+if [ ! -f $in ]; then echo "$in does not exist."; exit 77; fi
+
+
+
+
+
+# Actual test script
+# ==================
+#
+# `check_with_program' can be something like `Valgrind' or an empty
+# string. Such programs will execute the command if present and help in
+# debugging when the developer doesn't have access to the user's system.
+cat $in | $check_with_program $execname
- [gnuastro-commits] master 2c6c7e0 076/113: Imported recent work, no conflicts, (continued)
- [gnuastro-commits] master 2c6c7e0 076/113: Imported recent work, no conflicts, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master e2aba57 083/113: Recent work in master imported, conflicts fixed, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master e92fd9c 039/113: Merged recent work in master, conflicts fixed, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 54b099d 043/113: Neighbors options doc generalized for different dimensions, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 5cd3c1e 044/113: Recent work in master merged, small conflicts fixed, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 1b0046b 051/113: Recent additions in master imported here, no conflicts, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master b1ea932 054/113: Separate Segment program now available in 3D, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master f537368 066/113: Segment's default 3D configuration file in tarball, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 133e277 071/113: Imported recent work from master, conflicts fixed, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master a42ac37 085/113: Fixed MakeProfiles loop in checking radius values, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 1f9d941 090/113: Imported recent work in master, conflicts fixed,
Mohammad Akhlaghi <=
- [gnuastro-commits] master 737a0cd 046/113: Merged recent corrections in master branch, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 486f915 067/113: Imported recent work in master, conflicts fixed, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master d6051a1 074/113: Upperlimit magnitude in 3D, corrected creation of check table, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 5dac3c3 075/113: Recent work in master imported, small conflict in book fixed, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 3882956 077/113: Narrow-band areas from MakeCatalog in 3D datasets, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master abe82cb 086/113: Imported recent work in master, no conflicts, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 924787b 098/113: Updated copyright year in files specific to this branch, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master fbbc2fc 100/113: Imported recent work in master, minor conflict fixed, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 22d0fba 089/113: Imported recent work in master, no conflicts, Mohammad Akhlaghi, 2021/04/16
- [gnuastro-commits] master 8edc804 092/113: Imported recent work in master, minor conflic fixed, Mohammad Akhlaghi, 2021/04/16