[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[gnuastro-commits] master 66a9f60: Query: can now get dataset informatio
|
From: |
Mohammad Akhlaghi |
|
Subject: |
[gnuastro-commits] master 66a9f60: Query: can now get dataset information, and other new features |
|
Date: |
Sun, 17 Jan 2021 19:45:14 -0500 (EST) |
branch: master
commit 66a9f609900c1c58db6161612ada6b26a47e1877
Author: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Commit: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Query: can now get dataset information, and other new features
Until now it was necessary to know the dataset information (within the
database) a-priori (before running Query). In the case of Gaia, this wasn't
a major problem because it isn't too large with one dominant
dataset. However, in the case of VizieR (with tens of thousands of
datasets), this was very annoying! Fortunately Francois Ochsenbein helped
alot in this aspect and gave me the necessary commands for obtaining this
information and I was able to generalize them.
With this commit, the necessary low-level changes in Query have been made
to allow printing the information of all datasets within a database, or
columns within a dataset. Therefore, it is now easy to first query the
database and get the list of datasets, then query the dataset and get the
list of columns, and then extract the exact data you want (not necessarily
all the columns). Also, it is now possible to specify which columns to use
for coordinate matching. The newly added options to Query are:
--ccol: to specify the coordinates used for RA and Dec.
--information: to get dataset and column information.
--limitinfo: to constrain the printed information.
--head: only print top rows.
In the process, the documentaion of Query was also fully edited and mostly
re-written to reflect the new changes. The following changes were made in
other parts.
- In the FITS library, two new functions have been added to allow the
programs to add COMMENT keywords to the output:
- gal_fits_key_list_comment_add
- gal_fits_key_list_comment_add_end
These were necessary due to the new feature to write the full
downloading command into the header of the downloaded file. Also the
function to write keywrods ('gal_fits_key_write_in_ptr') accounts for
writing comments.
- In the internal option library, the new 'gal_options_merge_list_of_csv'
function has been added to merge a list of strings (where each may be a
CSV string) into one list with all values separated by commas being
sparate.
- A bug in 'gal_table_print_info' was fixed: when the input has no
columns, it won't crash with a segmentation fault, it will just return
(because there is nothing to print).
---
NEWS | 24 +--
bin/query/args.h | 67 +++++--
bin/query/gaia.c | 81 +++++----
bin/query/main.h | 5 +
bin/query/query.c | 304 ++++++++++++++++++++++++++++---
bin/query/tap.c | 330 +++++++++++++++++++++++----------
bin/query/tap.h | 3 +
bin/query/ui.c | 52 +++++-
bin/query/ui.h | 8 +-
bin/query/vizier.c | 92 ++++++----
bin/table/ui.c | 2 +-
doc/gnuastro.texi | 394 ++++++++++++++++++++++++----------------
lib/data.c | 60 +++---
lib/fits.c | 77 +++++++-
lib/gnuastro-internal/options.h | 3 +
lib/gnuastro/fits.h | 22 ++-
lib/options.c | 57 ++++++
lib/table.c | 9 +-
18 files changed, 1175 insertions(+), 415 deletions(-)
diff --git a/NEWS b/NEWS
index 7dd5a49..a0c7625 100644
--- a/NEWS
+++ b/NEWS
@@ -17,15 +17,16 @@ See the end of the file for license conditions.
"Arithmetic operators" subsection of the Arithmetic program's section.
New program:
- - Query ('astquery') is a new program to allow easy submission of
- queries to external datasets on the command-line. The basic spatial
- query can be managed without knowing the query language of the
- database. For example with amiliar options (like '--center' or
- '--width'), or an input image with WCS (the image's sky coverage will
- be calculated internally). You can also directly write the database's
- query statement yourself. Currently Query supports these databases:
- VizieR (containing +20500 datasets, making it the largest database in
- astronomy), as well as ESA's Gaia.
+ - Query ('astquery') is a new program to query to external datasets and
+ retrieve the resulting datasets from the command-line. It is possible
+ to get list of datasets or column names from the databases. The basic
+ spatial query (finding objects in the vicinity of a certain point) can
+ be managed without knowing the query language of the database: with
+ basic options (like '--center' with '--radius' or '--width'), or an
+ input image that has WCS. Currently Query supports VizieR (containing
+ +20500 datasets, making it the largest database in astronomy), as well
+ as ESA's Gaia. See the new "Query" section of the book (under the
+ "Data containers" chapter) for more.
All programs:
- Plain text table inputs can have floating point columns that are in
@@ -140,8 +141,9 @@ See the end of the file for license conditions.
- gal_fits_hdu_is_healpix: Return 1 if HDU is a HEALpix grid.
- gal_fits_hdu_datasum: calculate DATASUM of given HDU in given FITS file.
- gal_fits_hdu_datasum_ptr: calculate DATASUM of opened FITS file pointer.
- - gal_pointer_allocate_ram_or_mmap: allocate space either in RAM or as a
- memory-mapped file.
+ - gal_fits_key_list_comment_add: add a COMMENT keyword to the keyword list.
+ - gal_fits_key_list_comment_add_end: add a COMMENT keyword to end of list.
+ - gal_pointer_allocate_ram_or_mmap: allocate in RAM or memory-mapped file.
- gal_pointer_mmap_free: "free" (actually delete) the memory-mapped file.
- gal_wcs_create: create WCSLIB-compatible WCS from raw values.
- gal_wcs_coverage: Return the sky coverage of given image HDU.
diff --git a/bin/query/args.h b/bin/query/args.h
index 632d7e7..d647e43 100644
--- a/bin/query/args.h
+++ b/bin/query/args.h
@@ -31,6 +31,25 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
/* Array of acceptable options. */
struct argp_option program_options[] =
{
+ /* Input options */
+ {
+ "ccol",
+ UI_KEY_CCOL,
+ "STR,STR",
+ 0,
+ "Coordinate (RA, Dec) column names in dataset.",
+ GAL_OPTIONS_GROUP_INPUT,
+ &p->ccol,
+ GAL_TYPE_STRLL,
+ GAL_OPTIONS_RANGE_ANY,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
+
+
+
+
+
/* Output related options. */
{
"keeprawdownload",
@@ -45,25 +64,38 @@ struct argp_option program_options[] =
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
-
-
-
-
-
- /* Database and dataset. */
{
- "database",
- UI_KEY_DATABASE,
+ "information",
+ UI_KEY_INFORMATION,
+ 0,
+ 0,
+ "Print database or dataset information.",
+ GAL_OPTIONS_GROUP_OUTPUT,
+ &p->information,
+ GAL_OPTIONS_NO_ARG_TYPE,
+ GAL_OPTIONS_RANGE_0_OR_1,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
+ {
+ "limitinfo",
+ UI_KEY_LIMITINFO,
"STR",
0,
- "Name of database (e.g., 'esa').",
+ "Only retrieve dataset info. with this string.",
GAL_OPTIONS_GROUP_INPUT,
- &p->databasestr,
+ &p->limitinfo,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
GAL_OPTIONS_NOT_MANDATORY,
- GAL_OPTIONS_NOT_SET,
+ GAL_OPTIONS_NOT_SET
},
+
+
+
+
+
+ /* Database and dataset. */
{
"query",
UI_KEY_QUERY,
@@ -183,6 +215,19 @@ struct argp_option program_options[] =
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
+ {
+ "head",
+ UI_KEY_HEAD,
+ "INT",
+ 0,
+ "Only download given number of top rows.",
+ UI_GROUP_BYCENTER,
+ &p->head,
+ GAL_TYPE_SIZE_T,
+ GAL_OPTIONS_RANGE_ANY,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET,
+ },
diff --git a/bin/query/gaia.c b/bin/query/gaia.c
index 462d1cb..b012637 100644
--- a/bin/query/gaia.c
+++ b/bin/query/gaia.c
@@ -32,39 +32,42 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include "main.h"
#include "ui.h"
+#include "tap.h"
-void
-gaia_prepare(struct queryparams *p)
+/* Gaia-specific: use simpler names for the commonly used Gaia
+ datasets. This is relevant anytime that '--dataset' has been called. */
+static void
+gaia_sanity_checks(struct queryparams *p)
{
- /* Make sure that atleast one type of constraint is specified. */
- if(p->query==NULL && p->center==NULL && p->overlapwith==NULL)
- error(EXIT_FAILURE, 0, "no '--query', '--center' or '--overlapwith' "
- "specified. At least one of these options are necessary in the "
- "Gaia dataset");
-
- /* If '--center' is given, '--radius' is also necessary. */
- if(p->center || p->overlapwith)
+ /* FIRST CHECK (BEFORE SETTING DEFAULT DATASET): Gaia datasets are large
+ and it doesn't allow downloading of the full dataset anonymously! You
+ need to contact them for that. So if no constraints are given for the
+ rows, a warning will be printed. */
+ if(p->information==0
+ && p->datasetstr
+ && p->query==NULL
+ && p->center==NULL
+ && p->range==NULL)
+ error(EXIT_FAILURE, 0, "no constraints specified!"
+ "In other words, you are asking for all the rows within this "
+ "dataset! Gaia datasets have billions of rows, therefore it "
+ "has a limit on the number of rows downloaded anonymously. "
+ "For bulk download access, you should contact "
+ "'gaia-helpdesk@cosmos.esa.int'. Alternatively, you can "
+ "constrain your search to a certain spatial region with "
+ "'--center=RA,DEC' (supplemented by '--radius' or '--width' in "
+ "degrees) or use '--overlapwith' to only download rows that "
+ "overlap with the provided image. See the documentation for "
+ "more with this command: 'info astquery' (press 'q' to return "
+ "to the command-line).");
+
+ /* Fix the summarized names. */
+ if(p->datasetstr)
{
- /* Make sure the radius is given, and that it isn't zero. */
- if(p->overlapwith==NULL && p->radius==NULL && p->width==NULL)
- error(EXIT_FAILURE, 0, "the '--radius' ('-r') or '--width' ('-w') "
- "options are necessary with the '--center' ('-C') option");
-
- /* If no dataset is explicitly given, then use default one and let
- the user know. */
- if( p->datasetstr==NULL)
- {
- gal_checkset_allocate_copy("edr3", &p->datasetstr);
- error(EXIT_SUCCESS, 0, "using '%s' dataset since no dataset "
- "was explicitly requested (with '--dataset')",
- p->datasetstr);
- }
-
- /* Use simpler names for the commonly used datasets. */
if( !strcmp(p->datasetstr, "edr3") )
{
free(p->datasetstr);
@@ -91,12 +94,28 @@ gaia_prepare(struct queryparams *p)
gal_checkset_allocate_copy("public.tycho2", &p->datasetstr);
}
}
+}
+
+
+
+
+
+void
+gaia_prepare(struct queryparams *p)
+{
+ /* Gaia-specific settings. */
+ gaia_sanity_checks(p);
/* Set the URLs, note that this is a simply-linked list, so we need to
- reverse it in the end to have the same order here. */
- gal_list_str_add(&p->urls, "https://gea.esac.esa.int/tap-server/tap/sync";,
0);
+ reverse it in the end (with 'gal_list_str_reverse') to have the same
+ order here. */
+ gal_list_str_add(&p->urls,
+ "https://gea.esac.esa.int/tap-server/tap/sync";, 0);
+
+ /* Name of default RA Dec columns. */
+ if(p->ra_name==NULL) p->ra_name="ra";
+ if(p->dec_name==NULL) p->dec_name="dec";
- /* Name of RA Dec columns to use in Gaia. */
- p->ra_name="ra";
- p->dec_name="dec";
+ /* Basic sanity checks. */
+ tap_sanity_checks(p);
}
diff --git a/bin/query/main.h b/bin/query/main.h
index fd0e427..66b777e 100644
--- a/bin/query/main.h
+++ b/bin/query/main.h
@@ -45,9 +45,13 @@ struct queryparams
/* From command-line */
struct gal_options_common_params cp; /* Common parameters. */
uint8_t keeprawdownload; /* Keep raw downloaded file. */
+ uint8_t information; /* Print information on database. */
+ char *limitinfo; /* Limit retried dataset information. */
int database; /* ID of database to use. */
char *datasetstr; /* ID of dataset in database to use. */
+ size_t head; /* The number of top rows. */
char *overlapwith; /* Image to use instead of center. */
+ gal_list_str_t *ccol; /* Coordinate column names in dataset.*/
gal_data_t *center; /* Center position of query. */
gal_data_t *radius; /* Radius around center. */
gal_data_t *range; /* Range of magnitudes to query. */
@@ -62,6 +66,7 @@ struct queryparams
gal_list_str_t *urls; /* List of URLs to use. */
char *ra_name; /* Name of RA column. */
char *dec_name; /* Name of Dec columns. */
+ char *finalcommand; /* The final command used. */
/* Output: */
time_t rawtime; /* Starting time of the program. */
diff --git a/bin/query/query.c b/bin/query/query.c
index 112d402..b1c1a93 100644
--- a/bin/query/query.c
+++ b/bin/query/query.c
@@ -41,6 +41,246 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
+
+/* Go over the columns and see if a given column exists. */
+static int
+query_output_meta_col(gal_list_str_t **cols, gal_data_t *allcols,
+ size_t numcols, char *string)
+{
+ size_t i;
+ for(i=0; i<numcols; ++i)
+ if( !strcmp(allcols[i].name, string) )
+ {
+ gal_list_str_add(cols, string, 0);
+ return 1;
+ }
+ return 0;
+}
+
+
+
+
+
+/* Read the downloaded metadata for all the tables (datasets) within a
+ database and print them in an easy to read format. */
+static void
+query_output_meta_database(struct queryparams *p)
+{
+ int tableformat;
+ size_t i, *size=NULL;
+ size_t numcols, numrows;
+ gal_list_str_t *cols=NULL;
+ char **name, **type, **desc;
+ gal_data_t *table, *allcols, *scol=NULL;
+
+ /* Get the downloaded metadata column information so we can ask for the
+ proper columns. */
+ allcols=gal_table_info(p->downloadname, "1", NULL, &numcols,
+ &numrows, &tableformat);
+
+ /* Parse the column information to set the necessary columns. */
+ if( query_output_meta_col(&cols, allcols, numcols, "table_name") == 0 )
+ { error(EXIT_SUCCESS, 0, "no 'table_name' found, but this "
+ "is required by the IVOA TAP standard"); return; }
+ if( query_output_meta_col(&cols, allcols, numcols, "description") == 0 )
+ { error(EXIT_SUCCESS, 0, "no 'description' found, but this "
+ "is required by the IVOA TAP standard"); return; }
+ if( query_output_meta_col(&cols, allcols, numcols, "table_type") == 0 )
+ { error(EXIT_SUCCESS, 0, "no 'table_type' found, but this "
+ "is required by the IVOA TAP standard"); return; }
+ query_output_meta_col(&cols, allcols, numcols, "size");
+
+ /* Read the necessary columns in the desired order, we just need to
+ reverse the list first since it is last-in-first-out. */
+ gal_list_str_reverse(&cols);
+ table=gal_table_read(p->downloadname, "1", NULL, cols,
+ GAL_TABLE_SEARCH_NAME, 1, p->cp.minmapsize,
+ p->cp.quietmmap, NULL);
+
+ /* Set the basic columns for easy reading. */
+ name=table->array;
+ desc=table->next->array;
+ type=table->next->next->array;
+ if(table->next->next->next)
+ {
+ scol=gal_data_copy_to_new_type_free(table->next->next->next,
+ GAL_TYPE_SIZE_T);
+ table->next->next->next=scol;
+ size=scol->array;
+ }
+
+ /* Print all the information for those tables that have a type of
+ 'table'. If I understood the TAP standard properly, the 'view' ones
+ aren't relevant for non-webpage users. */
+ printf("\nRETRIEVED DATASET INFORMATION\n"
+ "=============================\n");
+ printf("Database: %s (URL: %s)\n", p->databasestr, p->urls->v);
+ if(p->limitinfo)
+ printf("Only datasets containing string below in description "
+ "(case sensitive): '%s'\n", p->limitinfo);
+ if(table->size)
+ {
+ for(i=0;i<table->size;++i)
+ if( !strcmp(type[i],"table") )
+ {
+ printf("\n%zu of %zu\n"
+ "==================\n"
+ "DATASET NAME: %s\n"
+ "------------------\n", i+1, table->size, name[i]);
+ if(size)
+ printf("DATASET SIZE (number of rows): %zu\n"
+ "------------------\n", size[i]);
+ printf("DATASET DESCRIPTION:\n%s\n"
+ "==================\n", desc[i]);
+ }
+ }
+ else printf("\nNO DATASET FOUND!\n");
+
+ /* Clean up and return. */
+ gal_list_data_free(table);
+ gal_data_array_free(allcols, numcols, 0);
+}
+
+
+
+
+
+static uint8_t
+query_type_from_tap(char *typestr)
+{
+ uint8_t type;
+ if( !strcmp(typestr,"BOOLEAN") ) type=GAL_TYPE_UINT8;
+ else if( !strcmp(typestr,"BIGINT") ) type=GAL_TYPE_INT64;
+ else if( !strcmp(typestr,"REAL") ) type=GAL_TYPE_FLOAT32;
+ else if( !strcmp(typestr,"DOUBLE") ) type=GAL_TYPE_FLOAT64;
+ else if( !strcmp(typestr,"SMALLINT")
+ || !strcmp(typestr,"INTEGER") ) type=GAL_TYPE_INT32;
+ else if( !strcmp(typestr,"VARCHAR")
+ || !strcmp(typestr,"STRING")
+ || !strncmp(typestr, "CHAR", 4) ) type=GAL_TYPE_STRING;
+ else type=GAL_TYPE_INVALID;
+ return type;
+}
+
+
+
+
+/* Read the downloaded metadata for all the columns within a table
+ (dataset) and print them in an easy to read format. */
+static void
+query_output_meta_dataset(struct queryparams *p)
+{
+ size_t i;
+ int tableformat;
+ size_t numcols, numrows;
+ gal_list_str_t *cols=NULL;
+ gal_data_t *table, *allcols;
+ char **name, **type, **unit, **desc;
+
+ /* Get the downloaded metadata column information so we can ask for the
+ proper columns. */
+ allcols=gal_table_info(p->downloadname, "1", NULL, &numcols,
+ &numrows, &tableformat);
+
+ /* Parse the column information to set the necessary columns. */
+ if( query_output_meta_col(&cols, allcols, numcols, "column_name") == 0 )
+ { error(EXIT_SUCCESS, 0, "no 'column_name' found, but this "
+ "is required by the IVOA TAP standard"); return; }
+ if( query_output_meta_col(&cols, allcols, numcols, "datatype") == 0 )
+ { error(EXIT_SUCCESS, 0, "no 'datatype' found, but this "
+ "is required by the IVOA TAP standard"); return; }
+ if( query_output_meta_col(&cols, allcols, numcols, "description") == 0 )
+ { error(EXIT_SUCCESS, 0, "no 'description' found, but this "
+ "is required by the IVOA TAP standard"); return; }
+ if( query_output_meta_col(&cols, allcols, numcols, "unit") == 0 )
+ { error(EXIT_SUCCESS, 0, "no 'unit' found, but this "
+ "is required by the IVOA TAP standard"); return; }
+
+ /* Read the necessary columns in the desired order, we just need to
+ reverse the list first since it is last-in-first-out. */
+ gal_list_str_reverse(&cols);
+ table=gal_table_read(p->downloadname, "1", NULL, cols,
+ GAL_TABLE_SEARCH_NAME, 1, p->cp.minmapsize,
+ p->cp.quietmmap, NULL);
+
+ /* It may happen that the required dataset name isn't recognized by the
+ database. In this case, 'table' will have 0 rows. */
+ if(table->size)
+ {
+ /* Free the initial 'allcols' and set the array pointers. */
+ name=table->array;
+ type=table->next->array;
+ desc=table->next->next->array;
+ unit=table->next->next->next->array;
+ gal_data_array_free(allcols, numcols, 0);
+
+ /* Allocate the new 'allcols' and fill it with the metadata. */
+ numcols=table->size;
+ allcols=gal_data_array_calloc(numcols);
+ for(i=0;i<numcols;++i)
+ {
+ allcols[i].type=query_type_from_tap(type[i]);
+ gal_checkset_allocate_copy(name[i], &allcols[i].name);
+ gal_checkset_allocate_copy(desc[i], &allcols[i].comment);
+ if( !strcmp(unit[i]," ") ) allcols[i].unit=NULL;
+ else gal_checkset_allocate_copy(unit[i], &allcols[i].unit);
+ }
+
+ /* Print the basic information. */
+ printf("\n--------\ndatabase: %s (URL: %s)\ndataset: %s\n",
+ p->databasestr, p->urls->v, p->datasetstr);
+ gal_table_print_info(allcols, numcols, GAL_BLANK_SIZE_T);
+ }
+ else
+ {
+ /* We are using 'error' to have the progam name at the start, and so
+ it goes to 'stderr'. But we aren't exiting with 'EXIT_FAILURE',
+ because Query still has work to do (for example deleting the
+ temporarily downloaded file). */
+ printf("\n");
+ error(EXIT_SUCCESS, 0, "no '%s' dataset found in the '%s' database. "
+ "For the list of datasets within this database, please run the "
+ "command below (put any search word or phrase in 'SEARCH' to "
+ "find your dataset more easily):\n\n"
+ " astquery %s --information --limitinfo=\"SEARCH\"\n\n",
+ p->datasetstr, p->databasestr, p->databasestr);
+ }
+
+ /* Clean up and return. */
+ gal_list_data_free(table);
+ gal_data_array_free(allcols, numcols, 0);
+}
+
+
+
+
+
+/* Read the raw download data, and write them into the output file with
+ Gnuastro's own library. */
+static void
+query_output_data(struct queryparams *p)
+{
+ gal_data_t *table;
+
+ /* Read the table and write it into a clean output (in case the
+ downloaded table is compressed in any special FITS way). */
+ table=gal_table_read(p->downloadname, "1", NULL, NULL,
+ GAL_TABLE_SEARCH_NAME, 1, p->cp.minmapsize,
+ p->cp.quietmmap, NULL);
+ gal_table_write(table, NULL, NULL, p->cp.tableformat,
+ p->cp.output ? p->cp.output : p->cp.output,
+ "QUERY", 0);
+
+ /* Get basic information about the table and free it. */
+ p->outtableinfo[0]=table->size;
+ p->outtableinfo[1]=gal_list_data_number(table);
+ gal_list_data_free(table);
+}
+
+
+
+
+
void
query_check_download(struct queryparams *p)
{
@@ -48,7 +288,6 @@ query_check_download(struct queryparams *p)
int status=0;
char *logname;
fitsfile *fptr;
- gal_data_t *table;
/* Open the FITS file and if the status value is still zero, it means
everything worked properly. */
@@ -58,19 +297,13 @@ query_check_download(struct queryparams *p)
/* Close the FITS file pointer. */
fits_close_file(fptr, &status);
- /* Read the table and write it into a clean output (in case the
- downloaded table is compressed in any special FITS way). */
- table=gal_table_read(p->downloadname, "1", NULL, NULL,
- GAL_TABLE_SEARCH_NAME, 1, p->cp.minmapsize,
- p->cp.quietmmap, NULL);
- gal_table_write(table, NULL, NULL, p->cp.tableformat,
- p->cp.output ? p->cp.output : p->cp.output,
- "QUERY", 0);
-
- /* Get basic information about the table and free it. */
- p->outtableinfo[0]=table->size;
- p->outtableinfo[1]=gal_list_data_number(table);
- gal_list_data_free(table);
+ /* Prepare the output dataset. */
+ if(p->information)
+ {
+ if(p->datasetstr) query_output_meta_dataset(p);
+ else query_output_meta_database(p);
+ }
+ else query_output_data(p);
/* Delete the raw downloaded file if necessary. */
if(p->keeprawdownload==0) remove(p->downloadname);
@@ -87,16 +320,21 @@ query_check_download(struct queryparams *p)
know. */
rename(p->downloadname, logname);
if(p->cp.quiet==0) printf("\n");
- error(EXIT_FAILURE, 0, "the requested dataset could not be retrieved! "
- "For more, please see '%s'", logname);
+ error(EXIT_FAILURE, 0, "the requested dataset could not be "
+ "retrieved! For more, please see '%s'", logname);
}
/* Add the query keywords to the first extension (if the output was a
FITS file). */
- if( gal_fits_name_is_fits (p->cp.output) )
- gal_fits_key_write_config(&p->cp.okeys, "Query settings",
- "QUERY-CONFIG", p->cp.output, "0");
-
+ if( p->information==0 && gal_fits_name_is_fits(p->cp.output) )
+ {
+ gal_fits_key_list_title_add_end(&p->cp.okeys,
+ "Constructed query command", 0);
+ gal_fits_key_list_fullcomment_add_end(&p->cp.okeys,
+ p->finalcommand, 1);
+ gal_fits_key_write_config(&p->cp.okeys, "Query settings",
+ "QUERY-CONFIG", p->cp.output, "0");
+ }
}
@@ -112,9 +350,9 @@ query(struct queryparams *p)
case QUERY_DATABASE_GAIA: gaia_prepare(p); break;
case QUERY_DATABASE_VIZIER: vizier_prepare(p); break;
default:
- error(EXIT_FAILURE, 0, "%s: a bug! Please contact us at %s to address "
- "the problem. '%d' is not a recognized database code", __func__,
- PACKAGE_BUGREPORT, p->database);
+ error(EXIT_FAILURE, 0, "%s: a bug! Please contact us at %s to "
+ "address the problem. '%d' is not a recognized database "
+ "code", __func__, PACKAGE_BUGREPORT, p->database);
}
/* Download the requested query. */
@@ -127,13 +365,21 @@ query(struct queryparams *p)
/* Let the user know that things went well. */
if(p->cp.quiet==0)
{
- printf("\nQuery resulted in %zu rows and %zu columns.\n",
- p->outtableinfo[0], p->outtableinfo[1]);
+ if(p->information==0)
+ printf("\nQuery resulted in %zu rows and %zu columns.\n",
+ p->outtableinfo[0], p->outtableinfo[1]);
if(p->keeprawdownload)
- printf("Query's raw downloaded file: %s\n", p->downloadname);
- printf("Query's final output: %s\n", p->cp.output);
- printf("TIP: use the command below for column information:\n"
- " asttable %s --info\n", p->cp.output);
+ {
+ if(p->information) printf("\n");
+ printf("Query's raw downloaded file: %s\n", p->downloadname);
+ }
+ if(p->information==0)
+ {
+ printf("Query's final output: %s\n", p->cp.output);
+ printf("TIP: use the command below for more on the "
+ "downloaded table:\n"
+ " asttable %s --info\n", p->cp.output);
+ }
}
/* Clean up. */
diff --git a/bin/query/tap.c b/bin/query/tap.c
index e46e99a..9150945 100644
--- a/bin/query/tap.c
+++ b/bin/query/tap.c
@@ -29,20 +29,66 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include <gnuastro/wcs.h>
+#include <gnuastro-internal/checkset.h>
+
#include "main.h"
#include "ui.h"
+
+
+/* Basic sanity checks necessary in all TAP-based databases. */
+void
+tap_sanity_checks(struct queryparams *p)
+{
+ /* Checks in case a raw query isn't given. */
+ if(p->query==NULL)
+ {
+ /* If '--center' is given, '--radius' is also necessary. */
+ if(p->center || p->overlapwith)
+ {
+ /* Make sure the radius is given, and that it isn't zero. */
+ if(p->center && p->radius==NULL && p->width==NULL)
+ error(EXIT_FAILURE, 0, "the '--radius' ('-r') or "
+ "'--width' ('-w') options are necessary with "
+ "the '--center' ('-C') option");
+ }
+
+ /* If no dataset is explicitly given, let the user know that a
+ catalog reference is necessary. */
+ if( p->information==0 && p->datasetstr==NULL )
+ error(EXIT_FAILURE, 0, "no '--dataset' specified! To get the "
+ "list of available datasets (tables) in this database, "
+ "please run with '--information' (or '-i'). Note that some "
+ "databases (like VizieR) have (tens of) thousands of "
+ "datasets. Hence, for a fast result, its best to limit the "
+ "downloaded and displayed information list by also adding "
+ "--limitinfo=\"SEARCH\" (where 'SEARCH' can be any string "
+ "in the description of the dataset, usually project or "
+ "author names) for example:\n\n"
+ " astquery %s -i --limitinfo=\"SEARCH_STRING\"\n\n"
+ "For more, see the documentation of 'astquery' and the "
+ "\"Available databases\" section of the book for more:\n\n"
+ " info astquery\n"
+ " info gnuastro \"Available databases\"\n",
+ p->databasestr);
+ }
+}
+
+
+
+
+
/* The database string needs to be quoted if it contains a slash. */
-static void
-tap_database_quote_if_necessary(struct queryparams *p)
+static char *
+tap_dataset_quote_if_necessary(struct queryparams *p)
{
int quote=0;
- char *c, *new;
+ char *c, *quoted;
- /* Parse the string. */
+ /* Parse the string for bad characters */
for(c=p->datasetstr; *c!='\0'; ++c)
if(*c=='/') { quote=1; break; }
@@ -50,123 +96,225 @@ tap_database_quote_if_necessary(struct queryparams *p)
if(quote)
{
/* Allocate the new string with quotes. */
- if( asprintf(&new, "\"%s\"", p->datasetstr)<0 )
- error(EXIT_FAILURE, 0, "%s: asprintf allocation ('new')",
+ if( asprintf("ed, "\"%s\"", p->datasetstr)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('quoted')",
__func__);
+ }
+ else quoted=p->datasetstr;
+
+ /* Return the possibly quoted string. */
+ return quoted;
+}
+
- /* Free the old one, and replace it. */
- free(p->datasetstr);
- p->datasetstr=new;
+
+
+
+/* Construct the query for metadata download. */
+static char *
+tap_query_construct_meta(struct queryparams *p)
+{
+ char *querystr;
+
+ /* If a dataset is given, build the query to download the metadata of
+ that dataset. Otherwise, get the metadata of the full database. */
+ if(p->datasetstr)
+ {
+ if( asprintf(&querystr, "\"SELECT * FROM TAP_SCHEMA.columns "
+ "WHERE table_name = '%s'\"", p->datasetstr)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('querystr')",
+ __func__);
}
+ else
+ {
+ if(p->limitinfo)
+ {
+ if( asprintf(&querystr, "\"SELECT * FROM TAP_SCHEMA.tables "
+ "WHERE description LIKE '%%%s%%'\"", p->limitinfo)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('querystr')",
+ __func__);
+ }
+ else
+ gal_checkset_allocate_copy("\"SELECT * FROM TAP_SCHEMA.tables\"",
+ &querystr);
+ }
+
+ /* Clean up and return. */
+ return querystr;
}
-void
-tap_download(struct queryparams *p)
+/* Construct the spatial-constraints criteria if necessary. */
+static char *
+tap_query_construct_spatial(struct queryparams *p)
{
size_t ndim;
- gal_data_t *tmp;
- gal_list_str_t *url;
double width2, *center, *darray;
- char *tmpstr, *regionstr, *rangestr=NULL;
- char *command, *columns, allcols[]="*", *querystr;
+ char *regionstr, *spatialstr=NULL;
double *ocenter=NULL, *owidth=NULL, *omin=NULL, *omax=NULL;
- /* If the raw query has been given, use it. */
- if(p->query)
- querystr=p->query;
- else
+ /* If the user wanted an overlap with an image, then calculate it. */
+ if(p->overlapwith)
{
- /* If certain columns have been requested use them, otherwise
- download all existing columns.*/
- columns = p->columns ? ui_strlist_to_str(p->columns) : allcols;
+ /* Calculate the Sky coverage of the overlap dataset. */
+ gal_wcs_coverage(p->overlapwith, p->cp.hdu, &ndim, &ocenter,
+ &owidth, &omin, &omax);
+
+ /* Make sure a WCS existed in the file. */
+ if(owidth==NULL)
+ error(EXIT_FAILURE, 0, "%s (hdu %s): contains no WCS to "
+ "derive the sky coverage", p->overlapwith, p->cp.hdu);
+ }
- /* If the user wanted an overlap with an image, then calculate it. */
- if(p->overlapwith)
- {
- /* Calculate the Sky coverage of the overlap dataset. */
- gal_wcs_coverage(p->overlapwith, p->cp.hdu, &ndim, &ocenter,
- &owidth, &omin, &omax);
-
- /* Make sure a WCS existed in the file. */
- if(owidth==NULL)
- error(EXIT_FAILURE, 0, "%s (hdu %s): contains no WCS to "
- "derive the sky coverage", p->overlapwith, p->cp.hdu);
- }
+ /* For easy reading. */
+ center = p->overlapwith ? ocenter : p->center->array;
- /* For easy reading. */
- center = p->overlapwith ? ocenter : p->center->array;
+ /* Write the region. */
+ if(p->radius)
+ {
+ darray=p->radius->array;
+ if( asprintf(®ionstr, "CIRCLE('ICRS', %.8f, %.8f, %g)",
+ center[0], center[1], darray[0])<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('regionstr')",
+ __func__);
+ }
+ else if(p->width || p->overlapwith)
+ {
+ darray = p->overlapwith ? owidth : p->width->array;
+ width2 = ( (p->overlapwith || p->width->size==2)
+ ? darray[1] : darray[0] );
+ if( asprintf( ®ionstr, "BOX('ICRS', %.8f, %.8f, %.8f, %.8f)",
+ center[0], center[1], darray[0], width2 )<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('regionstr')",
+ __func__);
+ }
- /* Write the region. */
- if(p->radius)
- {
- darray=p->radius->array;
- if( asprintf(®ionstr, "CIRCLE('ICRS', %.8f, %.8f, %g)",
- center[0], center[1], darray[0])<0 )
- error(EXIT_FAILURE, 0, "%s: asprintf allocation ('regionstr')",
- __func__);
- }
- else if(p->width || p->overlapwith)
- {
- darray = p->overlapwith ? owidth : p->width->array;
- width2 = ( (p->overlapwith || p->width->size==2)
- ? darray[1] : darray[0] );
- if( asprintf( ®ionstr, "BOX('ICRS', %.8f, %.8f, %.8f, %.8f)",
- center[0], center[1], darray[0], width2 )<0 )
- error(EXIT_FAILURE, 0, "%s: asprintf allocation ('regionstr')",
- __func__);
- }
+ /* Build the final spatial constraint query string. Note on the
+ quotations: the final query is surrounded by single-quotes
+ ('). However, we need the single quotes around 'ICRS' in this command
+ (both in the final string below and the ones above). So just before
+ the first 'ICRS', we end the single-quote and start a double quote and
+ keep it until the end. Finally, we add a single quote again so all
+ other components of the query can assume that the single-quote
+ environment is active.*/
+ if( asprintf(&spatialstr,
+ "WHERE 1=CONTAINS( POINT('\"'ICRS', %s, %s), %s )\"'",
+ p->ra_name, p->dec_name, regionstr)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('querystr')",
+ __func__);
+
+
+ /* Clean up and return. */
+ free(regionstr);
+ if(p->overlapwith)
+ { free(ocenter); free(owidth); free(omin); free(omax); }
+ return spatialstr;
+}
- /* Set the range criteria on the requested columns. */
- if(p->range)
- for(tmp=p->range; tmp!=NULL; tmp=tmp->next)
+
+
+
+
+/* Construct the query for data download. */
+static char *
+tap_query_construct_data(struct queryparams *p)
+{
+ double *darray;
+ gal_data_t *tmp;
+ char *headstr=NULL, allcols[]="*";
+ char *datasetstr, *rangestr, *prevrange;
+ char *querystr, *columns, *spatialstr=NULL;
+
+ /* If the dataset has special characters (like a slash) it needs to
+ be quoted. */
+ datasetstr=tap_dataset_quote_if_necessary(p);
+
+ /* If certain columns have been requested use them, otherwise
+ download all existing columns.*/
+ columns = p->columns ? ui_strlist_to_str(p->columns) : allcols;
+
+ /* If the user only wants the top few rows, only return them. */
+ if(p->head!=GAL_BLANK_SIZE_T)
+ if( asprintf(&headstr, "TOP %zu", p->head)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('head')",
+ __func__);
+
+ /* If the user has asked for a spatial constraint. */
+ if(p->overlapwith || p->center)
+ spatialstr=tap_query_construct_spatial(p);
+
+ /* Set the range criteria on the requested columns. */
+ prevrange=NULL;
+ if(p->range)
+ for(tmp=p->range; tmp!=NULL; tmp=tmp->next)
+ {
+ /* Write 'rangestr'. */
+ darray=tmp->array;
+ if(prevrange)
{
- darray=tmp->array;
- if( asprintf(&tmpstr, "%s%sAND %s>=%g AND %s<=%g",
- rangestr==NULL ? "" : rangestr,
- rangestr==NULL ? "" : " ",
+ if( asprintf(&rangestr, "%s AND %s>=%g AND %s<=%g", prevrange,
tmp->name, darray[0], tmp->name, darray[1]) < 0 )
- error(EXIT_FAILURE, 0, "%s: asprintf allocation ('tmpstr')",
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('rangestr')",
__func__);
- free(rangestr);
- rangestr=tmpstr;
+ free(prevrange);
}
+ else
+ if( asprintf(&rangestr, "%s>=%g AND %s<=%g",
+ tmp->name, darray[0], tmp->name, darray[1]) < 0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('rangestr')",
+ __func__);
- /* If the dataset has special characters (like a slash) it needs to
- be quoted. */
- tap_database_quote_if_necessary(p);
+ /* Put the 'rangestr' in previous-range string for the next
+ round.*/
+ prevrange=rangestr;
+ }
+
+ /* Write the automatically generated query string. */
+ if( asprintf(&querystr, "'SELECT %s %s FROM %s %s %s %s'",
+ headstr ? headstr : "",
+ columns,
+ datasetstr,
+ spatialstr ? spatialstr : "",
+ ( rangestr && spatialstr
+ ? "AND"
+ : rangestr ? "WHERE" : "" ),
+ rangestr ? rangestr : "")<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('querystr')",
+ __func__);
+
+ /* Clean up and return. */
+ if(datasetstr!=p->datasetstr) free(datasetstr);
+ if(columns!=allcols) free(columns);
+ return querystr;
+}
- /* Write the automatically generated query string. */
- if( asprintf(&querystr, "SELECT %s "
- "FROM %s "
- "WHERE 1=CONTAINS( POINT('\"'ICRS', %s, %s), %s ) %s",
- columns, p->datasetstr, p->ra_name, p->dec_name, regionstr,
- rangestr ? rangestr : "")<0 )
- error(EXIT_FAILURE, 0, "%s: asprintf allocation ('querystr')",
- __func__);
- /* Clean up. */
- free(regionstr);
- if(columns!=allcols) free(columns);
- if(p->overlapwith)
- {free(ocenter); free(owidth); free(omin); free(omax);}
- }
+
+
+void
+tap_download(struct queryparams *p)
+{
+ gal_list_str_t *url;
+ char *command, *querystr;
+
+ /* If the raw query has been given, use it. */
+ querystr = ( p->query
+ ? p->query
+ : ( p->information
+ ? tap_query_construct_meta(p)
+ : tap_query_construct_data(p) ) );
/* Go over the given URLs for this server. */
for(url=p->urls; url!=NULL; url=url->next)
{
- /* Build the calling command. Note the quotations around the value of
- QUERY: we start the values with a single quote, because in ADQL,
- we need double quotes to comment dataset names. However, inside
- 'queryst', we finish the single quotes and switch to double quotes
- because we need the single quotes around 'IRCS'. So here, while we
- started it with a single quote, we finish with double quotes. */
+ /* Build the calling command. Note that the query quotes are
+ included by the function building it. */
if( asprintf(&command, "curl%s -o%s --form LANG=ADQL "
"--form FORMAT=fits --form REQUEST=doQuery "
- "--form QUERY='%s\" %s", p->cp.quiet ? " -s" : "",
+ "--form QUERY=%s %s", p->cp.quiet ? " -s" : "",
p->downloadname, querystr, url->v)<0 )
error(EXIT_FAILURE, 0, "%s: asprintf allocation ('command')",
__func__);
@@ -193,6 +341,6 @@ tap_download(struct queryparams *p)
"if you don't use the option '--quiet', or '-q')");
}
- /* Clean up. */
- free(command);
+ /* Keep the executed command (to put in the final file's meta-data). */
+ p->finalcommand=command;
}
diff --git a/bin/query/tap.h b/bin/query/tap.h
index 7f31b73..02b7c9c 100644
--- a/bin/query/tap.h
+++ b/bin/query/tap.h
@@ -26,6 +26,9 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include "main.h"
void
+tap_sanity_checks(struct queryparams *p);
+
+void
tap_download(struct queryparams *p);
#endif
diff --git a/bin/query/ui.c b/bin/query/ui.c
index 82a431c..988c587 100644
--- a/bin/query/ui.c
+++ b/bin/query/ui.c
@@ -59,7 +59,7 @@ const char *
argp_program_bug_address = PACKAGE_BUGREPORT;
static char
-args_doc[] = "ASTRdata";
+args_doc[] = "DATABASE";
const char
doc[] = GAL_STRINGS_TOP_HELP_INFO PROGRAM_NAME" is just a place holder "
@@ -111,6 +111,9 @@ ui_initialize_options(struct queryparams *p,
cp->program_authors = PROGRAM_AUTHORS;
cp->coptions = gal_commonopts_options;
+ /* Program-specific initializations. */
+ p->head = GAL_BLANK_SIZE_T;
+
/* Modify common options. */
for(i=0; !gal_options_is_last(&cp->coptions[i]); ++i)
{
@@ -144,6 +147,17 @@ ui_initialize_options(struct queryparams *p,
+/* Fixed string */
+#define UI_NODATABASE "Please use the '--database' ('-d') option to " \
+ "specify your desired database, see manual ('info gnuastro " \
+ "astquery' command) for the current databases, here is the list " \
+ "of acceptable values (with their web-based search URLs):\n\n" \
+ " vizier http://vizier.u-strasbg.fr/viz-bin/VizieR\n"; \
+ " gaia https://gea.esac.esa.int/archive\n";
+
+
+
+
/* Parse a single option: */
error_t
parse_opt(int key, char *arg, struct argp_state *state)
@@ -171,7 +185,7 @@ parse_opt(int key, char *arg, struct argp_state *state)
/* Read the non-option tokens (arguments): */
case ARGP_KEY_ARG:
- argp_error(state, "no input arguments are needed");
+ p->databasestr=arg;
break;
@@ -255,10 +269,7 @@ ui_read_check_only_options(struct queryparams *p)
/* See if database has been specified. */
if(p->databasestr==NULL)
- error(EXIT_FAILURE, 0, "no input database.\n\n"
- "Please use the '--database' ('-d') option to specify your "
- "desired database, see manual ('info gnuastro astquery' "
- "command) for the current databases");
+ error(EXIT_FAILURE, 0, "no input database! " UI_NODATABASE);
/* Convert the given string into a code. */
if( !strcmp(p->databasestr, "gaia") ) p->database=QUERY_DATABASE_GAIA;
@@ -268,6 +279,35 @@ ui_read_check_only_options(struct queryparams *p)
"For the full list of recognized databases, please see the "
"documentation (with the command 'info astquery')", p->databasestr);
+ /* If '--limitinfo' is given, but the string is empty (possibly due to a
+ shell variable that wasn't set), remove it. */
+ if(p->limitinfo && p->limitinfo[0]=='\0')
+ {
+ free(p->limitinfo);
+ p->limitinfo=NULL;
+ for(i=0; !gal_options_is_last(&p->cp.poptions[i]); ++i)
+ if( p->cp.poptions[i].key == UI_KEY_LIMITINFO )
+ p->cp.poptions[i].set=GAL_OPTIONS_NOT_SET;
+ }
+
+ /* If '--ccol' is given, first merge all possible calls to it, confirm
+ that there are only two values and put them into the 'ra_name' and
+ 'dec_name' variables. */
+ if(p->ccol)
+ {
+ gal_options_merge_list_of_csv(&p->ccol);
+ if(gal_list_str_number(p->ccol)!=2)
+ error(EXIT_FAILURE, 0, "2 values should be given to '--ccol', "
+ "but you have given %zu values (possibly in multiple calls "
+ "to '--ccols'). Recall that '--ccol' is the coordinate "
+ "column (usually RA and Dec). You can either put them in "
+ "one call (for example '--ccol=ra,dec') or in two (for "
+ "example '--ccol=ra --ccol=dec')",
+ gal_list_str_number(p->ccol));
+ p->ra_name=p->ccol->v;
+ p->dec_name=p->ccol->next->v;
+ }
+
/* Make sure that '--query' and '--center' are not called together. */
if(p->query && (p->center || p->overlapwith) )
error(EXIT_FAILURE, 0, "the '--query' option cannot be called together "
diff --git a/bin/query/ui.h b/bin/query/ui.h
index 248bba0..0a71ea4 100644
--- a/bin/query/ui.h
+++ b/bin/query/ui.h
@@ -42,13 +42,15 @@ enum program_args_groups
/* Available letters for short options:
- a b e f i j m n p t u x y z
- A B E G H J L R W X Y
+ a b e f j m n p t u x y z
+ A B E G J R W X Y
*/
enum option_keys_enum
{
/* With short-option version. */
UI_KEY_KEEPRAWDOWNLOAD = 'k',
+ UI_KEY_INFORMATION = 'i',
+ UI_KEY_LIMITINFO = 'L',
UI_KEY_DATABASE = 'd',
UI_KEY_QUERY = 'Q',
UI_KEY_DATASET = 's',
@@ -58,9 +60,11 @@ enum option_keys_enum
UI_KEY_RANGE = 'g',
UI_KEY_COLUMN = 'c',
UI_KEY_WIDTH = 'w',
+ UI_KEY_HEAD = 'H',
/* Only with long version (start with a value 1000, the rest will be set
automatically). */
+ UI_KEY_CCOL = 1000,
};
diff --git a/bin/query/vizier.c b/bin/query/vizier.c
index 4bb4b00..e8196e1 100644
--- a/bin/query/vizier.c
+++ b/bin/query/vizier.c
@@ -32,6 +32,37 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include "main.h"
#include "ui.h"
+#include "tap.h"
+
+
+static void
+vizier_sanity_checks(struct queryparams *p)
+{
+ /* VizieR specific: if the user has asked for '--information', but
+ without '--limitinfo', print a notice to introduce 'limitinfo'. */
+ if(p->information && p->limitinfo==NULL)
+ {
+ fprintf(stderr, "\n--------------------\n");
+ error(EXIT_SUCCESS, 0, "WARNING: The full VizieR metadata "
+ "(information) is more than 20Mb, and contains tens of "
+ "thousands entries. You can use '--limitinfo=XXXX' to "
+ "constrain the downloaded and displayed metadata to "
+ "those that have 'XXXX' in the description (for example "
+ "a certain author, or a certain project name). This will "
+ "greatly improve the speed of your search");
+ fprintf(stderr, "--------------------\n");
+ }
+
+ /* Set the summarized names. */
+ if(p->datasetstr)
+ {
+ if( !strcmp(p->datasetstr, "gaiaedr3") )
+ {
+ free(p->datasetstr);
+ gal_checkset_allocate_copy("I/350/gaiaedr3", &p->datasetstr);
+ }
+ }
+}
@@ -40,45 +71,28 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
void
vizier_prepare(struct queryparams *p)
{
- /* Make sure that atleast one type of constraint is specified. */
- if(p->query==NULL && p->center==NULL && p->overlapwith==NULL)
- error(EXIT_FAILURE, 0, "no '--query', '--center' or '--overlapwith' "
- "specified. At least one of these options are necessary in the "
- "Gaia dataset");
-
- /* If '--center' is given, '--radius' is also necessary. */
- if(p->center || p->overlapwith)
- {
- /* Make sure the radius is given, and that it isn't zero. */
- if(p->overlapwith==NULL && p->radius==NULL && p->width==NULL)
- error(EXIT_FAILURE, 0, "the '--radius' ('-r') or '--width' ('-w') "
- "options are necessary with the '--center' ('-C') option");
-
- /* If no dataset is explicitly given, let the user know that a
- catalog reference is necessary. */
- if( p->datasetstr==NULL)
- error(EXIT_FAILURE, 0, "no '--dataset' specified. You can "
- "use the URL below to search existing datasets "
- "(catalogs):\n\n"
- " http://cdsarc.u-strasbg.fr/viz-bin/cat";);
- }
+ /* VizieR-specific. */
+ vizier_sanity_checks(p);
/* Set the URLs, note that this is a simply-linked list, so we need to
- reverse it in the end to have the same order here. */
- gal_list_str_add(&p->urls,
"http://tapvizier.u-strasbg.fr/TAPVizieR/tap/sync";, 0);
-
- /* These don't seem to be working as of January 2021.
- gal_list_str_add(&p->urls,
"https://vizier.cfa.harvard.edu/TAPVizieR/tap/sync";, 0);
- gal_list_str_add(&p->urls, "http://vizier.hia.nrc.ca/TAPVizieR/tap/sync";, 0);
- gal_list_str_add(&p->urls, "http://vizier.nao.ac.jp/TAPVizieR/tap/sync";, 0);
- gal_list_str_add(&p->urls, "http://data.bao.ac.cn/TAPVizieR/tap/sync";, 0);
- gal_list_str_add(&p->urls, "http://vizier.ast.cam.ac.uk/TAPVizieR/tap/sync";,
0);
- gal_list_str_add(&p->urls,
"http://www.ukirt.jach.hawaii.edu/TAPVizieR/tap/sync";, 0);
- gal_list_str_add(&p->urls, "http://vizier.inasan.ru/TAPVizieR/tap/sync";, 0);
- */
- gal_list_str_reverse(&p->urls);
-
- /* Name of RA Dec columns to use in Gaia. */
- p->ra_name="RAJ2000";
- p->dec_name="DEJ2000";
+ reverse it in the end (with 'gal_list_str_reverse') to have the same
+ order here.
+
+ Other possible VizieR TAP servers that don't seem to be working now
+ (extracted from the 'visquery' script):
+ http://vizier.cfa.harvard.edu/TAPVizieR/tap/sync
+ http://vizier.nao.ac.jp/TAPVizieR/tap/sync
+ http://data.bao.ac.cn/TAPVizieR/tap/sync
+ http://vizier.ast.cam.ac.uk/TAPVizieR/tap/sync
+ http://www.ukirt.jach.hawaii.edu/TAPVizieR/tap/sync
+ http://vizier.inasan.ru/TAPVizieR/tap/sync */
+ gal_list_str_add(&p->urls,
+ "http://tapvizier.u-strasbg.fr/TAPVizieR/tap/sync";, 0);
+
+ /* Name of default RA Dec columns. */
+ if(p->ra_name==NULL) p->ra_name="RAJ2000";
+ if(p->dec_name==NULL) p->dec_name="DEJ2000";
+
+ /* Basic sanity checks. */
+ tap_sanity_checks(p);
}
diff --git a/bin/table/ui.c b/bin/table/ui.c
index eb7c38a..74d6451 100644
--- a/bin/table/ui.c
+++ b/bin/table/ui.c
@@ -556,8 +556,8 @@ ui_print_info_exit(struct tableparams *p)
static void
ui_columns_prepare(struct tableparams *p)
{
- char *c, **strarr;
gal_data_t *strs;
+ char *c, **strarr;
size_t i, totcalled=0;
struct column_pack *node, *last;
gal_list_str_t *tmp, *toread=NULL;
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index 528d647..d9ab86a 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -415,6 +415,7 @@ Table
Query
+* Available databases:: List of available databases to Query.
* Invoking astquery:: Inputs, outputs and configuration of Query.
Data manipulation
@@ -10342,83 +10343,194 @@ Finally, if you already have a FITS table by other
means (for example by downloa
@node Query, , Table, Data containers
@section Query
+@cindex IVOA
@cindex Query
+@cindex TAP (Table Access Protocol)
+@cindex ADQL (Astronomical Data Query Language)
+@cindex Astronomical Data Query Language (ADQL)
There are many astronomical databases available for downloading astronomical
data.
-Each has its own interface, which is usually very well documented in their own
webpages.
-However, remembering the exact URLs and interface of each database is not
easily possible and using the graphic web interface (which is commonly easier
to use), is not an automatic process.
+Most follow the International Virtual Observatory Alliance (IVOA,
@url{https://ivoa.net}) standards (and in particular the Table Access Protocol,
or TAP@footnote{@url{https://ivoa.net/documents/TAP}}).
+With TAP, it is possible to submit your queries via a command-line downloader
(for example @command{curl}) to only get specific tables, targets (rows in a
table) or measurements (columns in a table): you don't have to download the
full table (which can be very large in some cases)!
+These customizations are done through the Astronomical Data Query Language
(ADQL@footnote{@url{https://ivoa.net/documents/ADQL}}).
-Gnuastro's Query program is designed to address these problems: it has a
common high-level interface for general operations that are common between its
recognized databases.
-For example, importing a catalog of objects within a certain distance of a
given coordinate.
+Therefore, if you are sufficiently familiar with TAP and ADQL, you can easily
custom-download any part of an online dataset.
+However, you also need to keep a record of the URLs of each database and in
many cases, the commands will become long and hard/buggy to type on the
command-line.
+On the other hand, most astronomers don't know TAP or ADQL at all, and are
forced to go to the database's webpage which is slow (it needs to download so
many images, and has too much annoying information), requires manual
interaction (further making it slow and buggy), and can't be automated.
+
+Gnuastro's Query program is designed to be the middle-man in this process: it
provides a simple high-level interface to let you specify your constraints on
what you want to download.
+It then internally constructs the command to download the data based on your
inputs and runs it to download your desired data.
+Query also prints the full command before it executes it (if not called with
@option{--quiet}).
+Also, if you ask for a FITS output table, the full command is written into its
0-th extension along with other input parameters to query (all Gnuastro
programs generally keep their input configuration parameters as FITS keywords
in the zero-th output).
+
+With the full command used to download the dataset, if you only have a minimal
knowledge of ADQL and want to do lower-level customizations on your downloaded
dataset, you can simply copy that command and change the parts you want: ADQL
is very powerful!
+For example you can ask the server to do mathematic operations on the columns
and apply selections after those operations, or combine/match multiple datasets
and etc.
+We will try to add high-level interfaces for such capabilities, but generally,
don't limit yourself to the high-level operations (that can't cover
everything!).
@menu
+* Available databases:: List of available databases to Query.
* Invoking astquery:: Inputs, outputs and configuration of Query.
@end menu
-@node Invoking astquery, , Query, Query
+@node Available databases, Invoking astquery, Query, Query
+@subsection Available databases
+
+The current list of databases supported by Query are listed at the end of this
section.
+To get the list of available datasets within each database, you can use the
@option{--information} option.
+For example with the command below you can get a list of the roughly 100
datasets that are available within the ESA Gaia server with their description:
+
+@example
+$ astquery gaia --information
+@end example
+
+@noindent
+However, other databases like VizieR host many more datasets (tens of
thousands!).
+Therefore it is very inconvenient to get the @emph{full} information everytime
you want to find your dataset of interest (the full metadata file VizieR is
more than 20Mb).
+In such cases, you can limit the downloaded and displayed information with the
@code{--limitinfo} option.
+For example with the first command below, you can get all datasets relating to
the MUSE (an instrument on the Very Large Telescope), and those that include
Roland Bacon (Principle Investigator of MUSE) as an author (@code{Bacon, R.}).
+Recall that @option{-i} is the short format of @option{--information}.
+
+@example
+$ astquery vizier -i --limitinfo=MUSE
+$ astquery vizier -i --limitinfo="Bacon R."
+@end example
+
+Once you find the recognized name of your desired dataset, you can see the
column information of that dataset with adding the dataset name.
+For example, with the command below you can see the column metadata in the
@code{J/A+A/608/A2/udf10} dataset (one of the datasets in the search above)
using this command:
+
+@example
+$ astquery vizier --dataset=J/A+A/608/A2/udf10 -i
+@end example
+
+For very popular datasets of a database, Query provides an easier-to-remember
short name that you can feed to @option{--dataset}.
+This short name will map to the officially recognized name of the dataset on
the server.
+For example in the VizieR and Gaia databases, the recognized name for the
early data release 3 data is respectively @code{I/350/gaiaedr3} and
@code{gaiaedr3.gaia_source}.
+These technical names can be hard to remember.
+Therefore Query provides @code{gaiaedr3} (for VizieR) and @code{edr3} (for
ESA's Gaia) shortcuts which you can give to @option{--dataset} instead.
+They will be directly mapped to the fully recognized name by Query.
+In the list below that describes the available databases, the available short
names are also listed.
+
+The list of databases recognized by Query (and their names in Query) is
described below.
+Since Query is a new member of the Gnuastro family (first available in
Gnuastro 0.14), this list will hopefully grow significantly in the next
releases.
+If you have any particular datasets in mind, please let us know by sending an
email to @code{bug-gnuastro@@gnu.org}.
+If the dataset supports IVOA's TAP (Table Access Protocol), it should be very
easy to add.
+
+
+@table @code
+@item vizier
+@cindex VizieR
+@cindex CDS, VizieR
+@cindex Catalog, Vizier
+@cindex Database, VizieR
+Vizier (@url{https://vizier.u-strasbg.fr}) is arguably the largest catalog
database in astronomy: containing more than 20500 catalogs as of mid January
2021.
+Almost all published catalogs in major projects, and even the tables in many
papers are archived and accessible here.
+For example VizieR also has a full copy of the Gaia database mentioned below,
with some additional standardized columns (like RA and Dec in J2000).
+
+The current implementation of @option{--limitinfo} only looks into the
description of the datasets, but since VizieR is so large, there is still a lot
of room for improvement.
+Until then, if @option{--limitinfo} isn't sufficient, you can use VizieR's own
web-based search for your desired dataset:
@url{http://cdsarc.u-strasbg.fr/viz-bin/cat}
+
+
+Because VizieR curates such a diverse set of data from tens of thousands of
projects and aims for interoperability between them, the column names in VizieR
may not be identical to the column names in the surveys' own databases (Gaia in
the example above).
+A query to @code{vizier} is submitted to
@code{http://tapvizier.u-strasbg.fr/TAPVizieR/tap/sync}.
+
+Here is the list of short names for popular datasets within VizieR:
+@itemize
+@item
+@code{gaiaedr3 --> I/350/gaiaedr3}
+@end itemize
+
+@item gaia
+@cindex Gaia catalog
+@cindex Catalog, Gaia
+@cindex Database, Gaia
+The Gaia project (@url{https://www.cosmos.esa.int/web/gaia}) database which is
a large collection of star positions on the celestial sphere, as well as
peculiar velocities, paralloxes and magnitudes in some bands among many others.
+Besides scientific studies (like studying resolved stellar populations in the
Galaxy and its halo), Gaia is also invaluable for raw data calibrations, like
astrometry.
+A query to @code{gaia} is submitted to
@code{https://gea.esac.esa.int/tap-server/tap/sync}.
+
+Here is the list of short names for popular datasets within Gaia:
+@itemize
+@item
+@code{edr3 --> gaiaedr3.gaia_source}
+@item
+@code{dr2 --> gaiadr2.gaia_source}
+@item
+@code{dr1 --> gaiadr1.gaia_source}
+@item
+@code{tycho2 --> public.tycho2}
+@item
+@code{hipparcos --> public.hipparcos}
+@end itemize
+@end table
+
+
+
+
+@node Invoking astquery, , Available databases, Query
@subsection Invoking Query
Query provides a high-level interface to downloading subsets of data from
databases.
The executable name is @file{astquery} with the following general template
@example
-$ astquery [OPTION...] ...
+$ astquery DATABASE-NAME [OPTION...] ...
@end example
@noindent
One line examples:
@example
-## Import all the columns of all entries in the Gaia early DR3
-## catalog from VizieR (CDS) within 20 arc-minutes of the center.
-$ astquery --database=vizier --dataset=I/350/gaiaedr3 \
- --center=113.8729761,31.9027152 --radius=20/60 \
- --output=gaia-on-vizier.fits
-## Import all the columns of all entries in the Gaia early DR3
-## catalog (default) within 20 arc-minutes of the given coordinate.
-$ astquery --database=gaia --output=my-gaia.fits \
- --center=113.8729761,31.9027152 --radius=20/60
+## Information about all datasets in ESA's GAIA database:
+$ astquery gaia --information
-## Similar to above, but for objects with magnitude range 10 to 15.
-## Note that in Vizier, this column is called 'Gmag'.
-$ astquery --database=gaia --output=my-gaia.fits \
- --center=113.8729761,31.9027152 --width=30/60 \
- --range=phot_g_mean_mag,10:15
+## Only get catalogs in VizieR that have 'MUSE' in their description.
+## The '-i' is short for '--information'.
+$ astquery vizier -i --limitinfo=MUSE
-## Similar to first example, but only import the ID, RA, Dec and G-band
-## magnitude of the sources (not all the columns).
-$ astquery --database=gaia --output=my-gaia.fits \
- --center=113.8729761,31.9027152 --radius=0.1 \
- --column=source_id,ra,dec,phot_g_mean_mag
+## Get the list of columns in 'J/A+A/608/A2/udf10' (one of the above).
+$ astquery vizier --dataset=J/A+A/608/A2/udf10 -i
## Find the ID, RA and Dec of all Gaia sources within an image.
-$ astquery --database=gaia --overlapwith=image.fits
+$ astquery gaia --dataset=edr3 --overlapwith=image.fits \
-csource_id,ra,dec
-## Use a custom query to extract entries in the Gaia early DR3 catalog.
-## The 'XXXX YYYY' can be a query of any size on the command-line.
-$ astquery --database=gaia --query="XXXX YYYY" --output=my-gaia.fits
+## Only extract the RA, DEC and spec. redshift columns (all rows).
+$ astquery vizier --dataset=J/A+A/608/A2/udf10 --column=RAJ2000 \
+ --column=DEJ2000,zMUSE
+
+## All columns of all entries in the Gaia eDR3 catalog (hosted at
+## VizieR) within 1 arc-minute of the given coordinate.
+$ astquery vizier --dataset=I/350/gaiaedr3 --output=my-gaia.fits \
+ --center=113.8729761,31.9027152 --radius=1/60 \
+
+## Similar to above, but only ID, RA and Dec columns for objects with
+## magnitude range 10 to 15. In VizieR, this column is called 'Gmag'.
+$ astquery vizier --dataset=I/350/gaiaedr3 --output=my-gaia.fits \
+ --center=113.8729761,31.9027152 --radius=1/60 \
+ --range=Gmag,10:15 -cEDR3Name,RAJ2000,DEJ2000
@end example
-Query doesn't take any input argument, because the main goal is to retreive
data from external sources.
-The main input to Query is the @option{--database} option which specifies
which database should be contacted for submitting the query.
-There are two methods to query the database, each is more fully discussed in
its option's description below.
+Query takes a single argument which is the name of the database.
+For the full list of available databases and accessing them, see
@ref{Available databases}.
+There are two methods to query the databases, each is more fully discussed in
its option's description below.
@itemize
@item
+@strong{Low-level:}
With @option{--query} you can directly give a raw query statement that is
recognized by the database.
-This is very low level and will require some knowledge of the database's query
language, but of course, it is much more powerful.
-If this option is given, the raw string is directly passed to the server and
all other constraints/options are ignored.
+This is very low level and will require a good knowledge of the database's
query language, but of course, it is much more powerful.
+If this option is given, the raw string is directly passed to the server and
all other constraints/options (for Query's high-level interface) are ignored.
@item
-With the @option{--center}, @option{--radius} and other constraining options
below, the low-level query will be constructed automatically for the particular
database.
+@strong{High-level:}
+With the high-level options (like @option{--column}, @option{--center},
@option{--radius} and other constraining options below), the low-level query
will be constructed automatically for the particular database.
This method is only limited to the generic capabilities that Query provides
for all servers.
So @option{--query} is more powerful, however, in this mode, you don't need
any knowledge of the database's query language.
-When query is run, before contacting the server, it will print the full
command that it executes which contains the raw server query that is
constructed.
+You can see the interlaly generated query on the terminal (if @option{--quiet}
is not used) or in the 0-th extension of the output (if its a FITS file).
+This full command contains the internally generated query.
@end itemize
The name of the downloaded output file can be set with @option{--output}.
-The requested output format can any of the @ref{Recognized table formats}
(currently @file{.txt} or @file{.fits}).
-Like all Gnuastro programs, if the output is a FITS file, the zero-th/first
HDU of the output will contain all the command-line options given to Query.
-If @option{--output} is not set, the output name will be in the format of
@file{NAME-STRING.fits}, where @file{NAME} is the name of the database (same
value given to @option{--database}), and @file{STRING} is a randomly selected
6-character set of numbers and alphabetic characters.
+The requested output format can have any of the @ref{Recognized table formats}
(currently @file{.txt} or @file{.fits}).
+Like all Gnuastro programs, if the output is a FITS file, the zero-th/first
HDU of the output will contain all the command-line options given to Query as
well as the full command used to access the server.
+When @option{--output} is not set, the output name will be in the format of
@file{NAME-STRING.fits}, where @file{NAME} is the name of the database and
@file{STRING} is a randomly selected 6-character set of numbers and alphabetic
characters.
With this feature, a second run of @command{astquery} that isn't called with
@option{--output} will not over-write an already downloaded one.
Generally, when calling Query more than once, it is recommended to set an
output name for each call based on your project's context.
@@ -10428,11 +10540,9 @@ After downloading is complete, the raw downloaded file
will be read into memory
The raw downloaded file will be deleted by default, but can be preserved with
the @option{--keeprawdownload} option.
This strategy avoids unnecessary surprises depending on database.
For example some databases can download a compressed FITS table, even though
we ask for FITS.
-But with the strategy above, the final output will be an uncompressed FITS
file if requested even be in plain text (@ref{Gnuastro text table format}).
-
-@strong{Under development, request for feedback:} Query is a new member of the
Gnuastro family of programs.
-It currently requires that the @command{curl} executable (for the cURL
downloading program) to be present on the host and the number of databases it
supports is still limited, see the list under the @option{--database} option
below.
-More downloader tools, and databases will be added in the near future as it is
used more often, so please don't hesitate to suggest any that you may need.
+But with the strategy above, the final output will be an uncompressed FITS
file.
+The metadata that is added by Query (including the full download command) is
also very useful for future usage of the downloaded data.
+Unfotunately many databases don't write the input queries into their generated
tables.
@table @option
@@ -10442,30 +10552,29 @@ Don't delete the raw downloaded file from the
database.
The name of the raw download will have a @file{OUTPUT-raw-download.fits}
format.
Where @file{OUTPUT} is either the base-name of the final output file (without
a suffix).
-@item -d STR
-@itemx --database=STR
-Identifer for the database for sending the query.
-The current list of databases are listed here:
+@item -i
+@itemx --information
+Print the information of all datasets (tables) within a database or all
columns within a database.
+When @option{--dataset} is specified, the latter mode (all column information)
is downloaded and printed and when its not defined, all dataset information
(within the database) is printed.
-@table @code
-@item vizier
-@cindex VizieR
-@cindex CDS, VizieR
-@cindex Catalog, Vizier
-@cindex Database, VizieR
-Vizier (@url{https://vizier.u-strasbg.fr}) is arguably the largest catalog
database in astronomy: containing more than 20500 catalogs as of mid January
2021.
-Almost all published catalogs in major projects, and even tables in papers are
archived and accessible here.
-For example VizieR also has a full copy of the Gaia database mentioned below,
with some additional standardized columns (like RA and Dec in J2000).
-Just note that because VizieR curates such a diverse set of data from tens of
thousands of projects and aims for interoperability between them, it is forced
to rename some columns compared to the original datasets of large projects.
+Some databases (like VizieR) contain tens of thousands of datasets, so you can
limit the downloaded and printed information for available databases with the
@option{--limitinfo} option (described below).
+Dataset descriptions are often large and contain a lot of text (unlike column
descriptions).
+Therefore when printing the information of all datasets within a database, the
information (e.g., database name) will be printed on separate lines before the
description.
+However, when printing column information, the output has the same format as a
similar option in Table (see @ref{Invoking asttable}).
-@item gaia
-@cindex Gaia catalog
-@cindex Catalog, Gaia
-@cindex Database, Gaia
-The Gaia project (@url{https://www.cosmos.esa.int/web/gaia}) database which is
a large collection of star positions on the celestial sphere, as well as
peculiar velocities, paralloxes and magnitudes in some bands among many others.
-Besides scientific studies (like studying resolved stellar populations in the
Galaxy and its halo), Gaia is also invaluable for raw data calibrations, like
astrometry.
-The query given to this database will be submitted to
@code{https://gea.esac.esa.int/tap-server/tap/sync}.
-@end table
+Important note to consider: the printed order of the datasets or columns is
just for displaying in the printed output.
+You cannot ask for datasets or columns based on the printed order, you need to
use dataset or column names.
+
+@item -L STR
+@itemx --limitinfo=STR
+Limit the information that is downloaded and displayed (with
@option{--information}) to those that have the string given to this option in
their description.
+Note that @emph{this is case-sensitive}.
+This option is only relevant when @option{--information} is also called.
+
+Databases may have thousands (or tens of thousands) of datasets.
+Therefore just the metadata (information) to show with @option{--information}
can be tens of megabytes (for example the full VizieR metadata file is about
23Mb as of January 2021).
+Once downloaded, it can also be hard to parse manually.
+With @option{--limitinfo}, only the metadata of datasets that contain this
string @emph{in their description} will be downloaded and displayed, greatly
improving the speed of finding your desired dataset.
@item -Q "STR"
@itemx --query="STR"
@@ -10475,58 +10584,29 @@ The queries will generally contain space and other
meta-characters, so we recomm
@item -s STR
@itemx --dataset=STR
The dataset to query within the database (not compatible with
@option{--query}).
-The reason for this is that many databases have different types of datasets,
for example different data releases (DRs), or various high-level calculations
on subsets of the database elements.
-For smaller databases like Gaia that have a clear ``main'' dataset, this
option is not mandatory: in such cases, the default database mentioned below
will be used.
-However, for larger databases like VizieR, this option is mandatory.
-
-You can either use the database's official name of the datasets, for example
@code{gaiaedr3.gaia_source} for the early data release 3 the Gaia database, or
a simplified version that maps to it (@code{edr3}) for easy typing on the
command-line.
-Below is a list of the simplified names for the databases that have them.
-
-@table @code
-@item vizier
-As mentioned under @option{--database}, VizieR contains +20000 datasets.
-It is therefore not possible to list them all here!
-The best solutions is to visit its own web-based catalog searching feature:
+This option is mandatory when @option{--query} or @option{--information}
aren't provided.
+You can see the list of available datasets within a database using
@option{--information} (possibly supplemented by @option{--limitinfo}).
+The output of @option{--information} will contain the recognized name of the
datasets within that database.
+You can pass the recognized name directly to this option.
+For more on finding and using your desired database, see @ref{Available
databases}.
-@itemize
-@item @url{http://cdsarc.u-strasbg.fr/viz-bin/cat}
-@end itemize
-
-For example if you want the VizieR version of Gaia early data release 3, you
should use @option{--dataset=I/350/gaiaedr3}.
-
-Alternatively, if you want the 80-row table published in Roman and Trujillo
(2017, @url{http://doi.org/10.1093/mnras/stx438}), you should use
@option{--dataset=J/MNRAS/468/703/tableb1}.
-Just note that individual paper datasets don't usually cover the whole sky and
are focused on a unique sub-set of objects, in one region of the sky.
-
-For example the objects in the paper above are all located around an RA and
Dec (in degrees, epoch 2000) of 18.8 and 0.33 (within a radius of less than 3
degrees).
-If you don't want a sub-set of the catalog, but the whole thing, you can use
the command below.
+@item -c STR
+@itemx --column=STR[,STR[,...]]
+The column name(s) to retrieve from the dataset in the given order (not
compatible with @option{--query}).
+If not given, all the dataset's columns for the selected rows will be queried
(which can be large!).
+This option can take multiple values in one instance (for example
@option{--column=ra,dec,mag}), or in multiple instances (for example
@option{-cra -cdec -cmag}), or mixed (for example @option{-cra,dec -cmag}).
-@example
-astquery --database=vizier \
- --dataset=J/MNRAS/468/703/tableb1 \
- --center=18,0 --radius=3 \
- --output=abell168-udgs.fits
-@end example
+In case, you don't know the full list of the dataset's column names a-priori,
and you don't want to download all the columns (which can greatly decrease your
download speed), you can use the @option{--information} option combined with
the @option{--dataset} option, see @ref{Available databases}.
-@item gaia
-Gaia is a single major project with only 5 datasets that are provided with a
shortcut that you can use.
-@itemize
-@item
-@code{edr3 --> gaiaedr3.gaia_source} (the default dataset)
-@item
-@code{dr2 --> gaiadr2.gaia_source}
-@item
-@code{dr1 --> gaiadr1.gaia_source}
-@item
-@code{tycho2 --> public.tycho2}
-@item
-@code{hipparcos --> public.hipparcos}
-@end itemize
-@end table
+@item -H INT
+@itemx --head=INT
+Only ask for the first @code{INT} rows of the finally selected columns, not
all the rows.
+This can be good when your search can result a large dataset, but before
downloading the full volume, you want to see the top rows and get a feeling of
what the whole dataset looks like.
@item -v STR
@itemx --overlapwith=STR
-File name containing image (in the HDU given by @option{--hdu}) to use for
identifying the region to query in the give database and dataset.
-Based on the image's WCS, the area it covers is estimated and values to the
@option{--center}, @option{--width} will be calculated internally.
+File name of FITS file containing an image (in the HDU given by
@option{--hdu}) to use for identifying the region to query in the give database
and dataset.
+Based on the image's WCS and pixel size, the sky coverage of the image is
estimated and values to the @option{--center}, @option{--width} will be
calculated internally.
Hence this option cannot be used with @code{--center}, @code{--width} or
@code{--radius}.
Also, since it internally generates the query, it can't be used with
@code{--query}.
@@ -10535,9 +10615,16 @@ Therefore the resulting catalog may not overlap, or
correspond to a larger/small
@item -C FLT,FLT
@itemx --center=FLT,FLT
-The center to use for the automatically generated query (not compatible with
@option{--query}).
-All objects within a certain distance of the requested center position will be
requested from the dataset.
-The distance can be specified with the @option{--radius} option.
+The spatial center position (mostly RA and Dec) to use for the automatically
generated query (not compatible with @option{--query}).
+The given values will be compared to two columns in the database to
find/return rows within a certain region around this center position will be
requested and downloaded.
+Pre-defined RA and Dec column names are defined in Query for every database,
however you can use @option{--ccol} to select other columns to use instead.
+The region can either be a circle and the point (configured with
@option{--radius}) or a box/rectangle around the point (configured with
@option{--width}).
+
+@item --ccol=STR,STR
+The name of the coordinate-columns in the dataset to compare with the values
given to @option{--center}.
+Query will use its internal defaults for each dataset (for example
@code{RAJ2000} and @code{DEJ2000} for VizieR data).
+But each dataset is treated separately and it isn't guaranteed that these
columns exist in all datasets.
+Also, more than one coordinate system/epoch may be present in a dataset and
you can use this option to construct your spatial constraint based on the
others coordinate systems/epochs.
@item -r FLT
@itemx --radius=FLT
@@ -10549,33 +10636,18 @@ For example if you want a radius of 20 arc-minutes or
20 arc-seconds, you can us
@itemx --width=FLT[,FLT]
The square (or rectangle) side length (width) about the requested center to
use for the automatically generated query (not compatible with
@option{--query}).
If only one value is given to @code{--width} the region will be a square, but
if two values are given, the widths of the query box along each dimension will
be different.
-The value(s) is(are) in units of degrees, but you can use simple division for
each value directly on the command-line, see the description of
@option{--radius} for more on using division.
+The value(s) is (are) in the same units as the coordinate column (see
@option{--ccol}, usually RA and Dec which are degrees).
+You can use simple division for each value directly on the command-line if you
want relatively small (and more human-friendly) sizes.
+For example if you want your box to be 1 arc-minutes along the RA and 2
arc-minutes along Dec, you can use @option{--width=1/60,2/60}.
@item -g STR,FLT,FLT
@itemx --range=STR,FLT,FLT
The column name and numerical range (inclusive) of acceptable values in that
column (not compatible with @option{--query}).
This option can be called multiple times for applying range limits on many
columns in one call (thus greatly reducing the download size).
-For example @code{--range=phot_g_mean_mag,10:15} will only query for rows that
have a value between 10 and 15 (inclusive) in the @code{phot_g_mean_mag} column
(from the Gaia catalog).
+For example when used on the ESA gaia database, you can use
@code{--range=phot_g_mean_mag,10:15} to only get rows that have a value between
10 and 15 (inclusive on both sides) in the @code{phot_g_mean_mag} column.
If you want the interval to not be inclusive on both sides, you can run
@code{astquery} once and get the command that it executes.
-Then you can edit it to be non-inclusive on one side.
-
-@item -c STR
-@itemx --column=STR[,STR[,...]]
-The optional column name(s) to retrieve from the dataset in the automatically
generated query (not compatible with @option{--query}).
-If not given, all the dataset's columns for the selected rows will be used
(which can be large!).
-This option can take multiple values in one instance (for example
@option{--column=ra,dec,mag}), or in multiple instances (for example
@option{-cra -cdec -cmag}), or mixed (for example @option{-cra,dec -cmag}).
-
-In case, you don't know the full list of the dataset's column names, one
solution is to run this program once without any @option{--column} and setting
@option{--radius=0} (to have few, or no, rows and download fast).
-You can then inspect the column names with @ref{Table}.
-In case the database doesn't support a search radius of 0, or can't return a
table with no rows, you can increase the radius.
-Here is an example on the Gaia DR2 catalog to show the full list of available
columns:
-
-@example
-$ astquery --database=gaia --dataset=edr3 --center=0,0 \
- --radius=0 --output=gaia-edr3-cols.fits
-$ asttable gaia-edr3-cols.fits --info
-@end example
+Then you can edit it to be non-inclusive on your desired side.
@end table
@@ -23388,15 +23460,18 @@ for adding elements to the list.
@example
typedef struct gal_fits_list_key_t
@{
- int kfree; /* ==1, free name. */
- int vfree; /* ==1, free value. */
- int cfree; /* ==1, free comment. */
- uint8_t type; /* Keyword value type. */
- char *keyname; /* Keyword Name. */
- void *value; /* Keyword value. */
- char *comment; /* Keyword comment. */
- char *unit; /* Keyword unit. */
- struct gal_fits_list_key_t *next; /* Next keyword. */
+ int tfree; /* ==1, free title string. */
+ int kfree; /* ==1, free keyword name. */
+ int vfree; /* ==1, free keyword value. */
+ int cfree; /* ==1, free comment. */
+ int ufree; /* ==1, free unit. */
+ uint8_t type; /* Keyword value type. */
+ char *title; /* !=NULL, only print title. */
+ char *keyname; /* Keyword Name. */
+ void *value; /* Keyword value. */
+ char *comment; /* Keyword comment. */
+ char *unit; /* Keyword unit. */
+ struct gal_fits_list_key_t *next; /* Pointer next keyword. */
@} gal_fits_list_key_t;
@end example
@end deftp
@@ -23530,18 +23605,14 @@ strarr = keysll[i].array;
str = strarray[0];
@end example
-If CFITSIO is unable to read a keyword for any reason the @code{status}
-element of the respective @code{gal_data_t} will be non-zero. If it is
-zero, then the keyword was found and successfully read. Otherwise, it is a
-CFITSIO status value. You can use CFITSIO's error reporting tools or
-@code{gal_fits_io_error} (see @ref{FITS macros errors filenames}) for
-reporting the reason of the failure. A tip: when the keyword doesn't exist,
-CFITSIO's status value will be @code{KEY_NO_EXIST}.
+If CFITSIO is unable to read a keyword for any reason the @code{status}
element of the respective @code{gal_data_t} will be non-zero.
+If it is zero, then the keyword was found and successfully read.
+Otherwise, it is a CFITSIO status value.
+You can use CFITSIO's error reporting tools or @code{gal_fits_io_error} (see
@ref{FITS macros errors filenames}) for reporting the reason of the failure.
+A tip: when the keyword doesn't exist, CFITSIO's status value will be
@code{KEY_NO_EXIST}.
-CFITSIO will start searching for the keywords from the last place in the
-header that it searched for a keyword. So it is much more efficient if the
-order that you ask for keywords is based on the order they are stored in
-the header.
+CFITSIO will start searching for the keywords from the last place in the
header that it searched for a keyword.
+So it is much more efficient if the order that you ask for keywords is based
on the order they are stored in the header.
@end deftypefun
@@ -23574,10 +23645,21 @@ Use this function if you want the keywords to be
written in the same order that
@deftypefun void gal_fits_key_list_title_add (gal_fits_list_key_t
@code{**list}, char @code{*title}, int @code{tfree})
Add a special ``title'' keyword (with the @code{title} string) to the top of
the keywords list.
+If @code{cfree} is non-zero, the space allocated for @code{comment} will be
freed immediately after writing the keyword (in another function).
@end deftypefun
@deftypefun void gal_fits_key_list_title_add_end (gal_fits_list_key_t
@code{**list}, char @code{*title}, int @code{tfree})
-Add a special ``title'' keyword (with the @code{title} string) to the
end/bottom of the keywords list.
+Similar to @code{gal_fits_key_list_title_add}, but put the comments at the end
of the list.
+@end deftypefun
+
+@deftypefun void gal_fits_key_list_comment_add (gal_fits_list_key_t
@code{**list}, char @code{*comment}, int @code{fcfree})
+Add a @code{COMMENT} keyword to the top of the keywords list.
+If the comment is longer than 70 characters, CFITSIO will automatically break
it into multiple @code{COMMENT} keywords.
+If @code{fcfree} is non-zero, the space allocated for @code{comment} will be
freed immediately after writing the keyword (in another function).
+@end deftypefun
+
+@deftypefun void gal_fits_key_list_comment_add_end (gal_fits_list_key_t
@code{**list}, char @code{*comment}, int @code{fcfree})
+Similar to @code{gal_fits_key_list_comment_add}, but put the comments at the
end of the list.
@end deftypefun
@deftypefun void gal_fits_key_list_reverse (gal_fits_list_key_t @code{**list})
diff --git a/lib/data.c b/lib/data.c
index ab1c762..f8eb11b 100644
--- a/lib/data.c
+++ b/lib/data.c
@@ -157,8 +157,10 @@ gal_data_initialize(gal_data_t *data, void *array, uint8_t
type,
data->size=1;
for(i=0;i<ndim;++i)
{
- /* Size along a dimension cannot be negative. */
- if(dsize[i] == 0)
+ /* Size along a dimension cannot be 0 if we are in a
+ multi-dimensional dataset. In a single-dimensional dataset, we
+ can have an empty dataset. */
+ if(ndim>1 && dsize[i] == 0)
error(EXIT_FAILURE, 0, "%s: dsize[%zu]==0. The size of a "
"dimension cannot be zero", __func__, i);
@@ -833,32 +835,34 @@ gal_data_copy_to_allocated(gal_data_t *in, gal_data_t
*out)
gal_checkset_allocate_copy(in->comment, &out->comment);
/* Do the copying. */
- switch(out->type)
- {
- case GAL_TYPE_UINT8: COPY_OT_SET( uint8_t ); break;
- case GAL_TYPE_INT8: COPY_OT_SET( int8_t ); break;
- case GAL_TYPE_UINT16: COPY_OT_SET( uint16_t ); break;
- case GAL_TYPE_INT16: COPY_OT_SET( int16_t ); break;
- case GAL_TYPE_UINT32: COPY_OT_SET( uint32_t ); break;
- case GAL_TYPE_INT32: COPY_OT_SET( int32_t ); break;
- case GAL_TYPE_UINT64: COPY_OT_SET( uint64_t ); break;
- case GAL_TYPE_INT64: COPY_OT_SET( int64_t ); break;
- case GAL_TYPE_FLOAT32: COPY_OT_SET( float ); break;
- case GAL_TYPE_FLOAT64: COPY_OT_SET( double ); break;
- case GAL_TYPE_STRING: data_copy_to_string(in, out); break;
-
- case GAL_TYPE_BIT:
- case GAL_TYPE_STRLL:
- case GAL_TYPE_COMPLEX32:
- case GAL_TYPE_COMPLEX64:
- error(EXIT_FAILURE, 0, "%s: copying to %s type not yet supported",
- __func__, gal_type_name(out->type, 1));
- break;
-
- default:
- error(EXIT_FAILURE, 0, "%s: type %d not recognized for 'out->type'",
- __func__, out->type);
- }
+ if(in->array)
+ switch(out->type)
+ {
+ case GAL_TYPE_UINT8: COPY_OT_SET( uint8_t ); break;
+ case GAL_TYPE_INT8: COPY_OT_SET( int8_t ); break;
+ case GAL_TYPE_UINT16: COPY_OT_SET( uint16_t ); break;
+ case GAL_TYPE_INT16: COPY_OT_SET( int16_t ); break;
+ case GAL_TYPE_UINT32: COPY_OT_SET( uint32_t ); break;
+ case GAL_TYPE_INT32: COPY_OT_SET( int32_t ); break;
+ case GAL_TYPE_UINT64: COPY_OT_SET( uint64_t ); break;
+ case GAL_TYPE_INT64: COPY_OT_SET( int64_t ); break;
+ case GAL_TYPE_FLOAT32: COPY_OT_SET( float ); break;
+ case GAL_TYPE_FLOAT64: COPY_OT_SET( double ); break;
+ case GAL_TYPE_STRING: data_copy_to_string(in, out); break;
+
+ case GAL_TYPE_BIT:
+ case GAL_TYPE_STRLL:
+ case GAL_TYPE_COMPLEX32:
+ case GAL_TYPE_COMPLEX64:
+ error(EXIT_FAILURE, 0, "%s: copying to %s type not yet supported",
+ __func__, gal_type_name(out->type, 1));
+ break;
+
+ default:
+ error(EXIT_FAILURE, 0, "%s: type %d not recognized for 'out->type'",
+ __func__, out->type);
+ }
+ else out->array=NULL;
/* Correct the sizes of the output to be the same as the input. If it is
equal, there is no problem, if not, the size information will be
diff --git a/lib/fits.c b/lib/fits.c
index fad6261..a53a426 100644
--- a/lib/fits.c
+++ b/lib/fits.c
@@ -1359,6 +1359,7 @@ gal_fits_key_list_add(gal_fits_list_key_t **list, uint8_t
type,
/* Write the given values into the key structure. */
newnode->title=NULL;
+ newnode->fullcomment=NULL;
newnode->type=type;
newnode->keyname=keyname;
newnode->value=value;
@@ -1393,6 +1394,7 @@ gal_fits_key_list_add_end(gal_fits_list_key_t **list,
uint8_t type,
/* Write the given values into the key structure. */
newnode->title=NULL;
+ newnode->fullcomment=NULL;
newnode->type=type;
newnode->keyname=keyname;
newnode->value=value;
@@ -1446,6 +1448,7 @@ gal_fits_key_list_title_add(gal_fits_list_key_t **list,
char *title, int tfree)
+
/* Put the title keyword in the end. */
void
gal_fits_key_list_title_add_end(gal_fits_list_key_t **list, char *title,
@@ -1483,6 +1486,70 @@ gal_fits_key_list_title_add_end(gal_fits_list_key_t
**list, char *title,
+/* Only set this key to be a full comment */
+void
+gal_fits_key_list_fullcomment_add(gal_fits_list_key_t **list,
+ char *fullcomment, int fcfree)
+{
+ gal_fits_list_key_t *newnode;
+
+ /* Allocate space (and initialize to 0/NULL) for the new node and fill it
+ in. */
+ errno=0;
+ newnode=calloc(1, sizeof *newnode);
+ if(newnode==NULL)
+ error(EXIT_FAILURE, errno, "%s: allocating new node", __func__);
+
+ /* Set the arguments. */
+ newnode->fullcomment=fullcomment;
+ newnode->fcfree=fcfree;
+
+ /* Set the next pointer. */
+ newnode->next=*list;
+ *list=newnode;
+}
+
+
+
+
+
+/* Put the title keyword in the end. */
+void
+gal_fits_key_list_fullcomment_add_end(gal_fits_list_key_t **list,
+ char *fullcomment, int fcfree)
+{
+ gal_fits_list_key_t *tmp, *newnode;
+
+ /* Allocate space (and initialize to 0/NULL) for the new node and fill it
+ in. */
+ errno=0;
+ newnode=calloc(1, sizeof *newnode);
+ if(newnode==NULL)
+ error(EXIT_FAILURE, errno, "%s: allocating new node", __func__);
+
+ /* Set the arguments. */
+ newnode->fullcomment=fullcomment;
+ newnode->fcfree=fcfree;
+
+ /* Set the next pointer. */
+ if(*list) /* List is already full, add this node to the end */
+ {
+ /* After this line, tmp points to the last node. */
+ tmp=*list; while(tmp->next!=NULL) tmp=tmp->next;
+ tmp->next=newnode;
+ newnode->next=NULL;
+ }
+ else /* List is empty */
+ {
+ newnode->next=*list;
+ *list=newnode;
+ }
+}
+
+
+
+
+
void
gal_fits_key_list_reverse(gal_fits_list_key_t **list)
{
@@ -1722,6 +1789,12 @@ gal_fits_key_write_in_ptr(gal_fits_list_key_t **keylist,
fitsfile *fptr)
gal_fits_key_write_title_in_ptr(tmp->title, fptr);
if(tmp->tfree) free(tmp->title);
}
+ else if (tmp->fullcomment)
+ {
+ if( fits_write_comment(fptr, tmp->fullcomment, &status) )
+ gal_fits_io_error(status, NULL);
+ if(tmp->fcfree) free(tmp->fullcomment);
+ }
else
{
/* Write the basic key value and comments. */
@@ -3104,9 +3177,9 @@ gal_fits_tab_read(char *filename, char *hdu, size_t
numrows,
/* Correct the array and sizes. */
out->size=0;
+ out->array=NULL;
+ out->dsize[0]=0;
free(out->array);
- free(out->dsize);
- out->dsize=out->array=NULL;
}
}
diff --git a/lib/gnuastro-internal/options.h b/lib/gnuastro-internal/options.h
index dc48185..fdb7478 100644
--- a/lib/gnuastro-internal/options.h
+++ b/lib/gnuastro-internal/options.h
@@ -295,6 +295,9 @@ void *
gal_options_parse_csv_strings(struct argp_option *option, char *arg,
char *filename, size_t lineno, void *junk);
+void
+gal_options_merge_list_of_csv(gal_list_str_t **list);
+
void *
gal_options_parse_sizes_reverse(struct argp_option *option, char *arg,
char *filename, size_t lineno, void *params);
diff --git a/lib/gnuastro/fits.h b/lib/gnuastro/fits.h
index 3f4e31a..24c119b 100644
--- a/lib/gnuastro/fits.h
+++ b/lib/gnuastro/fits.h
@@ -77,17 +77,19 @@ __BEGIN_C_DECLS /* From C++ preparations */
/* To create a linked list of headers. */
typedef struct gal_fits_list_key_t
{
- int tfree; /* ==1, free title string. */
- int kfree; /* ==1, free keyword name. */
- int vfree; /* ==1, free keyword value. */
- int cfree; /* ==1, free comment. */
- int ufree; /* ==1, free unit. */
- uint8_t type; /* Keyword value type. */
char *title; /* !=NULL, only print title. */
+ char *fullcomment; /* Fully COMMENT (no name). */
char *keyname; /* Keyword Name. */
void *value; /* Keyword value. */
+ uint8_t type; /* Keyword value type. */
char *comment; /* Keyword comment. */
char *unit; /* Keyword unit. */
+ int tfree; /* ==1, free title string. */
+ int fcfree; /* ==1, free full comment. */
+ int kfree; /* ==1, free keyword name. */
+ int vfree; /* ==1, free keyword value. */
+ int cfree; /* ==1, free comment. */
+ int ufree; /* ==1, free unit. */
struct gal_fits_list_key_t *next; /* Pointer next keyword. */
} gal_fits_list_key_t;
@@ -218,6 +220,14 @@ gal_fits_key_list_title_add_end(gal_fits_list_key_t
**list, char *title,
int tfree);
void
+gal_fits_key_list_fullcomment_add(gal_fits_list_key_t **list,
+ char *fullcomment, int fcfree);
+
+void
+gal_fits_key_list_fullcomment_add_end(gal_fits_list_key_t **list,
+ char *fullcomment, int fcfree);
+
+void
gal_fits_key_list_reverse(gal_fits_list_key_t **list);
void
diff --git a/lib/options.c b/lib/options.c
index fe7f40d..e029072 100644
--- a/lib/options.c
+++ b/lib/options.c
@@ -916,6 +916,61 @@ gal_options_parse_csv_strings(struct argp_option *option,
char *arg,
+/* Some options can be called multiple times on the command-line, but
+ within the program, all the values must be merged into one. For example
+ '--column=1 --column=2,3 --column=4,5,6'. In this example, the output
+ 'gal_list_str_t' will have 3 nodes, but we actually want a list that has
+ 6 nodes. */
+void
+gal_options_merge_list_of_csv(gal_list_str_t **list)
+{
+ size_t i;
+ gal_data_t *strs;
+ char *c, **strarr;
+ gal_list_str_t *tmp, *in=*list, *out=NULL;
+
+ /* Go over each input and add it to the list. */
+ for(tmp=in; tmp!=NULL; tmp=tmp->next)
+ {
+ /* Remove any possibly commented new-line where we have a backslash
+ followed by a new-line character (replace the two characters with
+ two single space characters). This can happen with options have
+ have longer scripts and the user is forced to break the line with
+ a '\' followed by newline. */
+ for(c=tmp->v;*c!='\0';++c)
+ if(*c=='\\' && *(c+1)=='\n') { *c=' '; *(++c)=' '; }
+
+ /* Read the different comma-separated strings into an array (within a
+ 'gal_data_t'). */
+ strs=gal_options_parse_csv_strings_raw(tmp->v, NULL, 0);
+ strarr=strs->array;
+
+ /* Go through all the items and add the pointers to the output
+ list. We won't re-allocate the string, we'll just set it to NULL
+ in the array. */
+ for(i=0;i<strs->size;++i)
+ {
+ gal_list_str_add(&out, strarr[i], 0);
+ strarr[i]=NULL;
+ }
+
+ /* Clean up. */
+ gal_data_free(strs);
+ }
+
+ /* Free the input list and reverse the output list to be in the same
+ input order. */
+ gal_list_str_free(in, 1);
+ gal_list_str_reverse(&out);
+
+ /* Reset the input pointer. */
+ *list=out;
+}
+
+
+
+
+
/* Parse the given string into a series of size values (integers, stored as
an array of size_t). The output array will be stored in the 'value'
element of the option. The last element of the array is
@@ -2396,6 +2451,8 @@ option_is_printable(struct argp_option *option)
case GAL_OPTIONS_KEY_LASTCONFIG:
return 0;
}
+
+ /* Everything is fine, print the option. */
return 1;
}
diff --git a/lib/table.c b/lib/table.c
index f875929..eb65630 100644
--- a/lib/table.c
+++ b/lib/table.c
@@ -90,6 +90,9 @@ gal_table_print_info(gal_data_t *allcols, size_t numcols,
size_t numrows)
int Nw=3, nw=4, uw=5, tw=4; /* Initial width from label's width */
char *name, *unit, *comment;
+ /* If there aren't any columns, there is no need to print anything. */
+ if(numcols==0) return;
+
/* Set the widths to print the column information. The width for the
column number can easily be identified from the logarithm of the
number of columns. */
@@ -126,12 +129,14 @@ gal_table_print_info(gal_data_t *allcols, size_t numcols,
size_t numrows)
printf("%-*zu%-*s%-*s%-*s%s\n", Nw, i+1,
nw, name ? name : GAL_BLANK_STRING ,
uw, unit ? unit : GAL_BLANK_STRING ,
- tw, gal_type_name(allcols[i].type, 1),
+ tw,
+ allcols[i].type ? gal_type_name(allcols[i].type, 1) : "--",
comment ? comment : GAL_BLANK_STRING);
}
/* Print the number of rows. */
- printf("--------\nNumber of rows: %zu\n--------\n", numrows);
+ if(numrows!=GAL_BLANK_SIZE_T)
+ printf("--------\nNumber of rows: %zu\n--------\n", numrows);
}
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [gnuastro-commits] master 66a9f60: Query: can now get dataset information, and other new features,
Mohammad Akhlaghi <=