[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[gnuastro-commits] master 6010f00: Query: new program to extract data fr
|
From: |
Mohammad Akhlaghi |
|
Subject: |
[gnuastro-commits] master 6010f00: Query: new program to extract data from external databases |
|
Date: |
Fri, 11 Sep 2020 07:34:44 -0400 (EDT) |
branch: master
commit 6010f004863ee091162664cca7f69e3989a11cb4
Author: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Commit: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Query: new program to extract data from external databases
Many online databases exist in astronomy that provide command-line access
to their data, enabling users to submit queries and extract a sub-set of
them. Usually people do this on the webpages of the various datasets, but
due to the varying format of each dataset, it is annoying.
Hence with this commit, Gnuastro builds a new program called 'astquery', it
has an internal set of databases and provides a high-level interface to
directly download the desired data on the command-line.
---
Makefile.am | 13 +-
NEWS | 21 +++
README | 6 +-
bin/query/Makefile.am | 50 ++++++
bin/query/args.h | 157 +++++++++++++++++
bin/query/astquery.conf | 20 +++
bin/query/authors-cite.h | 38 +++++
bin/query/main.c | 58 +++++++
bin/query/main.h | 58 +++++++
bin/query/query.c | 260 ++++++++++++++++++++++++++++
bin/query/query.h | 36 ++++
bin/query/ui.c | 434 +++++++++++++++++++++++++++++++++++++++++++++++
bin/query/ui.h | 73 ++++++++
configure.ac | 52 +++---
doc/Makefile.am | 20 ++-
doc/gnuastro.texi | 153 ++++++++++++++++-
16 files changed, 1415 insertions(+), 34 deletions(-)
diff --git a/Makefile.am b/Makefile.am
index 0034fd2..f435839 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -83,6 +83,9 @@ endif
if COND_NOISECHISEL
MAYBE_NOISECHISEL = bin/noisechisel
endif
+if COND_QUERY
+ MAYBE_QUERY = bin/query
+endif
if COND_SEGMENT
MAYBE_SEGMENT = bin/segment
endif
@@ -113,12 +116,12 @@ endif
## conditions above). When 'MAYBE_TEMPLATE' is not defined, then Make will
## see it as a blank string and igonore it, so there is no problem with
## having an uncommented 'MAYBE_TEMPLATE' as a value in 'SUBDIRS'.
-SUBDIRS = bootstrapped/lib $(MAYBE_GNULIBCHECK) lib $(MAYBE_ARITHMETIC) \
- $(MAYBE_BUILDPROG) $(MAYBE_CONVERTT) $(MAYBE_CONVOLVE) \
- $(MAYBE_COSMICCAL) $(MAYBE_CROP) $(MAYBE_FITS) $(MAYBE_MATCH) \
+SUBDIRS = bootstrapped/lib $(MAYBE_GNULIBCHECK) lib $(MAYBE_ARITHMETIC) \
+ $(MAYBE_BUILDPROG) $(MAYBE_CONVERTT) $(MAYBE_CONVOLVE) \
+ $(MAYBE_COSMICCAL) $(MAYBE_CROP) $(MAYBE_FITS) $(MAYBE_MATCH) \
$(MAYBE_MKCATALOG) $(MAYBE_MKNOISE) $(MAYBE_MKPROF) $(MAYBE_NOISECHISEL) \
- $(MAYBE_SEGMENT) $(MAYBE_STATISTICS) $(MAYBE_TABLE) $(MAYBE_TEMPLATE) \
- $(MAYBE_WARP) bin/script doc tests
+ $(MAYBE_QUERY) $(MAYBE_SEGMENT) $(MAYBE_STATISTICS) $(MAYBE_TABLE) \
+ $(MAYBE_TEMPLATE) $(MAYBE_WARP) bin/script doc tests
diff --git a/NEWS b/NEWS
index 58253e1..95beb5e 100644
--- a/NEWS
+++ b/NEWS
@@ -3,6 +3,27 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
Copyright (C) 2015-2020 Free Software Foundation, Inc.
See the end of the file for license conditions.
+* Noteworthy changes in release 0.XX (library XX.X.X) (XXXX-XX-XX) [stable]
+
+** New features
+
+ New program:
+ - Query ('astquery') is a new program to allow easy submission of
+ queries to extenral datasets and retrieve the result as a FITS file,
+ directly on the command-line. You can either directly write the
+ database's query statement yourself, or use Query's high-level
+ interface to avoid having to learn it.
+
+** Removed features
+
+** Changed features
+
+** Bugs fixed
+
+
+
+
+
* Noteworthy changes in release 0.13 (library 11.0.0) (2020-09-06) [stable]
** New features
diff --git a/README b/README
index 8fb5619..1e17be0 100644
--- a/README
+++ b/README
@@ -85,6 +85,10 @@ categories/chapters.
in noise (galaxies in the sky), using thresholds that are below the Sky
value (see arXiv:1505.01664).
+ - Query (astquery): High-level interface to query pre-defined remote, or
+ external, databases and directly download the required sub-tables on the
+ command-line.
+
- Segment (astsegment): Segment a detection based on the structure of
signal within it.
@@ -254,4 +258,4 @@ Copyright (C) 2015-2020 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document under
the terms of the GNU Free Documentation License, Version 1.3 or any later
version published by the Free Software Foundation; with no Invariant
-Sections, with no Front-Cover Texts, and with no Back-Cover Texts.
\ No newline at end of file
+Sections, with no Front-Cover Texts, and with no Back-Cover Texts.
diff --git a/bin/query/Makefile.am b/bin/query/Makefile.am
new file mode 100644
index 0000000..89f1f10
--- /dev/null
+++ b/bin/query/Makefile.am
@@ -0,0 +1,50 @@
+## Process this file with automake to produce Makefile.inx
+##
+## Original author:
+## Mohammad akhlaghi <mohammad@akhlaghi.org>
+## Contributing author(s):
+## Copyright (C) 2016-2020, Free Software Foundation, Inc.
+##
+## Gnuastro is free software: you can redistribute it and/or modify it
+## under the terms of the GNU General Public License as published by
+## the Free Software Foundation, either version 3 of the License, or
+## (at your option) any later version.
+##
+## Gnuastro is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Gnuastro. If not, see <http://www.gnu.org/licenses/>.
+
+
+## Necessary pre-processer and linker flags.
+AM_LDFLAGS = -L\$(top_builddir)/lib
+AM_CPPFLAGS = -I\$(top_srcdir)/bootstrapped/lib -I\$(top_srcdir)/lib
+
+if COND_NORPATH
+ MAYBE_NORPATH = $(CONFIG_LDADD)
+endif
+
+
+## Program definition (name, linking, sources and headers)
+bin_PROGRAMS = astquery
+
+## We are linking with 'libgnu' first eventhough 'libgnu' is already linked
+## to 'libgnuastro'. The reason is that some linkers (Mac OS for example),
+## don't keep external variables (needed in Argp) after the first link. So
+## the 'libgnu' (that is indirectly linked through 'libgnuastro') can't see
+## those variables. We thus need to explicitly link with 'libgnu' first.
+astquery_LDADD = $(top_builddir)/bootstrapped/lib/libgnu.la -lgnuastro \
+ $(MAYBE_NORPATH)
+
+astquery_SOURCES = main.c ui.c query.c
+
+EXTRA_DIST = main.h authors-cite.h args.h ui.h query.h
+
+
+
+## The configuration file (distribute and install).
+## NOTE: the man page is created in doc/Makefile.am
+dist_sysconf_DATA = astquery.conf
diff --git a/bin/query/args.h b/bin/query/args.h
new file mode 100644
index 0000000..7eebcd1
--- /dev/null
+++ b/bin/query/args.h
@@ -0,0 +1,157 @@
+/*********************************************************************
+Query - Retreive data from a remote data server.
+Query is part of GNU Astronomy Utilities (Gnuastro) package.
+
+Original author:
+ Mohammad akhlaghi <mohammad@akhlaghi.org>
+Contributing author(s):
+Copyright (C) 2020, Free Software Foundation, Inc.
+
+Gnuastro is free software: you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation, either version 3 of the License, or (at your
+option) any later version.
+
+Gnuastro is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Gnuastro. If not, see <http://www.gnu.org/licenses/>.
+**********************************************************************/
+#ifndef ARGS_H
+#define ARGS_H
+
+
+
+
+
+
+/* Array of acceptable options. */
+struct argp_option program_options[] =
+ {
+ /* Database and dataset. */
+ {
+ "database",
+ UI_KEY_DATABASE,
+ "STR",
+ 0,
+ "Name of database (e.g., 'esa').",
+ GAL_OPTIONS_GROUP_INPUT,
+ &p->database,
+ GAL_TYPE_STRING,
+ GAL_OPTIONS_RANGE_ANY,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET,
+ ui_parse_database
+ },
+ {
+ "dataset",
+ UI_KEY_DATASET,
+ "STR",
+ 0,
+ "Name of dataset in database (e.g., 'gaiadr2').",
+ GAL_OPTIONS_GROUP_INPUT,
+ &p->datasetstr,
+ GAL_TYPE_STRING,
+ GAL_OPTIONS_RANGE_ANY,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET,
+ },
+ {
+ "query",
+ UI_KEY_QUERY,
+ "STR",
+ 0,
+ "The raw query as a simple string.",
+ GAL_OPTIONS_GROUP_INPUT,
+ &p->query,
+ GAL_TYPE_STRING,
+ GAL_OPTIONS_RANGE_ANY,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET,
+ },
+
+
+
+
+
+ /* Query by center. */
+ {
+ 0, 0, 0, 0,
+ "Calculate query by center:",
+ UI_GROUP_BYCENTER,
+ },
+ {
+ "center",
+ UI_KEY_CENTER,
+ "FLT[,...]",
+ 0,
+ "Central coordinates of the query.",
+ UI_GROUP_BYCENTER,
+ &p->center,
+ GAL_TYPE_STRING,
+ GAL_OPTIONS_RANGE_ANY,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET,
+ gal_options_parse_csv_float64
+ },
+ {
+ "radius",
+ UI_KEY_RADIUS,
+ "FLT",
+ 0,
+ "Radius around center to select targets.",
+ UI_GROUP_BYCENTER,
+ &p->radius,
+ GAL_TYPE_FLOAT64,
+ GAL_OPTIONS_RANGE_ANY,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET,
+ },
+ {
+ "column",
+ UI_KEY_COLUMN,
+ "STR",
+ 0,
+ "Column names to download from catalog.",
+ UI_GROUP_BYCENTER,
+ &p->columns,
+ GAL_TYPE_STRLL,
+ GAL_OPTIONS_RANGE_ANY,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
+
+
+
+
+ {0}
+ };
+
+
+
+
+
+/* Define the child argp structure
+ -------------------------------
+
+ NOTE: these parts can be left untouched.*/
+struct argp
+gal_options_common_child = {gal_commonopts_options,
+ gal_options_common_argp_parse,
+ NULL, NULL, NULL, NULL, NULL};
+
+/* Use the child argp structure in list of children (only one for now). */
+struct argp_child
+children[]=
+{
+ {&gal_options_common_child, 0, NULL, 0},
+ {0, 0, 0, 0}
+};
+
+/* Set all the necessary argp parameters. */
+struct argp
+thisargp = {program_options, parse_opt, args_doc, doc, children, NULL, NULL};
+#endif
diff --git a/bin/query/astquery.conf b/bin/query/astquery.conf
new file mode 100644
index 0000000..dd5917b
--- /dev/null
+++ b/bin/query/astquery.conf
@@ -0,0 +1,20 @@
+# Default parameters (System) for Query.
+# Query is part of GNU Astronomy Utilities.
+#
+# Use the long option name of each parameter followed by a value. The name
+# and value should be separated by atleast one white-space character (for
+# example ' '[space], or tab). Lines starting with '#' are ignored.
+#
+# For more information, please run these commands:
+#
+# $ astquery --help # Full list of options, short doc.
+# $ astquery -P # Print all options and used values.
+# $ info astquery # All options and input/output.
+# $ info gnuastro "Configuration files" # How to use configuration files.
+#
+# Copyright (C) 2020 Free Software Foundation, Inc.
+#
+# Copying and distribution of this file, with or without modification, are
+# permitted in any medium without royalty provided the copyright notice and
+# this notice are preserved. This file is offered as-is, without any
+# warranty.
diff --git a/bin/query/authors-cite.h b/bin/query/authors-cite.h
new file mode 100644
index 0000000..b0b7aad
--- /dev/null
+++ b/bin/query/authors-cite.h
@@ -0,0 +1,38 @@
+/*********************************************************************
+Query - Retreive data from a remote data server.
+Query is part of GNU Astronomy Utilities (Gnuastro) package.
+
+Original author:
+ Mohammad akhlaghi <mohammad@akhlaghi.org>
+Contributing author(s):
+Copyright (C) 2020, Free Software Foundation, Inc.
+
+Gnuastro is free software: you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation, either version 3 of the License, or (at your
+option) any later version.
+
+Gnuastro is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Gnuastro. If not, see <http://www.gnu.org/licenses/>.
+**********************************************************************/
+#ifndef AUTHORS_CITE_H
+#define AUTHORS_CITE_H
+
+/* When any specific citation is necessary, please add its BibTeX (from ADS
+ hopefully) to this variable along with a title decribing what this
+ paper/book does for the progarm in a short line. In the following line
+ put a row of '-' with the same length and then put the BibTeX.
+
+ This macro will be used in 'gal_options_print_citation' function of
+ 'lib/options.c' (from the top Gnuastro source code directory). */
+
+#define PROGRAM_BIBTEX ""
+
+#define PROGRAM_AUTHORS "Mohammad Akhlaghi"
+
+#endif
diff --git a/bin/query/main.c b/bin/query/main.c
new file mode 100644
index 0000000..70668bb
--- /dev/null
+++ b/bin/query/main.c
@@ -0,0 +1,58 @@
+/*********************************************************************
+Query - Retreive data from a remote data server.
+Query is part of GNU Astronomy Utilities (Gnuastro) package.
+
+Original author:
+ Mohammad akhlaghi <mohammad@akhlaghi.org>
+Contributing author(s):
+Copyright (C) 2020, Free Software Foundation, Inc.
+
+Gnuastro is free software: you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation, either version 3 of the License, or (at your
+option) any later version.
+
+Gnuastro is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Gnuastro. If not, see <http://www.gnu.org/licenses/>.
+**********************************************************************/
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include <gnuastro-internal/timing.h>
+
+#include "main.h"
+
+#include "ui.h"
+#include "query.h"
+
+
+/* Main function */
+int
+main (int argc, char *argv[])
+{
+ struct timeval t1;
+ struct queryparams p={{{0},0},0};
+
+ /* Set the starting time. */
+ time(&p.rawtime);
+ gettimeofday(&t1, NULL);
+
+ /* Read the input parameters. */
+ ui_read_check_inputs_setup(argc, argv, &p);
+
+ /* Run Query */
+ query(&p);
+
+ /* Free all non-freed allocations. */
+ ui_free_report(&p, &t1);
+
+ /* Return successfully.*/
+ return EXIT_SUCCESS;
+}
diff --git a/bin/query/main.h b/bin/query/main.h
new file mode 100644
index 0000000..319212e
--- /dev/null
+++ b/bin/query/main.h
@@ -0,0 +1,58 @@
+/*********************************************************************
+Query - Retreive data from a remote data server.
+Query is part of GNU Astronomy Utilities (Gnuastro) package.
+
+Original author:
+ Mohammad akhlaghi <mohammad@akhlaghi.org>
+Contributing author(s):
+Copyright (C) 2020, Free Software Foundation, Inc.
+
+Gnuastro is free software: you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation, either version 3 of the License, or (at your
+option) any later version.
+
+Gnuastro is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Gnuastro. If not, see <http://www.gnu.org/licenses/>.
+**********************************************************************/
+#ifndef MAIN_H
+#define MAIN_H
+
+/* Include necessary headers */
+#include <gnuastro/data.h>
+
+#include <gnuastro-internal/options.h>
+
+/* Progarm names. */
+#define PROGRAM_NAME "query" /* Program full name. */
+#define PROGRAM_EXEC "astquery" /* Program executable name. */
+#define PROGRAM_STRING PROGRAM_NAME" (" PACKAGE_NAME ") " PACKAGE_VERSION
+
+
+
+
+
+
+
+/* Main program parameters structure */
+struct queryparams
+{
+ /* From command-line */
+ struct gal_options_common_params cp; /* Common parameters. */
+ int database; /* ID of database to use. */
+ char *datasetstr; /* ID of dataset in database to use. */
+ gal_data_t *center; /* Center position of query. */
+ double radius; /* Radius around center. */
+ char *query; /* Raw query string. */
+ gal_list_str_t *columns; /* Columns to extract from database. */
+
+ /* Output: */
+ time_t rawtime; /* Starting time of the program. */
+};
+
+#endif
diff --git a/bin/query/query.c b/bin/query/query.c
new file mode 100644
index 0000000..819f3df
--- /dev/null
+++ b/bin/query/query.c
@@ -0,0 +1,260 @@
+/*********************************************************************
+Query - Retreive data from a remote data server.
+Query is part of GNU Astronomy Utilities (Gnuastro) package.
+
+Original author:
+ Mohammad akhlaghi <mohammad@akhlaghi.org>
+Contributing author(s):
+Copyright (C) 2020, Free Software Foundation, Inc.
+
+Gnuastro is free software: you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation, either version 3 of the License, or (at your
+option) any later version.
+
+Gnuastro is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Gnuastro. If not, see <http://www.gnu.org/licenses/>.
+**********************************************************************/
+#include <config.h>
+
+#include <stdio.h>
+#include <errno.h>
+#include <error.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <gnuastro/pointer.h>
+
+#include <gnuastro-internal/checkset.h>
+
+#include "main.h"
+#include "query.h"
+
+
+
+
+
+char *
+query_strlist_to_str(gal_list_str_t *input)
+{
+ char *out=NULL;
+ gal_list_str_t *node;
+ size_t n, nn, nnodes=0, alllen=0;
+
+ /* First calculate the full length of all nodes. */
+ for(node=input; node!=NULL; node=node->next)
+ {
+ /* We'll add two extra for each. One for the ',' that must come in
+ between it and the next one. One just for a buffer, incase we
+ haven't accounted for something. */
+ alllen += strlen(node->v) + 2;
+ ++nnodes;
+ }
+
+ /* Allocate the output string. */
+ out=gal_pointer_allocate(GAL_TYPE_STRING, alllen, 1, "out", __func__);
+
+ /* Write all the strings into the allocated space. */
+ n=nn=0;
+ for(node=input; node!=NULL; node=node->next)
+ {
+ if(nn++==nnodes-1)
+ sprintf(out+n, "%s", node->v);
+ else
+ n += sprintf(out+n, "%s,", node->v);
+ }
+
+ /* Return the merged string. */
+ return out;
+}
+
+
+
+
+
+/* Gaia database. */
+void
+query_gaia_sanitycheck(struct queryparams *p)
+{
+ /* Make sure that atleast one type of constraint is specified. */
+ if(p->center==NULL && p->query==NULL)
+ error(EXIT_FAILURE, 0, "no '--center' or '--query' specified. At least "
+ "one of these options are necessary in the Gaia dataset");
+
+ /* If '--center' is given, '--radius' is also necessary. */
+ if(p->center)
+ {
+ /* Make sure the radius is given, and that it isn't zero. */
+ if( isnan(p->radius) )
+ error(EXIT_FAILURE, 0, "the '--radius' ('-r') option is necessary "
+ "with the '--center' ('-c') option");
+
+ /* Make sure a dataset is also given. */
+ if( p->datasetstr==NULL)
+ error(EXIT_FAILURE, 0, "the '--dataset' ('-s') option is necessary "
+ "with the '--center' ('-c') option");
+
+
+ /* Use simpler names for the commonly used datasets. */
+ if( !strcmp(p->datasetstr, "dr2") )
+ {
+ free(p->datasetstr);
+ gal_checkset_allocate_copy("gaiadr2.gaia_source", &p->datasetstr);
+ }
+ else if( !strcmp(p->datasetstr, "dr1") )
+ {
+ free(p->datasetstr);
+ gal_checkset_allocate_copy("gaiadr1.gaia_source", &p->datasetstr);
+ }
+ else if( !strcmp(p->datasetstr, "hipparcos") )
+ {
+ free(p->datasetstr);
+ gal_checkset_allocate_copy("public.hipparcos", &p->datasetstr);
+ }
+ else if( !strcmp(p->datasetstr, "tyco2") )
+ {
+ free(p->datasetstr);
+ gal_checkset_allocate_copy("public.tyco2", &p->datasetstr);
+ }
+ }
+}
+
+
+
+
+
+void
+query_gaia(struct queryparams *p)
+{
+ double *center;
+ char *command, *columns, allcols[]="*", *querystr;
+
+ /* Make sure everything is fine. */
+ query_gaia_sanitycheck(p);
+
+
+ /* If the raw query has been given, use it. */
+ if(p->query)
+ querystr=p->query;
+ else
+ {
+ /* For easy reading. */
+ center=p->center->array;
+
+ /* If certain columns have been requested use them, otherwise
+ download all existing columns.
+
+ columns="source_id,ra,dec,phot_g_mean_mag";
+ */
+ columns = p->columns ? query_strlist_to_str(p->columns) : allcols;
+
+ /* Write the automatically generated query string. */
+ if( asprintf(&querystr, "SELECT %s "
+ "FROM %s "
+ "WHERE 1=CONTAINS( "
+ "POINT('ICRS', ra, dec), "
+ "CIRCLE('ICRS', %.8f, %.8f, %g) )", columns,
+ p->datasetstr, center[0], center[1], p->radius)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('querystr')",
__func__);
+
+ /* Clean up. */
+ if(columns!=allcols) free(columns);
+ }
+
+
+ /* Build the calling command. */
+ if( asprintf(&command, "curl -o%s --form LANG=ADQL --form FORMAT=fits "
+ "--form REQUEST=doQuery --form QUERY=\"%s\" "
+ "https://gea.esac.esa.int/tap-server/tap/sync";, p->cp.output,
+ querystr)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation ('command')", __func__);
+
+
+ /* Print the calling command for the user to know. */
+ if(p->cp.quiet==0)
+ printf("Running: %s\n", command);
+ //exit(0);
+
+ /* Run the command. */
+ if(system(command))
+ error(EXIT_FAILURE, 0, "the query download command %sfailed%s\n",
+ p->cp.quiet==0 ? "printed above " : "",
+ p->cp.quiet==0 ? "" : " (the command can be printed "
+ "if you don't use the option '--quiet', or '-q')");
+}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+void
+query_check_download(struct queryparams *p)
+{
+ size_t len;
+ int status=0;
+ char *logname;
+ fitsfile *fptr;
+
+ /* Open the FITS file and if the status value is still zero, it means
+ everything worked properly. */
+ fits_open_file(&fptr, p->cp.output, READONLY, &status);
+ if(status==0) fits_close_file(fptr, &status);
+ else
+ {
+ /* Add a '.log' suffix to the output filename. */
+ len=strlen(p->cp.output);
+ logname=gal_pointer_allocate(GAL_TYPE_STRING, len+10, 1,
+ __func__, "logname");
+ sprintf(logname, "%s.log", p->cp.output);
+
+ /* Rename the output file to the logname file and let the user
+ know. */
+ rename(p->cp.output, logname);
+ error(EXIT_FAILURE, 0, "the requested dataset could not be retreived! "
+ "For more, please see '%s'", logname);
+ }
+}
+
+
+
+
+
+void
+query(struct queryparams *p)
+{
+ /* Download the dataset. */
+ switch(p->database)
+ {
+ case QUERY_DATABASE_GAIA: query_gaia(p); break;
+ default:
+ error(EXIT_FAILURE, 0, "%s: a bug! Please contact us at %s to address "
+ "the problem. '%d' is not a recognized database code", __func__,
+ PACKAGE_BUGREPORT, p->database);
+ }
+
+ /* Make sure that the result is a readable FITS file, otherwise, abort
+ with an error. */
+ query_check_download(p);
+
+ /* Let the user know that things went well. */
+ if(p->cp.quiet==0)
+ printf("Query output written to: %s\n", p->cp.output);
+}
diff --git a/bin/query/query.h b/bin/query/query.h
new file mode 100644
index 0000000..dc5409b
--- /dev/null
+++ b/bin/query/query.h
@@ -0,0 +1,36 @@
+/*********************************************************************
+Query - Retreive data from a remote data server.
+Query is part of GNU Astronomy Utilities (Gnuastro) package.
+
+Original author:
+ Mohammad akhlaghi <mohammad@akhlaghi.org>
+Contributing author(s):
+Copyright (C) 2020, Free Software Foundation, Inc.
+
+Gnuastro is free software: you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation, either version 3 of the License, or (at your
+option) any later version.
+
+Gnuastro is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Gnuastro. If not, see <http://www.gnu.org/licenses/>.
+**********************************************************************/
+#ifndef query_H
+#define query_H
+
+enum query_databases
+{
+ QUERY_DATABASE_INVALID,
+ QUERY_DATABASE_GAIA,
+};
+
+
+void
+query(struct queryparams *p);
+
+#endif
diff --git a/bin/query/ui.c b/bin/query/ui.c
new file mode 100644
index 0000000..fc050ce
--- /dev/null
+++ b/bin/query/ui.c
@@ -0,0 +1,434 @@
+/*********************************************************************
+Query - Retreive data from a remote data server.
+Query is part of GNU Astronomy Utilities (Gnuastro) package.
+
+Original author:
+ Mohammad akhlaghi <mohammad@akhlaghi.org>
+Contributing author(s):
+Copyright (C) 2020, Free Software Foundation, Inc.
+
+Gnuastro is free software: you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation, either version 3 of the License, or (at your
+option) any later version.
+
+Gnuastro is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Gnuastro. If not, see <http://www.gnu.org/licenses/>.
+**********************************************************************/
+#include <config.h>
+
+#include <argp.h>
+#include <errno.h>
+#include <error.h>
+#include <stdio.h>
+#include <string.h>
+
+#include <gnuastro/fits.h>
+
+#include <gnuastro-internal/timing.h>
+#include <gnuastro-internal/options.h>
+#include <gnuastro-internal/checkset.h>
+#include <gnuastro-internal/fixedstringmacros.h>
+
+#include "main.h"
+
+#include "ui.h"
+#include "query.h"
+#include "authors-cite.h"
+
+
+
+
+
+/**************************************************************/
+/********* Argp necessary global entities ************/
+/**************************************************************/
+/* Definition parameters for the Argp: */
+const char *
+argp_program_version = PROGRAM_STRING "\n"
+ GAL_STRINGS_COPYRIGHT
+ "\n\nWritten/developed by "PROGRAM_AUTHORS;
+
+const char *
+argp_program_bug_address = PACKAGE_BUGREPORT;
+
+static char
+args_doc[] = "ASTRdata";
+
+const char
+doc[] = GAL_STRINGS_TOP_HELP_INFO PROGRAM_NAME" is just a place holder "
+ "used as a minimal set of files and functions necessary for a program in "
+ "Gnuastro. It can be used for learning or as a template to build new "
+ "programs.\n"
+ GAL_STRINGS_MORE_HELP_INFO
+ /* After the list of options: */
+ "\v"
+ PACKAGE_NAME" home page: "PACKAGE_URL;
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+/**************************************************************/
+/********* Initialize & Parse command-line **************/
+/**************************************************************/
+static void
+ui_initialize_options(struct queryparams *p,
+ struct argp_option *program_options,
+ struct argp_option *gal_commonopts_options)
+{
+ size_t i;
+ struct gal_options_common_params *cp=&p->cp;
+
+
+ /* Set the necessary common parameters structure. */
+ cp->program_struct = p;
+ cp->poptions = program_options;
+ cp->program_name = PROGRAM_NAME;
+ cp->program_exec = PROGRAM_EXEC;
+ cp->program_bibtex = PROGRAM_BIBTEX;
+ cp->program_authors = PROGRAM_AUTHORS;
+ cp->coptions = gal_commonopts_options;
+
+ /* Program-specific initializations. */
+ p->radius = NAN;
+
+ /* Modify common options. */
+ for(i=0; !gal_options_is_last(&cp->coptions[i]); ++i)
+ {
+ /* Select individually. */
+ switch(cp->coptions[i].key)
+ {
+ case GAL_OPTIONS_KEY_HDU:
+ case GAL_OPTIONS_KEY_LOG:
+ case GAL_OPTIONS_KEY_TYPE:
+ case GAL_OPTIONS_KEY_SEARCHIN:
+ case GAL_OPTIONS_KEY_QUIETMMAP:
+ case GAL_OPTIONS_KEY_IGNORECASE:
+ case GAL_OPTIONS_KEY_NUMTHREADS:
+ case GAL_OPTIONS_KEY_MINMAPSIZE:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
+ case GAL_OPTIONS_KEY_KEEPINPUTDIR:
+ cp->coptions[i].flags=OPTION_HIDDEN;
+ break;
+ }
+
+ /* Select by group. */
+ switch(cp->coptions[i].group)
+ {
+ case GAL_OPTIONS_GROUP_TESSELLATION:
+ cp->coptions[i].doc=NULL; /* Necessary to remove title. */
+ cp->coptions[i].flags=OPTION_HIDDEN;
+ break;
+ }
+ }
+}
+
+
+
+
+
+/* Parse a single option: */
+error_t
+parse_opt(int key, char *arg, struct argp_state *state)
+{
+ struct queryparams *p = state->input;
+
+ /* Pass 'gal_options_common_params' into the child parser. */
+ state->child_inputs[0] = &p->cp;
+
+ /* In case the user incorrectly uses the equal sign (for example
+ with a short format or with space in the long format, then 'arg'
+ start with (if the short version was called) or be (if the long
+ version was called with a space) the equal sign. So, here we
+ check if the first character of arg is the equal sign, then the
+ user is warned and the program is stopped: */
+ if(arg && arg[0]=='=')
+ argp_error(state, "incorrect use of the equal sign ('='). For short "
+ "options, '=' should not be used and for long options, "
+ "there should be no space between the option, equal sign "
+ "and value");
+
+ /* Set the key to this option. */
+ switch(key)
+ {
+
+ /* Read the non-option tokens (arguments): */
+ case ARGP_KEY_ARG:
+ argp_error(state, "no input arguments are needed");
+ break;
+
+
+ /* This is an option, set its value. */
+ default:
+ return gal_options_set_from_key(key, arg, p->cp.poptions, &p->cp);
+ }
+
+ return 0;
+}
+
+
+
+
+
+/* Parse the database. */
+void *
+ui_parse_database(struct argp_option *option, char *arg,
+ char *filename, size_t lineno, void *junk)
+{
+ char *str;
+ int database;
+
+ /* We want to print the stored values. */
+ if(lineno==-1)
+ {
+ /* Based on the value, write the database. */
+ database=*(int *)(option->value);
+ switch(database)
+ {
+ case QUERY_DATABASE_GAIA:
+ gal_checkset_allocate_copy("gaia", &str);
+ break;
+ default:
+ error(EXIT_FAILURE, 0, "%s: a bug! Please contact us at %s to "
+ "address the problem. The code %d isn't recognized for "
+ "'p->database'", __func__, PACKAGE_BUGREPORT, database);
+ }
+ return str;
+ }
+ else
+ {
+ /* Convert the given string into a code. */
+ if( !strcmp(arg, "gaia") ) database=QUERY_DATABASE_GAIA;
+ else
+ error(EXIT_FAILURE, 0, "'%s' is not a recognized database.\n\n"
+ "For the full list of recognized databases, please see the "
+ "documentation (with the command 'info astquery')", arg);
+
+ /* Write the database code into the option. */
+ *(int *)(option->value)=database;
+
+ /* Our job is done, return NULL. */
+ return NULL;
+ }
+}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+/**************************************************************/
+/*************** Sanity Check *******************/
+/**************************************************************/
+/* Read and check ONLY the options. When arguments are involved, do the
+ check in 'ui_check_options_and_arguments'. */
+static void
+ui_read_check_only_options(struct queryparams *p)
+{
+ /* See if database has been specified. */
+ if(p->database==0)
+ error(EXIT_FAILURE, 0, "no input dataset.\n\n"
+ "Please use the '--database' ('-d') option to specify your "
+ "desired database, see manual ('info gnuastro astquery' "
+ "command) for the current databases");
+
+ /* Make sure that '--query' and '--center' are not called together. */
+ if(p->center && p->query)
+ error(EXIT_FAILURE, 0, "the '--center' and '--query' options cannot be "
+ "called together (they are parallel ways to define a query)");
+
+ /* If radius is given, it should be positive. */
+ if( !isnan(p->radius) && p->radius<0 )
+ error(EXIT_FAILURE, 0, "the '--radius' option value cannot be negative");
+
+ /* If an output name isn't given, set one. */
+ if(p->cp.output)
+ {
+ /* Make sure its a FITS table. */
+ if( gal_fits_name_is_fits(p->cp.output)==0 )
+ error(EXIT_FAILURE, 0, "'%s' is not a FITS file. Currently only "
+ "FITS file outputs are supported", p->cp.output);
+ }
+ else
+ gal_checkset_automatic_output(&p->cp, "./query.fits", ".fits");
+}
+
+
+
+
+
+static void
+ui_check_options_and_arguments(struct queryparams *p)
+{
+
+}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+/**************************************************************/
+/*************** Preparations *******************/
+/**************************************************************/
+static void
+ui_preparations(struct queryparams *p)
+{
+
+}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+/**************************************************************/
+/************ Set the parameters *************/
+/**************************************************************/
+void
+ui_read_check_inputs_setup(int argc, char *argv[], struct queryparams *p)
+{
+ struct gal_options_common_params *cp=&p->cp;
+
+
+ /* Just to avoid warning on no minmapsize. It is irrelevant here. */
+ p->cp.minmapsize=-1;
+
+
+ /* Include the parameters necessary for argp from this program ('args.h')
+ and for the common options to all Gnuastro ('commonopts.h'). We want
+ to directly put the pointers to the fields in 'p' and 'cp', so we are
+ simply including the header here to not have to use long macros in
+ those headers which make them hard to read and modify. This also helps
+ in having a clean environment: everything in those headers is only
+ available within the scope of this function. */
+#include <gnuastro-internal/commonopts.h>
+#include "args.h"
+
+
+ /* Initialize the options and necessary information. */
+ ui_initialize_options(p, program_options, gal_commonopts_options);
+
+
+ /* Read the command-line options and arguments. */
+ errno=0;
+ if(argp_parse(&thisargp, argc, argv, 0, 0, p))
+ error(EXIT_FAILURE, errno, "parsing arguments");
+
+
+ /* Read the configuration files and set the common values. */
+ gal_options_read_config_set(&p->cp);
+
+
+ /* Read the options into the program's structure, and check them and
+ their relations prior to printing. */
+ ui_read_check_only_options(p);
+
+
+ /* Print the option values if asked. Note that this needs to be done
+ after the option checks so un-sane values are not printed in the
+ output state. */
+ gal_options_print_state(&p->cp);
+
+
+ /* Check that the options and arguments fit well with each other. Note
+ that arguments don't go in a configuration file. So this test should
+ be done after (possibly) printing the option values. */
+ ui_check_options_and_arguments(p);
+
+
+ /* Read/allocate all the necessary starting arrays. */
+ ui_preparations(p);
+}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+/**************************************************************/
+/************ Free allocated, report *************/
+/**************************************************************/
+void
+ui_free_report(struct queryparams *p, struct timeval *t1)
+{
+ /* Free the allocated arrays: */
+ free(p->cp.hdu);
+ free(p->cp.output);
+}
diff --git a/bin/query/ui.h b/bin/query/ui.h
new file mode 100644
index 0000000..d8e3ce7
--- /dev/null
+++ b/bin/query/ui.h
@@ -0,0 +1,73 @@
+/*********************************************************************
+Query - Retreive data from a remote data server.
+Query is part of GNU Astronomy Utilities (Gnuastro) package.
+
+Original author:
+ Mohammad akhlaghi <mohammad@akhlaghi.org>
+Contributing author(s):
+Copyright (C) 2020, Free Software Foundation, Inc.
+
+Gnuastro is free software: you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation, either version 3 of the License, or (at your
+option) any later version.
+
+Gnuastro is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Gnuastro. If not, see <http://www.gnu.org/licenses/>.
+**********************************************************************/
+#ifndef UI_H
+#define UI_H
+
+/* For common options groups. */
+#include <gnuastro-internal/options.h>
+
+
+
+
+
+/* Option groups particular to this program. */
+enum program_args_groups
+{
+ UI_GROUP_BYCENTER = GAL_OPTIONS_GROUP_AFTER_COMMON,
+};
+
+
+
+
+
+/* Available letters for short options:
+
+ a b e f g i j k m n p t u v x y z
+ A B E G H J L R W X Y
+*/
+enum option_keys_enum
+{
+ /* With short-option version. */
+ UI_KEY_DATABASE = 'd',
+ UI_KEY_QUERY = 'Q',
+ UI_KEY_DATASET = 's',
+ UI_KEY_CENTER = 'c',
+ UI_KEY_RADIUS = 'r',
+ UI_KEY_COLUMN = 'C',
+ UI_KEY_WIDTH = 'w',
+
+ /* Only with long version (start with a value 1000, the rest will be set
+ automatically). */
+};
+
+
+
+
+
+void
+ui_read_check_inputs_setup(int argc, char *argv[], struct queryparams *p);
+
+void
+ui_free_report(struct queryparams *p, struct timeval *t1);
+
+#endif
diff --git a/configure.ac b/configure.ac
index f90ea54..a7c6a00 100644
--- a/configure.ac
+++ b/configure.ac
@@ -718,103 +718,109 @@ ayes=false
AC_ARG_ENABLE([arithmetic],
[AS_HELP_STRING([--enable-arithmetic],
[Install Arithmetic and other enabled programs.])],
- [AS_IF([test "x$enable_arithmetic" != xno],
+ [AS_IF([test "x$enable_arithmetic" != xno],
[enable_arithmetic=yes; ayes=true])],
[enable_arithmetic=notset])
AC_ARG_ENABLE([buildprog],
[AS_HELP_STRING([--enable-buildprog],
[Install BuildProgram and other enabled programs.])],
- [AS_IF([test "x$enable_buildprog" != xno],
+ [AS_IF([test "x$enable_buildprog" != xno],
[enable_buildprog=yes; ayes=true])],
[enable_buildprog=notset])
AC_ARG_ENABLE([convertt],
[AS_HELP_STRING([--enable-convertt],
[Install ConvertType and other enabled programs.])],
- [AS_IF([test "x$enable_convertt" != xno],
+ [AS_IF([test "x$enable_convertt" != xno],
[enable_convertt=yes; ayes=true])],
[enable_convertt=notset])
AC_ARG_ENABLE([convolve],
[AS_HELP_STRING([--enable-convolve],
[Install Convolve and other enabled programs.])],
- [AS_IF([test "x$enable_convolve" != xno],
+ [AS_IF([test "x$enable_convolve" != xno],
[enable_cognvolve=yes; ayes=true])],
[enable_convolve=notset])
AC_ARG_ENABLE([cosmiccal],
[AS_HELP_STRING([--enable-cosmiccal],
[Install CosmicCalculator and other enabled programs.])],
- [AS_IF([test "x$enable_cosmiccal" != xno],
+ [AS_IF([test "x$enable_cosmiccal" != xno],
[enable_cosmiccal=yes; ayes=true])],
[enable_cosmiccal=notset])
AC_ARG_ENABLE([crop],
[AS_HELP_STRING([--enable-crop],
[Install Crop and other enabled programs.])],
- [AS_IF([test "x$enable_crop" != xno],
+ [AS_IF([test "x$enable_crop" != xno],
[enable_crop=yes; ayes=true])],
[enable_crop=notset])
AC_ARG_ENABLE([fits],
[AS_HELP_STRING([--enable-fits],
[Install Fits and other enabled programs.])],
- [AS_IF([test "x$enable_fits" != xno],
+ [AS_IF([test "x$enable_fits" != xno],
[enable_fits=yes; ayes=true])],
[enable_fits=notset])
AC_ARG_ENABLE([match],
[AS_HELP_STRING([--enable-match],
[Install Match and other enabled programs.])],
- [AS_IF([test "x$enable_match" != xno],
+ [AS_IF([test "x$enable_match" != xno],
[enable_match=yes; ayes=true])],
[enable_match=notset])
AC_ARG_ENABLE([mkcatalog],
[AS_HELP_STRING([--enable-mkcatalog],
[Install MakeCatalog and other enabled programs.])],
- [AS_IF([test "x$enable_mkcatalog" != xno],
+ [AS_IF([test "x$enable_mkcatalog" != xno],
[enable_mkcatalog=yes; ayes=true])],
[enable_mkcatalog=notset])
AC_ARG_ENABLE([mknoise],
[AS_HELP_STRING([--enable-mknoise],
[Install MakeNoise and other enabled programs.])],
- [AS_IF([test "x$enable_mknoise" != xno],
+ [AS_IF([test "x$enable_mknoise" != xno],
[enable_mknoise=yes; ayes=true])],
[enable_mknoise=notset])
AC_ARG_ENABLE([mkprof],
[AS_HELP_STRING([--enable-mkprof],
[Install MakeProfile and other enabled programs.])],
- [AS_IF([test "x$enable_mkprof" != xno],
+ [AS_IF([test "x$enable_mkprof" != xno],
[enable_mkprof=yes; ayes=true])],
[enable_mkprof=notset])
AC_ARG_ENABLE([noisechisel],
[AS_HELP_STRING([--enable-noisechisel],
[Install NoiseChisel and other enabled programs.])],
- [AS_IF([test "x$enable_noisechisel" != xno],
+ [AS_IF([test "x$enable_noisechisel" != xno],
[enable_noisechisel=yes; ayes=true])],
[enable_noisechisel=notset])
+AC_ARG_ENABLE([query],
+ [AS_HELP_STRING([--enable-query],
+ [Install query and other enabled packages.])],
+ [AS_IF([test "x$enable_query" != xno],
+ [enable_query=yes; ayes=true])],
+ [enable_query=notset])
AC_ARG_ENABLE([segment],
[AS_HELP_STRING([--enable-segment],
[Install Segment and other enabled programs.])],
- [AS_IF([test "x$enable_segment" != xno],
+ [AS_IF([test "x$enable_segment" != xno],
[enable_segment=yes; ayes=true])],
[enable_segment=notset])
AC_ARG_ENABLE([statistics],
[AS_HELP_STRING([--enable-statistics],
[Install Statistics and other enabled programs.])],
- [AS_IF([test "x$enable_statistics" != xno],
+ [AS_IF([test "x$enable_statistics" != xno],
[enable_statistics=yes; ayes=true])],
[enable_statistics=notset])
AC_ARG_ENABLE([table],
[AS_HELP_STRING([--enable-table],
[Install Table and other enabled programs.])],
- [AS_IF([test "x$enable_table" != xno],
+ [AS_IF([test "x$enable_table" != xno],
[enable_table=yes; ayes=true])],
[enable_table=notset])
#AC_ARG_ENABLE([TEMPLATE],
# [AS_HELP_STRING([--enable-TEMPLATE],
# [Install TEMPLATE and other enabled packages.])],
-# [AS_IF([test "x$enable_TEMPLATE" != xno],
+# [AS_IF([test "x$enable_TEMPLATE" != xno],
# [enable_TEMPLATE=yes; ayes=true])],
# [enable_TEMPLATE=notset])
AC_ARG_ENABLE([warp],
[AS_HELP_STRING([--enable-warp],
[Install Warp and other enabled programs.])],
- [AS_IF([test "x$enable_warp" != xno],
+ [AS_IF([test "x$enable_warp" != xno],
[enable_warp=yes; ayes=true])],
[enable_warp=notset])
@@ -840,6 +846,7 @@ AS_IF([test $ayes = true ],
AS_IF([test $enable_mknoise = notset], [enable_mknoise=no])
AS_IF([test $enable_mkprof = notset], [enable_mkprof=no])
AS_IF([test $enable_noisechisel = notset], [enable_noisechisel=no])
+ AS_IF([test $enable_query = notset], [enable_query=no])
AS_IF([test $enable_segment = notset], [enable_segment=no])
AS_IF([test $enable_statistics = notset], [enable_statistics=no])
AS_IF([test $enable_table = notset], [enable_table=no])
@@ -860,6 +867,7 @@ AS_IF([test $ayes = true ],
AS_IF([test $enable_mknoise = notset], [enable_mknoise=yes])
AS_IF([test $enable_mkprof = notset], [enable_mkprof=yes])
AS_IF([test $enable_noisechisel = notset], [enable_noisechisel=yes])
+ AS_IF([test $enable_query = notset], [enable_query=yes])
AS_IF([test $enable_segment = notset], [enable_segment=yes])
AS_IF([test $enable_statistics = notset], [enable_statistics=yes])
AS_IF([test $enable_table = notset], [enable_table=yes])
@@ -893,6 +901,7 @@ AM_CONDITIONAL([COND_MKCATALOG], [test $enable_mkcatalog
= yes])
AM_CONDITIONAL([COND_MKNOISE], [test $enable_mknoise = yes])
AM_CONDITIONAL([COND_MKPROF], [test $enable_mkprof = yes])
AM_CONDITIONAL([COND_NOISECHISEL], [test $enable_noisechisel = yes])
+AM_CONDITIONAL([COND_QUERY], [test $enable_query = yes])
AM_CONDITIONAL([COND_SEGMENT], [test $enable_segment = yes])
AM_CONDITIONAL([COND_STATISTICS], [test $enable_statistics = yes])
AM_CONDITIONAL([COND_TABLE], [test $enable_table = yes])
@@ -921,12 +930,13 @@ AS_ECHO(["linking flags (LDADD) ... $LDADD"])
AC_CONFIG_FILES([Makefile
doc/Makefile
lib/Makefile
- tests/Makefile
+ tests/Makefile
bin/crop/Makefile
bin/fits/Makefile
bin/warp/Makefile
bin/table/Makefile
bin/match/Makefile
+ bin/query/Makefile
bin/mkprof/Makefile
bin/script/Makefile
bin/mknoise/Makefile
@@ -935,9 +945,9 @@ AC_CONFIG_FILES([Makefile
bin/convolve/Makefile
bin/buildprog/Makefile
bin/cosmiccal/Makefile
- bin/mkcatalog/Makefile
+ bin/mkcatalog/Makefile
bin/arithmetic/Makefile
- bin/statistics/Makefile
+ bin/statistics/Makefile
bin/noisechisel/Makefile
bootstrapped/lib/Makefile
bootstrapped/tests/Makefile
@@ -957,7 +967,7 @@ AC_CONFIG_FILES([Makefile
AC_ARG_ENABLE([guide-message],
[AS_HELP_STRING([--disable-guide-message],
[No messages after each build step.])],
- [], [enable_guide_message=yes])
+ [], [enable_guide_message=yes])
AC_SUBST(GUIDEMESSAGE, [$enable_guide_message])
diff --git a/doc/Makefile.am b/doc/Makefile.am
index 820627d..4aae045 100644
--- a/doc/Makefile.am
+++ b/doc/Makefile.am
@@ -117,6 +117,9 @@ endif
if COND_NOISECHISEL
MAYBE_NOISECHISEL_MAN = man/astnoisechisel.1
endif
+if COND_QUERY
+ MAYBE_QUERY_MAN = man/astquery.1
+endif
if COND_SEGMENT
MAYBE_SEGMENT_MAN = man/astsegment.1
endif
@@ -132,12 +135,13 @@ endif
#if COND_TEMPLATE
# MAYBE_TEMPLATE_MAN = man/astTEMPLATE.1
#endif
-dist_man_MANS = $(MAYBE_ARITHMETIC_MAN) $(MAYBE_BUILDPROG_MAN) \
- $(MAYBE_CONVERTT_MAN) $(MAYBE_CONVOLVE_MAN) $(MAYBE_COSMICCAL_MAN) \
- $(MAYBE_CROP_MAN) $(MAYBE_FITS_MAN) $(MAYBE_MATCH_MAN) \
- $(MAYBE_MKCATALOG_MAN) $(MAYBE_MKNOISE_MAN) $(MAYBE_MKPROF_MAN) \
- $(MAYBE_NOISECHISEL_MAN) $(MAYBE_SEGMENT_MAN) $(MAYBE_STATISTICS_MAN) \
- $(MAYBE_TABLE_MAN) $(MAYBE_WARP_MAN) man/astscript-sort-by-night.1
+dist_man_MANS = $(MAYBE_ARITHMETIC_MAN) $(MAYBE_BUILDPROG_MAN) \
+ $(MAYBE_CONVERTT_MAN) $(MAYBE_CONVOLVE_MAN) $(MAYBE_COSMICCAL_MAN) \
+ $(MAYBE_CROP_MAN) $(MAYBE_FITS_MAN) $(MAYBE_MATCH_MAN) \
+ $(MAYBE_MKCATALOG_MAN) $(MAYBE_MKNOISE_MAN) $(MAYBE_MKPROF_MAN) \
+ $(MAYBE_NOISECHISEL_MAN) $(MAYBE_QUERY_MAN) $(MAYBE_SEGMENT_MAN) \
+ $(MAYBE_STATISTICS_MAN) $(MAYBE_TABLE_MAN) $(MAYBE_WARP_MAN) \
+ man/astscript-sort-by-night.1
## See if help2man is present or not. When help2man doesn't exist, we don't
@@ -204,6 +208,10 @@ man/astnoisechisel.1: $(top_srcdir)/bin/noisechisel/args.h
$(ALLMANSDEP)
$(MAYBE_HELP2MAN) -n "detect signal in a noisy image" \
--libtool $(toputildir)/noisechisel/astnoisechisel
+man/astquery.1: $(top_srcdir)/bin/query/args.h $(ALLMANSDEP)
+ $(MAYBE_HELP2MAN) -n "query remote data servers and download" \
+ --libtool $(toputildir)/query/astquery
+
man/astscript-sort-by-night.1: $(top_srcdir)/bin/script/sort-by-night.in \
$(ALLMANSDEP)
$(MAYBE_HELP2MAN) -n "Sort input FITS files by night" \
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index f010539..d017516 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -565,6 +565,7 @@ Noise basics
High-level calculations
* CosmicCalculator:: Calculate cosmological variables
+* Query:: Import data from external databases
CosmicCalculator
@@ -17622,9 +17623,10 @@ Even higher-level analysis is still needed to convert
the observed magnitudes, s
@menu
* CosmicCalculator:: Calculate cosmological variables
+* Query:: Import data from external databases
@end menu
-@node CosmicCalculator, , High-level calculations, High-level calculations
+@node CosmicCalculator, Query, High-level calculations, High-level calculations
@section CosmicCalculator
To derive higher-level information regarding our sources in extra-galactic
astronomy, cosmological calculations are necessary.
@@ -18293,6 +18295,150 @@ In the latter (when a number is given), the returned
value is the same units of
+@node Query, , CosmicCalculator, High-level calculations
+@section Query
+
+There are many astronomical databases available for downloading astronomical
data.
+Each has its own interface, which is usually very well documented in their own
webpages.
+However, remembering the exact URLs and interface of each database is not
easily possible and using the graphic web interface (which is commonly easier
to use), is not an automatic process.
+
+Gnuastro's Query program is designed to address these problems: it has a
common high-level interface for general operations that are common between its
recognized databases.
+For example, importing a catalog of objects within a certain distance of a
given coordinate.
+
+@node Invoking astquery
+@subsection Invoking Query
+
+Query provides a high-level interface to downloading subsets of data from
databases.
+The executable name is @file{astquery} with the following general template
+
+@example
+$ astquery [OPTION...] ...
+@end example
+
+@noindent
+One line examples:
+
+@example
+## Import all the columns of all entries in the Gaia DR2 catalog within
+## 0.1 degrees of the given coordinate.
+$ astquery --database=gaia --dataset=dr2 --output=sub-gaia.fits \
+ --center=113.8729761,31.9027152 --radius=0.1
+
+## Similar to above, but only import the ID, RA, Dec and G-band
+## magnitude of the sources (not all the columns).
+$ astquery --database=gaia --dataset=dr2 --output=sub-gaia.fits \
+ --center=113.8729761,31.9027152 --radius=0.1 \
+ --column=source_id,ra,dec,phot_g_mean_mag
+
+## Use a custom query to extract entries in the Gaia DR2 catalog.
+## The 'XXXX YYYY ZZZZ' can be a query of any size on the command-line.
+$ astquery --database=esa --query="XXXX YYYY ZZZZ" --output=my-gaia.fits
+@end example
+
+Query doesn't take any input argument, because the main goal is to retreive
data from external sources.
+The main input to Query is the @option{--database} option which specifies
which database should be contacted for submitting the query.
+The name of the downloaded output file can optionally be set with
@option{--output}.
+If @option{--output} is not set, an automatically generated name will be used.
+
+There are two methods to query the database: 1) with @option{--query} you can
directly give a raw query statement that is recognized by the database, 2) with
the @option{--center} and @option{--radius}, the low-level query will
constructed automatically for the particular database.
+The former is very low level and will require some knowledge of the database's
query language, but of course, it is much more powerful.
+The latter is much more limited in terms of capabilities (the only constraint
is the location of the objects compared to the given center), but doesn't
require any knowledge of the database's query language.
+
+@cartouche
+@noindent
+@strong{Under development, request for feedback:} Query is a new member of the
Gnuastro family of programs.
+It currently requires that the @command{curl} executable (for the cURL
downloading program) to be present on the host and the number of databases it
supports is still limited, see the list under the @option{--database} option
below.
+More downloader tools, and databases will be added in the near future as it is
used more often, so please don't hesitate to suggest any that you may need.
+@end cartouche
+
+@table @option
+
+@item -d STR
+@itemx --database=STR
+Identifer for the database for sending the query.
+The current list of databases are listed here:
+
+@table @code
+@item gaia
+The Gaia project (@url{https://www.cosmos.esa.int/web/gaia}) database which is
a large collection of star positions on the celestial sphere, as well as
peculiar velocities, paralloxes and magnitudes in some bands among many others.
+Besides scientific studies (like studying resolved stellar populations in the
Galaxy and its halo), Gaia is also invaluable for raw data calibrations, like
astrometry.
+The query given to this database will be submitted to
@code{https://gea.esac.esa.int/tap-server/tap/sync}.
+@end table
+
+@item -Q "STR"
+@itemx --query="STR"
+Directly specify the query to be passed onto the database.
+The queries will generally contain space and other meta-characters, so we
recommend placing the query within quotations.
+
+@item -c FLT,FLT
+@itemx --center=FLT,FLT
+The center to use for the automatically generated query (not compatible with
@option{--query}).
+All objects within a certain distance of the requested center position will be
requested from the dataset.
+The distance can be specified with the @option{--radius} option.
+
+@item -r FLT
+@itemx --radius FLT
+The radius about the requested center to use for the automatically generated
query (not compatible with @option{--query}).
+Since the center is usually determined in units of degrees, the radius to
select sources is also in the same units.
+
+@item -s STR
+@itemx --dataset=STR
+The dataset to query within the database for the automatically generated query
(not compatible with @option{--query}).
+The reason for this is that many databases have different types of datasets,
for example different data releases (DRs), or various high-level calculations
on subsets of the database elements.
+For example when @option{--database=gaia}, you can set @option{--dataset=dr2}
to only select objects within the second data release (DR2).
+
+You can either use the database's official name of the datasets, for example
@code{gaiadr2.gaia_source} for the second data release of the Gaia database, or
a simplified version that maps to it (@code{dr2}) for easy typing on the
command-line.
+Below is a list of the simplified names for the databases that have them.
+
+@table @code
+@item gaia
+@itemize
+@item
+@code{dr2 --> gaiadr2.gaia_source}
+@item
+@code{dr2 --> gaiadr1.gaia_source}
+@item
+@code{tyco2 --> public.tyco2}
+@item
+@code{hipparcos --> public.hipparcos}
+@end itemize
+@end table
+
+@item -C STR
+@itemx --column=STR[,STR[,...]]
+The optional column name(s) to retrieve from the dataset in the automatically
generated query (not compatible with @option{--query}).
+If not given, all the dataset's columns for the selected rows will be used
(which can be large!).
+This option can take multiple values in one instance (for example
@option{--column=ra,dec,mag}), or in multiple instances (for example
@option{-cra -cdec -cmag}), or mixed (for example @option{-cra,dec -cmag}).
+
+In case, you don't know the full list of the dataset's column names, one
solution is to run this program once without any @option{--column} and setting
@option{--radius=0} (to have few, or no, rows and download fast).
+You can then inspect the column names with @ref{Table}.
+In case the database doesn't support a search radius of 0, or can't return a
table with no rows, you can increase the radius.
+Here is an example on the Gaia DR2 catalog to show the full list of available
columns:
+
+@example
+$ astquery --database=gaia --dataset=dr2 --center=0,0 \
+ --radius=0 --output=gaia-dr2-cols.fits
+$ asttable gaia-dr2-cols.fits --info
+@end example
+@end table
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
@node Library, Developing, High-level calculations, Top
@@ -28097,6 +28243,11 @@ uses a technique to detect very faint and diffuse,
irregularly shaped
signal in noise (galaxies in the sky), using thresholds that are below the
Sky value, see @url{http://arxiv.org/abs/1505.01664, arXiv:1505.01664}.
+@item Query
+(@file{astquery}, see @ref{Query}) High-level interface to query
+pre-defined remote, or external databases, and directly download the
+required sub-tables on the command-line.
+
@item Segment
(@file{astsegment}, see @ref{Segment}) Segment detected regions based on
the structure of signal and the input dataset's noise properties.
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [gnuastro-commits] master 6010f00: Query: new program to extract data from external databases,
Mohammad Akhlaghi <=