[patch 17/19] Start writing developers guide.

[blp]

Move data file format, portable file format, q2c input format chapters
into developer's guide.

diff --git a/doc/.cvsignore b/doc/.cvsignore
index 6d2576f..06eb3bb 100644
--- a/doc/.cvsignore
+++ b/doc/.cvsignore
@@ -1,6 +1,9 @@
 Makefile
 Makefile.in
 ni.texi
+pspp-dev.info
 pspp.info*
+stamp-1
 stamp-vti
+version-dev.texi
 version.texi
diff --git a/doc/automake.mk b/doc/automake.mk
index 301757c..b0056ea 100644
--- a/doc/automake.mk
+++ b/doc/automake.mk
@@ -1,13 +1,12 @@
 ## Process this file with automake to produce Makefile.in  -*- makefile -*-
 
-info_TEXINFOS = doc/pspp.texinfo 
+info_TEXINFOS = doc/pspp.texinfo doc/pspp-dev.texinfo
 
 doc_pspp_TEXINFOS = doc/version.texi \
        doc/bugs.texi \
        doc/command-index.texi \
        doc/concept-index.texi \
        doc/configuring.texi \
-       doc/data-file-format.texi \
        doc/data-io.texi \
        doc/data-selection.texi \
        doc/expressions.texi \
@@ -21,8 +20,6 @@ doc_pspp_TEXINFOS = doc/version.texi \
        doc/license.texi \
        doc/ni.texi \
        doc/not-implemented.texi \
-       doc/portable-file-format.texi \
-       doc/q2c.texi \
        doc/statistics.texi \
        doc/transformation.texi \
        doc/regression.texi \
@@ -30,9 +27,18 @@ doc_pspp_TEXINFOS = doc/version.texi \
        doc/variables.texi \
        doc/fdl.texi 
 
+doc_pspp_dev_TEXINFOS = doc/version-dev.texi \
+       doc/dev-intro.texi \
+       doc/dev-concepts.texi \
+       doc/dev-syntax.texi \
+       doc/dev-data.texi \
+       doc/dev-output.texi \
+       doc/data-file-format.texi \
+       doc/portable-file-format.texi \
+       doc/q2c.texi
+
 EXTRA_DIST += doc/pspp.man \
-       doc/get-commands.pl \
-       $(doc_pspp_TEXINFOS)
+       doc/get-commands.pl
 
 doc/ni.texi: $(top_srcdir)/src/language/command.def doc/get-commands.pl
        @$(MKDIR_P)  doc
diff --git a/doc/data-file-format.texi b/doc/data-file-format.texi
index f1dfce8..dcde545 100644
--- a/doc/data-file-format.texi
+++ b/doc/data-file-format.texi
@@ -135,7 +135,7 @@ format and using 24-hour time.  If the time is not 
available then this
 field is arbitrarily set to @samp{00:00:00}.
 
 @item char file_label[64];
-Set the file label declared by the user, if any (@pxref{FILE LABEL}).
+Set the file label declared by the user, if any (@pxref{FILE LABEL,,,pspp}).
 Padded on the right with spaces.
 
 @item char padding[3];
diff --git a/doc/dev-concepts.texi b/doc/dev-concepts.texi
new file mode 100644
index 0000000..bf26440
--- /dev/null
+++ b/doc/dev-concepts.texi
@@ -0,0 +1,383 @@
address@hidden Basic Concepts
address@hidden Basic Concepts
+
+This chapter introduces basic data structures and other concepts
+needed for developing in PSPP.
+
address@hidden
+* Values::                      
+* Variables::                   
+* Cases::                       
+* Dictionaries::                
+* Data Sets::                   
+* Coding Conventions::          
address@hidden menu
+
address@hidden Values
address@hidden Values
+
address@hidden value
+The unit of data in PSPP is a @dfn{value}.  Definitions related to
+values are provided in @file{data/value.h}.
+
address@hidden width
address@hidden string value
address@hidden numeric value
address@hidden MAX_STRING
+Values are classified by @dfn{type} and @dfn{width}.  The
+type of a value is either @dfn{numeric} or @dfn{string} (sometimes
+called alphanumeric).  The width of a string value ranges from 1 to
address@hidden bytes.  The width of a numeric value is artificially
+defined to be 0; thus, the type of a value can be inferred from its
+width.
+
+Some support is provided for working with value types and widths, in
address@hidden/val-type.h}:
+
address@hidden Macro int MAX_STRING
+Maximum width of a string value, in bytes, currently 32,767.
address@hidden deftypefn
+
address@hidden bool val_type_is_valid (enum val_type @var{val_type})
+Returns true if @var{val_type} is a valid value type, that is,
+either @code{VAL_NUMERIC} or @code{VAL_STRING}.  Useful for
+assertions.
address@hidden deftypefun
+
address@hidden {enum val_type} val_type_from_width (int @var{width})
+Returns @code{VAL_NUMERIC} if @var{width} is 0 and thus represents the
+width of a numeric value, otherwise @code{VAL_STRING} to indicate that
address@hidden is the width of a string value.
address@hidden deftypefun
+
+The following subsections describe how values of each type are
+represented.
+
address@hidden
+* Numeric Values::              
+* String Values::               
+* Runtime Typed Values::        
address@hidden menu
+
address@hidden Numeric Values
address@hidden Numeric Values
+
+A value known to be numeric at compile time is represented as a
address@hidden  PSPP provides three values of @code{double} for
+special purposes, defined in @file{data/val-type.h}:
+
address@hidden Macro double SYSMIS
+The @dfn{system-missing value}, used to represent a datum whose true
+value is unknown, such as a survey question that was not answered by
+the respondent, or undefined, such as the result of division by zero.
+PSPP propagates the system-missing value through calculations and
+compensates for missing values in statistical analyses.  @xref{Missing
+Observations,,,pspp}, for a PSPP user's view of missing values.
+
+PSPP currently defines @code{SYSMIS} as @code{-DBL_MAX}, that is, the
+greatest finite negative value of @code{double}.  It is best not to
+depend on this definition, because PSPP may transition to using an
+IEEE NaN (not a number) instead at some point in the future.
address@hidden deftypefn
+
address@hidden Macro double LOWEST
address@hidden Macro double HIGHEST
+The greatest finite negative (except for @code{SYSMIS}) and positive
+values of @code{double}, respectively.  These values do not ordinarily
+appear in user data files.  Instead, they are used to implement
+endpoints of open-ended ranges that are occasionally permitted in PSPP
+syntax, e.g.@: @code{5 THRU HI} as a range of missing values
+(@pxref{MISSING VALUES,,,pspp}).
address@hidden deftypefn
+
address@hidden String Values
address@hidden String Values
+
+A value known at compile time to have string type is represented as an
+array of @code{char}.  String values do not necessarily represent
+readable text strings and may contain arbitrary 8-bit data, including
+null bytes, control codes, and bytes with the high bit set.  Thus,
+string values are not null-terminated strings, but rather opaque
+arrays of bytes.
+
address@hidden, @code{LOWEST}, and @code{HIGHEST} have no equivalents
+as string values.  Usually, PSPP fills an unknown or undefined string
+values with spaces, but PSPP does not treat such a string as a special
+case when it processes it later.
+
address@hidden MAX_STRING
address@hidden, the maximum length of a string value, is defined in
address@hidden/val-type.h}.
+
address@hidden Runtime Typed Values
address@hidden Runtime Typed Values
+
+When a value's type is only known at runtime, it is often
+represented as a @union{value}, which has two members: a
address@hidden named @samp{f} to store a numeric value and an array of
address@hidden named @samp{s} to a store a string value.  A @union{value}
+does not identify the type or width of the data it contains.  Code
+that works with @union{values}s must therefore have external knowledge
+of its content, often through the type and width of a
address@hidden (@pxref{Variables}).
+
address@hidden MAX_SHORT_STRING
address@hidden short string
address@hidden long string
address@hidden string value
+The array of @code{char} in @union{value} has only a small, fixed
+capacity of @code{MAX_SHORT_STRING} bytes.  A value that
+fits within this capacity is called a @dfn{short string}.  Any wider
+string value, which must be represented by more than one
address@hidden, is called a @dfn{long string}.
+
address@hidden Macro int MAX_SHORT_STRING
+Maximum width of a short string value, never less than 8 bytes.  It is
+wider than 8 bytes on systems where @code{double} is either larger
+than 8 bytes or has stricter alignment than 8 bytes.
address@hidden deftypefn
+
address@hidden Macro int MIN_LONG_STRING
+Minimum width of a long string value, that is, @code{MAX_SHORT_STRING
++ 1}.
address@hidden deftypefn
+
+Long string variables are slightly harder to work with than short
+string values, because they cannot be conveniently and efficiently
+allocated as block scope variables or structure members.  The PSPP
+language exposes this inconvenience to the user: there are many
+circumstances in PSPP syntax where short strings are allowed but not
+long strings.  Short string variables, for example, may have
+user-missing values, but long string variables may not (@pxref{Missing
+Observations,,,pspp}).
+
+PSPP provides a few functions for working with @union{value}s.  The
+most useful are described below.  To use these functions, recall that
+a numeric value has a width of 0.
+
address@hidden size_t value_cnt_from_width (int @var{width})
+Returns the number of consecutive @union{value}s that must be
+allocated to store a value of the given @var{width}.  For a numeric or
+short string value, the return value is 1; otherwise, for long string
+variables, it is greater than 1.
address@hidden deftypefun
+
address@hidden void value_copy (union value address@hidden, @
+                             const union value address@hidden, @
+                             int @var{width})
+Copies a value of the given @var{width} from the @union{value} array
+starting at @var{src} to the one starting at @var{dst}.  The two
+arrays must not overlap.
address@hidden deftypefun
+
address@hidden void value_set_missing (union value address@hidden, int 
@var{width})
+Sets a numeric value to @code{SYSMIS} or a string value to all spaces,
+according to @var{width}.  @var{value} must point to the start of a
address@hidden array of the given @var{width}.
address@hidden deftypefun
+
address@hidden Variables
address@hidden Variables
+
+A PSPP variable is represented by @struct{variable}, an opaque type
+declared in @file{data/variable.h} along with related declarations.
address@hidden,,,pspp}, for a description of PSPP variables from a
+user perspective.
+
+PSPP is unusual among computer languages in that, by itself, a PSPP
+variable does not have a value.  Instead, a variable in PSPP takes on
+a value only in the context of a case, which supplies one value for
+each variable in a set of variables (@pxref{Cases}).  The set of
+variables in a case, in turn, are ordinarily part of a dictionary
+(@pxref{Dictionaries}).
+
+Every variable has several attributes, most of which correspond
+directly to one of the variable attributes visible to PSPP users
+(@pxref{Attributes,,,pspp}).
+
+The following sections describe variable-related functions and macros.
+
address@hidden Variable Creation and Destruction
address@hidden Variable Creation and Destruction
+
+Only rarely should PSPP code create or destroy variables directly.
+Ordinarily, variables are created within a dictionary and destroying
+by individual deletion from the dictionary or by destroying the entire
+dictionary at once.  The functions here enable the exceptional case,
+of creation and destruction of variables that are not associated with
+any dictionary.  These functions are used internally in the dictionary
+implementation.
+
address@hidden {struct variable *} var_create (const char address@hidden, int 
@var{width})
+Creates and returns a new variable with the given @var{name} and
address@hidden  The new variable is not part of any dictionary.  Use
address@hidden, instead, to create a variable in a dictionary
+(@pxref{Dictionary Creating Variables}).
+
address@hidden must be a ``plausible'' variable name (@pxref{Variable
+Name}).  @var{width} must be between 0 and @code{MAX_STRING},
+inclusive (@pxref{Values}).
+
+The new variable has no user missing values, value labels, or variable
+label.  Numeric variables initially have F8.2 print and write formats,
+right-justified display alignment, and scale level of measurement.
+String variables are created with A print and write formats,
+left-justified display alignment, and nominal level of measurement.
+Regardless of width, the new variable's display width is 8 columns.
+
+The new variable initially has no short name (@pxref{Variable Short
+Names}) and no auxiliary data (@pxref{Variable Auxiliary data}).
address@hidden deftypefun
+
address@hidden {struct variable *} var_clone (const struct variable 
address@hidden)
+Creates and returns a new variable with the same attributes as
address@hidden, with a few exceptions.  First, the new variable is not
+part of any dictionary, regardless of whether @var{old_var} was in a
+dictionary.  Use @func{dict_clone_var} to add a clone of a variable to
+a dictionary.
+
+Second, the new variable is not given any short name, even if
address@hidden had a short name.  This is because the new variable is
+likely to be immediately renamed, in which case the short name would
+be incorrect (@pxref{Variable Short Names}).
+
+Finally, @var{old_var}'s auxiliary data, if any, is not copied to the
+new variable (@pxref{Variable Auxiliary Data}).
address@hidden deftypefun
+
address@hidden {void} var_destroy (struct variable address@hidden)
+Destroys @var{var} and frees all associated storage, including its
+auxiliary data, if any.  @var{var} must not be part of a dictionary.
+To delete a variable from a dictionary and destroy it, use
address@hidden (@pxref{Dictionary Deleting Variables}).
address@hidden deftypefun
+
address@hidden Variable Name
address@hidden Variable Name
+
+A variable name is a string between 1 and @code{VAR_NAME_LEN} bytes
+long that satisfies the rules for PSPP identifiers
+(@pxref{Tokens,,,pspp}).  Variable names are mixed-case and treated
+case-insensitively.
+
address@hidden Macro int VAR_NAME_LEN
+Maximum length of bytes of a variable name, in bytes, currently 64.
address@hidden deftypefn
+
+Only one commonly useful function relates to variable names:
+
address@hidden {const char *} var_get_name (const struct variable 
address@hidden)
+Returns @var{var}'s variable name as a C string.
address@hidden deftypefun
+
+A few other functions are much more rarely used.  These functions are
+used internally in the dictionary implementation:
+
address@hidden {void} var_set_name (struct variable address@hidden, const char 
address@hidden)
+Changes the name of @var{var} to @var{new_name}, which must be a
+``plausible'' name as defined below.
+
+This function cannot be applied to a variable that is part of a
+dictionary.  Use @var{dict_rename_var} instead (@pxref{Dictionary
+Renaming Variables}).
address@hidden deftypefun
+
address@hidden {bool} var_is_valid_name (const char address@hidden, bool 
@var{issue_error})
address@hidden {bool} var_is_plausible_name (const char address@hidden, bool 
@var{issue_error})
+Tests @var{name} for validity or ``plausibility.''  Returns true if
+the name is acceptable, false otherwise.  If the name is not
+acceptable and @var{issue_error} is true, also issues an error message
+explaining the violation.
+
+A valid name is one that fully satisfies all of the requirements for
+variable names (@pxref{Tokens,,,pspp}).  A ``plausible'' name is
+simply a string whose length is in the valid range and that is not a
+reserved word.  PSPP accepts plausible but invalid names as variable
+names in some contexts where the character encoding scheme is
+ambiguous, as when reading variable names from system files.
address@hidden deftypefun
+
address@hidden Variable Type and Width
address@hidden Variable Type and Width
+
+A variable's type and width are the type and width of its values
+(@pxref{Values}).
+
+
+
address@hidden Cases
address@hidden Cases
+
+A 
+
address@hidden Dictionaries
address@hidden Dictionaries
+
+Each data file in memory or on disk has an associated dictionary,
+whose primary purpose is to describe the data in the file.  A data
+file stored in a PSPP format, either as a system or portable file, has
+a representation of its dictionary embedded in it.  Data files of
+other types are usually not self-describing enough to construct a
+dictionary unassisted, so the dictionaries for these files must be
+specified explicitly with PSPP commands such as @cmd{DATA LIST}.  
+
+The primary data-description content of a dictionary is its set of
+variables.  This set is ordered, and each variable's name is unique
+within the set.
+
+A dictionary also contains a mapping from each of its variables to a
+location within a case.
+
+System variables are not members of a dictionary's set of variables
+(@pxref{System Variables,,,pspp}).
+
+A @struct{dictionary} is the data structure that corresponds closely
+to the notion of a dictionary that is exposed to PSPP users
+(@pxref{Variables,,,pspp}).
+
+  
+
+  The set of variables is ordered
+
+A @struct{dictionary} binds together the following:
+
address@hidden @bullet
address@hidden
+An ordered set of variables.  @xref{Variables}, below.
+
address@hidden
+Optionally, a subset of the dictionary's variables that were named on
+the most recent SPLIT FILE command.  SPLIT FILE use is somewhat
+uncommon in practice, so this set is usually empty.  @xref{SPLIT
+FILE,,,pspp}.
+
address@hidden
+Optionally, a weighting variable.  @xref{WEIGHT,,,pspp}.
+
address@hidden
+Optionally, a filtering variable.  @xref{FILTER,,,pspp}.
+
+
address@hidden itemize
+
address@hidden Data Sets
address@hidden Data Sets
+
+A @dfn{data set} is 
+
+Missing sections here:
+
address@hidden
+** Dictionaries
+** Variables
+** Data sets
+** Coding conventions
+** Pools
address@hidden example
+
address@hidden Coding Conventions
address@hidden Coding Conventions
+
+Every @file{.c} file should have @samp{#include <config.h>} as its
+first non-comment line.  No @file{.h} file should include
address@hidden
diff --git a/doc/dev-data.texi b/doc/dev-data.texi
new file mode 100644
index 0000000..87687f7
--- /dev/null
+++ b/doc/dev-data.texi
@@ -0,0 +1,47 @@
address@hidden Processing Data
address@hidden Processing Data
+
+Developer's Guide
+
+Proposed outline:
+
address@hidden
+* Introduction
+* Basic concepts
+** Data sets
+** Variables
+** Dictionaries
+** Coding conventions
+** Pools
+* Syntax parsing
+* Data processing
+** Reading data
+*** Casereaders generalities
+*** Casereaders from data files
+*** Casereaders from the active file
+*** Other casereaders
+** Writing data
+*** Casewriters generally
+*** Casewriters to data files
+*** Modifying the active file 
+**** Modifying cases obtained from active file casereaders has no real effect
+**** Transformations; procedures that transform
+** Transforming data
+*** Sorting and merging
+*** Filtering
+*** Grouping
+**** Ordering and interaction of filtering and grouping
+*** Multiple passes over data
+*** Counting cases and case weights
+** Best practices
+*** Multiple passes with filters versus single pass with loops
+*** Sequential versus random access
+*** Managing memory
+*** Passing cases around
+*** Renaming casereaders
+*** Avoiding excessive buffering
+*** Propagating errors
+*** Avoid static/global data
+*** Don't worry about null filters, groups, etc.
+*** Be aware of reference counting semantics for cases
address@hidden example
diff --git a/doc/dev-intro.texi b/doc/dev-intro.texi
new file mode 100644
index 0000000..a9b656f
--- /dev/null
+++ b/doc/dev-intro.texi
@@ -0,0 +1,21 @@
address@hidden Introduction
address@hidden Introduction
+
+This manual is a guide to PSPP internals.  Its intended audience is
+developers who wish to modify or extend PSPP's capabilities.  The use
+of PSPP is documented in a separate manual.  @xref{Top,,,pspp}.
+
+This manual is both a tutorial and a reference manual for PSPP
+developers.  It is ultimately intended to cover everything that
+developers who wish to implement new PSPP statistical procedures and
+other commands should know.  It is currently incomplete, partly
+because existing developers have not yet spent enough time on writing,
+and partly because the interfaces not yet documented are not yet
+mature enough to making documenting them worthwhile.
+
+PSPP developers should have some familiarity with the basics of PSPP
+from a user's perspective.  This manual attempts to refer to the PSPP
+user manual's descriptions of concepts that PSPP users should find
+familiar at the time of their first reference.  However, it is
+probably a good idea to at least skim the PSPP manual before reading
+this one, if you are not already familiar with PSPP.
diff --git a/doc/dev-output.texi b/doc/dev-output.texi
new file mode 100644
index 0000000..6fb8b60
--- /dev/null
+++ b/doc/dev-output.texi
@@ -0,0 +1,2 @@
address@hidden Presenting Output
address@hidden Presenting Output
diff --git a/doc/dev-syntax.texi b/doc/dev-syntax.texi
new file mode 100644
index 0000000..81d34c0
--- /dev/null
+++ b/doc/dev-syntax.texi
@@ -0,0 +1,2 @@
address@hidden Parsing Command Syntax
address@hidden Parsing Command Syntax
diff --git a/doc/pspp-dev.texinfo b/doc/pspp-dev.texinfo
new file mode 100644
index 0000000..81c7461
--- /dev/null
+++ b/doc/pspp-dev.texinfo
@@ -0,0 +1,112 @@
+\input texinfo @c -*- texinfo -*-
address@hidden %**start of header
address@hidden pspp-dev.info
address@hidden PSPP
address@hidden For double-sided printing, uncomment:
address@hidden @setchapternewpage odd
address@hidden %**end of header
+
address@hidden FIXME: remove the following line when references resolve 
correctly
address@hidden
+
address@hidden version-dev.texi
+
address@hidden cmd{CMDNAME}
+\CMDNAME\
address@hidden macro
address@hidden struct{TAG}
address@hidden \TAG\}
address@hidden macro
address@hidden union{TAG}
address@hidden \TAG\}
address@hidden macro
address@hidden func{NAME}
address@hidden
address@hidden macro
+
address@hidden
address@hidden
address@hidden iftex
+
address@hidden Math
address@hidden
+* PSPP Developer Guide: (pspp-dev). Tutorial and reference for PSPP developers.
address@hidden direntry
+
address@hidden
+This manual is for GNU PSPP version @value{VERSION},
+software for statistical analysis.
+
+Copyright @copyright{} 1997, 1998, 2004, 2005, 2007 Free Software Foundation, 
Inc.
+
address@hidden
+Permission is granted to copy, distribute and/or modify this document
+under the terms of the GNU Free Documentation License, Version 1.1 or
+any later version published by the Free Software Foundation; with no
+Invariant Sections, with the Front-Cover Texts being ``A GNU Manual,''
+and with the Back-Cover Texts as in (a) below.  A copy of the
+license is included in the section entitled ``GNU Free Documentation
+License.''
+
+(a) The FSF's Back-Cover Text is: ``You have freedom to copy and modify
+this GNU Manual, like GNU software.  Copies published by the Free
+Software Foundation raise funds for GNU development.''
address@hidden quotation
address@hidden copying
+
address@hidden
address@hidden GNU PSPP Developers Guide
address@hidden Edition @value{EDITION}, for PSPP version @value{VERSION}
address@hidden
address@hidden 0pt plus 1filll
address@hidden
address@hidden titlepage
+
address@hidden
+
+
address@hidden
address@hidden Top
address@hidden GNU PSPP Developers Guide
+
address@hidden
address@hidden ifnottex
+
address@hidden
+* Introduction::                Introduction to PSPP development.
+* Basic Concepts::              Data structures and concepts.
+* Parsing Command Syntax::      How to parse command syntax.
+* Processing Data::             Data input, output, and processing.
+* Presenting Output::           Producing machine- and human-readable output.
+
+* Function Index::              Index of PSPP functions.
+* Concept Index::               Index of concepts.
+
+* Portable File Format::        Format of PSPP portable files.
+* Data File Format::            Format of PSPP system files.
+* q2c Input Format::            Format of syntax accepted by q2c.
+
+* GNU Free Documentation License:: License for copying this manual.
address@hidden menu
+
address@hidden dev-intro.texi
address@hidden dev-concepts.texi
address@hidden dev-syntax.texi
address@hidden dev-data.texi
address@hidden dev-output.texi
+
address@hidden function-index.texi
address@hidden concept-index.texi
+
address@hidden portable-file-format.texi
address@hidden data-file-format.texi
address@hidden q2c.texi
+
address@hidden fdl.texi
+
address@hidden
+
address@hidden Local Variables:
address@hidden use (texinfo-multiple-files-update "pspp-dev.texinfo")  in emacs 
to keep these files consistent
address@hidden compile-command: "makeinfo pspp-dev.texinfo"
address@hidden End:
diff --git a/doc/pspp.texinfo b/doc/pspp.texinfo
index 4ecffa4..5b5769f 100644
--- a/doc/pspp.texinfo
+++ b/doc/pspp.texinfo
@@ -87,10 +87,6 @@ Software Foundation raise funds for GNU development.''
 
 * Configuration::               Configuring PSPP.
 
-* Portable File Format::        Format of PSPP portable files.
-* Data File Format::            Format of PSPP system files.
-* q2c Input Format::            Format of syntax accepted by q2c.
-
 * GNU Free Documentation License:: License for copying this manual.
 @end menu
 
@@ -117,10 +113,6 @@ Software Foundation raise funds for GNU development.''
 
 @include configuring.texi
 
address@hidden portable-file-format.texi
address@hidden data-file-format.texi
address@hidden q2c.texi
-
 @include fdl.texi
 
 @bye
-- 
1.4.4.3

--