[taler-docs] branch master updated: adding DD-11

[gnunet]

This is an automated email from the git hooks/post-receive script.

grothoff pushed a commit to branch master
in repository docs.

The following commit(s) were added to refs/heads/master by this push:
     new 3b866c6  adding DD-11
3b866c6 is described below

commit 3b866c6ca19ebdd6a4354b7c0774bd6481afe59a
Author: Christian Grothoff <christian@grothoff.org>
AuthorDate: Fri Jan 1 15:26:55 2021 +0100

    adding DD-11
---
 coin.png                                  | Bin 0 -> 96492 bytes
 design-documents/010-exchange-helpers.rst |   4 +-
 design-documents/011-auditor-db-sync.rst  | 121 ++++++++++++++++++++++++++++++
 design-documents/index.rst                |   1 +
 4 files changed, 124 insertions(+), 2 deletions(-)

diff --git a/coin.png b/coin.png
new file mode 100644
index 0000000..5628070
Binary files /dev/null and b/coin.png differ
diff --git a/design-documents/010-exchange-helpers.rst 
b/design-documents/010-exchange-helpers.rst
index 9603004..a299948 100644
--- a/design-documents/010-exchange-helpers.rst
+++ b/design-documents/010-exchange-helpers.rst
@@ -1,5 +1,5 @@
-Exchange crypto helper design
-#############################
+Design Doc 010: Exchange crypto helper design
+#############################################
 
 Summary
 =======
diff --git a/design-documents/011-auditor-db-sync.rst 
b/design-documents/011-auditor-db-sync.rst
new file mode 100644
index 0000000..1088976
--- /dev/null
+++ b/design-documents/011-auditor-db-sync.rst
@@ -0,0 +1,121 @@
+Design Doc 011: Auditor-Exchange Database Synchronization
+#########################################################
+
+Summary
+=======
+
+Ways for the auditor to obtain a current copy of the exchange database (to
+verify that the exchange is operating correctly) are discussed.
+
+
+Motivation
+==========
+
+The Taler auditor is expected to check that the exchange is operating
+correctly.  For this, it needs (read-only) access to the exchange database and
+to the bank account of the exchange.  Bank account access is a matter of
+setting up an additional user with limited rights with the bank and is out of
+the scope of this document.  For database access, the auditor should not trust
+the exchange. In particular, the auditor must assume that the exchange may
+violate basic database constraints (like foreign keys) or delete/alter records
+maliciously. However, we also do not want to complicate the auditor logic to
+cope with with violations of well-formedness constraints (like foreign keys or
+non-NULL values or size constraints on fields).  Finally, the mechanism by
+which the auditor obtains the database must provide a reasonably current
+database and the process must perform reasonably well.
+
+
+Requirements
+============
+
+* The solution must allow data to be copied incrementally.
+* The solution must tolerate network outages and recover after connectivity
+  between exchange and auditor is restored.
+* The solution must enable the auditor database to serve as a full backup
+  of the exchange's database (even if possibly slightly outdated due to
+  asynchronous replication or network outages).
+* The solution must scale, in particular if the exchange shards the database,
+  the auditor must be also able to use the same kind of sharding and the
+  synchronization should be possible per shard.
+* The synchronization mechanism must not allow an attacker controlling the
+  exchange database to delete or modify arbitrary data from the auditor's copy
+  via the synchronization mechanism (in other words, some tables are
+  append-only and unalterable).
+* The solution must support database schema updates. Those may require some
+  downtime and closely coordinated work between exchange and auditor.
+* The solution must enable eventual garbage collection at the exchange to
+  be permitted and replicated at the auditor (e.g. DELETE on usually 
append-only
+  tables due to a CASCADE from expired denomination keys).
+* The synchronization mechanism should raise an alert if the exchange violates 
basic
+  constraints (unexpected schema changes, deletion/motification on append-only
+  tables) and then NOT replicate those changes. It may then soft-fail until the
+  exchange has rectified the problem.
+* A good solution would work independently of the specific database used.
+
+
+Proposed Solution
+=================
+
+* Use "common" incremental database replication (whichever is
+  approproate for the exchange database setup, synchronous
+  or asynchronous) to make a 1:1 copy of the exchange database
+  at the auditor.  This should work for any full-featured
+  modern database. The copy cannot be trusted, as constraint
+  violations or deletions would also be replicated.
+* Use helper process to SELECT against the local copy (by
+  SERIAL ID => make sure all append-only tables have one!)
+  to copy append-only tables to 2nd auditor-controlled copy
+  of the database.  Order (or transactionally group) SELECT
+  statements to ensure foreign key constraints are maintained.
+  For mutable tables (basically, only current reserve balance)
+  do not make another copy, but do have logic to recompute mutable
+  tables from other data *if* we need to recover from backup.
+* On schema migration, halt exchange, once auditor DB has
+  synchronized, update all DB schema, then resume DB synchronization
+  and then restart exchange.
+* For GC, simply run GC logic also on auditor's "secure" copy.
+  (The synchronization mechanism will take care of the primary copy,
+   and the helper to copy should not be disturbed by the DELETE operations
+   anyway.)
+
+
+
+Alternatives
+============
+
+* Copy the Postgres WAL, filter it for "illegal" operations
+  and then apply it at the auditor end.  Disadvantages: WAL
+  filtering is not a common operation (format documented?),
+  this would be highly Postgres-specific, and would require
+  complex work to write the filter.  Also unsure how one
+  could later recover gracefully from transient errors
+  (say where the exchange recified a bogus DELETE).
+* Directly SELECT against the (remote) exchange DB and then
+  INSERT/UPDATE at the auditor's local copy. Disadvantages:
+  remote SELECT likely very expensive due to high latency.
+  Diagnostics more difficult. May expose exchange to additional
+  risks from auditor, such as attacks exhausting DB resources
+  by running expensive SELECTs.
+
+
+
+
+Drawbacks
+=========
+
+* SERIAL IDs required in all tables that are "append-only" / immutable.
+* Additional custom logic required to recompute mutable tables
+  on-demand.
+* Limited ability to cope with mutable tables, imposes restrictions
+  on future exchange database evolution.
+* Helper logic to SELECT data in batches that will certainly
+  maintain invariants may be a bit tricky, but in principle
+  the foreign key constraints should form a DAG, simply dictating
+  the order in which new entries are to be copied.  It may also
+  be that simply running "big" transactions across all tables
+  is the answer, to be investigated what performs better.
+
+
+
+Discussion / Q&A
+================
diff --git a/design-documents/index.rst b/design-documents/index.rst
index 6b66dd2..1c3b97d 100644
--- a/design-documents/index.rst
+++ b/design-documents/index.rst
@@ -20,3 +20,4 @@ and protocol.
   008-fees
   009-backup
   010-exchange-helpers
+  011-auditor-db-sync

-- 
To stop receiving notification emails like this one, please contact
gnunet@gnunet.org.