Re: [ANNOUNCE] GNU fileutils 4.1

[Dave Dittrich]

Paul and Matt,

On Sun, 27 May 2001, Paul Eggert wrote:

> > Date: Sun, 27 May 2001 20:46:15 -0700
> > From: Matt Schalit <address@hidden>
> >
> > I'm still wondering if this hypothetical
> > md5sum -C option would output checksums
> > for every byte.

Not every *byte*, one for the entire stream and one per block (at the
same time).  That's why I started adding these features to dd (and am
trying to get the changes adopted), so that block size is controlled
by "ibs" or "bs".  I guess I could modify md5sum to do blocking, but
then it seems I'm re-writing parts of dd in md5sum.

> Since it's hypothetical, it would do whatever you want it to do.
> Perhaps you'll need to add more checksum-related options, but that's
> OK: that's what md5sum is for.

Yes, but md5sum won't do per-block checksums.  It just checksums the
file/stdin and that is it.

> I still don't quite follow why you're using MD5 for a new archival
> checksum application, though.  All the authorities I've consulted
> suggest that MD5 may not be secure enough for new applications of this
> type.

I think you miss the point that this is for Unix forensic analysis
applications.  It doesn't matter that SHA1 or RIPEM are more robust
against attacks when the existing programs that are widely used in the
Windows world (e.g., Encase) only (to my knowledge) supports a 32 bit
CRC (!) and MD5.  I don't really care that we aren't now supporting
something far beyond that, since the current "best of breed" barely
supports MD5:

        http://www.encase.com/encase/features_detail.html#file_sig

I guess I should possibly think longer-term (make a quantum leap above
what existing Windows commercial forensic tools provide), but the
short term goal is to get the same functionality that they have
*without* having to re-write them and try to get the resulting program
accepted by the Unix community so the legal community also can accept
it.  (This is not the typical GNU "please-the-masses" kind of scenario
here, and I can understand why it confuses you if you aren't familiar
with it.  There's a link to the Jan 2001 DoJ document on evidence
collection that cites legal cases, if you care to look at the
requirements.  See the Forensics section of my home page.)

> The latest textutils has an "sha1sum" program that you could use
> instead; it computes SHA-1, which is a better fit.  I think the source
> code is structured so that any options you need for "md5sum" can also
> be added easily to "sha1sum".

Now the question is, can it do per-block checksums?  Can it do both
per-block checksums and a complete stream checksum at the same time?
I *REALLY* want to get away from having to do checksum operations and
dd read operations multiple times to get checksums of both the bits
read from disc and the bits written to tape/disc (or at least minimize
them, and be able to handle the situation when media goes bad by
showing which blocks match the original and which don't.)  The goal is
to prove you have the same bits as the original, plus be able to
handle the situation where one or more blocks are corrupt (either on
read from a faulty disc, or read from a faulty tape copy) are not
consistant.

I think you should perhaps read the "Basic Steps in Forensic Analysis
of Unix Systems" and results of the Honeynet Project Forensic
Challenge to see what I'm trying to do:

        http://staff.washington.edu/dittrich/misc/forensics/
        http://project.honeynet.org/challenge/

--
Dave Dittrich                           Computing & Communications
address@hidden             University Computing Services
http://staff.washington.edu/dittrich    University of Washington

PGP key      http://staff.washington.edu/dittrich/pgpkey.txt
Fingerprint  FE 97 0C 57 08 43 F3 EB 49 A1 0C D0 8E 0C D0 BE C8 38 CC B5