help-gsl
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Checking GSL for Spectroscopy


From: Mike Marchywka
Subject: Re: Checking GSL for Spectroscopy
Date: Thu, 18 Mar 2021 10:21:55 +0000

Thanks. I've looked briefly at a lot of different kinds of "spectra" -
audio, solar, image fft, distributions,  xps, even Raman that may evolve
with time - 
 and
as you suggest you may not be interested so much in some abstract 
comparison as in extracting some model information. Comparing spectra
 may be with the intent of resolving a given one into component
pieces- how much of each basis element is  in the  measured thing.
Generally you have lines with some profile- gauss and lorentz would
be well known - and then a continuum which could be anything 
with blackbody and I guess fluorescence as examples.  Then you have
instrument issues to resolve- baseline and maybe broadening could
be factors for a library.  

You could imagine developing a language around common things-
consider maybe writing "R" packages that use GSL.  CRAN's R
may be a good open replacement for MATLAB. 

I played with python briefly and any language that enforces white space,
and IIRC earlier distinguished space and tab lol, is a bit
of a suspect ... 

I've also run into various language-vs-library issues and thinking about 
business
issues. I've got one "program" to make downloading citation information
less distracting from diverse sources targeted at academics or anyone doing 
internet
research ( this could be companies writing white papers or technical reports 
for their own products
compared to competitors,  political hacks writing position or policy papers if 
the
internet sites supply Bibtex for their works ). The code itself is almost the 
opposite
of science- it is a collection of hacks tried in the order in which I discovered
they may be useful to try to download citation information without bothering
the user much. After looking at maybe 100's of hacks, some patterns emerged
and in the conversion from an awful bash script to c/c++ it looked
like you could come up with a mini-language based on "subroutine"
or method calls.  The dev version uses readline for interaction which appears
to have some licensing issues but since I almost always just write for 
myself I don't usually notice stuff like that. 

btw, as their are likely academics here if you have your own horror or success
stories getting citation information for your publication efforts please share
as appropriate here or on the texhax list . Thanks. 

note new address
 Mike Marchywka 306 Charles Cox Drive Canton, GA 30115
 2295 Collinworth  Drive Marietta GA 30062.  formerly 487 Salem Woods Drive 
Marietta GA 30067 404-788-1216 (C)<- leave message 989-348-4796 (P)<- emergency


________________________________________
From: Fritz Sonnichsen <sonnichs@gmail.com>
Sent: Tuesday, March 16, 2021 9:53 AM
To: Mike Marchywka
Cc: help-gsl@gnu.org
Subject: Re: Checking GSL for Spectroscopy

Mark
  I am converting someone's MATLAB code so I am not sure what he is doing 
yet--but several years ago I did spectral analysis in MATLAB and probably very 
similar. This is for Raman and LIBS spectra.
1) "Usually" I apply a high pass filter to the spectrum. This gets rid of the 
noise I need control over this since as you would expect the signal and noise 
can get pretty close! Intuition comes into play here.
2) Next I baseline the spectra. This removes any constant bias.  For LIBS I was 
usually able to further filter "spikes" and then take a mean of the remaining 
line, subtracting this from the overall spectrum. Raman can get a bit more 
difficult-I am, at least,  subtracting the fluorescent line which can have a 
lot of features (e.g. spikes). At times, if you know this background you can 
subtract it first but you get all types of complications from normalization. 
Again--intuition comes into play.
3) The resulting spectrum needs to be compared to a database. For LIBS the 
latter is quite small--mostly atomic/elemental data such as NIST. I could 
generally do a discrete comparison of the spike locations using a peak-finder, 
align them with the known examples and get a pretty high hit rate. This was for 
qualitative data.
Raman is, again, much more complex. The data I was using was constrained and 
simpler but the case in hand here is much more complex. We are doing mixed 
plastics at the moment. My colleague found the best matches by taking a stats 
correlation with 44000 entries and pulling out the values closest to "one". It 
works remarkably well.

I don't think there is much above that cannot be written in C in a reasonable 
amount of time. But we are looking ahead and would like to draw on the 
collective experience of the science community. This type of analysis is quite 
common and there are enough new wheels out there that we don't want to 
re-invent old ones!
    Very important is that "intuition" part. I would think a lot of this issue 
has been better solved since I was doing this. There are a lot of adjustments 
that could be made-for example iterating trial baselines, rejecting noise at 
varied levels etc. Processors are faster now and the AI movement has brought in 
PCA and a lot of other techniques that begin to transcend my current state of 
knowledge (I work more on the physics end of things and would prefer to use 
routines from the communities if possible to save time).

Thanks for your interest Mark!
Fritz


On Tue, Mar 16, 2021 at 9:25 AM Mike Marchywka 
<marchywka@hotmail.com<mailto:marchywka@hotmail.com>> wrote:
Can you comment on how you compare spectra? Just for my own
personal interest, not sure if will further the thread here however..
Not sure a "dot product" in the conventional sense would help much.
You could imagine comparing peak positions and relative heights
or a fit to a continuum for example.  Peaks plus black body in some
vector comparison?

note new address
 Mike Marchywka 306 Charles Cox Drive Canton, GA 30115
 2295 Collinworth  Drive Marietta GA 30062.  formerly 487 Salem Woods Drive 
Marietta GA 30067 404-788-1216 (C)<- leave message 989-348-4796 (P)<- emergency


________________________________________
From: Help-gsl 
<help-gsl-bounces+marchywka=hotmail.com@gnu.org<mailto:hotmail.com@gnu.org>> on 
behalf of Fritz Sonnichsen <sonnichs@gmail.com<mailto:sonnichs@gmail.com>>
Sent: Tuesday, March 16, 2021 9:15 AM
To: help-gsl@gnu.org<mailto:help-gsl@gnu.org>
Subject: Checking GSL for Spectroscopy

I am preparing to convert MATLAB code to something more general. The new
code will run on LInux and ARM processors.
   For a lot of reasons I am not going to use Python. We also want to
keep this project "close" to scientists and do not want to turn it into a
full time computer programming job. So the final word is that I am looking
for something that can be called by (and hopefully is written) in C. Worse
case I will just write the code myself but would prefer to start
integrating our systems into something with a lot of pre-written and vetted
routines.

GSL looks like a good choice. Maybe R comes next. We have a mix of needs
but I will point out a few:
1) Baselining a spectrum
2) Finding peaks in that spectrum
3) using Pearson correlation to compare the spectrum QUICKLY to
about 50,000 recorded examples.

We also have some uses with basic statistics and we do some image
processing.

So my question is--does GSL position itself in these areas? MATLAB (with
packages) does them all.
     I am not sure how active GSL, if it is keeping up with AI, imaging and
spectroscopy--or is it fading or giving way to popular languages for
example. I was surprised that the 600+ page manual did not seem to show
anything relating to the simple spectral analysis described above for
example. Certainly I can search the web for others' code but at some point
if I cannot attach to a well established product I will just write it
myself.

Any comments appreciated
thanks
Fritz



reply via email to

[Prev in Thread] Current Thread [Next in Thread]