Re: [help-GIFT] Fwd: Big database on GIFT. gift-generate-inverted-file error.

[Wolfgang Müller]

> 1. imagemagick convert to generated thumbnail and ppm files.
> 2. gift-extract-features and gift-write-feature-descs to get features
> and creat .fts files.
> 3. gift-generate-inverted-file to generate InvertedFile.db.

Yes, and things do not need to remain that way. In fact, David McG. Squire 
(founder of the Viper project) who wrote the feature extraction had his 
"home-made" PPM routines and used them for feature extraction. I did not 
really want to touch his code, so I wrote the perl wrapper around that which 
uses image magick convert. In fact, much of the time the feature extraction 
takes is due to startup of convert and startup of gift-feature-extraction. 
Both of these programs have to be started for each image. The're both C, so 
dynamic linking is fast compared to c++, but it still takes non-negligible 
time of the whole feature extraction process.

If you're interested in speed, I would suggest you to extend 
gift-feature-extraction such that you combine with the libmagick library. I 
had some stuff doing that (among other things) for my thesis, but I never 
released it due to feature creep and due to libhell (I had so many pluggable 
components that things became a maintenance nightmare. Nice to work with at 
home, but you needed real practice to install it and due to my new job I did 
not have the time to get it ready for other people.)

However, if you just want fast feature extraction for one case (images, 
feature extraction using viper features), then it should be easy to link up 
image magick to gift-feature-extraction, and to do also the conversion to 
thumbnails while doing the feature extraction. I can send some starter code 
to this list if you wish.

(parts already answered snipped) 

> Actually it's part of my master thesis work. I'm try to put GIFT with
> big medial image database into a grid enviroment. I hope the grid
> technology can give me a reasonable speed to run the GIFT with the large
> medial image database. So I'd like to have a measurement of the speed of
> 1 nodes vs. 4, 8 16 or even 32 nodes.

I see.

> I think the gift-add-collection.pl can speed up by put into distrubuted
> cluster. 

Yes, when the time needed to transfer the image is shorter than the time 
needed for the feature extraction. In a local, Gbit Ethernet setting this 
might work.

> And the idea to me is to seperate the big InvertedFile.db into
> small one over lots of nodes. Then query these nodes (by
> gift-mrml-client.pl) and get those result for finally processing.

Yes, sounds cute.

> So can you please tell me what's the error message mean and give me some
> suggesstions.

I think this has been answered, too. I hope things work fine by now. Please 
don't hesitate to ask further questions.

Cheers,
Wolfgang