Hi all,

I want to call exiftool from a c image viewer program (gThumb), to get the metadata for hundreds of images (as part of the thumbnailing process).

exiftool works extremely well, except that it takes ~0.3s per file. I think that most of the time is actually spent initializing the program and perl, rather than actually generating the metadata.

Phil, have you considered adding a "daemon mode" to exiftool, so that it could be launched once and run persistently? Has anyone experimented with that?

- Mike

Hi Mike,

The exiftool script does have powerful multi-file processing abilities. A daemon would only be useful if you want interactive processing of multiple files. Right now to do this you would need to write your own script using calls to the ExifTool functions. It wouldn't be too difficult to set this up if you know a bit of Perl, and would definitely avoid the start-up cost of loading Perl, ExifTool and all the associated libraries. But in the end I'm not sure how much this will speed things up. You could run some tests on your system by processing a large number of files in a single directory to see if the speed benefits will be worth it.

Exiftool used to be a lot quicker, but for each new piece of information that it extracts, it slows down just a little bit more. And now the amount of information extracted from some images is really crazy.

- Phil

Hi Mike,

Ah, so you are using the stand-alone Windows version. Right.

Try running the tests I suggested, and if you see enough speed benefit then I will look into adding a CLI option for you.

- Phil

Using a directory of mixed files (jpgs, tiffs, RAWs, other), I get these benchmarks:

ls -1 | xargs -n1 exiftool -S -a -e -G1

~ 35 seconds

ls -1 | xargs -n100 exiftool -S -a -e -G1

~ 13 seconds

Calling exiftool once per file is 2-3 times slower than passing a list of 100 files. So the initialization overhead is pretty severe.

-Mike

Hmm, I did another benchmark on a more typical folder of jpgs and avis from a digital camera (217 files, 2007-06 folder) and the benchmarks were:

one at a time: 1m 30s

batch mode: 1m 10s

which suggests there is not that much to be gained by a daemon mode. I'm a little puzzled why this folder shows much less improvement than the other one, which had a strange mix of raw/tiff/jpg/pdf/other files... I guess more research is needed before bothering with daemons.

-Mike

I should have thought of this.

When exiftool finds an unknown file, it must load all the modules one-by-one until it discovers what the file format is. And if it isn't a supported format, it has to load ALL modules. This gives you the highest possible initial overhead.

- Phil

Hmm. Interesting. gThumb already knows the mime type of the file when it wants metadata. Could the known mime type be supplied to exiftool as a parameter to speed up processing (ideally exiftool would quickly quit if it didn't recognize the specified mime type)?

I could add an array of mime types that exiftool is known to support, but then I'd have to manually update the program each time new file support was added.

We'd have to agree on mime type names for RAW files - I'm not sure if they are all standardized (some are).

- Mike

Thanks for all the comments, Phil!

I can gain speed by being more careful about not feeding exiftool unsupported formats. The only formats that gThumb supports that exiftool doesn't are the high dynamic range types (OpenEXR and Radiance rgbe - any plans for those? I don't even know if they carry metadata...).

The daemon idea probably wouldn't gain much speed after all.

- Mike

You got me thinking so I ran a quick test on my system here. I timed the following two commands:

The way ExifTool is implemented, the first command doesn't need to load any of the format-specific modules, and it takes 0.100 sec on my system. The second command loads all modules to determine the full group list, and takes 0.677 sec here. The difference is mainly due to the time required to load all the modules, but there is a bit more CPU work done by -listg, so I hacked the code to remove this and the time dropped to 0.622 sec. So on my system (a 1.83 GHz Intel Core Duo), the time to load all modules is 0.522 seconds. That's pretty hefty. (...and you want me to add more?... hehe)

- Phil