AlbumFetcher - Download multiple galleries w/ embedded captions and keywords

[wellman]

*Admin Edit*

Hey everyone unfortunately AlbumFetcher is no longer being supported and I've removed the link fro the Wiki page. Try out RapidFetcher instead, it works with Macs, Win or Linux! (Adobe Air app).

You can find RapidFetcher along with other downloaders here: http://wiki.smugmug.net/display/SmugMug/Hacks+and+Apps

*/Admin Edit*

**Mac User? Go here.**

I'm pleased to announce the release of AlbumFetcher! AlbumFetcher is a little pet project of mine which I've decided to make freely available to the SmugMug community. I've received amazing amounts of help (and ideas) through these forums, and this my way of saying thanks. It's a simple little app that serves two main functions:

1. Download one or many SmugMug galleries at once
2. Embed captions and keywords into your downloaded images' IPTC headers

AlbumFetcher should run on Win2000, WinXP and above. It has been reported to run under Wine on Linux and through Parallels on a Mac. It's a single file, so it's easily portable and doesn't need to be installed. AlbumFetcher is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This DGrin thread will be the project's home page. Feedback is extremely appreciated, and if there are bugs and/or other issues (as there always are), please let me know here. This is a hobby, so I can't hope to live up to SmugMug's standard of customer service, but I will try to help you out if you're having an issue with the software. One word of caution - beware of the total number of bytes you download. You could easily run up against your monthly bandwith limit.

Again, this is yours to use forever for free. However, if you're a fan of my little app and feel so inclined, you can send me the electronic equivalent of the adult beverage of your choice here.

Enjoy!
-Greg


Current version: v0.3.6 (July 8, 2009) - Zipped Application | Exe File (if you're afraid of zips) | Source

Old Drop.io links, just in case: Zipped Application | Exe File (if you're afraid of zips) | Source
(Clicking these links will take you to a drop.io page, where you can click "Download" to get the file.)


How Does it Work?

1. Download AlbumFetcher (zip file) with the link above.
2. Use WinZip or your favorite zip program to extract AlbumFetcher.exe (the program) to the folder of your choice (maybe the desktop).
3. Double-click the AlbumFetcher program icon.
4. If you're an account owner: In the top-right corner, enter your SmugMug ID and password. If you're a guest: Switch the login type selector to Guest. Then enter the SmugMug ID of the account you'd like to work with. You'll only have access to listed public galleries. (SmugMug ID means either your email address which is registered with SmugMug or the yourname part of yourname.smugmug.com. Your ID and password go directly to SmugMug; there's nobody listening in between.)
5. Click List Albums. A listing of all your albums will appear.
6. In the Download Options box (on the right), pick a folder to save albums to. Also pick the image sizes and other options as you wish.
7. Click the list to select albums for download. Use Ctrl-clicking or Shift-clicking to select more than one.
8. Click Fetch Albums. AlbumFetcher will now download your requested albums at the size you specified.
9. Have a tasty beverage.

UPDATE, Nov 14 2006:
I've incorporated some of Pat Kane's requests and some improvements of my own into v0.2. New features include:

1. Datetime stamp for each line in log file
2. Files with same original name are now handled successfully (an underscore is prepended to the second file's name).
3. Users can optionally skip over previously downloaded images (helpful when a download fails and you want to start over, but not re-download all the successful images).
4. Category/subcategory info is shown as a field in the list control.
5. A thumbnail of the image currently being downloaded is displayed.
6. Total time statistics are recorded when a job is completed successfully.
7. MD5 hashes are now compared. Files that fail the comparison are left alone, but a note is made in the log file.

Again, enjoy!
-Greg


UPDATE, Nov 16 2006:
Pat Kane pointed out a bug in the folder selector, which is now fixed. Current version is now 0.2.1. Thanks, Pat!


UPDATE, Jan 11 2008:
I'm pleased to announce the release of AlbumFetcher v0.3. There are several improvements in this release. Read on for details.

1. Multiple login options - Users may now login to SmugMug as account owner, guest, or guest with site password. If you're uncomfortable with other people being able to bulk-download your images, set your external linking option (in gallery customization) to "No."
2. Album-level password support - Guests may download password-protected galleries (with the proper password of course).
3. Selectable image size for download - Choose which size you want! AlbumFetcher will find the next largest available if logged in as guest and requested size is not available.
4. Video support - Choose which size you want! AlbumFetcher will find the next largest available if logged in as guest and requested size is not available. Support is also included for legacy mpeg-1 videos.
5. As inspired by SmugMug, AlbumFetcher is stretchy!
6. No more runtime errors - (Fingers crossed.) AlbumFetcher has essentially been completely rewritten. I found a few scenarios that caused the app to crash during my development process, and they've been eliminated. I've successfully downloaded thousands of images while testing. Let me know if you still have runtime problems.

UPDATE, Jan 12 2008:
Scott Walter pointed out a bug which caused the app to crash when downloading GIFs, which is now fixed. Current version is now 0.3.1. Thanks, Scott!


UPDATE, Jan 29 2008:
A few people were having similar issues. Basically AlbumFetcher was bombing out if it tried to download an image which was missing (for whatever rare reason) on SmugMug. This is now handled more gracefully in new Version 0.3.2 (link above) - AlbumFetcher will stop the download and report the error (instead of crashing).


UPDATE, Mar 3 2008:
AlbumFetcher is now compatible with SmugMug's security changes (use of keys).


UPDATE, Dec 15 2008:
It's been a pleasure over the past two years to see AlbumFetcher become a useful little download utility for members of the SmugMug family. The users of AlbumFetcher deserve a product that will continue to evolve and keep up with the latest SmugMug features (hidden photos, SmugVault, etc). To use the catchphrase circulating at work, I no longer have the personal bandwidth to keep AlbumFetcher current.

This thread is a call for help for someone to take the AlbumFetcher reins. The code is simple Microsoft Visual C++, using MFC. If there's anyone interested in taking the project over as-is or porting it to something cross-platform, let me know.

Thanks, and thanks again to those who've had kind words and future ideas for AlbumFetcher.


UPDATE, June 17 2009:
Version 0.3.4 is out. This fixes some issues uncovered by some API changes. AF has to be smarter about how it reads responses from SmugMug now. Also, a fix suggested by t0mt0m related to photos from iPhones has been incorporated. Thanks for your patience on this one; the issue was hard to find.


UPDATE, July 7 2009:
Version 0.3.5 is out. Thanks to derf for finding a bug which caused AlbumFetcher to die when the filename recorded by SmugMug contains invalid characters for a directory structure. Now fixed.


UPDATE, July 8 2009:
I forgot to update one of the version number lines of text in v0.3.5. To keep things straight, I updated everything to v0.3.6, which is now available via the links above.


UPDATE, March 23 2010:
Thanks to SamirD, AlbumFetcher has a more permanent home. Drop.io is great, but it's 1000 limit was making it a pain to keep the files served up consistently. Thanks for the space, SamirD! Links updated above...

[richW]

Greg, I tried it on public, private, passworded galleries. Works great..
Fast and easy.
Thanks....

[Barb]

Well, if this isn't the neatest little app!!

Really nice job - it works wonderful!

[devbobo]

Hey Greg,

Sounds pretty sweet mate

I haven't had a chance to try it out yet but I will when I get a chance.

Thanks for contributing.

David

[pat.kane]

Wow, very cool. Thanks for making this available. Fortunately, I've not had a need to pull data back off of smugmug, but I'll definitely keep this in mind in case I do at a later time.

I just tried the application and downloaded one gallery. Total download was about 55-MBytes and it only took a few minutes.

I'm pleased to say that the program was easy to use and it worked w/out problems.

Here are a few feature requests to consider for a later version:

1. Add a date/time stamp to the log file in front of each recorded action, e.g.,

2006-11-12 18:44:22 Totaling bytes for album Test.
2006-11-12 18:44:34 Downloading album description for Test.
2006-11-12 18:44:40 Downloading IMG_1117.JPG from Test (1 of 25 total).

2. Once a download is complete, present some statistics in the Status window, e.g., Successfully downloaded 25 images, 55-Mbytes total in 3 minutes 14 seconds.

3. Show a thumbnail of the image currently being downloaded.

4. Allow a pick list for the e-mail address box (handy for users with more than one account)

5. Allow the list of Galleries (Albums) to be sorted.

6. Retain the Category > Subcategory > Album structure from the user's site and allow that to be shown where Album Title is currently shown.

These are just suggestions. Even without any of them being implemented, this is still a very nice utility. Thanks again for sharing.

Regards,
Pat

[pat.kane]

A couple of questions:

1. What happens to an original image's IPTC data (assuming it had data when it was uploaded) if the "Embed IPTC data" box is checked? Is the original data appended, overwritten, other?

2. Are you using an existing, well-established library to write the IPTC data to the originals?

[wellman]

Rich. Barb, David, and Pat:
Thanks for the kind words! I'm glad you like it (or in devbobo's case, the idea of it ).

Pat, thanks for the great suggestions; I'll get some of those worked into v0.2. Let me see if I can answer some of your questions...

Originally Posted by pat.kane

1. What happens to an original image's IPTC data (assuming it had data when it was uploaded) if the "Embed IPTC data" box is checked? Is the original data appended, overwritten, other?

Any IPTC data that is pre-existing in the image is preserved, with the exception of caption and keyword, which are overwritten with the SmugMug values.

For example, say you captioned an image in Bridge or Lightroom and put some geographical data in an IPTC field, too. Now you upload to SmugMug. SM should read the caption out for you and place it in SM's database. Now say you edit the pre-populated caption to something else. AlbumFetcher (if the IPTC box is checked) will overwrite the original's embedded caption with the one from SmugMug, but that geographical data will be untouched. Make sense?

Originally Posted by pat.kane

2. Are you using an existing, well-established library to write the IPTC data to the originals?

I'm using the Exiv2 library (with some edits to get it to compile under MS VC++ 2005). It's GPL'ed, which is why I GPL'ed my project, too.

[wellman]

Version 0.2 of AlbumFetcher is out. Details in the first post...

[pat.kane]

Wow, that was fast. Thanks for picking up some of my suggestions. I really like the thumbnail showing while each image is downloading. Now just to get the Album Title and Category fields to be sortable ;-)

Here's a minor bug -- I selected Cancel in the Download to Folder Select dialogue box and the previously selected folder name was replaced with a back slash. On Cancel, I would have expected the folder name to be preserved.

IPTC explanation received and understood. I figured that this would be the behaviour. I add quite a bit of metadata to my images and even though smugmug doesn't currently use it all (e.g., headline, city, state, etc.), I didn't see any reason for it not to be preserved on download. Fortunately, you confirmed that it would be.

Thanks for the pointer to the Exiv2 library. Their web site says the right types of things I was looking for, e.g., "a sophisticated write algorithm avoids corrupting the MakerNote."

Since you brought it up :) here's another suggestion for a future release: fill in the EXIF GPSInfo tags on download with any latitude/longitude assignment made within smugmug. I'm not sure if this information even available via the smugmug API, but it would be handy for those that don't geocode their images prior to upload. This isn't something I'm requesting for myself (I geocode prior to upload), but it seems like it would be a nice feature for those that use smugmug's tools. I'm sure this would be low on the priority list of things to do, but I figured I'd throw it out there anyway!

Check your paypal for a few drinks on me. You've earned them.

[wellman]

Originally Posted by pat.kane

Wow, that was fast. Thanks for picking up some of my suggestions. I really like the thumbnail showing while each image is downloading. Now just to get the Album Title and Category fields to be sortable ;-)

No problem. I agree that sorting would be nice - it's top on the list for v0.3. We're about to head out on vacation, so v0.3 won't be turned around quite as quickly.

Originally Posted by pat.kane

Here's a minor bug -- I selected Cancel in the Download to Folder Select dialogue box and the previously selected folder name was replaced with a back slash. On Cancel, I would have expected the folder name to be preserved.

Oops. Thanks for the heads-up. I'll take care of that one.

Originally Posted by pat.kane

IPTC explanation received and understood. I figured that this would be the behaviour. I add quite a bit of metadata to my images and even though smugmug doesn't currently use it all (e.g., headline, city, state, etc.), I didn't see any reason for it not to be preserved on download. Fortunately, you confirmed that it would be.

Thanks for the pointer to the Exiv2 library. Their web site says the right types of things I was looking for, e.g., "a sophisticated write algorithm avoids corrupting the MakerNote."

If you want to verify this for yourself, Google a free app called Exifer. It will show you (and allow you to edit) all EXIF and IPTC tags. I did some testing to make sure other IPTC fields were left alone, but I wouldn't mind some secondary confirmation.

Originally Posted by pat.kane

Since you brought it up :) here's another suggestion for a future release: fill in the EXIF GPSInfo tags on download with any latitude/longitude assignment made within smugmug. I'm not sure if this information even available via the smugmug API, but it would be handy for those that don't geocode their images prior to upload. This isn't something I'm requesting for myself (I geocode prior to upload), but it seems like it would be a nice feature for those that use smugmug's tools. I'm sure this would be low on the priority list of things to do, but I figured I'd throw it out there anyway!

That's a fantastic idea. That will also go into the list for v0.3. My goal here was to make a "true" Smugmug backup tool that would save all my caption/keyword work, since I do most of mine on the site. The GPSInfo is a further extension of the same principle, so I will be happy to work that in.

Originally Posted by pat.kane

Check your paypal for a few drinks on me. You've earned them.

Sweet! Thank you! Thanks also for your suggestions.

[pat.kane]

Originally Posted by wellman

If you want to verify this for yourself, Google a free app called Exifer. It will show you (and allow you to edit) all EXIF and IPTC tags. I did some testing to make sure other IPTC fields were left alone, but I wouldn't mind some secondary confirmation.

I still have Exifer on the hard drive. I used to use it to rename my photos to a date/time stamp, but that job is now taken care of by Downloader Pro.

It will probably be a couple of weeks before I get a chance, but I'd be glad to run some images through the process to see if any metadata gets clipped.

[wellman]

Originally Posted by pat.kane

Here's a minor bug -- I selected Cancel in the Download to Folder Select dialogue box and the previously selected folder name was replaced with a back slash. On Cancel, I would have expected the folder name to be preserved.

Fixed in v0.2.1.

Originally Posted by pat.kane

Since you brought it up :) here's another suggestion for a future release: fill in the EXIF GPSInfo tags on download with any latitude/longitude assignment made within smugmug.

I took a look at the API documentation, and I don't see this as being available, yet. Once Smugmug makes it available, I can use the Exiv2 library to get this implemented. I'll make an official feature request.

[andyatkinson]

AWESOME app. I downloaded a gallery at around 400kbps. I think it added some titles as keywords though, but I'll need to do some more through testing. I am using 2.1. It looks like at least in one case a photo named "Picture 022.jpg" was tagged with a keyword "22". Have you ever seen this?

Has anyone seem the same tool for Flickr? Have you worked with the Flickr API to do the same thing? Specifically I am interested in using Flickr as a geotagging interface because it works so well. Here is a relevant link...

http://www.flickr.com/forums/ideas/28952/

[wellman]

Originally Posted by andyatkinson

AWESOME app. I downloaded a gallery at around 400kbps.

Thanks! Glad you liked it.

Originally Posted by andyatkinson

I think it added some titles as keywords though, but I'll need to do some more through testing. I am using 2.1. It looks like at least in one case a photo named "Picture 022.jpg" was tagged with a keyword "22". Have you ever seen this?

I'm not exactly sure what might be going on here, but I have some ideas. It's possible that whatever image managemant program you're using (before you upload to SmugMug) is sticking the image number in as a keyword (I've seen this before), and that the same thing is happening with titles showing up in keywords.

Nothing too fancy is going on with AlbumFetcher. It just overwrites the original image's caption and keyword IPTC fields with the strings provided by SM's API. If you'd like to check out what data is embedded in an image before AlbumFetcher does its work, Exifer is a good little free tool.

If there really does seem to be something squirrelly going on with AlbumFectcher, you can send me a link to an original image that will work as an example (you can PM me the link if you like). That being said, I'm on holiday and away from my development tools for a few days.

Originally Posted by andyatkinson

Has anyone seem the same tool for Flickr? Have you worked with the Flickr API to do the same thing? Specifically I am interested in using Flickr as a geotagging interface because it works so well. Here is a relevant link...

http://www.flickr.com/forums/ideas/28952/

I haven't ever used Flickr, and honestly don't have plans to in the near future. It's a good idea, though. Hopefully SM will beef up their geotagging interface soon.

Thanks again for the kind words!
-Greg

[pat.kane]

Originally Posted by andyatkinson

It looks like at least in one case a photo named "Picture 022.jpg" was tagged with a keyword "22". Have you ever seen this?

Yes and it's a smugmug behaviour. smugmug tries to make keywords from file names. Sometimes it makes sense to do so, e.g., "Jim and Steve.jpg", and other times it does not, e.g., "Picture 022.jpg".

You probably don't see 22 as a keyword on your smugmug page as smugmug is programmed to ignore it; however, if you go to that image on smugmug and click on Edit next to your Keywords, you'll see 22 as one of the entries. Yes, it is there even if it doesn't show up.

Because it is there, AlbumFetcher adds it as a keyword on download.

[wellman]

Originally Posted by pat.kane

Yes and it's a smugmug behaviour. smugmug tries to make keywords from file names. Sometimes it makes sense to do so, e.g., "Jim and Steve.jpg", and other times it does not, e.g., "Picture 022.jpg".

Thanks for the great explanation!

[ghealy]

Great little utility!

I needed to grab several images the other day. This would have been a great tool to have had. At least, I'll have it next time.

Any chance that future development might include multiple individual images within a gallery/album?

Thank you so much for your efforts.

[wellman]

Originally Posted by ghealy

Great little utility!

Thanks!

Originally Posted by ghealy

Any chance that future development might include multiple individual images within a gallery/album?

It would take a little work, but that's a good idea. It will probably be a good while before I get back around to adding new features, but I'll put this on the list.

[Andy]

Greg:

This is so great. Thanks for doing this!

[wellman]

Originally Posted by Andy

Greg:

This is so great. Thanks for doing this!

You're quite welcome. It's been nice to "give back" a little, after all the great help I've received from this forum.