SyncMug
http://www.digitalmediashelf.com/syncmug/



This is a file synchronization program for uploading and downloading images and albums between your local PC and SmugMug. It currently has add and remove support, with update support coming very soon.

I currently have a Windows executable available, and I should have a Mac OS X package available shortly.

Because I am the only one that has tested this code so far, and I was in a bit of a hurry trying to meet the October 31st deadline, I recommend using caution with the program. I consider the application in a beta stage right now until more extensive testing is done. You can run the program to find find differences between your local storage and SmugMug without it trying to execute the difference tasks. I recommend at least checking it before you let it loose to make sure it's not doing anything you aren't expecting.

The program can be downloaded from my site http://www.digitalmediashelf.com/syncmug/ and there is also a quick tutorial there to better explain some things about the program.

I have now released SyncMug for Mac OS X
http://www.digitalmediashelf.com/syncmug/

I wasn't able to test it on any other computer besides the one I built the binary on, and that systems I borrowed, so please let me know if it doesn't run.

I am interested in any feedback on the use of the program.

Originally Posted by mjohnsonperl I have now released SyncMug for Mac OS X http://www.digitalmediashelf.com/syncmug/ I wasn't able to test it on any other computer besides the one I built the binary on, and that systems I borrowed, so please let me know if it doesn't run. I am interested in any feedback on the use of the program.

I can't really test it right now, but it opens without crashing! w00t!



Thanks for supporting OSX.

Originally Posted by DavidTO I can't really test it right now, but it opens without crashing! w00t! Thanks for supporting OSX.

Well opening is great, that means a lot of it is working. You can save your login and run a sync without the "Execute Differences" checked and it won't try and upload or download anything, it will just log what the differences are without actually executing the difference tasks.

When I built the Windows binary there were issues with my package including certain dll libraries in the binary, and that took a little to figure out, and I was hoping I wouldn't have those kind of problems on OS X build.

The system I built the OS X binary on was OS X 10.4 Tiger running on a Mac Book Pro with an intel processor, so I don't know what it does on Leapord, or previous versions of OS X, or the Power PC processors.

Hi Matt,
I'm not very technically inclined but I was interested to see that Syncmug enables bulk downloading from SmugMug. I have installed it on my mac but I can't see how to download ?
Please, in nice simple words could you explain :))
Thanks
Caroline

Originally Posted by mjohnsonperl SyncMug http://www.digitalmediashelf.com/syncmug/ This is a file synchronization program for uploading and downloading images and albums between your local PC and SmugMug. It currently has add and remove support, with update support coming very soon. I currently have a Windows executable available, and I should have a Mac OS X package available shortly. Because I am the only one that has tested this code so far, and I was in a bit of a hurry trying to meet the October 31st deadline, I recommend using caution with the program. I consider the application in a beta stage right now until more extensive testing is done. You can run the program to find find differences between your local storage and SmugMug without it trying to execute the difference tasks. I recommend at least checking it before you let it loose to make sure it's not doing anything you aren't expecting. The program can be downloaded from my site http://www.digitalmediashelf.com/syncmug/ and there is also a quick tutorial there to better explain some things about the program.

Originally Posted by caroline Hi Matt, I'm not very technically inclined but I was interested to see that Syncmug enables bulk downloading from SmugMug. I have installed it on my mac but I can't see how to download ? Please, in nice simple words could you explain :)) Thanks Caroline

I put together a quick tutorial online to demonstrate how the program works:
http://www.digitalmediashelf.com/syncmug/tutorial/osx/

When you start the program for the first time the local catalog is empty so the first time you sync it will download all albums and images from the website to your computer. Once it has synced for the first time it will keep your computer and your SmugMug site synchronized with new or removed images and albums.

You'll want to make sure you check the "Execute Differences" checkbox so that it actually execute all the difference tasks it identifies when you run the sync.

Hi Matt,
I left it running when I went to bed, woke after a couple of hours and it had stopped with this message:-

Died at /</Applications/SyncMug.app/Contents/MacOS/SyncMug>SyncMug_Lib.pm line 1100.

Died at /</Applications/SyncMug.app/Contents/MacOS/SyncMug>SyncMug_Lib.pm line 1100.

while executing
"::perl::CODE(0x181a8f4)"
invoked from within
".f2.b invoke"
("uplevel" body line 1)
invoked from within
"uplevel #0[list $w invoke]"
(procedure "tk::ButtonUp" line 23)
invoked from within
"tk::ButtonUp .f2.b"
(command bound to event)

I clicked sync and it restarted, is still running now, in total has downloaded around 750 mb so far.

UPDATE 11.40am GMT
After a couple more stops it ran with no problems and I now have just what I wanted - all my SM pics in one place on my HD. In total 3gb, - thanks very much Matt.
G4Mac 10.4.10

Caroline

Originally Posted by caroline Hi Matt, I left it running when I went to bed, woke after a couple of hours and it had stopped with this message:- Died at /</Applications/SyncMug.app/Contents/MacOS/SyncMug>SyncMug_Lib.pm line 1100.

I'm glad it finally got everything for you. Each time it stopped was it the same error "SyncMug_Lib.pm line 1100"?

I'm going to try and hammer out these bugs, at least you can retry to finally get it all.

THANKS FOR THE FEEDBACK!

Do you have any suggestions for improvements?
Are you primarily using it for the bulk downloading capabilities, or are you wanting to actually sync your images?

Matt

Originally Posted by mjohnsonperl I'm glad it finally got everything for you. Each time it stopped was it the same error "SyncMug_Lib.pm line 1100"? I'm going to try and hammer out these bugs, at least you can retry to finally get it all. THANKS FOR THE FEEDBACK! Do you have any suggestions for improvements? Are you primarily using it for the bulk downloading capabilities, or are you wanting to actually sync your images? Matt

Hi Matt,

The error was the same each time as above. I would use SyncMug primarily for downloading until such time as I've got thungs sorted better to want to sync my images properly.

I don't have any suggestions for you but I'm sure some of the more experienced members may have some input - DavidTo are you there ? :))

If its of interest, I had uploaded a video which has been downloaded as an mpeg.4 , it was originally 2 AVI files from my Canon A610 which I stitched and converted using VisualHub.
Hope this helps :)

Caroline

Matt -

I am very interested in SyncMug...

I am building a family photo album. I have setup a strict 3-level local folder structure to mimic the SmugMug category, subcategory, and gallery. Roughly the 3 levels correspond to Family, Date range, and Event.

I expect the photo album may ultimately reach 5000 photos or so (small compared to some collections I have heard about). However, I expect a pretty heavy amount of editing, changing of captions, and re-organizing of the photos as I get input and information from other members of the family.

I am hoping to use the SmugMug "add comment" functionality to enable other family members to chime in with their remembrances about certain photos. I can then use some of this info to update the captions and perhaps reorganize photos as I feel is necessary.

Getting to the point...

As I mention, I expect alot of editing and reorganizing. This will take place on both Smugmug (captions) and locally (adding folders, moving photos betwen folders, changing captions, cropping photos perhaps, etc.). Other folks may also contribute photos by email into Smugmug.

I am looking for an app that will enable me to keep the local folders and contents of these folders synchronized with the 3-level Smugmug organization regardless of where I make my edits. Without me having to remember where I made my edits. This would be impossible to manage manually.

1) Is this the problem you are solving with SyncMug?

2) How does SyncMug determine a match/duplicate? Same filename and folder? or does it compare the dates of files and choose the newer?

3) What happens if I make separate but conflicting changes on SmugMug and the local files? How does SyncMug resolve?

4) I get the sense that SyncMug doesn't actually delete any photos from either SmugMug or local folders, but instead stores "deleted" files out of the way. A good approach. I see that SyncMug has you specify where to store locally "deleted" files. Where are deleted files stored on the Smugmug side?

5) I don't really know much about how captions/keywords/EXIF/IPTC work. I believe some is stored within the actual jpeg and some may be stored in Smugmug itself and associated with the file. Does SyncMug ensure that this information is sync'ed as well as the photo itself? Is all of this info stored into the jpeg? (If it matters, I use Picasa locally, but may move to Elements)

6) At the moment, my local folders hold what I consider to be the gold production version of my 3-level structure and photos. What I have on Smugmug is largely out of sync with my local folders. I would like to start by running a process which essentially overwrites Smugmug with my local structure/photos and does not bring any of the existing mess down from Smugmug. Can I do this 1st step with SyncMug? or do I need to first identify and delete the messy parts from SmugMug?

My thanks in advance...

Tom Kirkpatrick

Originally Posted by Tomkirk23 I am very interested in SyncMug...

I'll try and answer your questions this evening.

Matt

Matt,

I gave SyncMug a test last night and today but I'm having some fairly major problems that (I think) I understand.

How does SyncMug cope with punctuation in gallery names?

I've tried several times (with some renaming tests) and it appears that at the very least its dying on the first image of galleries with ' (apostrophe) or . (period) in the name.

Does that make sense?

Another thing, if you're taking feature requests, it would be fantastic if partial sync'ing could be implemented. I've got 4000+ images (more tha 6GB) and it would be nice to be able to do it in sections.

Thanks for a really great utility.

Originally Posted by Tomkirk23 Matt - I am very interested in SyncMug... 1) Is this the problem you are solving with SyncMug?

Yes, that's my exact intent. I want to have the ability to manage the content from any location, and be able to have it all sync. I haven't yet had the time to code the parts where it updates images if they have changed. Right now it just adds new images, or removes deleted images.

Originally Posted by Tomkirk23 2) How does SyncMug determine a match/duplicate? Same filename and folder? or does it compare the dates of files and choose the newer?

It basically just uses the album title as the folder name. This presents a problem if you have multiple albums with the same name however. I need to decide what to do about that, any suggestions? For files I'm storing the file name in an XML catalog, and once I sync up the image, I have the filename on the local drive referenced to the image ID up on SmugMug, and that's how it knows it's synced that image. If one the image ID disapears up on SmugMug, or the file disapears on the hard drive then I know at one point it did exist on both sides and was synced, but now someone has removed one of the copies, so it deletes the one on the other side syncing things up.

Originally Posted by Tomkirk23 3) What happens if I make separate but conflicting changes on SmugMug and the local files? How does SyncMug resolve?

Right now it doesn't identify updates, so if a file changes on either side it doesn't do anything. That is unless you change the name on the local hard drive, then it thinks it's a new file. I am planning on coding it so that it checks for changes and then uploads a change file to SmugMug, or downloads the change to your local drive. I plan on implementing this via the MD5 checksum that is used, because then no matter what if the contents of the file change in any way, it will detect it. I'll also be checking for modified date changes, and the version incrementor up on SmugMug. I know there is a way to upload an update to an image on SmugMug so that it doesn't change the comments or anything else up on the site, just the image. They have a version number to track this.

Originally Posted by Tomkirk23 4) I get the sense that SyncMug doesn't actually delete any photos from either SmugMug or local folders, but instead stores "deleted" files out of the way. A good approach. I see that SyncMug has you specify where to store locally "deleted" files. Where are deleted files stored on the Smugmug side?

What I'm doing right now is if the image is deleted locally it will download an image from SmugMug and dump it into the local deleted folder. If the image is deleted from SmugMug it will move an image from the local folder to the local deleted folder. I did this as a safety net incase accidents happen, or code behaves funny. It needs a bit more work, but I imagine it will catch most instances. It will end up overwritting a file locally if it has the same file name... but other then that I think it works ok.

Originally Posted by Tomkirk23 5) I don't really know much about how captions/keywords/EXIF/IPTC work. I believe some is stored within the actual jpeg and some may be stored in Smugmug itself and associated with the file. Does SyncMug ensure that this information is sync'ed as well as the photo itself? Is all of this info stored into the jpeg? (If it matters, I use Picasa locally, but may move to Elements)

Yeah, I'm not certain how this is stored yet either. It probably wouldn't be too hard to test if it's stored in the file. After a sync move the image to some other place on your hard drive... resync, and SmugMug will do a delete, then add the file back, and SyncMug will upload it as a new file. If that data appears up on SmugMug, then it's stored with the file, otherwise I'll need to store it in the local XML catalog or something.

Originally Posted by Tomkirk23 6) At the moment, my local folders hold what I consider to be the gold production version of my 3-level structure and photos. What I have on Smugmug is largely out of sync with my local folders. I would like to start by running a process which essentially overwrites Smugmug with my local structure/photos and does not bring any of the existing mess down from Smugmug. Can I do this 1st step with SyncMug? or do I need to first identify and delete the messy parts from SmugMug?

I can think of two ways you could do this. You could wipe out everything up on SmugMug first, then delete your local XML catalog file, that way it doesn't try and do a local delete of all your local files seeing everything disapear from the SmugMug site. Then you could sync, and it would upload all your images up to SmugMug.

This may take longer (depending on how much you have up on SmugMug), but it might be simpler. You could sync everything into a clean spot on your local hard drive... then after it's all synced... delete the local files, and sync up, and it will wipe all the files off the SmugMug site... though I think it might download them all to the deleted folder, so you'd escentially be downloading everything twice... but that would put you at a clean slate on both places.

I would like to see an exampel of what you're calling a 3-level structure. Right now SyncMug just creates a list of albums as folders in the local sync folder.

Originally Posted by Tomkirk23 My thanks in advance... Tom Kirkpatrick

You are VERY welcome!

I'm glad to know that the work I put into this project has a use for someone, even if it means I need to put in a bit more work still.

Originally Posted by mbellot How does SyncMug cope with punctuation in gallery names?

Uh... it probably doesn't. I imagine it chokes when it gets strnage characters like puncuation in gallery titles.

I've added the bug to my list, and will try and get this fixed.

Originally Posted by mbellot Another thing, if you're taking feature requests, it would be fantastic if partial sync'ing could be implemented. I've got 4000+ images (more tha 6GB) and it would be nice to be able to do it in sections.

The more and more I think about it, the more I love this idea. It would definately be great for testing purposes on an existing account, and I'm sure would make for a much better way to transition to syncing everything up. GREAT SUGGESTION! I have added it to the top of my feature list.

Originally Posted by mjohnsonperl Uh... it probably doesn't. I imagine it chokes when it gets strnage characters like puncuation in gallery titles.

wow....really ?

Originally Posted by devbobo wow....really ?

Bobo, I think he meant SyncMug.

I've got near 50 galleries with punctuation and other "special" characters and never had a problem.

SyncMug is probably not handling special characters well/properly when converting the gallery name to a local directory name (esp. since different OS'es have different issues).

Simple example is your linked gallery contains a "*" character. I'm not aware of any (popular) OS that allows that as a valid character in a directory name.

Originally Posted by mjohnsonperl The more and more I think about it, the more I love this idea. It would definately be great for testing purposes on an existing account, and I'm sure would make for a much better way to transition to syncing everything up. GREAT SUGGESTION! I have added it to the top of my feature list.

Awesome!

Originally Posted by mbellot Bobo, I think he meant SyncMug. I've got near 50 galleries with punctuation and other "special" characters and never had a problem. SyncMug is probably not handling special characters well/properly when converting the gallery name to a local directory name (esp. since different OS'es have different issues). Simple example is your linked gallery contains a "*" character. I'm not aware of any (popular) OS that allows that as a valid character in a directory name.

ahh, yeah...nvm

Originally Posted by mjohnsonperl It basically just uses the album title as the folder name. This presents a problem if you have multiple albums with the same name however. I need to decide what to do about that, any suggestions? For files I'm storing the file name in an XML catalog, and once I sync up the image, I have the filename on the local drive referenced to the image ID up on SmugMug, and that's how it knows it's synced that image. If one the image ID disapears up on SmugMug, or the file disapears on the hard drive then I know at one point it did exist on both sides and was synced, but now someone has removed one of the copies, so it deletes the one on the other side syncing things up. [snip] I would like to see an exampel of what you're calling a 3-level structure. Right now SyncMug just creates a list of albums as folders in the local sync folder.

For the "3 level" (four actually) structure implement local_storage as the "root", use subdirectories for categories, subcategories and galleries. It won't eliminate the chance of collisions, but it should reduce them by several magnitudes.

A brief example:

\local_storage (root)
\local_storage\Events (category)
\local_storage\Events\Christmas Show (subcategory)
\local_storage\Events\Christmas Show\Act I (gallery)

How's that for directory insanity.

Originally Posted by mjohnsonperl What I'm doing right now is if the image is deleted locally it will download an image from SmugMug and dump it into the local deleted folder. If the image is deleted from SmugMug it will move an image from the local folder to the local deleted folder. I did this as a safety net incase accidents happen, or code behaves funny. It needs a bit more work, but I imagine it will catch most instances. It will end up overwritting a file locally if it has the same file name... but other then that I think it works ok.

Another "nice to have" would either be (optional?) localized "deleted" folders so it was obvious where the file was deleted from (not always the case if its coming out of a gallery on SM). Either that, or add a text file to the deleted directory that maps the files back to where their directory.

Originally Posted by mjohnsonperl It basically just uses the album title as the folder name. This presents a problem if you have multiple albums with the same name however. I need to decide what to do about that, any suggestions? ..... I would like to see an exampel of what you're calling a 3-level structure. Right now SyncMug just creates a list of albums as folders in the local sync folder.

Thanks for the detailed answer....In regards to the 3-level structure and your question about multiple albums with the same name:

SmugMug has a built-in 3-level hierarchical structure (category/subcategory/gallery). I use the Windows folder structure on my local PC to mimic the SmugMug 3-level hierarchical structure.

My Smugmug site is a photo album for my extended family. I have all photos stored in a local folder named "Family Album".

Within "Family Album", I have subfolders for each individual family. (Think of these as SmugMug "categories").

Within Family Album/Jones, I have subfolders for given time periods. (Think of these as SmugMug "sub-categories").

Within Family Album/Jones/1980s, I have folders for given events. (Think of these SmugMug "galleries").

Within Family Album/Jones/1980s/Tom Wedding, I have my wedding pictures.


I think SyncMug should monitor the local folder structure and the SmugMug category/subcategory/gallery structure and ensure they are kept in sync. For instance, if the user moves an entire local subfolder to a different local folder, SyncMug should move the SmugMug subcategory to the corresponding other category.

If the user has folders more than 3-deep (e.g., "Family Album/Jones/1980s/Tom Wedding/Bachelor Party"), you can create a composite gallery name on SmugMug, like "Tom Wedding: Bachelor Party" under Category "Jones" and Sub-category "1980s". I would keep those pictures hidden.

I have plenty of "Miscellaneous" galleries/subfolders, but do not get into trouble because of the hierarchical structure. You should always think of a photo as being in a certain place within the hierarchical structure.


At the end of the day, I am looking for a tool which syncs my local 3-level folder structure and contents with my Smugmug category/subcategory/gallery structure and contents. This is my holy grail.

Am I explaining this well?

Thanks,

Tom Kirkpatrick