• Gear
  • Shots
  • Photo Craft
  • Video
  • Wide Angle
  • Support
  • New Stuff
  • More
Support SmugMug Support Bug Reporting Smugmug uploader makes a big mess (170 duplicates)

Need some help with your New SmugMug Site?

Dgrin Challenges

They're back! Our latest series of Dgrin Challenges, the OOYCZ Challenges (Out of Your Comfort Zone), is a series of challenges to help you improve your photography by stepping out of your comfort zone and trying something new. Our challenge leader says "Consider it like a photography class with no diplomas.". Get started with OOYCZ by reading the Challenge Guide.

Past DSS Challenge Winners, DSS Challenge Rules, and other important DSS Challenge information is here.

Looking for a little challenge? The Dgrin Mini-Challenges are a great way to challenge yourself. Take a moment to look through past winners and find the current Mini Challenge here.

From Around the Net

Enjoy a few of our favorite articles from around the 'net. If you have something you think we should see, post it here and we'll have a look.

Journeys

Landscapes

Sports

 
Thread Tools Display Modes
Page 2  of  2
1 2
Old May-21-2012, 06:20 AM
#21
SamirD is offline SamirD
Huntsville Car Scene.com
SamirD's Avatar
Quote:
Originally Posted by jfriend View Post
True dups will have the same filename, but if you're shooting with more than one body or combining photos from more than one photographer and don't rename your files and you're not making sure they each have unique file numbers/names, then it's possible to have non-dups with the same filename. That was my point. I rename all camera generated filenames to have a timestamp in the filename so avoid this ever happening in my workflow. A generic tool meant for anyone to use it with could not make such assumptions about the workflow so it would have to be more careful about what was really a dup. That's one of the differences in the amount of work required developing robust software for anyone to use vs. developing software for one particular isolated use.
That's why in my implementation, I would show the thumbnail of each original duplicate in case they were actually different files. The final decision would be made by the user as to what gets deleted.

But you're absolutely right about development of tools like this. Lots of base cases when you expand the use to many users compared to individual use.
Quote:
Originally Posted by Nikolai View Post
I know it's probably too late and not very interesting for this thread primary posters, but I just deployed a new version of S*E that has image duplicates removal functionality, available to S*E Pro users via Albums4 Pro package, and to Pro+/Studio users as is.

It takes into account image file name, file size, image dimensions and original date, if available.

You can activate it from the context menu of both albums list and Category/Subcategory tree (so you can check the entire Category for dups:-)

FWIW, it was this thread that brought this problem to my attention, so hopefully somebody else will not suffer through what John and Samir did.

HTH
Nikolai
Thank you very much for the update Nikolai! It's been a while since I've visited SE, and now that I've got some newer computers, I'll have to take a look at it again.
__________________
Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
Want faster uploading? Vote for FTP!
Old Aug-13-2012, 09:06 AM
#22
f300 is offline f300
Beginner grinner
Gosh, it's killing me, this "Skip Duplicates" (mis)feature just doesn't work reliably. I have a gallery on the largish side (1600 pics or so) and first of all no matter what I do to it once I select and drop all the pictures it says it ignores 1000 of them and uploads the others. Fine. I try to upload them in 100 batches. Some work (as in you upload and then when you try to upload them again they're skipped). Some don't. You can drop them again and again and some 60-70 pics will reupload.
This is really getting way beyond annoying.
Old Aug-13-2012, 09:21 AM
#23
sujit1779 is offline sujit1779
Big grins
Quote:
Originally Posted by f300 View Post
Gosh, it's killing me, this "Skip Duplicates" (mis)feature just doesn't work reliably. I have a gallery on the largish side (1600 pics or so) and first of all no matter what I do to it once I select and drop all the pictures it says it ignores 1000 of them and uploads the others. Fine. I try to upload them in 100 batches. Some work (as in you upload and then when you try to upload them again they're skipped). Some don't. You can drop them again and again and some 60-70 pics will reupload.
This is really getting way beyond annoying.
Hey I don't know if it ok to write here. But we have a product PicBackMan which helps you to upload thousands of pictures without duplicates. You can download it at www.picbackman.com

Sujit
Disclosure : I am involved with PicBackMan as a developer.
Old Aug-13-2012, 09:49 PM
#24
Nikolai is offline Nikolai
Darth SLR
Nikolai's Avatar
Quote:
Originally Posted by f300 View Post
Gosh, it's killing me, this "Skip Duplicates" (mis)feature just doesn't work reliably. I have a gallery on the largish side (1600 pics or so) and first of all no matter what I do to it once I select and drop all the pictures it says it ignores 1000 of them and uploads the others. Fine. I try to upload them in 100 batches. Some work (as in you upload and then when you try to upload them again they're skipped). Some don't. You can drop them again and again and some 60-70 pics will reupload.
This is really getting way beyond annoying.
What uploader are you using?
__________________
"May the f/stop be with you!"
Star*Explorer: on Dgrin, home; Master Class: open;
Class is in session, My Facebook, @DarthSLR, #NiksTips
member: NAPP, PPA, partner: Adobe
Comprehending life, universe and everything - one pixel at a time
Old Aug-14-2012, 01:59 AM
#25
f300 is offline f300
Beginner grinner
I'm using the "default web-based one" which I think it's the HTML5. Tried firefox, chrome, also to change computers. And it's not like it happened once because "I was too fast", this is ongoing for more than one week already. I'm not using much Smugmug (I think these are the first uploads I try to do this year) and usually I used just to nuke the gallery and reupload but now it is just a big gallery and I'm not sure I can reupload "in one go". Also I don't fancy to split it in 5-10 batches and then upload and check each batch.

If only 1000 pics are read from the gallery and checked against the uploaded pics is fine, but I need to know it.

Are the other uploaders checking for duplicates (I guess I'll just try for myself anyway)?
Old Aug-14-2012, 02:52 AM
#26
rainforest1155 is online now rainforest1155
with a Heroes touch
rainforest1155's Avatar
Can you try the Simple uploader instead of the default html5 uploader to see if that works better in your case? The Simple uploader also has duplicate detection.

Note that the duplicate detection of our web uploaders only works for photos already in the gallery when the uploader is opened up. So to have it consider photos you just uploaded, go to the gallery and open the uploader again.
It won't take any photos into account that you just uploaded with the same or any other window.
__________________
Sebastian
SmugMug Support Hero

look on the bright side
http://www.SebastianHosche.com
Old Aug-14-2012, 08:55 AM
#27
Nikolai is offline Nikolai
Darth SLR
Nikolai's Avatar
Quote:
Originally Posted by f300 View Post
I'm using the "default web-based one" which I think it's the HTML5. Tried firefox, chrome, also to change computers. And it's not like it happened once because "I was too fast", this is ongoing for more than one week already. I'm not using much Smugmug (I think these are the first uploads I try to do this year) and usually I used just to nuke the gallery and reupload but now it is just a big gallery and I'm not sure I can reupload "in one go". Also I don't fancy to split it in 5-10 batches and then upload and check each batch.

If only 1000 pics are read from the gallery and checked against the uploaded pics is fine, but I need to know it.

Are the other uploaders checking for duplicates (I guess I'll just try for myself anyway)?
My Star*Explorer (http://www.starexplorer.com) does that and then some... And you don't have to break it down in batches by 100... :-) It has free 30 day trial, so as long as you're on Windows, you may try it and see it fit meets your needs...
__________________
"May the f/stop be with you!"
Star*Explorer: on Dgrin, home; Master Class: open;
Class is in session, My Facebook, @DarthSLR, #NiksTips
member: NAPP, PPA, partner: Adobe
Comprehending life, universe and everything - one pixel at a time
Old Aug-14-2012, 10:49 AM
#28
f300 is offline f300
Beginner grinner
Yea, thanks, actually I even went and tried your SE about one week ago but it said trial expired (it wasn't currently installed on my box for sure but I think I tried it once years ago, sorry I don't remember why I wasn't impressed...). FYI maybe if you had automatic order processing I might have bought it on the spot, just to forget about this duplicate mess :-)
But really for my limited use it is too much, in more ways than one.

I'm testing now the "simple" (Java) uploader, it is much slower (more than one second per duplicate skipped, on a quad no less...) but that isn't particularly bad. What's worse is that first time it crashed, then I checked my Java version and it seems it was behind (and it couldn't update automatically). So I updated Java and now it seems to be even slower (and it crashed just when starting first time).

Not giving up yet, if I find one way that works it's fine (I really don't want much, just basic functionality) but if not next year I'll have to move on from smugmug (oh, and get something with real nested categories support, oh, that would be a breath of fresh air).
Old Aug-15-2012, 12:32 AM
#29
rainforest1155 is online now rainforest1155
with a Heroes touch
rainforest1155's Avatar
What browser and operating system do you use the Simple uploader on? I haven't used it extensively in a while, but in the past I used it a lot and never had problems with it crashing on Firefox.
__________________
Sebastian
SmugMug Support Hero

look on the bright side
http://www.SebastianHosche.com
Old Aug-15-2012, 04:23 AM
#30
f300 is offline f300
Beginner grinner
XP and the latest stable firefox.
I wouldn't worry about this part (crashing) yet.
Last try yesterday it worked for hours (I said it takes more than one second to skip one duplicate - it's in fact way more than 1s). It did manage to skip everything fine (at least at first sight) yesterday but then I didn't have time to let it upload completely. Hopefully it will run through today and anyway even if it does I plan to run it once more to see if it skips everything as expected. I'll report back in any case.

Can somebody with a largish (1500+ pics) gallery and access to the original folder make a test in the meantime? Just drag+drop all files in the HTML5 uploader and see if it says (for example) 1600 duplicates or it starts again uploading something like 600 pics (if it starts uploading you can cancel it fast enough before it uploads anything, just keep an eye on it and kill it so you don't end up with duplicates as well).
Old Aug-24-2012, 07:44 AM
#31
SamirD is offline SamirD
Huntsville Car Scene.com
SamirD's Avatar
Quote:
Originally Posted by f300 View Post
Gosh, it's killing me, this "Skip Duplicates" (mis)feature just doesn't work reliably. I have a gallery on the largish side (1600 pics or so) and first of all no matter what I do to it once I select and drop all the pictures it says it ignores 1000 of them and uploads the others. Fine. I try to upload them in 100 batches. Some work (as in you upload and then when you try to upload them again they're skipped). Some don't. You can drop them again and again and some 60-70 pics will reupload.
This is really getting way beyond annoying.
I used to run into this problem a lot and developed a small 'duplicate detector' that I can share with you if you want to try it on your gallery, just PM me. It will help find the duplicates and give you a link to them so you can delete them.
Quote:
Originally Posted by f300 View Post
Can somebody with a largish (1500+ pics) gallery and access to the original folder make a test in the meantime? Just drag+drop all files in the HTML5 uploader and see if it says (for example) 1600 duplicates or it starts again uploading something like 600 pics (if it starts uploading you can cancel it fast enough before it uploads anything, just keep an eye on it and kill it so you don't end up with duplicates as well).
I regularly upload large batches like this and use simple uploader without any issues. I have 5Mb upload bandwidth, so that may make a difference as I don't know how SM uploaders work on less bandwidth. And skipping can take forever especially on large galleries. Videos make that even worse.

I always have to check the number of files uploaded though as the uploader can miss some or upload dupes. Finding a single dupe in a batch of 2000 is a real pain, so I developed a tool that can help me find it in a second. I used to use a technique that used the uploader, but that just takes waaaay too long on large galleries.
__________________
Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
Want faster uploading? Vote for FTP!
Old Aug-24-2012, 07:49 AM
#32
SamirD is offline SamirD
Huntsville Car Scene.com
SamirD's Avatar
Quote:
Originally Posted by sujit1779 View Post
Hey I don't know if it ok to write here. But we have a product PicBackMan which helps you to upload thousands of pictures without duplicates. You can download it at www.picbackman.com

Sujit
Disclosure : I am involved with PicBackMan as a developer.
Interesting software! Thank you for sharing.
__________________
Pictures and Videos of the Huntsville Car Scene: www.huntsvillecarscene.com
Want faster uploading? Vote for FTP!
Old Aug-24-2012, 09:37 AM
#33
sujit1779 is offline sujit1779
Big grins
Quote:
Originally Posted by SamirD View Post
Interesting software! Thank you for sharing.
Thanks Samir :-)
Old Dec-11-2012, 08:49 AM
#34
RolandP is offline RolandP
Big grins
I'm adding to this post even though it's been silent for a while. I recently shot a job in both JPG and RAW, uploaded the JPG files so the clients could see them sooner. My intention was to replace those JPG files in the gallery with ones created from the RAW files later. The uploader didn't recognize the new files, with identical names, as duplicates and proceeded to add 500 new files. Not good! Turns out that the original files were named *.JPG and the new files were *.jpg and the uploader is case sensitive. At least that is what I've been told by a support person.

Hopefully this helps somebody out there. I know I could have used that information last night while clients were looking at images and surely adding things to their shopping cart.
__________________
Ron

Chico Photographer - Ron Putnam | Facebook
Old Dec-12-2012, 06:38 AM
#35
Nikolai is offline Nikolai
Darth SLR
Nikolai's Avatar
Quote:
Originally Posted by RolandP View Post
I'm adding to this post even though it's been silent for a while. I recently shot a job in both JPG and RAW, uploaded the JPG files so the clients could see them sooner. My intention was to replace those JPG files in the gallery with ones created from the RAW files later. The uploader didn't recognize the new files, with identical names, as duplicates and proceeded to add 500 new files. Not good! Turns out that the original files were named *.JPG and the new files were *.jpg and the uploader is case sensitive. At least that is what I've been told by a support person.

Hopefully this helps somebody out there. I know I could have used that information last night while clients were looking at images and surely adding things to their shopping cart.
Get Star*Explorer trial and remove the duplicates.
__________________
"May the f/stop be with you!"
Star*Explorer: on Dgrin, home; Master Class: open;
Class is in session, My Facebook, @DarthSLR, #NiksTips
member: NAPP, PPA, partner: Adobe
Comprehending life, universe and everything - one pixel at a time
Old Jan-20-2013, 04:00 PM
#36
racer is offline racer
Major grins
Quote:
Originally Posted by jfriend View Post
The @#$%^& Smugmug uploader made a big mess today.
Your problem seems to be caused by a outdated version of the jfriend slideshow.

(sorry :P)
__________________
Todd - My Photos
Old Jan-20-2013, 04:07 PM
#37
jfriend is offline jfriend OP
Scripting dude-volunteer
Quote:
Originally Posted by racer View Post
Your problem seems to be caused by a outdated version of the jfriend slideshow.

(sorry :P)
Huh?
__________________
--John
Homepage Popular
JFriend's javascript customizations Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
Old Jan-20-2013, 05:12 PM
#38
racer is offline racer
Major grins
Quote:
Originally Posted by jfriend View Post
Huh?
It was a joke! (in reference to threads like the login issues thread)
__________________
Todd - My Photos
Page 2  of  2
1 2
Tell The World!  
Similar Threads Thread Starter Forum Replies Last Post
Announcing a New SmugMug Uploader: smupload dinsdale23432 SmugMug APIs, Hacks & Tricks 1 Aug-01-2008 04:52 AM
SnugUp--Alternate Windows uploader to SmugMug Ham1 SmugMug APIs, Hacks & Tricks 0 Mar-20-2008 04:56 PM
Can't upload using SmugMug Mac Uploader or drag and drop noeltykay SmugMug Support 7 Feb-19-2008 09:19 AM
Need SmugMug theme that doesn't mess with my colors Tomkirk23 Legacy SmugMug Customization 3 Feb-09-2008 07:21 PM
Smugmug uploader Andie SmugMug Support 3 Sep-11-2006 03:32 AM


Thread Tools
Display Modes

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump