SmugMug Slow / Problems / Status

onethumb · November 6, 2006

We posted a message about 3 hours ago, but since it's buried in another thread, it looks like some of you are missing it.

I don't really have time to go into detail, since I'm working on the problem, but here's what we know:

- We're having some trouble serving a percentage of photos in a reasonable amount of time. It looks like roughly 1% (1 in 100), but I don't have hard data to back that up.

- This can cause slow pages, 'photo not found' errors, and other problems.

- The problem is directly related to Amazon. They're having some systems issues, and we rely heavily on them for storage. As a result, we're rethinking just how heavily we want to rely on them, and will continue to always do so, just like we do with all of our other vendors: bandwidth, servers, datacenters, disk drives, printing, etc.

- The problem is more localized to recent uploads as compared to older photos.

We're well aware of the problem and are working on things as fast as we can. No service is perfect, SmugMug and Amazon included, so there's not much more I can do other than apologize and let you know we're doing what we can.

Thanks for your patience!

EDIT: Be sure to read my follow-up post, below, about Amazon.

cwphotos · November 6, 2006

Thank you so much for the update. I love you guys !

ehughes · November 6, 2006

Thanks for the update Don, I really appreciate all the hard work that goes into SmugMug...

onethumb · November 6, 2006

onethumb wrote:

- The problem is directly related to Amazon. They're having some systems issues, and we rely heavily on them for storage. As a result, we're rethinking just how heavily we want to rely on them, and will continue to always do so, just like we do with all of our other vendors: bandwidth, servers, datacenters, disk drives, printing, etc.

It's worth noting a few things about Amazon in this situation.

First, Amazon has saved us from many many outages in the last 6 months. Overall, their storage has been many times more reliable than ours. (Seriously, more than twice as reliable, easily). So while this particular issue is upsetting, please try to keep in mind that they've actually saved you heartache in the past.

Second, this is the first major issue we've had with them. I'm sure we'll have more. No-one is perfect, not our expensive networking hardware or expensive servers or any thing else we buy from anyone else. We've had other Amazon-related outages in the past, but either they were our fault, or we were making a bad assumption about their service, or it was fixed rapidly enough that I would call it a minor outage, rather than major.

Third, they haven't actually had what I'd call an "outage" per-se, just some fairly serious performance degradation. I'm not trying to excuse them, and you'd better believe they understand how upset we are, but when SmugMug has issues like these internally, it often results in an outage rather than degraded performance.

Hope that helps clarify things a bit.

onethumb · November 6, 2006

onethumb wrote:

We posted a message about 3 hours ago, but since it's buried in another thread, it looks like some of you are missing it.

Baldy has posted another follow-up in the other thread that closely matches what I've been saying, but check it out for another take on the situation.

agunther · November 6, 2006

Good to know things are being looked at.
I just created my pro account so pretty much all my galleries are affected.
Amazon is a pioneer in many areas and I am confident that they have the technical know how to deal with such issues.
Maybe we could make this update thread sticky until the problems have been solved?

digitalpins · November 7, 2006

Thanks for the info and letting us know, I was uploading last night and saw the "no photo found"... now I know what was causing that..... Thanks alot for the info

Baldy · November 7, 2006

Update: Onethumb just released some code that ensures we have the display copies of recent uploads immediately. So any images you upload from about 30 minutes ago should be fast and reliable.

The roughly 1 million images we received over the past few days are still a problem until either Amazon stops having problems or we get them firing off our local storage. Both companies are working on it.

Thanks,
Baldy

onethumb · November 7, 2006

Workaround in place
We have just recently (less than an hour ago) released a workaround for this problem.

Currently, the workaround affects roughly 107,000,000 photos, including any you upload from this moment on.

There are roughly 1,000,000 photos (the ones uploaded in the last week) which aren't affected, yet, but I'm working on that now.

Thanks again for your patience!

Mac Write · November 7, 2006

Suggestion, maybe you should have a third redundant with Google. They truly are the masters of data storage.

onethumb · November 7, 2006

Mac Write wrote:

Suggestion, maybe you should have a third redundant with Google. They truly are the masters of data storage.

If they offered it, we surely would investigate.

But my personal opinion is that most of that reputation is press spin, rather than actual fact. Every time they launch a new property, it gets inundated with traffic and almost always goes down.

Google is probably better at it than we are, in general, but I'm not sure anyone has earned the "masters of data storage" crown yet. I'd put Amazon ahead of Google, currently.

Baldy · November 7, 2006

Mac Write wrote:

Suggestion, maybe you should have a third redundant with Google. They truly are the masters of data storage.

We have a ton of respect for them and would love to see them offer storage like Amazon does, but we've had too many performance issues with their industrial products like Adwords and Analytics to trust their reliability.

dogwood · November 7, 2006

onethumb wrote:

We have just recently (less than an hour ago) released a workaround for this problem.

Currently, the workaround affects roughly 107,000,000 photos, including any you upload from this moment on.

There are roughly 1,000,000 photos (the ones uploaded in the last week) which aren't affected, yet, but I'm working on that now.

Thanks again for your patience!

Thank you so much for the updates.

Just curious, should we keep being patient or re-upload recent photos (from the weekend) and delete the old galleries?

marlinspike · November 7, 2006

onethumb wrote:

We have just recently (less than an hour ago) released a workaround for this problem.

Currently, the workaround affects roughly 107,000,000 photos, including any you upload from this moment on.

There are roughly 1,000,000 photos (the ones uploaded in the last week) which aren't affected, yet, but I'm working on that now.

Thanks again for your patience!

I'm guessng the work around is just hosting them from one of your backup servers? If so, your backup server kicks Amazon's butt. I can't remember the last time I saw things this fast.

cjyphoto · November 7, 2006

marlinspike wrote:

I'm guessng the work around is just hosting them from one of your backup servers? If so, your backup server kicks Amazon's butt. I can't remember the last time I saw things this fast.

It isn't going any faster here. In fact it has become worse. Hopefully things get better soon. Meanwhile I think I'll just :slosh and wait for things to get better.

marlinspike · November 7, 2006

cjyphoto wrote:

It isn't going any faster here. In fact it has become worse. Hopefully things get better soon. Meanwhile I think I'll just :slosh and wait for things to get better.

You broke it. Now it's bad again...thanks a lot.

cjyphoto · November 7, 2006

marlinspike wrote:

You broke it. Now it's bad again...thanks a lot.

Hopefully it gets fixed soon or I may just pass out. :snore

marlinspike · November 7, 2006

cjyphoto wrote:

Hopefully it gets fixed soon or I may just pass out. :snore

On the plus side, I have an excuse for not working on my 1200 photo big backlog.

GreatSave9 · November 8, 2006

marlinspike wrote:

On the plus side, I have an excuse for not working on my 1200 photo big backlog.

Ya but on the minus side, if you just shot photos on Sunday and people want them...

Little bump to get a status update perhaps on the servers....

cjyphoto · November 8, 2006

Hopefully I don't break anything but it does seem to be getting better here.

Frode · November 8, 2006

Previously it was apparently only the newly uploaded images that had issues (in my case, all, since I'm on trial). A fix was implemented that ensured all images uploaded from then on would be snappy.

So I deleted two of my three galleries and reuploaded them. It had a pretty noticable effect compared to the final gallery.

Today though, it's just the other way around. The two "fresh" galleries are now slow, while the third one that got uploaded on sunday is nice and fast.

Am I imagining things or has something changed?

Andy · November 8, 2006

Frode wrote:

Previously it was apparently only the newly uploaded images that had issues (in my case, all, since I'm on trial). A fix was implemented that ensured all images uploaded from then on would be snappy.

So I deleted two of my three galleries and reuploaded them. It had a pretty noticable effect compared to the final gallery.

Today though, it's just the other way around. The two "fresh" galleries are now slow, while the third one that got uploaded on sunday is nice and fast.

Am I imagining things or has something changed?

Can we have your smug site so we can look?

NikonGirl · November 8, 2006

I just uploaded one image - and 5 minutes later it is still in my queue. I thought the problem was fixed??

Andy · November 8, 2006

NikonGirl wrote:

I just uploaded one image - and 5 minutes later it is still in my queue. I thought the problem was fixed??

We've alerted onethumb, please standby.

aktse · November 8, 2006

I'm still having issues with everything that I uploaded on Sunday and haven't uploaded anything since then.

http://aktse.smugmug.com/gallery/2099197
http://aktse.smugmug.com/gallery/2099196
http://aktse.smugmug.com/gallery/2099198

It really does not like all thumbs, and I haven't sent the link to anyone because I've been having issues. It took me two days to get things squared and I acutally still haven't looked at any of these gallerys on-line; it's just too painful. I know that I need to rotate some photos, but I can't even get the thumbnails to show. I don't believe it's my connection because i have tried this on three computers (different locations, different networks, linux and windows). The only thing that I havne't tired is a Mac...

Maybe you'll have better luck.
I really love smugmug, and I was about to buy a the gift of smugmug for some new parents, but I'm now holding off. They are not computer people; they don't understand things like line test, messageboards, posting if there are problems, etc. If I would have bought them this on Sunday, they would have tried it once or twice, and would have just given up and stop using it.

Frode · November 8, 2006

Andy wrote:

Can we have your smug site so we can look?

It's frode.smugmug.com. The 2004 one is the speedy one, 2002 and 2003 are being lazy. All of them are passworded. If you need the password drop me a line and I'll pm it.

Edit: I tested again after reading this and speed is now up to normal for all three galleries. Whether coincidence or not I don't know, but for now things are looking good.

GreatSave9 · November 9, 2006

aktse wrote:

I'm still having issues with everything that I uploaded on Sunday and haven't uploaded anything since then.

http://aktse.smugmug.com/gallery/2099197
http://aktse.smugmug.com/gallery/2099196
http://aktse.smugmug.com/gallery/2099198

It really does not like all thumbs, and I haven't sent the link to anyone because I've been having issues. It took me two days to get things squared and I acutally still haven't looked at any of these gallerys on-line; it's just too painful. I know that I need to rotate some photos, but I can't even get the thumbnails to show. I don't believe it's my connection because i have tried this on three computers (different locations, different networks, linux and windows). The only thing that I havne't tired is a Mac...

Maybe you'll have better luck.
I really love smugmug, and I was about to buy a the gift of smugmug for some new parents, but I'm now holding off. They are not computer people; they don't understand things like line test, messageboards, posting if there are problems, etc. If I would have bought them this on Sunday, they would have tried it once or twice, and would have just given up and stop using it.

Same here, Im still having problems on the photos I uploaded on Sunday. They sometimes load, sometimes dont load, but 90% of the time dont load.
Especially in my AutoX galleries
http://www.greatsave9.smugmug.com

Andy · November 9, 2006

GreatSave9 wrote:

Same here, Im still having problems on the photos I uploaded on Sunday. They sometimes load, sometimes dont load, but 90% of the time dont load.
Especially in my AutoX galleries
http://www.greatsave9.smugmug.com

Don is hopefully going to be able to fix those. In the meantime, if it's not too many photos, you can always reupload. Sorry for the hassle!

GreatSave9 · November 9, 2006

Andy wrote:

Don is hopefully going to be able to fix those. In the meantime, if it's not too many photos, you can always reupload. Sorry for the hassle!

Ya thats what I was going to ask. just Sundays photos got messed up and dont load all the time. I can always reupload them, no problem.

So if I upload tonight, those galleries will be just fine... or should be in theory.
I assume just Sundays stuff got mixed up kinda bad. No problems. Happy bout the speed improvements

SmugMug Slow / Problems / Status

Comments