I also have been seeing very slow, bursty transfers of thumbnails and pictures for several days. I consistently see it both from my home computer (Comcast cable modem access) and my office (multiple T1's).
Additionally, today I have been seeing odd behavior where a very few photos -- correctly uploaded and viewable for the past week -- are suddenly replaced with a "photo not found" thumb. Click on the thumb, get an empty black image (but you can get the correct EXIF data). Come back an hour later, the thumb is still "photo not found" but if you click on it anyway the photo appears in the main window. Very odd; something appears to be actively going on with the photo database. I don't know whether or not this is related to the main sluggishness problem everyone is reporting.
Same Experience
I've been thinking it was my browser (IE 7) until i looked at the site using Firefox. With both browsers, the experience was horribly slow.
Just for claification: I'm using a 20mb pipe at my corporate office (I'm the VP of IT). Even bright and early in the morning when i was the only one around it was slow.
at 1050am pst, i ran a tracert.
Microsoft Windows [Version 6.0.5744]
Copyright (c) 2006 Microsoft Corporation. All rights reserved.
C:\Windows\system32>tracert www.froemke.com
Tracing route to froemke.smugmug.com [63.81.134.23]
over a maximum of 30 hops:
1 120 ms 10 ms 5 ms 172.16.1.71
2 <1 ms <1 ms <1 ms 74.8.94.254
3 13 ms 14 ms 13 ms ag-lieberma-serial1-12-1.anaca01.paetec.net [74.8.92.217]
4 15 ms 15 ms 15 ms 66.251.35.34
5 15 ms 15 ms 15 ms bb-laxca01-pos.laxca.paetec.net [64.80.253.21]
6 15 ms 15 ms 15 ms 66.251.30.4
7 15 ms 15 ms 15 ms ge-5-2-0.406.ar3.LAX1.gblx.net [207.138.128.165]
8 25 ms 30 ms 35 ms pri.r1-ge0-2-eq-sj.smugmug.com [206.223.117.70]
9 26 ms 25 ms 25 ms hera.smugmug.com [63.81.134.23]
Trace complete.
C:\Windows\system32>
I'm seeing speed problems also. A client just called and complained that a gallery with 195 photos in it, originals all < 500k, is slow to display. Browsing around myself at work (I work for a telecom giant and have extremely fast access here) I find that my older galleries all seem normal, but the new gallery I uploaded last night is slow to load.
I find that my older galleries all seem normal, but the new gallery I uploaded last night is slow to load.
Interesting...
I just did some browsing tests in my galleries and I see the same thing. Recent galleries (added within the past couple of weeks) are slow in showing either the thumbnails or medium size image when I select a thumbnail. Older galleries display the images right away... and I cleared my cache before this test...
Doesn't Smugmug use Amazon S3 to speed display of images? Maybe Amazon isn't doing something (distributing) the images since the last Smugmug maintenance?
It is taking forever for my galleries to come up. I am trying to rotate some photos in some galleries and it is literally taking 10 minutes for all the images to come up, and normally take about 30 seconds.
I have cleared cache, cookies, restarted computer, tried IE, firefox, etc.
My up speed is over 300kbs, down was about 3megs (usually 5).
I have a few more thousand images I am uploading that I still have to go rotate some. This has been going on since last night. Come on smugmug!!
"A photo is like a hamburger. You can get one from McDonalds for $1, one from Chili's for $5, or one from Ruth's Chris for $15. You usually get what you pay for, but don't expect a Ruth's Chris burger at a McDonalds price, if you want that, go cook it yourself." - me
I have 500 images in this one gallery and went to rotate the photos, one coming up then 3 seconds later the next, then 3 seconds later another, etc.
What is going on?????
"A photo is like a hamburger. You can get one from McDonalds for $1, one from Chili's for $5, or one from Ruth's Chris for $15. You usually get what you pay for, but don't expect a Ruth's Chris burger at a McDonalds price, if you want that, go cook it yourself." - me
Yup Im experiencing SLOOOOOOOOOOOOOOOOOOOW load times. Thats "IF" they load! I assume a server upgrade/crash/maintence? Hopefully its just short term. www.GreatSave9.Smugmug.com
Last week it was normal and fine. But Im getting complaints from people bout the new load times for an AutoX i just posted.
OMG I just found out that slow loading cost me freakin money!!!!!!!!! I thought I had set all my new galleries for Pro Pricing, however; I didnt get my HEAT 1 photos priced because the webpage timed out and I didnt notice...... so these guys found the photos and bought them (BEFORE i posted at another forum) and got the photos without me getting profits. SOOOOO PISSED ! FIX THE LOADING TIMES!!!
Still slow here also. Noticed my hit count on Statcounter has dropped like a rock in the last few days as well. I imagine people viewing my and others sites are giving up after attempting to view a gallery or two.
My Pictures : My Gear
I Reject Your Reality And Substitute My Own - Adam Savage
0
Options
BaldyRegistered Users, Super ModeratorsPosts: 2,853moderator
edited November 6, 2006
Hi All,
We're really sorry about the performance issues you're seeing on some galleries. The issue is Amazon is having performance issues so 1 of 100 images we're reading from them is coming back very slowly. Unfortunately, the performance issues are concentrated on recent images.
I could go into some detail about why just recent images are causing problems, but we're trying to focus on a solution. Amazon is telling us they don't have a quick fix so we're working on one of our own.
Oh well, things happen. Good to know that someone is on it. I am confident you will be able to resolve the issues with Amazon. It explains why I had some trouble reaching Amazon this weekend .
Sadly I just bought my pro account and uploaded a whole bunch of pictures, so effectively all my stuff is affected.
We're really sorry about the performance issues you're seeing on some galleries. The issue is Amazon is having performance issues so 1 of 100 images we're reading from them is coming back very slowly. Unfortunately, the performance issues are concentrated on recent images.
I totally admit to not fully (or even partially) understanding your data architecture, but when Don said this
Your priceless photos are stored in multiple datacenters, in multiple states, and at multiple companies. They're orders of magnitude more safe and secure.
in his blog, it lead me to believe that there were multiple copies of each photo stored on different servers. Specifically, I thought one copy was stored on Smugmug's servers and another on Amazon's. Therefore, if one system went down, the photo could be served up from the other. If I interpreted that correctly, then if Amazon is slow, why can't the images be served up quickly from Smugmug's servers?
I realize your priority right now is solving the problem, not explaining it. But, when everything's back up and running I (and I'm sure others) would be interested in hearing more.
I totally admit to not fully (or even partially) understanding your data architecture, but when Don said this
in his blog, it lead me to believe that there were multiple copies of each photo stored on different servers. Specifically, I thought one copy was stored on Smugmug's servers and another on Amazon's. Therefore, if one system went down, the photo could be served up from the other. If I interpreted that correctly, then if Amazon is slow, why can't the images be served up quickly from Smugmug's servers?
I realize your priority right now is solving the problem, not explaining it. But, when everything's back up and running I (and I'm sure others) would be interested in hearing more.
Thanks,
These are good questions. I'm curious about this as well. I too noticed a big drop in web hits since last week-- I've been getting 300-400 hits daily and over the weekend it dropped massively-- as in 50 unique hits a day. My guess is that surfers are leaving due to the slow loading too.
I know (well, least I'm crossing my fingers) that SM is all over this. The good news, I guess, is that SM has an interest in traffic coming to our sites since that leads to sales-- so hopefully everything will get figured out. Now where's that crossed fingers smilie?
BaldyRegistered Users, Super ModeratorsPosts: 2,853moderator
edited November 6, 2006
Amazon just updated us and predicted big speed improvements by tonight. They also said that their storage services are growing much faster than they anticipated so they're busy installing major infrastructure over, ahem, the next three weeks. I think what they're saying is we won't see the great performance that we were seeing for another 3 weeks... It could mean, however, that for all intents and purposes speed will be restored tonight and scaling to prevent a future occurence will be completed in 3 weeks.
Out of fairness to them, they've had the most reliable systems of any of our partners over the 7 or so months we've been using them. They've saved us many a slowdown or outage when we've been able to fail over to them.
And that, ironically, is what's causing the problems today. New images are the toughest for us to handle because they get hit so hard immediately after upload. They get viewed more frequently, posted to forums, rotated, cropped, color adjusted, watermarked, etc.
Since their storage has been so high performance and reliable, we formed a higher dependency on them for new images, which has helped us handle the huge bursts of high-pixel, low-compression images we get from event photographers with fast connections.
In other words, in the beginning, Amazon was just a failover. Later, they became the primary for new images and the failover for old ones. While we've failed over to them many times, I don't think we've had to fail over from them (which means handling images with our storage that are on fire).
I know it's a red alert at Amazon and here too. Don, Andrew and Craig are at the datacenter but I don't have an eta for a fix, unfortunately.
I totally admit to not fully (or even partially) understanding your data architecture, but when Don said this
in his blog, it lead me to believe that there were multiple copies of each photo stored on different servers. Specifically, I thought one copy was stored on Smugmug's servers and another on Amazon's. Therefore, if one system went down, the photo could be served up from the other. If I interpreted that correctly, then if Amazon is slow, why can't the images be served up quickly from Smugmug's servers?
I realize your priority right now is solving the problem, not explaining it. But, when everything's back up and running I (and I'm sure others) would be interested in hearing more.
Specifically, I thought one copy was stored on Smugmug's servers and another on Amazon's. Therefore, if one system went down, the photo could be served up from the other. If I interpreted that correctly, then if Amazon is slow, why can't the images be served up quickly from Smugmug's servers?
My guess would be that they have different agreements in place for accessibility of the images. With Amazon, the images are available for near real time viewing (when all is working well.) With the services that are storing the same photos in other places, the access to the images may take longer -- I read his statement as focusing on the safety of the images via redundant backup that can be restored if one set of servers (or one provider) fails, not that the same image is available for instant viewing from another server in order to support a network slow down by one provider over another. I'm sure their tech guys can clarify this, if they want (and if it isn't giving away any confidential info.)
I notice I can order a backup CD or DVD of my galleries from you guys. I don't care about photos themselves as I back them up myself. However, I don't want to lose all comments, descriptions, and all other text data. Does you backup service include such kind of data?
Amazon just updated us and predicted big speed improvements by tonight. They also said that their storage services are growing much faster than they anticipated so they're busy installing major infrastructure over, ahem, the next three weeks. I think what they're saying is we won't see the great performance that we were seeing for another 3 weeks... It could mean, however, that for all intents and purposes speed will be restored tonight and scaling to prevent a future occurence will be completed in 3 weeks.
Out of fairness to them, they've had the most reliable systems of any of our partners over the 7 or so months we've been using them. They've saved us many a slowdown or outage when we've been able to fail over to them.
And that, ironically, is what's causing the problems today. New images are the toughest for us to handle because they get hit so hard immediately after upload. They get viewed more frequently, posted to forums, rotated, cropped, color adjusted, watermarked, etc.
Since their storage has been so high performance and reliable, we formed a higher dependency on them for new images, which has helped us handle the huge bursts of high-pixel, low-compression images we get from event photographers with fast connections.
In other words, in the beginning, Amazon was just a failover. Later, they became the primary for new images and the failover for old ones. While we've failed over to them many times, I don't think we've had to failover from them (which means handling images that are on fire with our storage).
I know it's a red alert at Amazon and here too. Don, Andrew and Craig are at the datacenter but I don't have an eta for a fix, unfortunately.
I notice I can order a backup CD or DVD of my galleries from you guys. I don't care about photos themselves as I back them up myself. However, I don't want to lose all comments, descriptions, and all other text data. Does you backup service include such kind of data?
Thanks.
I'm sorry, but we currently don't offer that. The backup CD or DVD contains folders for each category/subcategory/gallery and the photos themselves, but no captions, comments, etc.
Great question! I hope Amazon S3 is not the only data center.
Actually they spread your photos across their own datacenters, which makes it great for us because we can pull from their East Coast location or Pacific Northwest. Datacenters close to customers are great for lots of speed and reliability reasons.
We have a failover fix in the works for the new images that we're testing right now on our test servers.
My guess would be that they have different agreements in place for accessibility of the images. With Amazon, the images are available for near real time viewing (when all is working well.) With the services that are storing the same photos in other places, the access to the images may take longer -- I read his statement as focusing on the safety of the images via redundant backup that can be restored if one set of servers (or one provider) fails, not that the same image is available for instant viewing from another server in order to support a network slow down by one provider over another. I'm sure their tech guys can clarify this, if they want (and if it isn't giving away any confidential info.)
Actually, the way it worked until recently was we could fetch images from our datacenter or one of two Amazon data centers whenever we want. Perceived performance from any of the three was very close.
The "until recently" proviso is that new images could only be fetched from the two Amazon data centers lately due to the performance requirements of the storage.
The changes we're testing now bring us back to triple redundancy even for new images. In practice, it will probably shake out that all three datacenters are delivering images most of the time, but any one could handle the load if the other two fail.
Actually, the way it worked until recently was we could fetch images from our datacenter or one of two Amazon data centers whenever we want. Perceived performance from any of the three was very close.
The "until recently" proviso is that new images could only be fetched from the two Amazon data centers lately due to the performance requirements of the storage.
The changes we're testing now bring us back to triple redundancy even for new images. In practice, it will probably shake out that all three datacenters are delivering images most of the time, but any one could handle the load if the other two fail.
Nice to know and makes my annual fee seem pretty cheap! Hopefully we'll see it soon and see a performance boost.
Comments
Additionally, today I have been seeing odd behavior where a very few photos -- correctly uploaded and viewable for the past week -- are suddenly replaced with a "photo not found" thumb. Click on the thumb, get an empty black image (but you can get the correct EXIF data). Come back an hour later, the thumb is still "photo not found" but if you click on it anyway the photo appears in the main window. Very odd; something appears to be actively going on with the photo database. I don't know whether or not this is related to the main sluggishness problem everyone is reporting.
I've been thinking it was my browser (IE 7) until i looked at the site using Firefox. With both browsers, the experience was horribly slow.
Just for claification: I'm using a 20mb pipe at my corporate office (I'm the VP of IT). Even bright and early in the morning when i was the only one around it was slow.
at 1050am pst, i ran a tracert.
Microsoft Windows [Version 6.0.5744]
Copyright (c) 2006 Microsoft Corporation. All rights reserved.
C:\Windows\system32>tracert www.froemke.com
Tracing route to froemke.smugmug.com [63.81.134.23]
over a maximum of 30 hops:
1 120 ms 10 ms 5 ms 172.16.1.71
2 <1 ms <1 ms <1 ms 74.8.94.254
3 13 ms 14 ms 13 ms ag-lieberma-serial1-12-1.anaca01.paetec.net [74.8.92.217]
4 15 ms 15 ms 15 ms 66.251.35.34
5 15 ms 15 ms 15 ms bb-laxca01-pos.laxca.paetec.net [64.80.253.21]
6 15 ms 15 ms 15 ms 66.251.30.4
7 15 ms 15 ms 15 ms ge-5-2-0.406.ar3.LAX1.gblx.net [207.138.128.165]
8 25 ms 30 ms 35 ms pri.r1-ge0-2-eq-sj.smugmug.com [206.223.117.70]
9 26 ms 25 ms 25 ms hera.smugmug.com [63.81.134.23]
Trace complete.
C:\Windows\system32>
Canon 5D Mk.2/Grip || Canon 7D Backup
17-40 f/4L || 70-200 f/2.8L IS || 100mm f/2.8L Macro || 24-70mm f/2.8L
Wedding Photographer
www.cwphotos.net
Hopefully this can be fixed soon.
Interesting...
I just did some browsing tests in my galleries and I see the same thing. Recent galleries (added within the past couple of weeks) are slow in showing either the thumbnails or medium size image when I select a thumbnail. Older galleries display the images right away... and I cleared my cache before this test...
Doesn't Smugmug use Amazon S3 to speed display of images? Maybe Amazon isn't doing something (distributing) the images since the last Smugmug maintenance?
A couple examples:
Recent gallery, slow display:
http://gallery.primarycolors.com/missionsjc
Older gallery, fast display:
http://gallery.primarycolors.com/wavecrest2006
I have cleared cache, cookies, restarted computer, tried IE, firefox, etc.
My up speed is over 300kbs, down was about 3megs (usually 5).
I have a few more thousand images I am uploading that I still have to go rotate some. This has been going on since last night. Come on smugmug!!
What is going on?????
www.GreatSave9.Smugmug.com
Last week it was normal and fine. But Im getting complaints from people bout the new load times for an AutoX i just posted.
OMG I just found out that slow loading cost me freakin money!!!!!!!!! I thought I had set all my new galleries for Pro Pricing, however; I didnt get my HEAT 1 photos priced because the webpage timed out and I didnt notice...... so these guys found the photos and bought them (BEFORE i posted at another forum) and got the photos without me getting profits. SOOOOO PISSED ! FIX THE LOADING TIMES!!!
I Reject Your Reality And Substitute My Own - Adam Savage
We're really sorry about the performance issues you're seeing on some galleries. The issue is Amazon is having performance issues so 1 of 100 images we're reading from them is coming back very slowly. Unfortunately, the performance issues are concentrated on recent images.
I could go into some detail about why just recent images are causing problems, but we're trying to focus on a solution. Amazon is telling us they don't have a quick fix so we're working on one of our own.
I'm sorry I don't have a better answer for you.
Chris
Canon 5D Mk.2/Grip || Canon 7D Backup
17-40 f/4L || 70-200 f/2.8L IS || 100mm f/2.8L Macro || 24-70mm f/2.8L
Wedding Photographer
www.cwphotos.net
Oh well, things happen. Good to know that someone is on it. I am confident you will be able to resolve the issues with Amazon. It explains why I had some trouble reaching Amazon this weekend
Sadly I just bought my pro account and uploaded a whole bunch of pictures, so effectively all my stuff is affected.
California Photo Scout
Travel Guides
I totally admit to not fully (or even partially) understanding your data architecture, but when Don said this in his blog, it lead me to believe that there were multiple copies of each photo stored on different servers. Specifically, I thought one copy was stored on Smugmug's servers and another on Amazon's. Therefore, if one system went down, the photo could be served up from the other. If I interpreted that correctly, then if Amazon is slow, why can't the images be served up quickly from Smugmug's servers?
I realize your priority right now is solving the problem, not explaining it. But, when everything's back up and running I (and I'm sure others) would be interested in hearing more.
Thanks,
www.ackersphotography.com
These are good questions. I'm curious about this as well. I too noticed a big drop in web hits since last week-- I've been getting 300-400 hits daily and over the weekend it dropped massively-- as in 50 unique hits a day. My guess is that surfers are leaving due to the slow loading too.
I know (well, least I'm crossing my fingers) that SM is all over this. The good news, I guess, is that SM has an interest in traffic coming to our sites since that leads to sales-- so hopefully everything will get figured out. Now where's that crossed fingers smilie?
Portland, Oregon Photographer Pete Springer
website blog instagram facebook g+
Out of fairness to them, they've had the most reliable systems of any of our partners over the 7 or so months we've been using them. They've saved us many a slowdown or outage when we've been able to fail over to them.
And that, ironically, is what's causing the problems today. New images are the toughest for us to handle because they get hit so hard immediately after upload. They get viewed more frequently, posted to forums, rotated, cropped, color adjusted, watermarked, etc.
Since their storage has been so high performance and reliable, we formed a higher dependency on them for new images, which has helped us handle the huge bursts of high-pixel, low-compression images we get from event photographers with fast connections.
In other words, in the beginning, Amazon was just a failover. Later, they became the primary for new images and the failover for old ones. While we've failed over to them many times, I don't think we've had to fail over from them (which means handling images with our storage that are on fire).
I know it's a red alert at Amazon and here too. Don, Andrew and Craig are at the datacenter but I don't have an eta for a fix, unfortunately.
Thanks,
Chris
My guess would be that they have different agreements in place for accessibility of the images. With Amazon, the images are available for near real time viewing (when all is working well.) With the services that are storing the same photos in other places, the access to the images may take longer -- I read his statement as focusing on the safety of the images via redundant backup that can be restored if one set of servers (or one provider) fails, not that the same image is available for instant viewing from another server in order to support a network slow down by one provider over another. I'm sure their tech guys can clarify this, if they want (and if it isn't giving away any confidential info.)
http://mkress65.zenfolio.com
http://mkress65.smugmug.com
I notice I can order a backup CD or DVD of my galleries from you guys. I don't care about photos themselves as I back them up myself. However, I don't want to lose all comments, descriptions, and all other text data. Does you backup service include such kind of data?
Thanks.
http://www.smugmug.com/prints/backup.mg
O.K. I admit it. It was me
California Photo Scout
Travel Guides
We have a failover fix in the works for the new images that we're testing right now on our test servers.
The "until recently" proviso is that new images could only be fetched from the two Amazon data centers lately due to the performance requirements of the storage.
The changes we're testing now bring us back to triple redundancy even for new images. In practice, it will probably shake out that all three datacenters are delivering images most of the time, but any one could handle the load if the other two fail.
Nice to know and makes my annual fee seem pretty cheap! Hopefully we'll see it soon and see a performance boost.
Thanks Baldy.
Matt
http://mkress65.zenfolio.com
http://mkress65.smugmug.com