Maintenance Alert: 8/28 10:00pm - 06:00am PDT

[Wireless]

This evening, we will be performing equipment moves that require us to take down SmugMug for a few hours. We will do everything possible to minimize the amount of time we will be down.

The work will be performed from 10:00pm to 06:00am Pacific time, Monday night 8/28 to Tuesday morning 8/29.

If tonight is a success, we will follow it up with a brief outage tomorrow, Tuesday night, to upgrade our load balancers. More information will be posted about that later.

Thanks for your patience, and sorry for the inconvenience!

[Sheaf]

This will also affect Dgrin I believe.

[Andy]

Bump.

[Baldy]

Here's some of the stuff we have to move:

http://zdnet.com.com/1606-2_2-6110331.html

[THE TOUCH]

Great video!

What will costumers see when they visit our Smugmug galleries while it's down? Should we put a notice up on our websites?

Thanks!

[Wireless]

We already have a couple of machines in the new datacenter that we plan to put a notice up on. Customers should see our standard Maintenance page when we completely shut down.

[Andy]

Bump

[scwalter]

Why doesn't the Amazon S3 thing pickup the slack during this move?

[onethumb]

As you can probably tell, it has begun.

There probably will be a 20-30 minute window where nothing may come up on any pages, but we'll try to minimize it as much as we can.

Thanks again for your patience!

Don

[mcgil]

I'd think that's because Amazon S3 only has a copy of the images ... not the whole rest (templates, CSS, software to call those images) ... all these things that are Smugmug and customers proprietary...

Why doesn't the Amazon S3 thing pickup the slack during this move?

[onethumb]

I'd think that's because Amazon S3 only has a copy of the images ... not the whole rest (templates, CSS, software to call those images) ... all these things that are Smugmug and customers proprietary...

Actually, it's because Amazon S3 is purely storage. It's not CPU or database or web servers or anything like that.

So if we have to take core parts of our infrastructure down, like we do tonight, we don't have the server logic necessary to fetch stuff from S3.

What is cool, though, is that we've been moving the storage nodes all month this month without taking down the site. And every storage node that has Amazon S3 redundancy hasn't had any problems and no customers have even noticed that we had stuff offline.

We managed to move 300TB of storage with almost no problems during primetime hours. Pretty amazing.

Don

[ballentphoto]

Actually, it's because Amazon S3 is purely storage. It's not CPU or database or web servers or anything like that.

So if we have to take core parts of our infrastructure down, like we do tonight, we don't have the server logic necessary to fetch stuff from S3.

What is cool, though, is that we've been moving the storage nodes all month this month without taking down the site. And every storage node that has Amazon S3 redundancy hasn't had any problems and no customers have even noticed that we had stuff offline.

We managed to move 300TB of storage with almost no problems during primetime hours. Pretty amazing.

Don

I would suggest that before taking down the system, that an e-mail be sent to all account holders so at least we know that it is happening. I only found out after trying to log in, and was met with a maintenance screen, so I came here to find out what was happening. :dunno An e-mail would at least let me know that it was looming.

[dcyphert]

Just a thought....

When maintenance of any kind is being performed, can a redirect script be in place that will direct from our site's "The page cannot be displayed" page, which really should be http://www.propointmedia.smugmug.com/
to the "Down for Maintenance" page on SmugMug's site. This would save allot of emails from customers asking why they can't view as well as eliminate the need to post something on our own sites saying it's down.

[onethumb]

Just a thought....

When maintenance of any kind is being performed, can a redirect script be in place that will direct from our site's "The page cannot be displayed" page, which really should be http://www.propointmedia.smugmug.com/
to the "Down for Maintenance" page on SmugMug's site. This would save allot of emails from customers asking why they can't view as well as eliminate the need to post something on our own sites saying it's down.

That's exactly what we do. :)

Don

[JohnR]

Good point because I had no idea that Smugmug was going offline tonight. I just tried to upload some more photos and it wouldn't log in.

[JohnR]

That's exactly what we do. :)

Don


Thanks! You guys are on the ball! :):

[I Simonius]

I would suggest that before taking down the system, that an e-mail be sent to all account holders so at least we know that it is happening. I only found out after trying to log in, and was met with a maintenance screen, so I came here to find out what was happening. :dunno An e-mail would at least let me know that it was looming.

yup!!!

Murphy's law dictates that just when you're about to do someting really important to your site and have just told someone to go look at it , that's when maintaenance hits

- then we go and see why at DGrin:smack

I have told two people of a shopping centre management to look at my site so they can determine whether or not my style is suitable for taking shots in their shopping centre so they can determine whether I can get permission and whether they are interested in talking to me a bout the possibility of setting up an exhibition on their site afterwards:bash

It would have helped me enormously not to be red faced if I had had a warning email of the outage:splat

[suchit]

That's exactly what we do. :)

Don

What is really needed is a simple e-mail message if anything major is happening.

Right now I came to my account http://photos.suchit.in and it won't work. After trying for 15 min I decided to check out DGRIN and found out what was happening. Went back and now I see the system maintainance message as well. Since we are in India (+5.5 GMT) its working hours for us. Behind the scene is fine but if something as major as couple of hours of shut down is going to take place it would help to let customers know (via e-mail) in advance so that they too can plan ahead.

We are about to hold a Photo Exhibition and sending out my URL to many who won't be able to access my site. If I had know ahead I would used my other domain - http://suchit.net which is hosted elsewhere.

~ Suchit

[elyobelyob]

I would suggest that before taking down the system, that an e-mail be sent to all account holders so at least we know that it is happening. I only found out after trying to log in, and was met with a maintenance screen, so I came here to find out what was happening. :dunno An e-mail would at least let me know that it was looming.

I couldn't agree more. It's pretty poor that I now have no time to make any changes to stave off the effects. I'll be cancelling smugmug and going in-house as this isn't acceptable for my needs.

[Andy]

I couldn't agree more. It's pretty poor that I now have no time to make any changes to stave off the effects. I'll be cancelling smugmug and going in-house as this isn't acceptable for my needs.
Hi,

We're sorry to hear this. We really hate excessive email, but in this case, I do agree with you, at least maybe we could have used our control panel messaging. Sorry about that. We do hope you won't be leaving, goodness - the whole point of this is to make SmugMug stronger, faster, better :D Sorry for the inconvenience, we hope to keep these sorts of things to a minimum. And we'll certainly talk about notification.

[onethumb]

I couldn't agree more. It's pretty poor that I now have no time to make any changes to stave off the effects. I'll be cancelling smugmug and going in-house as this isn't acceptable for my needs.

We just can't win. :(

We've gotten complaints from people in the past that our outage notifications were 'spam' and we claimed we didn't spam. Some cancelled.

We also got complaints that our email was caught in spam filters, so they didn't see it.

Those people asked us to post it on our message forum instead, so they could read it at their leisure instead of being spammed. So that's what we did.

I believe this is our only maintenance outage for all of 2006. That's pretty unheard of for any online company. I wish we didn't have to do it - but we do. I'm sorry.

Don

[I Simonius]

We just can't win. :(

We've gotten complaints from people in the past that our outage notifications were 'spam' and we claimed we didn't spam. Some cancelled.

We also got complaints that our email was caught in spam filters, so they didn't see it.

Those people asked us to post it on our message forum instead, so they could read it at their leisure instead of being spammed. So that's what we did.

I believe this is our only maintenance outage for all of 2006. That's pretty unheard of for any online company. I wish we didn't have to do it - but we do. I'm sorry.

Don

You CAN win:

Wouldn't a good solution for all be an email notification system that you have to opt IN for?

That way, part of the opting-in process would ensure that your spam filter didn't nab it ( e.g. ' Someone, possibly you has requested email notification of maintenance outages - click on this link to confirm that you want to have email notification etc')

BTW eyyobelyob - I don't think leaving SM for this one single outage makes sense because there is no guarantee that any new system you set up will be 100% outage free surely?

Notification is the key.

[elyobelyob]

You CAN win:

Wouldn't a good solution for all be an email notification system that you have to opt IN for?

That way, part of the opting-in process would ensure that your spam filter didn't nab it ( e.g. ' Someone, possibly you has requested email notification of maintenance outages - click on this link to confirm that you want to have email notification etc')

BTW eyyobelyob - I don't think leaving SM for this one single outage makes sense because there is no guarantee that any new system you set up will be 100% outage free surely?

Notification is the key.

At the moment I have two points of potential outage. One is smugmug, the other is my host. Actually, my domain registrar is another. However, this has just proved to me that I shouldn't really host my photos seperately, there's not enough of them to completely justify it, and also adds more latency when displaying pages. At least if either my registrar or webhost went down, the whole lot would be out. This just adds another point of failure which I can do without.

Nothing personal, but it's just tipped me over the edge. It'll take some work moving it all back, but I probably shouldn't have outsourced in the first place.

The good news is that I will still recommend smugmug for people wanting to do heavy photography work, it's just not the right solution for me anymore.

[Manny]

Oh my goodness, anyone complaining about an outage is a dweeb amateur ... get a grip and learn a thing or two about systems. Even your On Line Bank, the big old rich guys (Chase, Citibank, BoA) all go down. Only the really big powerful ones, the ones in the real know, like Amazon and eBay... they also go down.. a lot less, but they do. They have learned a lot of lessons and do a lot to minimize and eliminate downtime.

Sometimes it has nothing to do with having redundancy but just with how things are setup.... sometimes databases need TLC and it means doing a re-build and it means downtime.

Can it be zero? Well, never, but you can get it down to 99.99999 (that is 5 nines) which I can tell you costs a ton of money. I know because I am a Technical Architect at a large IT shot in NYC and when we look to go from 4 9's availability to 5 9's, like the telcos have to, it is exponentially more money.

So I go the cheapo route. I have a separate domain and website hosted outside SMUGMUG. This way these guys can do their maintenance and I am still up. It costs close to nothing. If they had to give me 5 9's reliability, my yearly fee would be at least $1,000.

Test it out, my website is up... http://www.mannyphoto.com/

My pictures are mostly at http://pictures.mannyphoto.com/ and that is down... but I can easily communicate with my customers/family/whatever through my domain site. If my main domain site goes down, Smugmug is up... win/win.

I hate downtime too, but you need to get a reality check and get a clue about how systems run, how they operate, what it takes to run a real database, especially one with millions of records, 24 hrs a day, 7 days a week, 365 days a year.

To the guys at Smugmug: Not that you need more work but one thing that has helped me in my shop is total disclosure. People now don;t complain as much. We post our entire outage windows and graph them against year to date, month to date and week to date... it clearly shows how we are doing in pretty colored charts. Once you hit that wonderful 364 days up, 1 day down, the graph speaks volumes.

Another thing we do on major apps/websites is post a countdown timer estimating when the outage will be over/site comes back... that helps MILLIONS more than a static page. I know it sucks but it works great for customer service. The number can easily be pushed back or forth as needed.

Good luck with your move, I am sure a lot of bullets were sweat already. Go get a lemonade and enjoy the success of the move. I'll be patiently waiting for my pics to come back :clap:clap:clap

Cheers!

[Manny]

At the moment I have two points of potential outage. One is smugmug, the other is my host. Actually, my domain registrar is another. However, this has just proved to me that I shouldn't really host my photos seperately, there's not enough of them to completely justify it, and also adds more latency when displaying pages. At least if either my registrar or webhost went down, the whole lot would be out. This just adds another point of failure which I can do without.

Nothing personal, but it's just tipped me over the edge. It'll take some work moving it all back, but I probably shouldn't have outsourced in the first place.

The good news is that I will still recommend smugmug for people wanting to do heavy photography work, it's just not the right solution for me anymore.

Can I see your pictures? Post your website URL... thanks...

[Wireless]

Almost there folks, sorry for the delay.

[marlinspike]

Oh my goodness, anyone complaining about an outage is a dweeb amateur ... get a grip and learn a thing or two about systems. Even your On Line Bank, the big old rich guys (Chase, Citibank, BoA) all go down.

Heh, my banks online banking went down just last night in fact. The servers of another guy I know, who maintains his own site, have been down for about 4 days now, so I don't think running your own site helps things.

[Andy]

Good luck with your move, I am sure a lot of bullets were sweat already. Go get a lemonade and enjoy the success of the move. I'll be patiently waiting for my pics to come back :clap:clap:clap

Cheers!
Thanks Manny, words like this go a long way to all the guys pulling the all-nighter.

[cdhames]

yup!!!

Murphy's law dictates that just when you're about to do someting really important to your site and have just told someone to go look at it , that's when maintaenance hits

amen. :rofl:rofl

So can we expect any glitches, minor problems like pages or images not being found for the next few days, after the transfer is complete? I'd just like to forwarn my customers if possible.

[bwg]

I told them printing out all the pictures and re-scanning them back in was a bad idea. But NOOOOO, nobody listens to the new guy.

[Barb]

I told them printing out all the pictures and re-scanning them back in was a bad idea. But NOOOOO, nobody listens to the new guy.

:rofl

[bham]

I told them printing out all the pictures and re-scanning them back in was a bad idea. But NOOOOO, nobody listens to the new guy.

:rofl :rofl :rofl :rofl

Yeah like 90,000,000 images in one night, that's amazing. They should be helping google with their book scanning project.

[chn]

Hope your move is going well and that no nuclear reactor move is getting in your way. When our data center's mainframe was moved to a new building we unexpectedly had to deal with road closures since some other department of the university was getting rid of their small research reactor.

Don't forget to take some pictures of the move and share them with us. :D

[samwise]

Don't forget to take some pictures of the move and share them with us. :D

http://www.laurel-and-hardy.com/goodies/vid-down/images/music9.jpg

:D

[Flash]

This evening, we will be performing equipment moves that require us to take down SmugMug for a few hours. We will do everything possible to minimize the amount of time we will be down.

The work will be performed from 10:00pm to 06:00am Pacific time, Monday night 8/28 to Tuesday morning 8/29.

If tonight is a success, we will follow it up with a brief outage tomorrow, Tuesday night, to upgrade our load balancers. More information will be posted about that later.

Thanks for your patience, and sorry for the inconvenience!

Few hours my butt :huh

[dcyphert]

Oh my goodness, anyone complaining about an outage is a dweeb amateur ... get a grip and learn a thing or two about systems. Even your On Line Bank, the big old rich guys (Chase, Citibank, BoA) all go down. Only the really big powerful ones, the ones in the real know, like Amazon and eBay... they also go down.. a lot less, but they do. They have learned a lot of lessons and do a lot to minimize and eliminate downtime.

Sometimes it has nothing to do with having redundancy but just with how things are setup.... sometimes databases need TLC and it means doing a re-build and it means downtime.

Can it be zero? Well, never, but you can get it down to 99.99999 (that is 5 nines) which I can tell you costs a ton of money. I know because I am a Technical Architect at a large IT shot in NYC and when we look to go from 4 9's availability to 5 9's, like the telcos have to, it is exponentially more money.

So I go the cheapo route. I have a separate domain and website hosted outside SMUGMUG. This way these guys can do their maintenance and I am still up. It costs close to nothing. If they had to give me 5 9's reliability, my yearly fee would be at least $1,000.

Test it out, my website is up... http://www.mannyphoto.com/

My pictures are mostly at http://pictures.mannyphoto.com/ and that is down... but I can easily communicate with my customers/family/whatever through my domain site. If my main domain site goes down, Smugmug is up... win/win.

I hate downtime too, but you need to get a reality check and get a clue about how systems run, how they operate, what it takes to run a real database, especially one with millions of records, 24 hrs a day, 7 days a week, 365 days a year.

To the guys at Smugmug: Not that you need more work but one thing that has helped me in my shop is total disclosure. People now don;t complain as much. We post our entire outage windows and graph them against year to date, month to date and week to date... it clearly shows how we are doing in pretty colored charts. Once you hit that wonderful 364 days up, 1 day down, the graph speaks volumes.

Another thing we do on major apps/websites is post a countdown timer estimating when the outage will be over/site comes back... that helps MILLIONS more than a static page. I know it sucks but it works great for customer service. The number can easily be pushed back or forth as needed.

Good luck with your move, I am sure a lot of bullets were sweat already. Go get a lemonade and enjoy the success of the move. I'll be patiently waiting for my pics to come back :clap:clap:clap

Cheers!

I couldn't agree more...and having been involved with and have seen IT evolve firsthand (since my Haze Gray Navy Daze of the 80's), I realize what a huge task it must be to manage. I too don't like down time but it's a necessary evil of the business. Good job!

[flyingdutchie]

Hi,
... at least maybe we could have used our control panel messaging ....

That would not work :D
If you can't log in, you can't view your control panel messages.

I would say: Why not an opt-in e-mail notification (much like whether you want to be notified when comments are posted)?

BTW: The move seems to take more than just a few hours.... :uhoh

[chn]

BTW: The move seems to take more than just a few hours.... :uhoh

It's probably due to the fact that they have to calm their customers, create a new notification system for downtimes, and take pictures. All things they didn't calculate for.

[Andy]

That would not work :D
If you can't log in, you can't view your control panel messages.

Hi Anton, I was referring to "advance notification." :D

[Scott_Quier]

This evening, we will be performing equipment moves that require us to take down SmugMug for a few hours. We will do everything possible to minimize the amount of time we will be down.

The work will be performed from 10:00pm to 06:00am Pacific time, Monday night 8/28 to Tuesday morning 8/29.

If tonight is a success, we will follow it up with a brief outage tomorrow, Tuesday night, to upgrade our load balancers. More information will be posted about that later.

Thanks for your patience, and sorry for the inconvenience!By my calculations, it's now 7:08AM PDT. As with all things computers (and I should know, it's how I pay for my addiction), this is taking just a mite longer than predicted. Oh well. Dgrin is working much better for me, much faster....

[papajay]

Thanks Manny, words like this go a long way to all the guys pulling the all-nighter.

Andy, as you know, I'm an amateur #1 (not a pro photog), and a tinkerer #2 (not a real "techie")...so feel free to consider my"observations" with both facts in mind.

I've been a Smugmug customer for about 8 months now, and while I've probably done more than my share of complaining about relatively minor stuff in the overall scheme of things, I can say this with conviction:

I HAVE NEVER EXPERIENCED MORE COMMITTED, PROACTIVE, EFFECTIVE, AND RESPONSIVE CUSTOMER SERVICE FROM ANY OTHER COMMERCIAL ENTERPRISE IN MY LIFE THAN THAT WHICH IS ROUTINELY PROVIDED BY SMUGMUG....BAR NONE!

Having said that, I'm one of those in the "opt-in" camp for notification (funny thing is I probably would have chosen NOT to opt-in initially, but would chose to do-so now that I know how infrequently these interruptions occur). I "invited" several folks early last evening to go check out my latest photographic endeavors. I'm certain most of them felt a sense of "let down" when they tried to access my site, only to see the Maintenance message.

I, too, felt even more dissapointment as recently as just this minute (7:00 AM PACIFIC on Tuesday, a full nine hours after I first saw the message myself). There is no indication of actual dates/times in the message, so anyone visiting right now (on Tuesday) might assume the plan is to be down until Wednesday morning:

"Tonight, we're moving a lot of servers to a new datacenter. We'll likely be offline most of the night. "

Careful, cautious, "one-size-fits-all" wording often conveys unintended meaning.

[Scott_Quier]

I HAVE NEVER EXPERIENCED MORE COMMITTED, PROACTIVE, EFFECTIVE, AND RESPONSIVE CUSTOMER SERVICE FROM ANY OTHER COMMERCIAL ENTERPRISE IN MY LIFE THAN THAT WHICH IS ROUTINELY PROVIDED BY SMUGMUG....BAR NONE!
:agree :agree :agree Was three of them enough?

[chuckice]

:agree :agree :agree Was three of them enough?

I'll 4th-ify it...:agree

[nickphoto123]

Hello,

New York, 10:26 AM, No Smugmug, what is the status?

Thank you.

[DJOrourke]

I'm one of those who really would have benefited from an email notification. Opt-in or forced would be fine with me. It just so happened that today was a day that I'm away from my personal backups and need to access originals sitting on Smugmug. Worked out to be terribly inconvenient for me. Some notice would have helped me avoid the situation.

[Andy]

Hello,

New York, 10:26 AM, No Smugmug, what is the status?

Thank you.
Hello Nick, the info is here in this thread, and keep an eye out for updates from Wireless or Onethumb.

http://www.dgrin.com/showthread.php?t=41408

http://www.dgrin.com/showthread.php?p=373401#post373401

[grannyrobin]

Andy, as you know, I'm an amateur #1 (not a pro photog), and a tinkerer #2 (not a real "techie")...so feel free to consider my"observations" with both facts in mind.

I've been a Smugmug customer for about 8 months now, and while I've probably done more than my share of complaining about relatively minor stuff in the overall scheme of things, I can say this with conviction:

I HAVE NEVER EXPERIENCED MORE COMMITTED, PROACTIVE, EFFECTIVE, AND RESPONSIVE CUSTOMER SERVICE FROM ANY OTHER COMMERCIAL ENTERPRISE IN MY LIFE THAN THAT WHICH IS ROUTINELY PROVIDED BY SMUGMUG....BAR NONE!

Having said that, I'm one of those in the "opt-in" camp for notification (funny thing is I probably would have chosen NOT to opt-in initially, but would chose to do-so now that I know how infrequently these interruptions occur). I "invited" several folks early last evening to go check out my latest photographic endeavors. I'm certain most of them felt a sense of "let down" when they tried to access my site, only to see the Maintenance message.

I, too, felt even more dissapointment as recently as just this minute (7:00 AM PACIFIC on Tuesday, a full nine hours after I first saw the message myself). There is no indication of actual dates/times in the message, so anyone visiting right now (on Tuesday) might assume the plan is to be down until Wednesday morning:

"Tonight, we're moving a lot of servers to a new datacenter. We'll likely be offline most of the night. "

Careful, cautious, "one-size-fits-all" wording often conveys unintended meaning.
I saw the message about moving the servers earlier this morning. i knew what it meant, and so it made perfect sense to me. :):

I read your message here, wondered what on earth? Went back to look at the message, and the light dawned. Oh! No wonder someone emailed to ask whether we'd be down until tomorrow.

Whether you're patient or impatient, we appreciate your bearing with us.

Thanks for the warning about wording! I think it may have been Baldy who said that SmugMug would not be the site that it is without the comments and suggestions we receive from customers.
-grannyrobin

[XHawkeye]

Making progress, the maintenance screen is gone and the cannot find server or DNS Error screen is now visible. Web page should be just around the corner.

[dancorder]

Making progress, the maintenance screen is gone and the cannot find server or DNS Error screen is now visible. Web page should be just around the corner.

Hurrah, mine is back up (rather slow at the moment but I'm sure that won't last)

Well done smugmug :thumb

[leebase]

FYI -- my photos are back online (hurray) -- but it's running real slow (awww).

Count me in as one who really appreciates the service -- and one who'd love an "opt in" notification of outages.

Lee

[jfriend]

We just can't win. :(

We've gotten complaints from people in the past that our outage notifications were 'spam' and we claimed we didn't spam. Some cancelled.

We also got complaints that our email was caught in spam filters, so they didn't see it.

Those people asked us to post it on our message forum instead, so they could read it at their leisure instead of being spammed. So that's what we did.

I believe this is our only maintenance outage for all of 2006. That's pretty unheard of for any online company. I wish we didn't have to do it - but we do. I'm sorry.

Don

Why not add a couple checkbox items to the control panel:

[X] Notify me via email before Smugmug site maintenance
[X] Notify me via email when significant new Smugmug features become available

And, then include a paragraph at the bottom of each email that explains that you can opt-out of these emails by changing the settings in your control panel.

It seems that you'd get the best of both worlds. Those who want notification can have it. Those who don't don't opt-in.

[dogwood]

Count me in as one who really appreciates the service -- and one who'd love an "opt in" notification of outages.

Lee

I check dgrin regularly so this outage wasn't a huge surprise (so don't ask me why I started an upload at 9:40 p.m. PT yesterday... just wanted to see if SM really would go down at 10-- you guys were a few minutes behind schedule :D)

Anyway, check the forums over at pbase-- wow. The concensus there is that pbase goes down randomly (it's been up and down for four days now), no one involved with pbase warns anyone anywhere (even on the forum), then refuses to respond to inquiries (even on the pbase forums). So just to add my two-cents to this discussion, I'll take SM service anyday over the lack of service at pbase (which I basically use as backup and forum linking). :clap

[rdlugosz]

FYI -- my photos are back online (hurray) -- but it's running real slow (awww).

Count me in as one who really appreciates the service -- and one who'd love an "opt in" notification of outages.

Lee

Seems to be alive, but my vanity domain is going to the SM main page and not my galleries...

[nickphoto123]

Seems to be alive, but my vanity domain is going to the SM main page and not my galleries...

Me too....And very SLOOOOOOW

Nicholas

[JamesJWeg]

Seems to be alive, but my vanity domain is going to the SM main page and not my galleries...

Vanity Domain? yeah all co-brands seem screwed, I'm sure they'll have it sorted out soon. Sounds like they might be eyeing the backups in the corner at this point. :D

James.

[NaturalEye]

I saw the message about moving the servers earlier this morning. i knew what it meant, and so it made perfect sense to me. :):

I read your message here, wondered what on earth? Went back to look at the message, and the light dawned. Oh! No wonder someone emailed to ask whether we'd be down until tomorrow.

Whether you're patient or impatient, we appreciate your bearing with us.

Thanks for the warning about wording! I think it may have been Baldy who said that SmugMug would not be the site that it is without the comments and suggestions we receive from customers.
-grannyrobin

Also bear in mind that some of us are in significantly different time zones - it may be night for you, but is probablt isn't for us!

Gary

[rdlugosz]

Vanity Domain? yeah all co-brands seem screwed, I'm sure they'll have it sorted out soon. Sounds like they might be eyeing the backups in the corner at this point. :D

James.

yea - as an IT guy myself I can empathize... I certainly don't envy the SM team right now!

[photofreak]

I, too have done my share of complaining...can I blame it on the fact that I have teens...oh, wait that explains the facial tic...the complaining, I think comes from my impatience. But, I have to say...I was in the middle of uploading last night when the maintenance screen came up. I took a deep breath, a technique I've learned by having teens, and decided to call it a night. I'm beginning to understand the vaule we all have here in SM. I came straight to DG to find out what was going on.
So...thank you, SM for all the quick responses, the never-ending patience, and the incredible way you all are willing to help those of us out here in "pictureland." I, for one, have learned a little patience goes along way.
I love my smugmug:lust

[JimM]

As a guy that is a partner in a web hosting company, I certainly feel their pain. To help matters, we had a hardware failure on one of our servers too. So I understand the complaints and frustrations from both sides of the fence.

Hang in their SM guys! No pressure from me, I have a customer waiting to see a retouched photo (that I haven't had time to do yet), so the site being down works as a good excuse for me!

[Manny]

I told them printing out all the pictures and re-scanning them back in was a bad idea. But NOOOOO, nobody listens to the new guy.
:D :rofl
Good one... made my morning LOL

[Tony Bonanno]

Geesh, I've got an event that my clients are scrambling to see and this morning I'm getting emails that they can't access the gallery, etc. y@#$.

I had no idea that this was scheduled. I finally arrived here at dgrin and found out what's going on. I don't think that I should have to lurk on this forum to get a head's up about the system being down. Many of us with PRO accounts are working photographers, and don't have time to be cruising DGRIN on a regular basis. I am very enthused about Smugmug, often recommend it to colleagues, but really.. there's got to be a better way so that we know this kind of thing is being scheduled.

Thank you.

Tony Bonanno

[Andy]

the fact that I have teens...oh, wait that explains the facial tic

:rofl :rofl :rofl :rofl

Three of them, here. I know the feeling :uhoh

[Andy]

. there's got to be a better way so that we know this kind of thing is being scheduled.

Thank you.

Tony Bonanno
Tony, I agree with you. You can bet that I'll be talking to the team about this. Thanks for taking the time to let us know how important this is to you!

[JimM]

Everything seems, up and good and fast to me now. :clap

[Tony Bonanno]

Thanks Andy !

Tony, I agree with you. You can bet that I'll be talking to the team about this. Thanks for taking the time to let us know how important this is to you!

[Flash]

You guys said this crap was not going to happen again and now the stats are out AGAIN.

[photogmomma]

Hmm, still having issues on www.tippiepics.com (http://www.tippiepics.com). My slideshow isn't working... (Worked last night.)

Just sitting here patiently waiting!

Oh, and I have to say (after working for a software company) that I'm blown away by the response and support from SmugMug. I am definitely one of the biggest supporters. You people ROCK. :clap

ETA: It's working now... thanks!

[Andy]

You guys said this crap was not going to happen again and now the stats are out AGAIN.
Ouch - I see you are frustrated - hang in there, we've no indication that stats will be affected - I'm sure it's just a matter of time for them to come back up.

[JimM]

I spoke too soon too. Site is down again. Obviously kinks are still being ironed out. No biggie!

[JamesJWeg]

I'm sure we will be told when they are done. I'm another vote for e-mail notification.

James.

[Andy]

I'm another vote for e-mail notification.

James.

+1 :D

[Wireless]

OK, we're done for now, sorry for the extra time it took.

We understand the inconvenience of no stats, however we had to make the call to restore as much of the site as possible. We'll continue to work on bringing stats back online ASAP.

Thanks for your patience everyone!

[Mike Lane]

OK, we're done for now, sorry for the extra time it took.

We understand the inconvenience of no stats, however we had to make the call to restore as much of the site as possible. We'll continue to work on bringing stats back online ASAP.

Thanks for your patience everyone!am I going to have to make another t-shirt???

[onethumb]

OK, we're done for now, sorry for the extra time it took.

We understand the inconvenience of no stats, however we had to make the call to restore as much of the site as possible. We'll continue to work on bringing stats back online ASAP.

Thanks for your patience everyone!

A little more information before I fall into a coma: we managed to move hundreds of servers, and hundreds of terabytes. But one of them (which happens to be a core part of the stats tracking) won't boot now.

We have spares if we can't fix it, so it's just a matter of taking an hour or so to make sure it's working and the data is intact - but we're all wiped out. And wiped out people messing with mission critical stuff = bad news.

So we'll get the stats back online as soon as we've gotten a little sleep.

Sorry for the delay!

Don

[photofreak]

OK, we're done for now, sorry for the extra time it took.

We understand the inconvenience of no stats, however we had to make the call to restore as much of the site as possible. We'll continue to work on bringing stats back online ASAP.

Thanks for your patience everyone!
I'm surprisingly calm over here...was about to start making phone calls about "yearbook photos" so, I can get started on those...glad I checked back. Should I shelve this for today? Do you think by this afternoon everything will be up?

[Wireless]

I'm surprisingly calm over here...was about to start making phone calls about "yearbook photos" so, I can get started on those...glad I checked back. Should I shelve this for today? Do you think by this afternoon everything will be up?

'Everything' is up besides stats, so if that component is critical to your work, you might want to consider rescheduling.

Otherwise, the bits are flying well around the new datacenter :)

[photofreak]

So we'll get the stats back online as soon as we've gotten a little sleep.



No...no, sleep for you...keep working.:baldy

[cabinetbuff]

Smugmug is really slow right now. It takes 20-30 seconds for any gallery to respond.

[photofreak]

'Everything' is up besides stats, so if that component is critical to your work, you might want to consider rescheduling.

Otherwise, the bits are flying well around the new datacenter :)

You guys rock!!!!! Thanks for all your hard work! As long as my customer and I can be looking at the same screen...we're good to go! :thumb

[photofreak]

Smugmug is really slow right now. It takes 20-30 seconds for any gallery to respond.
it's a comin'...patience...breathe in through the nose and out through the mouth...now, one more time. It should be there now.:D

[cabinetbuff]

it's a comin'...patience...breathe in through the nose and out through the mouth...now, one more time. It should be there now.:D
Actually according to smugmug support it's already there besides stats:


OK, we're done for now, sorry for the extra time it took.

I am just here to give them some feedback on what I am experiencing with my service, no need for come me down :D

[leebase]

Just fyi -- the site is NOT up for me: leebase.smugmug.com -- 11:17am Central. I wouldn't bother posting, but you are saying the site is up.

Lee

[Dramatapix®]

Is the migration/maintenance still happening? I'm having a great deal of difficulty maneuvering my site:

www.dramatpix.com (http://www.dramatpix.com)

I can hit the main page alright, but trying to navigate through the galleries is vewy vewy slow (when it actually goes to the page at all).

Is this a known issue? Is it being worked on? Am I just a moron?

Thanks
B~

[cabinetbuff]

Just fyi -- the site is NOT up for me: leebase.smugmug.com -- 11:17am Central. I wouldn't bother posting, but you are saying the site is up.

Lee

I can get to your site from here (So CAL) but the galleries are very slow just like on my site (20-30 sec response time).

[Scott_Quier]

I the "for what it's worth column" I also am experiencing strange behavior. The pages get served up either in zip time or the browser times out. There does not seem to be a pattern to the behavior.

Site: scottquier.smugmug.com

I guess I'll just sit back, breath in through the nose, out through the mouth ... ommmmmmm, ooooommmmmmmm Ah, all better now.

I'll check back in an hour or two to see if things have improved.

Thanks again for all your hard work. If SmugMug runs like dGrin is now, things are going to be very cool very soon!

[Rene`]

Looks like things are getting back to normal. Should everything be working now?

This evening, we will be performing equipment moves that require us to take down SmugMug for a few hours. We will do everything possible to minimize the amount of time we will be down.

The work will be performed from 10:00pm to 06:00am Pacific time, Monday night 8/28 to Tuesday morning 8/29.

If tonight is a success, we will follow it up with a brief outage tomorrow, Tuesday night, to upgrade our load balancers. More information will be posted about that later.

Thanks for your patience, and sorry for the inconvenience!

[dogwood]

am I going to have to make another t-shirt???

Not everyone has survived "the great maintenance of 2006" yet though-- so don't print 'em up just yet!

[photofreak]

[quote=cabinetbuff]Actually according to smugmug support it's already there besides stats:



I mean't breathe for the 20-30 seconds it takes to come up. Mine is working...just a tad slow.

[bham]

Mine too, having to click on things two or three times to get pages to change, etc.

I imagine with a big move everything may take some time to get smoothed out.

[steveL]

The video was really interesting and I was happy to see it. Just reinforces my thinking that I made the right decision to move to Smugmug.

My photography is just a little hobby-- but still important to me-- and I am glad that Smugmug takes my hobby as seriously as they do the pro's buisness.

Thanks.

[cjkphoto]

Murphy's law kicked in, badly I might add.
After a trial for the pro account, I was impressed.
My first client since I started digital (and a good friend of mine) wanted to view the pictures from a coffee shop last night. And we got the maint. page.
Needless to say, I was horrified. She wasn't impressed. It was a hard shoot for her and my new assistant. She wanted to show some the initial work to some people we met there.
Bad. Major maint. needs to go back on to email alerts. And default to 'on' in the CP. If people are really that bugged about one email every few months or so, let them.
I was lucky I copied a couple of the photos on my usb drive to play around with on PS.
BTW-It's still crawling with a goo of slowness. I'm sure that will be fixed in a while...and we'll all go back to uploading and downloading and having loads of fun, but this is seriously bad.

[renstar]

I'm going to Nth the request for an email notification during outages, though, I'm gonna request something slightly different.. I don't particularly care if smugmug is down for 2 minutes because of a hickup on their end or something up with the backbone providers or whatnot, or even small planned outtages to bring a server online (say, 5 mins), but when there is a planned outtage that will take on the order of hours and potentially longer if issues come up, I dont think any reasonable customer could see it as spam. (and if they do, let me know so i can kick them in the shins)

Glad you got (most) everything back up and running again. And get that deserved sleep, so I hope you miss this message for a while.

[Flash]

I'm sure we will be told when they are done. I'm another vote for e-mail notification.

James.

Why would they want to broadcast a cluster mess to us by email, that won't happen.
PROMISED less than 60 day since the stat lose and here we are again

[Seymore]

I concur with the email notification aspect. Even something as simple as adding a field to the Control Panel that allows the user to opt in or out to Admin email like the announcement that should have happened on this outage. That then links them to the bulk mail server for issues like this. You could also include announcements on upgrades. This could also open the door to other Smugmug plans down the road via email. (read: contests, newsletters, ect...)

But, I'm really surprised that you opt to reply to all the emails you get, instead of implementing a (granted bulk) system to inform. BTW, @ 2am (PST) all I got was a "The page cannot be displayed..." message on my browser. There was NO redirect.

And, I know the cost... and have not seen any talk about this... BUT, what about a mirror site, maintained by you guys? Can you say 0 (that's a ZERO, not an O) down time? Has that ever been considered?

[JamesJWeg]

Why would they want to broadcast a cluster mess to us by email, that won't happen.
PROMISED less than 60 day since the stat lose and here we are again

Dude, check the past, I've blown my lid on here to them defore about changes that pissed me, guess what, they listened to a large degree. Judging from past experiance they will implment some kinda of e-mail notification in the future. As for the stats, I am also annoyed by that, but give them a little time to see if they are really gone.

James.

[flyingdutchie]

[quote]

I mean't breathe for the 20-30 seconds it takes to come up. Mine is working...just a tad slow.

Same on my site.
Sometimes a gallery takes upto a minute to show up (when doing a 'view source', no data at all is downloaded during the wait), and then all the page's content is downloaded at once (really fast).

It seems that downloading content (images) is not slow at all; getting the first response from the server is sometimes very slow (generating HTML for galleries, etc).

[wslam]

For the past 24 hours, the site has been painfully slow. I am not writing for myself, but also on behalf of all my buddies here in HK who have signed up to smugmug due to my recommendation. It takes over a min for the site to respond at times. And sometimes, simply time out. I know overseas connections have a lot more bottlenecks, but the last 24 hours has definitely been worse then avg but a *lot*.

Ping report and Traceroute report below:

Pinging ws.smugmug.com [63.81.134.23] with 32 bytes of data:

Reply from 63.81.134.23: bytes=32 time=600ms TTL=52
Reply from 63.81.134.23: bytes=32 time=509ms TTL=52
Reply from 63.81.134.23: bytes=32 time=245ms TTL=52
Reply from 63.81.134.23: bytes=32 time=338ms TTL=52

Ping statistics for 63.81.134.23:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 245ms, Maximum = 600ms, Average = 423ms

C:\Documents and Settings\WSLam>tracert ws.smugmug.com

Tracing route to ws.smugmug.com [63.81.134.23]
over a maximum of 30 hops:

1 2 ms <1 ms <1 ms 192.168.0.1
2 294 ms 172 ms 165 ms pcd-vta2-1-rx.netvigator.com [203.218.32.254]
3 365 ms 326 ms 348 ms 218.102.57.70
4 31 ms 21 ms 73 ms 218.102.21.78
5 118 ms 160 ms 195 ms unknown.net.reach.com [202.40.142.9]
6 589 ms 585 ms 373 ms i-5-2.hht-dist02.net.reach.com [202.84.154.98]
7 273 ms 234 ms 228 ms unknown.net.reach.com [202.84.154.197]
8 239 ms 325 ms 283 ms i-6-0.paix-core01.net.reach.com [202.84.140.125]

9 192 ms 243 ms 397 ms i-3-3.paix05.net.reach.com [202.84.251.94]
10 191 ms 191 ms 203 ms GBLX.peer.paix05.net.reach.com [134.159.62.98]
11 239 ms 453 ms 662 ms so5-0-0-2488M.ar1.SJC2.gblx.net [67.17.67.154]
12 216 ms 196 ms 323 ms pri.r1-ge0-2-eq-sj.smugmug.com [206.223.117.70]

13 209 ms 355 ms 353 ms hera.smugmug.com [63.81.134.23]

Trace complete.

[Andy]

Why would they want to broadcast a cluster mess to us by email, that won't happen.
PROMISED less than 60 day since the stat lose and here we are again
I can see you are upset, Flash. The two stats problems are un-related, though. Sorry again for the trouble, and thank you in advance for your patience.

[Steve Cavigliano]

I was just responding to a thread about how slow SM is overseas. The thread disappeared while I was typing.....LOL

Anyhow, the site seems way slow today. It is taking 25 seconds to get to a gallery and 20+ seconds to load up an M size image for viewing :scratch

I sent some SM links to my wife and she sent me back an email asking if her PC was bad, since the pics were taking so long to load.

So what I want to know is, which one of youse breaka my Smugmug? :D

Steve

[Dalantech]

It's slow here in Italy too, but I wouldn't trust ping or traceroute as a trouble shooting tool. A router will drop ICMP packets in favor of "real" traffic -so a congested link between you and SmugMug could account for the high return trip counts. The RFC for traceroute is a joke, because decrementing the TTL counter is a "should" and not a must, so there could be hidden gateways that your traffic is passing through. Also a gateway can remain hidden because it doesn't have to send a TTL exceeded ICMP packet back to the source (another "should"). Finally, due to security reasons, a lot of sites are blocking ICMP all together...

[wslam]

It's slow here in Italy too, but I wouldn't trust ping or traceroute as a trouble shooting tool. A router will drop ICMP packets in favor of "real" traffic -so a congested link between you and SmugMug could account for the high return trip counts. The RFC for traceroute is a joke, because decrementing the TTL counter is a "should" and not a must, so there could be hidden gateways that your traffic is passing through. Also a gateway can remain hidden because it doesn't have to send a TTL exceeded ICMP packet back to the source (another "should"). Finally, due to security reasons, a lot of sites are blocking ICMP all together...

Yes I understand. I just want to post *some* data for ref. If I don't post, some people will ask, now I post, some people will say they are useless =) Can't win! =)

[Andy]

I was just responding to a thread about how slow SM is overseas. The thread disappeared while I was typing.....LOL

it did not disappear, it's right here:
http://www.dgrin.com/showthread.php?p=373620#post373620

Anyhow, the site seems way slow today. It is taking 25 seconds to get to a gallery and 20+ seconds to load up an M size image for viewing :scratch

I sent some SM links to my wife and she sent me back an email asking if her PC was bad, since the pics were taking so long to load.

So what I want to know is, which one of youse breaka my Smugmug? :D

Steve

Major data center move, as explained in this thread. Sorry for the trouble today!

And as for blame? HERE (http://www.dgrin.com/showthread.php?t=41424)

[papajay]

[quote=photofreak]

Same on my site.
Sometimes a gallery takes upto a minute to show up (when doing a 'view source', no data at all is downloaded during the wait), and then all the page's content is downloaded at once (really fast).

It seems that downloading content (images) is not slow at all; getting the first response from the server is sometimes very slow (generating HTML for galleries, etc).


Similar experience here...initial page load (connecting to site) is superfast...gallery page loads that follow are super slow.

One VERY encouraging observation:
Keyword searches at both Smugmug level and Site Owner level proceed with LIGHTNING SPEED!!!

[leebase]

My site is up now. Alternately fast and slow. On IE that is. For some reason, I can't view my site with Firefox. wierd.

Tuesday, 3pm Central

Lee

[Dramatapix®]

My site is up now. Alternately fast and slow. On IE that is. For some reason, I can't view my site with Firefox. wierd.

Tuesday, 3pm Central

Lee

4:17PM EDT still experiencing the very frustrating issues within the galleries. Unable to navigate the pictures, or taking 30 seconds or more at times to open if at all.

[Steve Cavigliano]

it did not disappear, it's right here:
http://www.dgrin.com/showthread.php?p=373620#post373620


Major data center move, as explained in this thread. Sorry for the trouble today!

And as for blame? HERE (http://www.dgrin.com/showthread.php?t=41424)

Thanks Andy :D

I think we have a semantics issue. Technically those threads did disappear. But only because you took the posts and incorporated them into this thread. That's fine with me. If I would have seen this thread to begin with, I wouldn't have started a new thread, duplicating the same topic. But it was on page 2 and I missed it :dunno

Sorry to add more ragging to an already over-ragged support staff...lol

I have the utmost confidence that the site will be up and smokin' as soon as possible :thumb

Steve

[DJKennedy]

My site is up now. Alternately fast and slow. On IE that is. For some reason, I can't view my site with Firefox. wierd.

Tuesday, 3pm Central

Lee
I agree - I only use Firefox, and my site is deadly slow. I mean its so slow that it would be faster for me to snail mail prints of my photos.

Derek

[Flash]

Paid my renewal fee this month and got a e mail to update my VISA info.

When it comes to SM getting an e mail to to me to make payment is sure the heck gets to me with no delays or site problems.
Hmm I wonder why, I wonder why :dunno

[Mike Lane]

Paid my renewal fee this month and got a e mail to update my VISA info.

When it comes to SM getting an e mail to to me to make payment is sure the heck gets to me with no delays or site problems.
Hmm I wonder why, I wonder why :dunnoI personally think that's a little unfair. The insinuation is that smugmug is greedy and cares more about its bottom line than its customers. From my experience on dgrin and with the owners of smugmug personally, I have to say that it seems the very opposite is true. The smugmug staff are busting their humps constantly every day (and every night mind you) to give their customers the best service they can. I've yet to see this greed you off-handedly insinuate of them. For instance just this week there have been 2 dgrinners that have very nearly busted their traffic limits for the month. What did smugmug do? They reset their limits. I've seen a number of times when someone wanted out of their smugmug accounts for whatever reason and smugmug simply refunded their money no questions asked.

I know things can get frustrating but that insinuation IMHO is over the top. But maybe that's just me.

[Dramatapix®]

I personally think that's a little unfair. The insinuation is that smugmug is greedy and cares more about its bottom line than its customers. From my experience on dgrin and with the owners of smugmug personally, I have to say that it seems the very opposite is true. The smugmug staff are busting their humps constantly every day (and every night mind you) to give their customers the best service they can. I've yet to see this greed you off-handedly insinuate of them. For instance just this week there have been 2 dgrinners that have very nearly busted their traffic limits for the month. What did smugmug do? They reset their limits. I've seen a number of times when someone wanted out of their smugmug accounts for whatever reason and smugmug simply refunded their money no questions asked.

I know things can get frustrating but that insinuation IMHO is over the top. But maybe that's just me.

I understand your wanting to come to the support of the people that run the service, but from my perspective, I'm new here... my account is less than a month old, and to see this kind or performance is a little frustrating. The thing that makes it more frustrating is the fact that I've yet to see anyone with authority offer some kind of explanation other than there has been a major database transfer. Now mind you I've not read every thread, but I've posted numerous messages in two threads regarding performance today and have not seen a reply.

Just one man's perspective.

Brett

[richW]

Originally Posted by Mike Lane
I personally think that's a little unfair. The insinuation is that smugmug is greedy and cares more about its bottom line than its customers.

:agree With Mike.

After reading this thread http://www.dgrin.com/showthread.php?t=39356 it confirmed to me SmugMug's level of service is very high.

I think I'll go out and take some pictures today instead of waiting for the servers to come up to speed.

BTW who gave the keys to the data center to The BigWebGuy?
Congratulations Lee....

[Mike Lane]

I understand your wanting to come to the support of the people that run the service, but from my perspective, I'm new here... my account is less than a month old, and to see this kind or performance is a little frustrating. The thing that makes it more frustrating is the fact that I've yet to see anyone with authority offer some kind of explanation other than there has been a major database transfer. Now mind you I've not read every thread, but I've posted numerous messages in two threads regarding performance today and have not seen a reply.

Just one man's perspective.

Brett

I've been with smugmug for geeze nearly a year-and-a-half. My personal experience is that there has been 2 or 3 instances unplanned downtime and a few planned outages. The unplanned events weren't anything major as I recall (I think I remember one being a fairly big deal) and the planned outages all went reasonably well. All in all my service has been up FAR more than it's been down. And from what little I know of running a datacenter (not much) I know that no datacenter has 100% uptime. My point about that is, Smugmug is very reliable in my experience.

As far as explanations, there have been 2 threads where wireless (the guy who is in charge of the datacenter) posted and one thread where onethumb (the CEO of smugmug) posted in fairly reasonable detail what happened and what to expect next. I understand your frustration (I'm working on a project right now that would go so much faster if my smugmug site were going full speed) but I know that these things happen to the best of sites. I'm looking forward to my site being its old self (hopefully better :wink) and I can guarantee that your concerns are not being ignored by the smugmug staff (of which I am not a member BTW).

That said, feel free to vent all you want in here and elsewhere. I'm not trying to stifle that by any means. I do however think that Flash was way out of line with his accusations.

[ivar]

I think I'll go out and take some pictures today instead of waiting for the servers to come up to speed.Now that sounds like a great idea!

[igoforit]

Here in Belgium also the response times are not workable. My links in forums to pictures in my albums are sometimes simply not showing up (red cross).
Hope it will be resolved soon.
Jos

[Andy]

The thing that makes it more frustrating is the fact that I've yet to see anyone with authority offer some kind of explanation other than there has been a major database transfer.

Hi Dramatapix, thanks for posting. Onethumb (our CEO and Chief Geek) and Wireless (our Director of Operations) are both crashed from staying up all night. They'll be back on soon and will provide an update, I'm sure.

Thanks for your continued patience - we're sure this will lead to an even stronger SmugMug!

[Scott_Quier]

A little more information before I fall into a coma: we managed to move hundreds of servers, and hundreds of terabytes. But one of them (which happens to be a core part of the stats tracking) won't boot now.

We have spares if we can't fix it, so it's just a matter of taking an hour or so to make sure it's working and the data is intact - but we're all wiped out. And wiped out people messing with mission critical stuff = bad news.

So we'll get the stats back online as soon as we've gotten a little sleep.

Sorry for the delay!

DonIt's been just about 6 hours (4 mintes shy of 6 hours, actually) and I'm thinking it's time to haul it out of the sack and finish the job or to, at least, proffer an idea of when the job is going to be finished. In the absense of a recent estimate (has there been one posted yet?) the slow response from the SmugMug sites is starting to wear just a bit.

How about it Andy. You'be been the most responsive in this thread. What have you heard?

You beat me to the punch by about a minute (while I was reviewing my post).

Thanks

[cdhames]

wow. i just checked my account.. whatever wireless and those boys are doing, KEEP DOING IT!!! my galleries just loaded so fast my eyeballs got singed.

[Andy]

I

How about it Andy. You'be been the most responsive in this thread. What have you heard?
We've actually just heard from Onethumb, who did a bit more to the system, and things should be dramatically improved - have a look and report back please :D

[Mike Lane]

We've actually just heard from Onethumb, who did a bit more to the system, and things should be dramatically improved - have a look and report back please :DOMG much faster! Nice.

[igoforit]

Here in Belgium also the response times are not workable. My links in forums to pictures in my albums are sometimes simply not showing up (red cross).
Hope it will be resolved soon.
Jos
It doesn't seem to be that bad consistently.... I tried a few things and then I thought it was a problem only with my album that I uploaded yesterday night. That now seems to go faster as well, but I assume that may be because the images have been cashed on my HD by now.
Hope that response times will be OK (or even better then before) once all issues have been solved.
Jos

[Scott_Quier]

We've actually just heard from Onethumb, who did a bit more to the system, and things should be dramatically improved - have a look and report back please :DNow that's the kind of response I was expecting of the stellar SmugMug staff.

Oh, and Andy, my site screams now.

[Scott_Quier]

It doesn't seem to be that bad consistently.... I tried a few things and then I thought it was a problem only with my album that I uploaded yesterday night. That now seems to go faster as well, but I assume that may be because the images have been cashed on my HD by now.
Hope that response times will be OK (or even better then before) once all issues have been solved.
JosHadn't thought of that so I cleared my cashe and did it again. Still screams!!!!:clap:clap:clap:clap:clap:clap

[igoforit]

We've actually just heard from Onethumb, who did a bit more to the system, and things should be dramatically improved - have a look and report back please :D
I confirm that things seem to be OK now.
Jos

[cjkphoto]

I understand your wanting to come to the support of the people that run the service, but from my perspective, I'm new here... my account is less than a month old, and to see this kind or performance is a little frustrating. The thing that makes it more frustrating is the fact that I've yet to see anyone with authority offer some kind of explanation other than there has been a major database transfer. Now mind you I've not read every thread, but I've posted numerous messages in two threads regarding performance today and have not seen a reply.

Just one man's perspective.

Brett
That is the frustrating thing about it. I was just 10 days into my 14 day trial, and upped the ante to the pro account over the weekend.
It's Tuesday, and that means that it's only about 80% uptime for me. Not good.
And things are still slow.
Wait a second. Update us again, I just checked and it's lighting fast.
Hopefully some lessons were learned on this. Please email us for major outages.

[dogwood]

I understand your wanting to come to the support of the people that run the service, but from my perspective, I'm new here... my account is less than a month old, and to see this kind or performance is a little frustrating. The thing that makes it more frustrating is the fact that I've yet to see anyone with authority offer some kind of explanation other than there has been a major database transfer. Now mind you I've not read every thread, but I've posted numerous messages in two threads regarding performance today and have not seen a reply.

Just one man's perspective.

Brett
Just to be clear, I'm not connected to SM in any way other than I am a happy customer... even with the issues today. Yeah, the slowness is driving me nuts, but I gotta point out (again) that over at pbase right now (another very popular photo site), people with paid accounts can't even upload photos (and this has been going on for four days now) and pbase is ignoring any questions about what is going on. It's rumored that the owners may be on their way to Burning Man, BTW.

But I look through the threads here, and I see several responses from the SM CEO himself (Don, you're not the Burning Man type? Or is there still time to head over?) And I gotta say Andy Williams is absolutely amazing when it comes to customer service-- yes, he's with SM-- and he's helped me out so much in the past it's amazing. And he's been posting here all day too.

So anyway, just wanted to offer up another man's perspective. I'll bet the SM folks are scrambling to fix the problems-- and honestly, I'd rather they work on that than waste time chiming in here! Anyways, long ago I complained once on dpreview about smugmug being slow and some guy responded that if I wasn't happy, I should set up my own website on my own server. Of course his comment steamed me-- but he was right. And when I researched my own site on a server and the design involved and the updating and cost and all that, well, smugmug started to seem like a mighty good deal again.

And just so you know, I've certainly posted my share of "WTF is going on with SM?!" and every time, I've had a SM person (sometimes even a customer) talk some sense into me. So use this downtime to catch up on post-production, design a postcard promo, call some potential clients... yeah, yeah easy for me to say, right? But ironically, photographers actually used to exist before websites.

So what did I do today since my website is basically too slow to use (and I can't upload anything to my backup-- pbase)? I actually met in person with a potential client and booked a paid shoot. She didn't care about my website problems-- she cared about my photography skills.

[Baldy]

I'll bet the SM folks are scrambling to fix the problems--They sure are. It was a shame that we had to move from one datacenter that ran out of space to a new one, and another painful shame that it triggered performance problems.

Five of us were on the move team last night including the three guys who are most critical for performance tuning. They're on the case as we speak but did take some hours for sleep the first half of today.

We have 26 new servers on the way that had to wait for the new data center before we could install them.

Thanks,
Baldy

[Mike Lane]

They sure are. It was a shame that we had to move from one datacenter that ran out of space to a new one, and another painful shame that it triggered performance problems.

Five of us were on the move team last night including the three guys who are most critical for performance tuning. They're on the case as we speak but did take some hours for sleep the first half of today.

We have 26 new servers on the way that had to wait for the new data center before we could install them.

Thanks,
Baldy26 new servers? :huh No wonder it's so fast now.

[I Simonius]

I know things can get frustrating but that insinuation IMHO is over the top. But maybe that's just me.

I'm with you there.

It has to be said that SM consistantly do their best.

That might not sound like much when things don't go smoothly- and I'll complain along with the rest when Murphy joins in as well, but that doesn't mean I don't appreciate what they do right, I do.

I understand things go wrong sometimes, and of course it causes much gnashing of the teeth, renting of raiment and so forth when it does, but I put it down to experience

SM consistently doing their best however means that they learn, and demonstrate that they learn, from their mistakes; are constantly striving to make SM a better experience ( and generally succeeding!) and show a very human face, (which has a lot of value AFAIC) are always innovating and do have the best customer service I have seen so far on the net, and are good value. Their best is, as far as I can see for the service they offer, the best on the net, (just ;-))

So their best is good enough for me:thumb

OH - and they are Mac afficionados:thumb :D :D :thumb

[cabinetbuff]

Just to be clear, I'm not connected to SM in any way other than I am a happy customer... even with the issues today. Yeah, the slowness is driving me nuts, but I gotta point out (again) that over at pbase right now (another very popular photo site), people with paid accounts can't even upload photos (and this has been going on for four days now) and pbase is ignoring any questions about what is going on. It's rumored that the owners may be on their way to Burning Man, BTW.
Good to know that we are not the only once to experience downtime but why measure to the worst of the crop?

I can tell you for example that I've had yahoo.com mail/dsl and hosting services and they never let me down -- so lets encourage SM to look up to the big boys not the garage type operations -- we will all benefit in the end.

I personally want SM to be the best of the best.

Things are working much better now by the way! :thumb

[aktse]

Wow! Lightning Fast!!! Way to go smumug!

I love it!:lust

[Wireless]

26 new servers? :huh No wonder it's so fast now.

Actually, those haven't arrived yet. But we have space for them now :)

[Mike Lane]

Actually, those haven't arrived yet. But we have space for them now :):jawdrop

[Dramatapix®]

UNBELIEVABLE!!!

Wow... I'm impressed and glad that I didn't get torn to shreds with my speaking my mind earlier. I'm glad that it was taken in the context it was meant.. (merely an observation from a newcomer).

The service and customer service is awesome.

Thanks SM for taking the initiative to make things better, and forgive me for my impatience and momentary lapse of trust. :)

Brett

[mcgil]

:clap Thanks a ton for the big upgrade in performance ! :clap

:thumb Happy customer !

[photo_mom]

Have had a smugmug account for over a year and lurked here once in awhile. Had to de-lurk to tell you that your uptime is very impressive, the maintenance periods are short and sweet, and everytime there's been maintenance it's been followed by a vast improvement. Every e-mail I've sent to support has been answered with speed and been very helpful. Keep up the good work and take this as a huge pat on the back for this very successful upgrade! :clap:thumb:clap:thumb

[ballentphoto]

FYI at least my stats have returned :thumb For the life of me I do not understand why people would consider notification spam :dunno but to each his/her own I suppose. I too work in IT and have pulled my own share of all nighters (as a matter of fact last night was one of them :wxwax) The service has been good so far for me (this is my first glitch since signing on).

So count me in for a little check box on the control panel to give me notifications. :wink

[cjkphoto]

Well...I do see how much the staff and dgrin moderators are dedicated to smugmug. It's a bummer for everyone involved.
But the intentions were for nothing but the best.
As I said before, lessons to be learned. Not everyone is perfect.
And uber photo hosting sites offering a huge variety of services isn't a defined science like say running a simple bulletin board.
It just happened to be really bad timing for me and others that have been vocal about it.
It looks like the fun factor returned, off to more uploading and browsing.

[Andy]

Have had a smugmug account for over a year and lurked here once in awhile. Had to de-lurk to tell you that your uptime is very impressive, the maintenance periods are short and sweet, and everytime there's been maintenance it's been followed by a vast improvement. Every e-mail I've sent to support has been answered with speed and been very helpful. Keep up the good work and take this as a huge pat on the back for this very successful upgrade! :clap:thumb:clap:thumb
:wave Hi photo_mom and thanks for de-lurking :D

[jerryr]

Performance seems to have increased - awesome job...
Being in the computer business, yep, downtime for maintenance
happens - not a big issue if the end results are benefitical !
(you can even www.askdrz.com (http://www.askdrz.com) - lol - ok - that was cheap but you can win some cool stuff - even a car !)
Good job smugmug - yea :clap

love the znet video - cool marketing stuff !

- jr

[leebase]

Just a note -- all is working well for me too....even with firefox. Thanks SM -- sorry you folks had such a difficult move.

Lee

[Flash]

Stats still not counting up :huh

[Dna]

Stats still not counting up :huhGo sign up for Google Analytics.

Dna

[Seymore]

I'll log in, and when I goto another page, I'm logged out again. And to view any hidden galleries, I have to log in again. This happen no matter where I go... and it happens on all IE windows.

BTW, cleared the cache, IE6 under W2k/SP4. I also rebooted my machine this morning. So, tell me that this is not on my end...


UPDATE: just got the following: I'm sorry, but you don't appear to be logged in, or don't have access to this gallery. Please try again. Hmmmm... but I got into it!!!

[Scott_Quier]

I'll log in, and when I goto another page, I'm logged out again. And to view any hidden galleries, I have to log in again. This happen no matter where I go... and it happens on all IE windows.

BTW, cleared the cache, IE6 under W2k/SP4. I also rebooted my machine this morning. So, tell me that this is not on my end...


UPDATE: just got the following: I'm sorry, but you don't appear to be logged in, or don't have access to this gallery. Please try again. Hmmmm... but I got into it!!!I just tested this on my site. No issues for me. Your password is maintained in a cookie. Are you SURE that you are allowing cookies?

Question to self: Would SmugMug alert if cookie settings are not correct?

[Andy]

Stats still not counting up :huh
Hi Dan, I can see your stats fine. Are you still having troubles? I have a screen grab if you'd like me to show it.

[joffun]

My site seems a lot quicker now - so the upgrade has definitely been a success. :):

That said, I would also be in favour of an 'opt-in' email before known downtime starts, plus I think that the suggestion of a countdown timer showing when the anticipated work will finish (which can be adjusted by SM as need dictates) would be really useful - especially for overseas users. PST doesn't mean a lot to many people in the UK - so it would be good to just have an indication of how long the site will be down for - thanks!

[Andy]

I'll log in, and when I goto another page, I'm logged out again. And to view any hidden galleries, I have to log in again. This happen no matter where I go... and it happens on all IE windows.

BTW, cleared the cache, IE6 under W2k/SP4. I also rebooted my machine this morning. So, tell me that this is not on my end...


UPDATE: just got the following: I'm sorry, but you don't appear to be logged in, or don't have access to this gallery. Please try again. Hmmmm... but I got into it!!!
Hi Sid, you're using moorephoto.smugmug.com ? I cannot replicate this behavior.

[Seymore]

I just tested this on my site. No issues for me. Your password is maintained in a cookie. Are you SURE that you are allowing cookies?

Question to self: Would SmugMug alert if cookie settings are not correct?
Yep... This I know Scott. But, I didn't have any issues till the move. Anyone know which COOKIE need to be re-written? (I'm not deleting all of them)



Hi Sid, you're using moorephoto.smugmug.com ? I cannot replicate this behavior.
It's www.mooreFoto.smugmug.com Andy...

[Andy]

Yep... This I know Scott. But, I didn't have any issues till the move. Anyone know which COOKIE need to be re-written? (I'm not deleting all of them)




It's www.mooreFoto.smugmug.com Andy...
Gaa! And I called you "Sid" not Seymore :) :bow sorry about that....

[Seymore]

Gaa! And I called you "Sid" not Seymore :) :bow sorry about that....
It's alright Greg. :huh No worries! :thumb

So, Any ideas on which cookies I need to deal with? I did find, and DEL, the following: seymore@smugmug[1].txt & seymore@moorefoto.smugmug[2].txt. However the moorefoto.smugmug has not reappeared. :cry Any thoughts for me Andy.



ps: I know you're a MAC guy. But let me know if you have any thoughts.

[mbrady]

Go sign up for Google Analytics.

Dna

Analytics (and others like them) can't track hits directly to a photo's URL.

[Andy]

It's alright Greg. :huh No worries! :thumb

So, Any ideas on which cookies I need to deal with? I did find, and DEL, the following: seymore@smugmug[1].txt & seymore@moorefoto.smugmug[2].txt. However the moorefoto.smugmug has not reappeared. :cry Any thoughts for me Andy.



ps: I know you're a MAC guy. But let me know if you have any thoughts.
Well, why not just delete all your smugmug.com cookies? :ear

[Seymore]

Well, why not just delete all your smugmug.com cookies? :ear
DISREGARD GREG! Errrr.... Andy. :D Looks like another re-boot resolved my current issues with SM. Thanks for the comm on this. :thumb

And, I know what you ment. But, you're the first to call me Sid. :huh

[Andy]

ps: I know you're a MAC guy. But let me know if you have any thoughts.

No worries - I'm a MAC NUT! But, like all of us here at SmugMug, I run Windows and IE6 too, so that we can support our customers :D

[dragon300zx]

No worries - I'm a MAC NUT! But, like all of us here at SmugMug, I run Windows and IE6 too, so that we can support our customers :D

Likely story. Your just a closet Wintel fanatic and you know it.

[Andy]

Likely story. Your just a closet Wintel fanatic and you know it.
:lol3

[Seymore]

Likely story. Your just a closet Wintel fanatic and you know it.
:lol3 :lol3 :lol3



No worries - I'm a MAC NUT! But, like all of us here at SmugMug, I run Windows and IE6 too, so that we can support our customers :D
Thanks again Andy... Glad to have you here.

[Baldy]

But the intentions were for nothing but the best.
As I said before, lessons to be learned. Not everyone is perfect.The biggest lesson I learned was something we should have known: we need opt-in email.

Great intentions are ironically what kept us from being able to see it; the last thing we wanted was to be like the annoying spammers our customers rail against.

We've heard so many times how much our customers hate receiving unsolicited email and how horrified they are to learn that at other photo sharing sites their visitors have to register, only to be spammed...

Anyway, we're working on opt-in email now so this doesn't happen again.

Thanks,
Baldy

[dogwood]

Anyway, we're working on opt-in email now so this doesn't happen again.

Thanks,
Baldy

This is the answer many, many of us are excited to hear! Thank you for listening. We all hate spam as much as the SM folks, but I'd hardly call an outage notification e-mail a "spam". Especially if I ask to recieve it!

[BenA2]

The biggest lesson I learned was something we should have known: we need opt-in email.

Isn't this precisely what "Message Center" was intended for? I thought the idea was to use message center to broadcast site info without creating the illusion of spam. I think it still should be considered as an option among "Opt Out" and "E-mail notification".

[flyingdutchie]

Isn't this precisely what "Message Center" was intended for? I thought the idea was to use message center to broadcast site info without creating the illusion of spam. I think it still should be considered as an option among "Opt Out" and "E-mail notification".

That would work, but not all the time.
If i'm not online all the time, but do check my e-mail from time to time, i get to read the e-mail even when (during) the system is down. When the system is down, i can not read messages in the message center...

[Mike Lane]

That would work, but not all the time.
If i'm not online all the time, but do check my e-mail from time to time, i get to read the e-mail even when (during) the system is down. When the system is down, i can not read messages in the message center...:agree Also my computer automagically tells me when I've got new mail. Plus I don't consider opt-in notifications of downtime, updates, and other info that could affect my site's status spam. I would probably even opt in to a monthly newsletter if there was one.

I'm just saying...

[mbrady]

The biggest lesson I learned was something we should have known: we need opt-in email.

Great intentions are ironically what kept us from being able to see it; the last thing we wanted was to be like the annoying spammers our customers rail against.

We've heard so many times how much our customers hate receiving unsolicited email and how horrified they are to learn that at other photo sharing sites their visitors have to register, only to be spammed...

Anyway, we're working on opt-in email now so this doesn't happen again.

Thanks,
Baldy

Here's a question to make your head spin:
For those who don't read dgrin or check release notes or anything like that and were blindsided by the maintenance outage, how will they know that there's a way to opt-in to get notification emails like that unless you send them an email saying that the opt-in is available?
:scratch:wink

[flyingdutchie]

Here's a question to make your head spin:
For those who don't read dgrin or check release notes or anything like that and were blindsided by the maintenance outage, how will they know that there's a way to opt-in to get notification emails like that unless you send them an email saying that the opt-in is available?
:scratch:wink
:rofl
How would they know there is such a thing as Smugmug, uh... i mean the internet, at all!!!! ????

[saurora]

WOW! I just viewed some galleries and was I impressed! The new conversion is really smokin'!!! I was able to fly through the galleries faster than a flying...well, you know!!! Thanks guys!!!!:clap :clap :clap

Question: Is the update officially over and is everyone back up and running???? The reason I ask is that a Dgrinner I communicate with (via PM) their site is entirely gone, they are no longer listed in the members roster at Dgrin, or Daily Photo, etc. I have no other means of reaching this member and was curious if there are still some problems???? :dunno

[Andy]

WOW! I just viewed some galleries and was I impressed! The new conversion is really smokin'!!! I was able to fly through the galleries faster than a flying...well, you know!!! Thanks guys!!!!:clap :clap :clap

Question: Is the update officially over and is everyone back up and running???? The reason I ask is that a Dgrinner I communicate with (via PM) their site is entirely gone, they are no longer listed in the members roster at Dgrin, or Daily Photo, etc. I have no other means of reaching this member and was curious if there are still some problems???? :dunno
The move is finished and all is back to normal production. Nobody's site was lost - perhaps they changed their site name? What was the site?

[rainforest1155]

The move is finished and all is back to normal production. Nobody's site was lost - perhaps they changed their site name? What was the site?
I believe Susan meant the page of Sayntbrigid. She's still in my friends list, but with a strange looking username (0985sayntbrigid) and when I try to access the new username it only gets redirected to the smugmug homepage. :dunno

I've already contacted her. Is her account completly gone? Maybe someone got access to it and closed it?

Thanks for looking into it,
Sebastian

[saurora]

Hi Sebastian...yes, I was looking for Sayntbrigid...

[Andy]

I believe Susan meant the page of Sayntbrigid. She's still in my friends list, but with a strange looking username (0985sayntbrigid) and when I try to access the new username it only gets redirected to the smugmug homepage. :dunno

I've already contacted her. Is her account completly gone? Maybe someone got access to it and closed it?

Thanks for looking into it,
Sebastian
It appears she has closed her account.

[rainforest1155]

It appears she has closed her account.
Thanks for letting me know. It's strange - hopefully she's getting my mail.

Sebastian

[saurora]

Thank you Andy for letting us know.

[DavidTO]

Thank you Andy for letting us know.


A google search for sayntbrigid takes you to her myspace page...