Options

11:49 ET - Is Smug MugDown? Outage Update

1356715

Comments

  • Options
    kygardenkygarden Registered Users Posts: 1,060 Major grins
    edited July 20, 2008
    Andy wrote:
    Yup, it does. Cancel the auction, and relist?

    Nah...most of this stuff is not high dollar and it eats up enough of my time to get it on there in the first place (selling off some of the wife's collectables - her idea)...but it's usually good for the bidder to be able to see a large version of the photo instead of the 1 free ebay photo so they know we're selling good stuff with no chips/cracks, etc. While I may not get as much as I hoped, it's still not enough to get me to contact all the bidders and re-list 7 or 8 auctions. They may think I'm up to some funny business for one thing.
  • Options
    pabsterpabster Registered Users Posts: 17 Big grins
    edited July 20, 2008
    rolii wrote:
    Please let us know when and if you do find that perfect, 0% downtime photo hosting service.

    Maybe not a fair comparison, but I can't recall a single time when Google's PicasaWeb has been out of service. And certainly not for hours on end. YMMV.
  • Options
    PHOTOlinkPHOTOlink Registered Users Posts: 150 Major grins
    edited July 20, 2008
    System outage of 7-20
    yaknow.. i just can't help but laugh at some of the over-reactivness (of which I used to be a vivid part of too) about this situation....

    about a year ago i stopped dumping on SM for just about everything, because although there are and will be continued problems, they have never let us down in terms of customer support, never mind for a moment if the fault is theirs or not....

    those of you that become spoiled by all the uptime they have given us shows how quickly we forget how responsive they are. Anyone who doesn't really understand the nature of the the internet and servers, or anyone who does but is in denial of it's inherent soft areas that make up it's underbelly, is plain just not being objective and fair to SM. After a recent upgrade, I found a problem with the page that lets one pick the order of the galleries. I wrote the help desk ONCE, they told me it was a bug, and although it took awhile, and was indeed a pain-in-the-neck to have happen, they did fix it, and i purposely didn't write again because i learned there is no need to. if they know there is a problem, it get's fixed.. perhaps not as quickly as we would like, but it DOES get done...... boys & girls, give SM a break and remember to appreciate the fact that they don't take their customers for granted....
  • Options
    DmitrySDmitryS Registered Users Posts: 29 Big grins
    edited July 20, 2008
    CSwinton wrote:
    When you find that perfect service let us know. thumb.gif
    I have been smugmug customer for several years and I can say for sure quality of serve became worse:
    - Email support. Yes if you write to help@smugmug.com you will get an asnwer very quick. However chances are it wil not be helpfull at all. So far, I have see two types of replies: "I will pass this info to our engineers" or "We wish we had resources to implement it ". It both cases it sounds like "get lost"
    - dgrin.com. Look at the "features suggestion" thread. It used to be a place where smugmugers could leave feedback and discuss possible impovements wiht Smugmug engeneers. Now it's a list of repeating suggestions and every 20th post is a message from Andy "Thanks for the suggestion clap.gif".
    - S3 storage. Guys, it was your decision to start storing data on S3. And it is unprofessional to blame amazon for your problems. I do remember several years ago you were praising your data center, then couple of years latter your were praising amazon's S3.
    Unfortunately it seems that Smugmug put it's resource into marketing not technology.
    While there are a lot of groupies that will praise smugmug here whatever happens, don't forget alternatives do exist: entry level users can go with fotki.com that provide phone support, users who need uninterrupted service will choose picasa.
  • Options
    RacetogRacetog Registered Users Posts: 37 Big grins
    edited July 20, 2008
    Anyone have an idea of when SM will be back online? Need to upload photos of a race I shot last night...
  • Options
    kevinchamplinkevinchamplin Registered Users Posts: 7 Beginner grinner
    edited July 20, 2008
    I run a web solutions company and host many sites. Stuff like this happens, it sucks but life goes on.

    I've only been using smugmug for a week or so now, but so far have been very pleased with the ease of use and the options available. I'm not about to let one outage shake me. Andy hads been very responsive and straight forward with the issue.

    Lets pray the service comes up soon, but until then go enjoy the day...its beautiful outside :)
  • Options
    pabsterpabster Registered Users Posts: 17 Big grins
    edited July 20, 2008
    DmitryS wrote:
    I have been smugmug customer for several years and I can say for sure quality of serve became worse:
    - Email support. Yes if you write to help@smugmug.com you will get an asnwer very quick. However chances are it wil not be helpfull at all. So far, I have see two types of replies: "I will pass this info to our engineers" or "We wish we had resources to implement it ". It both cases it sounds like "get lost"
    - dgrin.com. Look at the "features suggestion" thread. It used to be a place where smugmugers could leave feedback and discuss possible impovements wiht Smugmug engeneers. Now it's a list of repeating suggestions and every 20th post is a message from Andy "Thanks for the suggestion clap.gif".
    - S3 storage. Guys, it was your decision to start storing data on S3. And it is unprofessional to blame amazon for your problems. I do remember several years ago you were praising your data center, then couple of years latter your were praising amazon's S3.
    Unfortunately it seems that Smugmug put it's recurces into marketing, instead of technology.
    While there are a lot of groupies that will praise smugmug here whatever happens, don't forget alternatives do exist: entry level users can go with fotki.com that provide phone support, users who need uniterupted service will choose picasa.

    Tough, but to the point. I agree wholeheartedly, particularly on the S3 thing. Running around blaming third parties is rather unprofessional in my opinion. I love SmugMug (I really do!) and hell I've only been using it a few days. I'm willing to continue but I sure do hope that this is a wake-up call to action so that next time the service interruption is momentary rather than momentous.
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited July 20, 2008
    DmitryS wrote:
    I have been smugmug customer for several years and I can say for sure quality of serve became worse:
    - Email support. Yes if you write to help@smugmug.com you will get an asnwer very quick. However chances are it wil not be helpfull at all. So far, I have see two types of replies: "I will pass this info to our engineers" or "We wish we had resources to implement it ". It both cases it sounds like "get lost"
    Wow, this is not what we want at all. I'm going to personally investigate your emails to our help desk, so that I can take any corrective action as needed.
    - dgrin.com. Look at the "features suggestion" thread. It used to be a place where smugmugers could leave feedback and discuss possible impovements wiht Smugmug engeneers. Now it's a list of repeating suggestions and every 20th post is a message from Andy "Thanks for the suggestion clap.gif".
    Hm. I'll disagree with your strongly here, Dmitri. It's still a place where you can bring up and discuss improvements and we ALL , the whole company, read it. And we act. My posts thanking for the suggestions, well now, what exactly would you have me do, give a Bronx Cheer? headscratch.gif We also act on tons of customer feature requests. TONS. Keep in mind, probably only 20% of our FRs come to us via the Dgrin feature request thread, Dmitri :)
    - S3 storage. Guys, it was your decisions to start storing data on S3. And it is unprofessional to blame amazon for your problems. I do remember several years ago you were praising your data center, then couple of years latter your were praising amazon's S3.
    Unfortunately it seems that Smugmug put it's recurces into marketing, instead of technology.
    While there are a lot of groupies that will praise smugmug here whatever happens, don't forget alternatives do exist: entry level users can go with fotki.com that provide phone support, users who need uniterupted service will choose picasa.
    We're not blaming anyone. The fact that we are down is because Amazon is down. But in the end, its' SmugMug that is down, and we've never ever shied away from taking responsibility for our actions.

    I'm sorry today's outage sucked so bad. Very sorry. But I'm thrilled with how good our track record is, sustained and over the years, at being extremely available and protection our customers' priceless photos. I'll stack it up against any other commercial service.

    Thanks Dmitry for posting, and telling it like it is - we'd not want it any other way.
  • Options
    edmaineedmaine Registered Users Posts: 22 Big grins
    edited July 20, 2008
    any ETA yet from Amazon when the service can be restored???
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited July 20, 2008
    DmitryS wrote:
    I have been smugmug customer for several years and I can say for sure quality of serve became worse:
    - Email support. Yes if you write to help@smugmug.com you will get an asnwer very quick. However chances are it wil not be helpfull at all. So far, I have see two types of replies: "I will pass this info to our engineers" or "We wish we had resources to implement it ". It both cases it sounds like "get lost"
    Hi Dmitry, I went and looked, and only see one single email from you back in January 08. You had asked for a feauture request about more than one featured photo per gallery. And I replied to you - saying thanks and that we'd shared your request amongst the team. I also said that we reqret not being able to implement every single feauture request that comes in. I did pass your request around and no, we've not implemented it.

    I'm very, very sorry Dmitry.

    If there are other emails that I'm not aware of, please let me know by writing to me, ATTN: Andy.
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited July 20, 2008
    Racetog wrote:
    Anyone have an idea of when SM will be back online? Need to upload photos of a race I shot last night...
    No, I'm very sorry. Updates here:
    http://smugmug.wordpress.com/
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited July 20, 2008
    edmaine wrote:
    any ETA yet from Amazon when the service can be restored???
    No, I'm very sorry. Updates here:
    http://smugmug.wordpress.com/
  • Options
    MikeMcA²MikeMcA² Registered Users Posts: 177 Major grins
    edited July 20, 2008
    I'm staying with SM through this, and I'm very happy that my SM galleries have allowed me to make a few thousand dollars this year which I likely would not have made otherwise. And you guys are way smarter than I am with all this web-hosting stuff (but then I pay you to be).

    But I do have a small bit of concern when I read "we've been waiting for this and expect more in the future." What the...? I am also with a Tier I wireless provider, and redundancy and failover are the prime directive with us. Can Andy or anyone explain why "we've been waiting for this and expect more in the future"?

    I trust you guys with my images and know it will all be resolved, but would just like a little reassurance that this trust is well-placed. And yes, I understand that this place has been great and very reliable in the last year of my membership, but we've all got to face the fact that it's the 21st Century and the age of instant gratification, and more than any other time in our history the customer's credo is "What have you done for me lately?"
  • Options
    edmaineedmaine Registered Users Posts: 22 Big grins
    edited July 20, 2008
    Andy wrote:
    No, I'm very sorry. Updates here:
    http://smugmug.wordpress.com/

    Thanks, i check that and Amazon's status board, but no word on either, when the service can be restored. :help

    not good.
  • Options
    kriskroskriskros Registered Users Posts: 29 Big grins
    edited July 20, 2008
    This is MAJOR INTERRUPTION
    Andy wrote:
    We're not blaming anyone. The fact that we are down is because Amazon is down. But in the end, its' SmugMug that is down, and we've never ever shied away from taking responsibility for our actions.
    Andy, I think everyone wants now an inside scoop of what is going on. Maybe you should tell us your communication with Amazon. We would love to know exactly when this will be over. It has been 4 hours now and many missed opportunities during those times. This cannot be classified as temporary interruption. This is major interruption.

    Still, I believe. But get those photos back on service. People can say good things about SmugMug. Hard not to. But deep in their hearts, they know that this is a big business failure by SmugMug.

    BTW, Amazon is not really down. It's just their servers for SmugMug that are down. Amazon.com is still doing business.
  • Options
    swintonphotoswintonphoto Registered Users Posts: 1,664 Major grins
    edited July 20, 2008
    Everyone cool your jets. They are working on it. I have faith in Smugmug. I am sure they are doing all they can.
    Thanks for working so hard on the weekend to get this resolved.
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited July 20, 2008
    kriskros wrote:
    Andy, I think everyone wants now an inside scoop of what is going on. Maybe you should tell us your communication with Amazon. We would love to know exactly when this will be over
    I don't mean to be cute but it'll be over when it's over. I'm sorry I don't have faster, better, more precise answer than that :(

    It sucks and we're doing everything in our power to bring things back to normal as fast as we can.
  • Options
    TallahasseeTrailsTallahasseeTrails Registered Users Posts: 11 Big grins
    edited July 20, 2008
    Amazon S3 Update timeline
    Hopefully they're getting close.
    From http://status.aws.amazon.com/

    9:05 AM PDT
    We are currently experiencing elevated error rates with S3. We are investigating.
    9:26 AM PDT We're investigating an issue affecting requests. We'll continue to post updates here.
    9:48 AM PDT Just wanted to provide an update that we are currently pursuing several paths of corrective action.
    10:12 AM PDT We are continuing to pursue corrective action.
    10:32 AM PDT A quick update that we believe this is an issue with the communication between several Amazon S3 internal components. We do not have an ETA at this time but will continue to keep you updated.
    11:01 AM PDT We're currently in the process of testing a potential solution.
    11:22 AM PDT Testing is still in progress. We're working very hard to restore service to our customers.
    11:45 AM PDT We are still in the process of testing a series of configuration changes aimed at bringing the service back online.
    12:05 PM PDT We have now restored communication between a small subset of hosts. We are working on restoring internal communication across the rest of the fleet. Once communication is fully restored, then we will work to restore request processing.
    12:25 PM PDT We have restored communication between additional hosts and are continuing this work across the rest of the fleet. Thank you for your continued patience.
    12:51 PM PDT The restored hosts are stable and we are moving forward in restoring communication between additional hosts.

    While we're waiting, further reading about S3:
    http://en.wikipedia.org/wiki/Amazon_Simple_Storage_Service
    Note Smugmug mention, under the "Notable Uses" header:
    "Photo hosting service SmugMug has used S3 since April 2006. In November 2006, they claimed to be saving US$500,000 per year by using S3 instead of their own dedicated storage servers.<sup id="cite_ref-smugmug-smtm_5-0" class="reference">[6]</sup> SmugMug has noted some problems with S3<sup id="cite_ref-smugmug-problems_6-0" class="reference">[7]</sup> but continued to use the service and as of April 30, 2007, had saved almost $1 million in storage costs."
    That'll send you to Don MacAskill's blog posting:
    http://blogs.smugmug.com/don/2007/01/30/amazon-s3-outages-slowdowns-and-problems/
    That's enough. Hopefully, in the time it's taken me to type this, things are close to being back online.
    jk
  • Options
    wxwaxwxwax Registered Users Posts: 15,471 Major grins
    edited July 20, 2008
    DmitryS wrote:
    - S3 storage. Guys, it was your decision to start storing data on S3. And it is unprofessional to blame amazon for your problems.

    I think that's a very uncharitable interpretation.

    Smugmug's error page is informative and factually accurate. To characterize it as "blaming Amazon" isn't helpful. Would you have the error page be less informative? Remove the link to Amazon's Dashboard and its service updates? Kill the link to this thread?

    A major service interruption sucks. The class and concern the Smugmug team brings to pretty much everything they do, does *not* suck. Not at all.
    Sid.
    Catapultam habeo. Nisi pecuniam omnem mihi dabis, ad caput tuum saxum immane mittam
    http://www.mcneel.com/users/jb/foghorn/ill_shut_up.au
  • Options
    kriskroskriskros Registered Users Posts: 29 Big grins
    edited July 20, 2008
    Andy wrote:
    I don't mean to be cute but it'll be over when it's over. I'm sorry I don't have faster, better, more precise answer than that :(

    It sucks and we're doing everything in our power to bring things back to normal as fast as we can.

    Thanks Andy. My friends on flickr are just wondering why my site is down. But still, I am a believer.
  • Options
    Awais YaqubAwais Yaqub Registered Users Posts: 10,572 Major grins
    edited July 20, 2008
    1st time this is happening in 4 years of smooth service. There is always room for problems in every form of services you cannot completely avoid them.

    And i have seen requested features are introduced.

    On top of all everyone at smugmug is Photographer that also helps customers mwink.gif.
    Look at great photography meetups, workshops and this great community, this is all because of SmugMug and their love for Photography and photos.

    I bet no one at Picasa knows how to correct skin tones rolleyes1.gif
    Just kidding don't drag it into debate of SM vs Picasa lol
    Thine is the beauty of light; mine is the song of fire. Thy beauty exalts the heart; my song inspires the soul. Allama Iqbal

    My Gallery
  • Options
    bhambham Registered Users Posts: 1,303 Major grins
    edited July 20, 2008
    From Amazon, looks like they got a solution and slowly bringing things back to normal.


    9:05 AM PDT We are currently experiencing elevated error rates with S3. We are investigating.
    9:26 AM PDT We're investigating an issue affecting requests. We'll continue to post updates here.
    9:48 AM PDT Just wanted to provide an update that we are currently pursuing several paths of corrective action.
    10:12 AM PDT We are continuing to pursue corrective action.
    10:32 AM PDT A quick update that we believe this is an issue with the communication between several Amazon S3 internal components. We do not have an ETA at this time but will continue to keep you updated.
    11:01 AM PDT We're currently in the process of testing a potential solution.
    11:22 AM PDT Testing is still in progress. We're working very hard to restore service to our customers.
    11:45 AM PDT We are still in the process of testing a series of configuration changes aimed at bringing the service back online.
    12:05 PM PDT We have now restored communication between a small subset of hosts. We are working on restoring internal communication across the rest of the fleet. Once communication is fully restored, then we will work to restore request processing.
    12:25 PM PDT We have restored communication between additional hosts and are continuing this work across the rest of the fleet. Thank you for your continued patience.
    12:51 PM PDT The restored hosts are stable and we are moving forward in restoring communication between additional hosts.
    "A photo is like a hamburger. You can get one from McDonalds for $1, one from Chili's for $5, or one from Ruth's Chris for $15. You usually get what you pay for, but don't expect a Ruth's Chris burger at a McDonalds price, if you want that, go cook it yourself." - me
  • Options
    TonyLTonyL Registered Users Posts: 169 Major grins
    edited July 20, 2008
    As a new Dgrinner...
    I can't believe the outage. I found out from someone whose trial is about to expire and this sent him back to flickr.

    Figures, I just upgraded to the pro account, advertised in the local paper, etc. have business cards printed, alloted time to work on the site today, but instead I signed up here.thumb.gif

    I have used you guys for years and have faith you will get this up soon.
    -Anthony

    APL Photography || My Gear: Bunch of 4/3rds stuff
    Facebook: Friend / Fan || Twitter: @aplphoto
  • Options
    DrDavidDrDavid Registered Users Posts: 1,292 Major grins
    edited July 20, 2008
    cadguru wrote:
    I can't believe the outage. I found out from someone whose trial is about to expire and this sent him back to flickr.

    Figures, I just upgraded to the pro account, advertised in the local paper, etc. have business cards printed, alloted time to work on the site today, but instead I signed up here.thumb.gif

    I have used you guys for years and have faith you will get this up soon.
    Flickr has gone down before too FWIW...

    David
  • Options
    pecheneypecheney Registered Users Posts: 48 Big grins
    edited July 20, 2008
    Hahahahahahah
    I am in awe. I run about 8-10 websites with smugmug and vbulletin. I have about 250,000 photos online all with smugmug. Wow. This discussion board is letting people say too much. Smugmug rocks. Big deal. site went offline. it happens. relax. it will be back up momentarily. there is no better script out there. not even close. look at the positive. i have been using sm for maybe 4 years now it seems. i have a problem. I email. i get an answer. surreal customer service. I envy their customer service. it is a model for any of our companies to follow. try it. send en email to help@smugmug.com and ask for something. you will get a response. it will be helpful. i've had sm staff call my clients and my customers. how many dotcoms are on top of their game like this? the site is super flexible. almost flawless. every new thing they add to it is well thought out and very user friendly. I am a former UI engineer. They are doing one of the best jobs i have ever seen for UI design. It is SO user friendly now and always tweaking better and better. Big deal. SM is so tight if they went offline for a couple days I would still chill. And that is what you should do. chill. rolleyes1.gif
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited July 20, 2008
    BTW to those asking us to do more:
    SmugMug wrote:
    We’ve already spent the last few months investigating additional resources we can combine with Amazon to cover outage scenarios like the one we’re experiencing today, and have some promising prospects.

    From our Status Blog. http://smugmug.wordpress.com/
  • Options
    presencepresence Registered Users Posts: 54 Big grins
    edited July 20, 2008
    I'm a big fan of Smugmug and I know it's Amazon's fault but it's been several hours now with the outage.

    I'd be a bit less concerned if I knew that other commercial websites were affected as well as SM. Amazon.com doesn't seem to be down. Also, looking at the smugmug wordpress blog, the problems seem to be escalating over the past few months, from the odd post per month to 9 this month.

    Every website suffers downtime but it's been 3 hours for me headscratch.gif
  • Options
    iSparkiSpark Registered Users Posts: 119 Major grins
    edited July 20, 2008
    bham wrote:
    "A photo is like a hamburger. You can get one from McDonalds for $1, from Chili's for $5, or from Ruth's Chris for $15. You usually get what you pay for, but don't expect a Ruth's Chris burger at a McDonalds price, if you want that, go cook it yourself." - me

    Not here to rag SM, I do wish my site was online though. :)
    I just wanted to say I like that tag!!

    I wonder if bham would let me use it...lol
  • Options
    pabsterpabster Registered Users Posts: 17 Big grins
    edited July 20, 2008
    kriskros wrote:
    BTW, Amazon is not really down. It's just their servers for SmugMug that are down. Amazon.com is still doing business.

    Of course not. We'd be talking literally millions of dollars in losses if Amazon.com were down for an hour, let alone several hours. And, incidentally, they *have* had a few such outages in the past month-odd. Clearly S3 isn't the priority that Amazon.com is.
  • Options
    gman33gman33 Registered Users Posts: 279 Major grins
    edited July 20, 2008
    I am a true fan of SM and love the service they have been providing but a simple search on Google for "Amazon S3 technology" and just the first page is full of nightmares. I know that it saves SM money but money can never bring back a reputation.

    The only reason I am upset now because I had a huge potential client wanting to check out my pictures today and they got the error message..I know, bad timing, and I explained to them that this has not happened for several years.

    I am really hoping that SM changes from S3...probably the reason why things are slow sometimes...I do not want to look anywhere else to host my pictures and I know it is Amazon's issue but the trickle effect is becoming a flood.

    I know technology is not perfect but S3 has had so many issues for so long, I think the game plan needs to change and the "partnership" broken. I am keeping the faith and I know everything is in panic mode now.

    Just hope it is back online soon...the love is still there for SM...but not S3 ne_nau.gif

    Thanks
    Ed G - Philadelphia, PA
    http://ergphoto.smugmug.com
Sign In or Register to comment.