Baldy
Jun-01-2006, 01:20 PM
First, a disclaimer :D : I haven't had time to go through these threads because I've been immersed in upload issues, but for people who want to hear my personal perspective (which could be different from others at SmugMug), in the spirit of openness, here goes:
1. We installed new database servers some months back and most people loved the performance boost. They were big, gnarly, power-hungry, and expensive -- but worth it.
With the stunning growth in images, users and the increasing demands new features place on database servers, it was surprising how quickly we began to load those servers. So we went to order a lot of new ones.
The issue is the manufacturer was on the verge of releasing the new hotness. You know the story -- faster processors & disks & more memory -- much better. The question was to settle for the old or hold for the new.
We held a few more weeks to get the new hotness for the long term good.
2. Memorial weekend happened. Aaaiiieee we got hit with volume. And if an upload takes time to process, customers automatically upload it again (and again), causing the situation to spiral.
3. Something happened with uploaders that I'm not sure we completely understand because we can't replicate it on our test servers, only on the live site. That suggested to me that it was load-related. It was very hard to diagnose because for the customers who were reporting the issue, it generally wasn't repeatable and was sometimes mixed in with Comcast's packet loss problems or switching to a new uploader.
We knew a significant part had to do with us and we've been immersed in what it could be. I get the feeling that today has been good for uploading, no? So perhaps we've got it and Don may be able to say with authority what it was.
We're proceeding with the new hotness servers as quickly as the manufacturer can get them to us. We're sickened by the problems some of you saw and hope never to see them again.
Back to work for me and a hundred lashes.
:whip:whip:whip
Thanks,
Chris
1. We installed new database servers some months back and most people loved the performance boost. They were big, gnarly, power-hungry, and expensive -- but worth it.
With the stunning growth in images, users and the increasing demands new features place on database servers, it was surprising how quickly we began to load those servers. So we went to order a lot of new ones.
The issue is the manufacturer was on the verge of releasing the new hotness. You know the story -- faster processors & disks & more memory -- much better. The question was to settle for the old or hold for the new.
We held a few more weeks to get the new hotness for the long term good.
2. Memorial weekend happened. Aaaiiieee we got hit with volume. And if an upload takes time to process, customers automatically upload it again (and again), causing the situation to spiral.
3. Something happened with uploaders that I'm not sure we completely understand because we can't replicate it on our test servers, only on the live site. That suggested to me that it was load-related. It was very hard to diagnose because for the customers who were reporting the issue, it generally wasn't repeatable and was sometimes mixed in with Comcast's packet loss problems or switching to a new uploader.
We knew a significant part had to do with us and we've been immersed in what it could be. I get the feeling that today has been good for uploading, no? So perhaps we've got it and Don may be able to say with authority what it was.
We're proceeding with the new hotness servers as quickly as the manufacturer can get them to us. We're sickened by the problems some of you saw and hope never to see them again.
Back to work for me and a hundred lashes.
:whip:whip:whip
Thanks,
Chris