Caveat: I'm not from Smugmug, My opinions are my own, I don't know how Smugmug have build they're data architectures. I mearly hope to offer advice based on best guesses and hope they're helpful. Please don't consider them conclusive.
If re-writing smugmug search functionality is not possible because of complexity and volume issues, can we at least hide keywords and captions from Google searches via robot meta tags as mentioned in the previous post?
Some points to consider:
1. Robot tags are entirely optional, there is nothing to stop a robot completly ignoring them.
2. The 'standard' for Robot META Exclusion states:
You obviously should not specify conflicting or repeating directives such as:
[php]
<meta name="robots" content="INDEX,NOINDEX,NOFOLLOW,FOLLOW,FOLLOW">
[/php]
3. Google advocate the use of: <META NAME="Googlebot" CONTENT="nofollow">
4. Hiding from search engines such as Google without using strong protection such as access control is ultimately not going to succeed, you are too dependent on
- The whim of the search indexer, if it decides to follow your links and fake an IE 6 UserAgent, then there's not a lot you can do about it
- No body, anywhere on the internet linking to your site, ever (or these days, sending a link to anyone using a GMail account etc.)
In combination, the general advise to web developers is don't bother too hard about trying. Sorry... :uhoh
Does smugmug search rely upon the same robot crawler functionality as Google search?
I would say, that IMHO it was massively unlikely.
Crawlers are slow, nasty, messy, computationally and network expensive things to use. They also have a tendency to leave hole in semi-unconnected networks, need to be written very carefully to avoid dying in cyclic networks. In networks of shifting topology (the interconnected mess of webpages), they are not trivial things to write, if you want a good one...
Search engines only use Crawlers, because they have no other option. They can't ask for a structured database of all the information on a website.
By its very nature, Smugmug already has this, and indeed uses it to serve up the Smugmug pages. Hence there is little if any need for them to use crawlers, which would be costly, slow to update and generally a pain...
So I think it much more likely that they will build their search databases from their own production data, rather than re-crawling their own site to get the information.
That's how I think I'd do it anyway...
Having said which, if I was trying to build something the size of Smugmug, I'd probably give it more than the 2 minutes thought I just have, so I could well be wrong :
I'll just leave you with a final thought from Googlebot's description page:
6. Why is Googlebot downloading information from our "secret" web server?
It's almost impossible to keep a web server secret by not publishing any links to it. As soon as someone follows a link from your "secret" server to another web server, your "secret" URL may appear in the referrer tag and can be stored and published by the other web server in its referrer log. So, if there's a link to your "secret" web server or page on the web anywhere, it's likely that Googlebot and other web crawlers will find it.
Sorry I can't be more encouraging, hope this helps anyhow,
Luke
PS. Incidentally a good way of hiding from Google is by inducing the use of very common keywods, so that the search clauses will be swamped... Try googling for me to see what I mean: 'Luke Church'. http://www.google.com/search?hl=en&q=Luke+Church
The results are quite funny.
I think I've been very patient on this subject but I've seen only a little interest from Smugmug, and that was FOUR months ago. I like Smugmug and recommend it to friends but I feel that they're more interested in rolling out new features.
I to put it bluntly I want Smugmug to support the ability to turn off (1) search engine indexing and (2) searching of my photos from the main site or from other people's galleries. I would still want to be able to search my photos from within my gallery though.
I just don't want my stuff broadcast to the world and make it more isolated.
Does anyone else feel that same?
You only list two points, but there are really three, and the third one is the kicker:
1) No search engine indexing
2) No searching from the main site
3) But allowing people to search within your photos
It's #3 that's basically impossible today without a *lot* of work. I've been working on making #1 and #2 happen for quite awhile now, since our last discussion, but for some reason I either missed or misunderstood #3.
At this time, I just can't make #3 happen. I'd like to sometime in the future, if I can, but that's a long ways out, I'm afraid.
I am very interested in this topic as well. In this thread, others have well articulated my privacy issues/desires. I will only add that I believe this is a shared concern for many smugmug loyalists.
I don't want to encumber access with account passwords or private galleries. But I do want to localize the use of keywords and captions within my account (hide the visibility of keywords and captions from Google searches and smugmug global searches). This is what I call the "semi-private" option. I think this is what we are all talking about here.
Again, I hate to beat a dead horse, but this is a massive project. Keywords weren't built with this in mind, so they're not going to happen this way. I hate to say never, but this one is probably never.
"Searching only in your account" is possible, but a lot of work. I want to make this happen some day, but it's both not a high priority and a ton of work, meaning it's gonna be awhile.
Through experimentation, I have discovered that enabling an account password does not accomplish this goal. My keywords and captions are still visible to Google searches and smugmug global searches even with an account password enabled. An account password seems to be effective only for restricting access through the front door. Backdoor access remains wide open via direct URL from Google.
From what I can tell, hiding keywords and captions from Google searches and smugmug global searches is only possible by creating private galleries. For me, that kinda defeats the whole purpose of photo sharing.
Are you sure about this? If so, this is a *bug* and not expected behavior. Google cannot get to your pages to index them, so your pages will fall out of their index the next time they crawl.
As for smugmug, we check for user passwords and drop your results automagically from the search engine. (At least, that's what it's supposed to do and what we tested it to do. Bugs do happen, though .
Given this unfortunate reality, I am very interested to learn about any partial solutions. If re-writing smugmug search functionality is not possible because of complexity and volume issues, can we at least hide keywords and captions from Google searches via robot meta tags as mentioned in the previous post? Does smugmug search rely upon the same robot crawler functionality as Google search? Let's keep this discussion going please to learn how Google cloaking might be accomplished.
Robot tags would remove Google, yes, but we have a tag conflict problem.
As I stated in my last post, I'm working on making it so you can have Google not index your gallery pages on a gallery-by-gallery basis. Also allowing public galleries which do not show up in smugmug's search.
But the side effects are that keywords will cease to function in those galleries as well.
Caveat: I'm not from Smugmug, My opinions are my own, I don't know how Smugmug have build they're data architectures. I mearly hope to offer advice based on best guesses and hope they're helpful. Please don't consider them conclusive.
Some points to consider:
1. Robot tags are entirely optional, there is nothing to stop a robot completly ignoring them.
2. The 'standard' for Robot META Exclusion states:
You obviously should not specify conflicting or repeating directives such as:
[php]
<meta name="robots" content="INDEX,NOINDEX,NOFOLLOW,FOLLOW,FOLLOW">
[/php]
3. Google advocate the use of: <META NAME="Googlebot" CONTENT="nofollow">
4. Hiding from search engines such as Google without using strong protection such as access control is ultimately not going to succeed, you are too dependent on
- The whim of the search indexer, if it decides to follow your links and fake an IE 6 UserAgent, then there's not a lot you can do about it
- No body, anywhere on the internet linking to your site, ever (or these days, sending a link to anyone using a GMail account etc.)
In combination, the general advise to web developers is don't bother too hard about trying. Sorry... :uhoh
I would say, that IMHO it was massively unlikely.
Crawlers are slow, nasty, messy, computationally and network expensive things to use. They also have a tendency to leave hole in semi-unconnected networks, need to be written very carefully to avoid dying in cyclic networks. In networks of shifting topology (the interconnected mess of webpages), they are not trivial things to write, if you want a good one...
Search engines only use Crawlers, because they have no other option. They can't ask for a structured database of all the information on a website.
By its very nature, Smugmug already has this, and indeed uses it to serve up the Smugmug pages. Hence there is little if any need for them to use crawlers, which would be costly, slow to update and generally a pain...
So I think it much more likely that they will build their search databases from their own production data, rather than re-crawling their own site to get the information.
That's how I think I'd do it anyway...
Having said which, if I was trying to build something the size of Smugmug, I'd probably give it more than the 2 minutes thought I just have, so I could well be wrong :
I'll just leave you with a final thought from Googlebot's description page:
Sorry I can't be more encouraging, hope this helps anyhow,
Luke
PS. Incidentally a good way of hiding from Google is by inducing the use of very common keywods, so that the search clauses will be swamped... Try googling for me to see what I mean: 'Luke Church'. http://www.google.com/search?hl=en&q=Luke+Church
The results are quite funny.
Great points, Luke.
Let me point out, though, that all of the major search engines *do* adhere strictly to the META tags. I'm not aware of one that doesn't.
They have to, otherwise admins block their sites on their routers and they lose the ability to crawl at all.
So don't get all paranoid that your META tags will be ignored - they won't be.
Oh, and the only reason Google recommends putting "Googlebot" in the META tag is if you'd like to only restrict Google and not Yahoo/MSN/etc.
Let me point out, though, that all of the major search engines *do* adhere strictly to the META tags. I'm not aware of one that doesn't.
Agreed. Though I don't know how they behave with tag conflicts?
So don't get all paranoid that your META tags will be ignored - they won't be.
Sure, sorry I probably wasn't clear enough. My point is that META tags give 'privacy', they do not give 'security'. They will keep 99.99% of people away by hiding from major search engines, however they do not gaurentee anything.
But I agree, for this purpose they are fine.
If people want better than 99.99% then access protection is the only way to go, with its associated annoyances.
Oh, and the only reason Google recommends putting "Googlebot" in the META tag is if you'd like to only restrict Google and not Yahoo/MSN/etc.
I hate to say never, but this one is probably never. ""Searching only in your account" is possible, but a lot of work. I want to make this happen some day, but it's both not a high priority and a ton of work, meaning it's gonna be a while.
From the beginning of this thread (over six months ago), the idea of modifying smugmug search has been the focus of most of the discussion. I accept your conclusion that the effort required makes such smugmug search modifications improbable. I would rather shift the focus of the discussion to a partial solution that makes it possible to cloak keywords and captions from Google robot crawling for public galleries.
As I stated in my last post, I'm working on making it so you can have Google not index your gallery pages on a gallery-by-gallery basis. Also allowing public galleries which do not show up in smugmug's search. But the side effects are that keywords will cease to function in those galleries as well.
All sounds very good, except the part about disabling keywords - bummer. I love keywords and use them heavily.
Robot tags would remove Google, yes, but we have a tag conflict problem.
Instead of swallowing the whole elephant (modifying smugmug search), how about just a light snack? Perhaps something more simple that removes the meta tag conflict, and leaves alone the smugmug search and keywords functionality. Granted - it is not a complete solution to all the issues raised in this thread. However, it would be a relatively easy change that would address at least half of the privacy concerns. Just eliminating all the prying eyes that might stumble into my galleries from Google searches would be a big comfort. Sorry if I am talking like a simpleton - I don't mean to dismiss the complexities of such changes.
Again, I hate to beat a dead horse, but this is a massive project. Keywords weren't built with this in mind, so they're not going to happen this way. I hate to say never, but this one is probably never.
"Searching only in your account" is possible, but a lot of work. I want to make this happen some day, but it's both not a high priority and a ton of work, meaning it's gonna be awhile.
I know we're beating this but I'm still not convinced why this is not possible within a reasonable amount of work. We had discussed this months ago in this thread and I thought we were getting somewhere.
Is the resistance because there's a single database of public keywords that would require reworking?
Looking at it from my view I see that
1) If an album is made private, the keywords are not indexed in smugmug searches, and
2) I can already search only within my gallery for my photos without including anyone elses.
Why can search results not include photos that the users had made semi-private because they are not being searched from within that user's gallery? It appears that a possible solution could reside in a search results filter.
I urge Smugmug, again, to look at this with its user's privacy in mind and not to brush this off because "it's a lot of work" and "not a high priority." I'm sure with more analysis there's a solution that is meets both the requirements of the users without as much work as initially anticipated. I'm not trying to be overly anal about this but I'll bet that if enough Smugmug users were surveyed about this topic you would get a persuasive number of responses in the affirmative.
Through experimentation, I have discovered that enabling an account password does not accomplish this goal. My keywords and captions are still visible to Google searches and smugmug global searches even with an account password enabled. An account password seems to be effective only for restricting access through the front door. Backdoor access remains wide open via direct URL from Google.
Are you sure about this? If so, this is a *bug* and not expected behavior. Google cannot get to your pages to index them, so your pages will fall out of their index the next time they crawl.
Yep - I'm sure about the Google part. I just verified it again. It is very easy to get into our passworded smugmug account via backdoor URLs available from a Google search. If you are interested in learning more, I'll be happy to sit down with you for a few minutes to demonstrate.
As for smugmug, we check for user passwords and drop your results automagically from the search engine. (At least, that's what it's supposed to do and what we tested it to do. Bugs do happen, though .
My bad - you are correct on this point. Smugmug searches work exactly as your describe when an account password is set.
Don,
I urge Smugmug, again, to look at this with its user's privacy in mind and not to brush this off because "it's a lot of work" and "not a high priority." I'm sure with more analysis there's a solution that is meets both the requirements of the users without as much work as initially anticipated. I'm not trying to be overly anal about this but I'll bet that if enough Smugmug users were surveyed about this topic you would get a persuasive number of responses in the affirmative.
I totally agree, and let's be honest here, a lot of the new competitor photo sharing websites are probably reading this forum and throwing some cash at their site to implement this semi private style as we speak.
And I love smugmug and dont want to leave, but the full frontal or password being the only security option does bug the hell out of me.
I too would like to disable global search on my photos, but have public galleries that are visible to anyone if they go to my homepage. But I really want to still be able to use keywords on my photos.
If keeping the complexity down on the global search is the main concern (ie you don't want to add a "where allowSearch=true" clause), couldn't you have a separate database table for these photos' keywords?
I'm keen on smugmug because it has better privacy features than flickr, but not being able to have keyword search on photos if they aren't globally searchable is a bit of a killer!
Would it at least be feasible to give members the option to shut off all indexing in smugmug's meta tags? I have mostly password-protected albums, but had a few that I didn't protect also. This means that my smugmug home page gets indexed, which I'd prefer it not be. I saw the suggestion about putting in new meta tags in the customization page, but am not sure that that would work given that the first reference to robot tags is set to index, and a customized tag would simply create a conflict.
How about adding a choice to the account settings which would insert the no index / no follow robot tags into that member's pages regardless of whether or not they have password-protected albums?
I too would like to disable global search on my photos, but have public galleries that are visible to anyone if they go to my homepage. But I really want to still be able to use keywords on my photos.
If keeping the complexity down on the global search is the main concern (ie you don't want to add a "where allowSearch=true" clause), couldn't you have a separate database table for these photos' keywords?
I'm keen on smugmug because it has better privacy features than flickr, but not being able to have keyword search on photos if they aren't globally searchable is a bit of a killer!
Not a happy camper
I just spent hours and hours and hours applying keywords to all my photos only to realize that these keywords are going to be viewable by global search engines such as google, yahoo, msn, ect. not to mention to anyone who searches for photos through the smugmug homepage.
So to keep my privacy, my only option is to make all my galleries private through password protection, right? Well, that also makes my keywords on my homepage useless and a big pain in the a** for my friends and family who will have to search through my category, sub-category, gallery, password system to find photos that could be so easy to access through keywords.
This problem is a showstopper for me.
Basically, the way I see it, the only use for smugmug would be for pros or serious amateurs to sell or showcase "professional photos" that they don't hold as sacred and private. However, for someone who is looking to share photos with thier friends and family, there is a huge risk of advertising your private life (including the private life of your family and friends who are in your photos) to the whole world.
This is a serious issue.
I will cancel my account with smugmug if I do not see some serious progress to solve this problem ASAP.
And if you read through this thread, you'll see there are many other smugmug costomers who share my concerns.
Smugmug....you have a good reputation here....do not let this issue slide and jepordize the hard work you've accomplised to date with your relationships with your costomers.
Ouch, I'm so sorry you are not a happy camper when it comes to keywords.
I wish I had a better answer for you but I'm afraid there really isn't much smugmug can do in the near future to allow for keyword searching in private galleries without a complete redesign of the system. Making the changes to allow for the kind of functionality would be a very large and lengthy project.
I'm very sorry that there isn't more smugmug can do for you right now. We are discussing this feature and are very aware that this is an area that could use some improvement. Unfortunately, I don't know when we will be able to start and undertake a project this size.
I wish I had a better answer for you but I'm afraid there really isn't much smugmug can do in the near future to allow for keyword searching in private galleries without a complete redesign of the system. Making the changes to allow for the kind of functionality would be a very large and lengthy project.
John - with respect, I think you are missing the point of this thread. I don't think this group of concerned smugmuggers is asking for keyword searching of private galleries. What we want is to hide our public galleries and associated keywords/captions from Google searches and Smugmug global searches (while keeping keywords functional within a specific account). This is what we have been calling the "semi-private" option.
Over the course of this discussion, it has become clear that keyword functionality was designed specifically to enable global search capability. Some smugmug users however don't understand the global intent of keywords and desire to use them in a different manner (not to enable global audience access, but rather to facilitate local indexing of photos within a smugmug account). Keywords are a great piece of functionality, but Smugmug's intended use is not our desired use. This seems to be the sticking point.
John - with respect, I think you are missing the point of this thread. I don't think this group of concerned smugmuggers is asking for keyword searching of private galleries. What we want is to hide our public galleries and associated keywords/captions from Google searches and Smugmug global searches (while keeping keywords functional within a specific account). This is what we have been calling the "semi-private" option.
Over the course of this discussion, it has become clear that keyword functionality was designed specifically to enable global search capabilility. Some smugmug users however don't understand the global intent of keywords and desire to use them in a different manner (not to enable global audience access, but rather to facilitate local indexing of photos within a smugmug account). Keywords are a great piece of functionality, but Smugmug's intended use is not our desired use. This seems to be the sticking point.
Perfectly said gblotter. That is exactly what I want.
Perhaps, since smugmug has determined that this feature would require too many resources for the benifit, smugmug users should be forewarned about two very important facts regarding keywords before they spend hours and hours of thier time applying them to thousands of picture (bitter...no...):
1) Keywords can ONLY be applied to public pictures and galleries and will NOT work for galleries you want to remain private.
2) Applying keywords to pictures and galleriers will make them searchable on google, yahoo, msn...and the general smugmug keyword search for anyone to see.
I would also remind smugmug users that they might want to get the permission of thier friends and family before they decide to expose them to the searchable world wide web.
Just a thought....
Yikes, I am a :cry :cry :cry man right now. I wish it could be fixed. I don't want to leave smugmug......but I might have no choice.
I'm still searching for any possible solutions that will allow me to cloak my keywords/captions from Google crawlers. Even though this would only partially address my privacy concerns, it is certainly worth the effort to explore.
Previous posts in this thread provide helpful reference information about robot meta tags.
I can add meta tags to instruct the crawler to perform no indexing. However, others have pointed out that unprectable crawler behavior will result because of the conflict with smugmug's own meta tags.
I noticed that the URL references above also describe a robots.txt file as an alternate way to provide instructions to a crawler.
Question: Could a robots.txt file be used as a possible workaround solution for the meta tag conflict previously discussed?
As an experiment, I have placed the following line in my smugmug CSS customization text:
According the the Google reference, it takes 6-8 weeks for such changes to become effective (time waiting for the robot to re-crawl web pages). I will report back in November to let you know how the Google crawler behaved with the meta tag conflict.
FWIW: Google searches return a variety of backdoor links into my passworded smugmug account. However, a similar Yahoo search returns nothing. I guess that just proves how well Google does its job.
idea: smugmug spinoff?
My wife and I were discussing this privacy topic last night (she is even more concerned about Google visibility to our photos than I am, and is urging me to pull the smugmug plug now - sorry, this is a very tough issue in our family). She raised an idea that at first seemed odd, but makes more sense as I think about it.
I can see why it would be difficult for smugmug to change its meta tags to "nofollow, noindex, noarchive". To deliberately cloak Google searches would be very unpopular with professional photographers and others who use smugmug to run their business. They certainly want the publicity, traffic, and maximum exposure. In addition, smugmug itself wants as many Google hits as possible to expand their user base and grow their own business.
So my wife's idea was that smugmug could spinoff a new/separate website for families and other folks who are concerned with privacy issues. It could use all the same underlying technology as the existing smugmug except that meta tags and robot.txt files would be set for maximum privacy. As I think about it more, this could also be a good business opportunity for smugmug to branch out and develop sites with functionality that is more specific to the home photographer and the professional photographer. Isn't that what other businesses do all the time? - fragment their own product line to achieve maximum market penetration.
My wife and I were discussing this privacy topic last night (she is even more concerned about Google visibility to our photos than I am, and is urging me to pull the smugmug plug now - sorry, this is a very tough issue in our family).
Just so you know this is an option, but rather than pulling the smugmug plug, you can immediately set up an account password and tell all your family members what the password is. Even better, yo ucan make the password self describing to anyone who knows you, by using the password hint so that any family member would know the password as soon as they saw the prompt. As an example, set up an account password. Set the "hint text" to "Type the name of our dog" or "Type the name of our yougest daughter". And, then set the password to be that name that anyone who knows you should know.
I agree this isn't the ultimate solution as we'd rather have many of the other things in this thread, but if your wife is asking you to pull the plug, this should allay her concerns. I myself have a variety of different kinds of content in my site, some of which I want open to all, some of which I need protected with an "event" password for school/children privacy reasons and some of which I want to protect with a "family" password. What's bumming me about this is that I can't use keywords and search if I want any form of privacy. Some of smugmug's best features go away if you want any privacy.
... rather than pulling the smugmug plug, you can immediately set up an account password and tell all your family members what the password is.
Thanks for the suggestion - been there, done that. We already have an account password. The problem is that Google search still shows backdoor links to my password-protected photos. I obviously don't want to further publicize those backdoor URLs here, so I won't be more specific. However, I have communicated directly with smugmug support about the details so that they are at least aware of the backdoor access issue. On the plus side, account passwords appear to be a completely effective solution for Smugmug global searches.
Private galleries are *private*, that's what the option is for.
Given current smugmug technology, private galleries seem to be the only real solution. If it is necessary to impose restrictive access controls (i.e. private galleries) in order to shield my photos from Google's prying eyes, I am then left to answer a few fundamental questions. Does that kind of access restriction defeat the whole purpose of photo sharing? Is it worth $50 per year to host photos that cannot be easily shared? As I said before, this is a tough issue for our family. We have invested a lot of time and energy in building and customizing a nice smugmug site for our family photos. Aside from the privacy issues, we are thrilled with the entire smugmug offering. We have promoted smugmug to many friends and family members. However, when the photos and names of our very young children end up plastered all over Google searches - well, that is kind of a trump card for us. In addition to regular Google search, there is also a separate Google IMAGE search. Try out Google IMAGE search - it can be very jarring to learn that the names and associated face photos of your young children are so easily accessible to the entire world. My wife almost started crying when she saw the search results come back. Thus, you can see that we are faced with some very difficult decisions.
Now a Happy Camper I would like everyone to know how I have set up my site to accommodate public visitors and protect the privacy of my family and friends. It may help some of you who are having problems with organization and security.
Note, I use PhotoshopBridge to organize my files on my computer.
1) Applied ratings to all of my pictures from 1 to 5 (this info does not get attached to the pictures permanent metadata)
2) Applied keywords to all of my pictures including the words like "friends", "family", "cats", "architecture", "1 star", "2 stars", ect. (keywords ARE embedded in the metadata)
3) Organized photos in the following folder structure on my computer:
2002 January Cats and Dogs Trip to Australia February ..... .... 2003 .... ....
4) Using the same file structure that I set up on my computer, in smugmug I used 2002, 2003, ect for my categories, and months for my sub-categories...and then the description of the photos for gallery names.
5) Made all of these galleries PRIVATE thus making their keywords useless in smugmug. (OUCH)
6) Created a Public Gallery and uploaded any 4 or 5 star pictures (I think about 50 of them) which I wanted to share to the world. I figured only friends and family would ever want to see my 3, 2, or 1 star photos.
7) Created links to keywords in my bio for easy access to special interest galleries - ex. 5 stars, 4 stars, black and white, architecture, ect...
Now I just need to change the colors of my links and get rid of the underline and I'll be happy with that...
8) One more thing which I haven't done yet, is caption each of my public photos with information on which gallery that photo came from and information on how to get there (ex. if you see a picture of a sunset it will say - 2002, June, Nelson Sunsets). This way friends and family can find the private gallery to see more pictures.
I think that's it for now. It is the closest balance I can find to making my photos private while also giving some tools for anyone to find the pictures I want to share with the world.
Obviously, there is a little more to it then this simplified version, so if you have any questions go to my site and click "contact" to email me with your questions.
Thanks and I have to apologize to smugmug and it's loyal customers for my uncontrolled rants yesterday. Needless to say, I will NOT be canceling my smugmug account. I am now a happy camper.
another idea: channel smugmug content through https
I'm not giving up yet.
When considering a possible solution that will allow me to cloak my photos/keywords/captions from Google crawlers, could an alternate secure path through https be the solution?
Each port must have its own robots.txt file. In particular, if you serve content via both http and https, you'll need a separate robots.txt file for each of these protocols. For example, to allow Googlebot to index all http pages but no https pages, you'd use the robots.txt files below.
For your http protocol (http://yourserver.com/robots.txt):
If this is possible, could smugmug channel content through http or https depending on a user-specified account setting, and thereby provide options for users to shield photos/keywords/captions from Google crawlers through a robot.txt file? I know nothing about https, so please pardon if this is a completely silly idea - I'm searching for any kind of solution at this point. My preference is not to abandon smugmug.
When considering a possible solution that will allow me to cloak my photos/keywords/captions from Google crawlers, could an alternate secure path through https be the solution?
If this is possible, could smugmug channel content through http or https depending on a user-specified account setting, and thereby provide options for users to shield photos/keywords/captions from Google crawlers through a robot.txt file? I know nothing about https, so please pardon if this is a completely silly idea - I'm searching for any kind of solution at this point. My preference is not to abandon smugmug.
Would my solution not work for you? At least temporarily until smugmug finds a way to separate keyword searches in individual smugmug pages from google and the general smugmug searches?
Would my solution not work for you? At least temporarily until smugmug finds a way to separate keyword searches in individual smugmug pages from google and the general smugmug searches?
Or could you tell me why my setup doesn't work?
I just saw your post. I need to digest what you have done a bit more to understand it and determine if it would work for me. I'll be in touch.
I would like everyone to know how I have set up my site to accommodate public visitors and protect the privacy of my family and friends ...
... Made all of these galleries PRIVATE thus making their keywords useless in smugmug.
Are you confusing password-protected galleries with private galleries? After looking over your site, it appears to me that you have applied a password to each of your galleries, but you did not mark them private. If your galleries were marked private, I would not be able to see them at all when I browse your site.
That is one of the biggest problems with using private galleries to ensure that Google doesn't crawl over your personal information - private galleries also make it virtually impossible to share your photos with visitors to your smugmug site because private galleries become invisible. The only way for you to share photos in private galleries is to send the exact cryptic URL for that gallery or photo (and the associated password if you have set one).
Let me know if I have misunderstood something about your configuration.
That is one of the biggest problems with using private galleries to ensure that Google doesn't crawl over your personal information - private galleries also make it virtually impossible to share your photos with visitors to your smugmug site because private galleries become invisible. The only way for you to share photos in private galleries is to send the exact cryptic URL for that gallery or photo (and the associated password if you have set one).
You can create a Sharegroup and put the private galleries in it. Consider this the starting point for anything sensitive that friends and family will visit. This is a bit of work, but allows you to keep photos private but easily accessible (instead of bookmarking blah.smugmug.com, they bookmark blah.smugmug.com/share/laksjdlakjsdlkj). You can even chain sharegroups by creating others to group galleries and putting links to them in the first sharegroup's description.
While this won't get around the fact that keywords will not work for the private/passworded albums, it will give you a way to hide your albums completely but share them with others.
Thanks asd...but I am yet to hear why my password protected galleries are not secure.
I would like to keep my site as I've set it up with password protected galleries, but I don't want to jeapordize my family and friends privacy.
Well, I think the answer depends on who you ask. In a previous posting within this thread, Onethumb said that Google should not be able to crawl your passworded galleries. However my smugmug account has a password, and I still find multiple Google links to my smugmug photos. Your configuration is different than mine, so I guess you should just give it the Google test after a couple of months. On the plus side, password protection appears to be completely effective with Smugmug global searches.
Comments
Some points to consider:
1. Robot tags are entirely optional, there is nothing to stop a robot completly ignoring them.
2. The 'standard' for Robot META Exclusion states:
You obviously should not specify conflicting or repeating directives such as:
[php]
<meta name="robots" content="INDEX,NOINDEX,NOFOLLOW,FOLLOW,FOLLOW">
[/php]
3. Google advocate the use of: <META NAME="Googlebot" CONTENT="nofollow">
you can read about their search engine here: http://www.google.com/bot.html
4. Hiding from search engines such as Google without using strong protection such as access control is ultimately not going to succeed, you are too dependent on
- The whim of the search indexer, if it decides to follow your links and fake an IE 6 UserAgent, then there's not a lot you can do about it
- No body, anywhere on the internet linking to your site, ever (or these days, sending a link to anyone using a GMail account etc.)
In combination, the general advise to web developers is don't bother too hard about trying. Sorry... :uhoh
I would say, that IMHO it was massively unlikely.
Crawlers are slow, nasty, messy, computationally and network expensive things to use. They also have a tendency to leave hole in semi-unconnected networks, need to be written very carefully to avoid dying in cyclic networks. In networks of shifting topology (the interconnected mess of webpages), they are not trivial things to write, if you want a good one...
Search engines only use Crawlers, because they have no other option. They can't ask for a structured database of all the information on a website.
By its very nature, Smugmug already has this, and indeed uses it to serve up the Smugmug pages. Hence there is little if any need for them to use crawlers, which would be costly, slow to update and generally a pain...
So I think it much more likely that they will build their search databases from their own production data, rather than re-crawling their own site to get the information.
That's how I think I'd do it anyway...
Having said which, if I was trying to build something the size of Smugmug, I'd probably give it more than the 2 minutes thought I just have, so I could well be wrong
I'll just leave you with a final thought from Googlebot's description page:
Sorry I can't be more encouraging, hope this helps anyhow,
Luke
PS. Incidentally a good way of hiding from Google is by inducing the use of very common keywods, so that the search clauses will be swamped... Try googling for me to see what I mean: 'Luke Church'. http://www.google.com/search?hl=en&q=Luke+Church
The results are quite funny.
SmugSoftware: www.smugtools.com
You only list two points, but there are really three, and the third one is the kicker:
1) No search engine indexing
2) No searching from the main site
3) But allowing people to search within your photos
It's #3 that's basically impossible today without a *lot* of work. I've been working on making #1 and #2 happen for quite awhile now, since our last discussion, but for some reason I either missed or misunderstood #3.
At this time, I just can't make #3 happen. I'd like to sometime in the future, if I can, but that's a long ways out, I'm afraid.
Don
Again, I hate to beat a dead horse, but this is a massive project. Keywords weren't built with this in mind, so they're not going to happen this way. I hate to say never, but this one is probably never.
"Searching only in your account" is possible, but a lot of work. I want to make this happen some day, but it's both not a high priority and a ton of work, meaning it's gonna be awhile.
Are you sure about this? If so, this is a *bug* and not expected behavior. Google cannot get to your pages to index them, so your pages will fall out of their index the next time they crawl.
As for smugmug, we check for user passwords and drop your results automagically from the search engine. (At least, that's what it's supposed to do and what we tested it to do. Bugs do happen, though
Robot tags would remove Google, yes, but we have a tag conflict problem.
As I stated in my last post, I'm working on making it so you can have Google not index your gallery pages on a gallery-by-gallery basis. Also allowing public galleries which do not show up in smugmug's search.
But the side effects are that keywords will cease to function in those galleries as well.
Don
Great points, Luke.
Let me point out, though, that all of the major search engines *do* adhere strictly to the META tags. I'm not aware of one that doesn't.
They have to, otherwise admins block their sites on their routers and they lose the ability to crawl at all.
So don't get all paranoid that your META tags will be ignored - they won't be.
Oh, and the only reason Google recommends putting "Googlebot" in the META tag is if you'd like to only restrict Google and not Yahoo/MSN/etc.
Don
Sure, sorry I probably wasn't clear enough. My point is that META tags give 'privacy', they do not give 'security'. They will keep 99.99% of people away by hiding from major search engines, however they do not gaurentee anything.
But I agree, for this purpose they are fine.
If people want better than 99.99% then access protection is the only way to go, with its associated annoyances.
Of couse, I should have mentioned that as well.
Cheers,
Luke
SmugSoftware: www.smugtools.com
All sounds very good, except the part about disabling keywords - bummer. I love keywords and use them heavily.
Instead of swallowing the whole elephant (modifying smugmug search), how about just a light snack? Perhaps something more simple that removes the meta tag conflict, and leaves alone the smugmug search and keywords functionality. Granted - it is not a complete solution to all the issues raised in this thread. However, it would be a relatively easy change that would address at least half of the privacy concerns. Just eliminating all the prying eyes that might stumble into my galleries from Google searches would be a big comfort. Sorry if I am talking like a simpleton - I don't mean to dismiss the complexities of such changes.
Like other users are saying, there needs to be a middle-of-the-road when it comes to privacy.
I know we're beating this but I'm still not convinced why this is not possible within a reasonable amount of work. We had discussed this months ago in this thread and I thought we were getting somewhere.
Is the resistance because there's a single database of public keywords that would require reworking?
Looking at it from my view I see that
1) If an album is made private, the keywords are not indexed in smugmug searches, and
2) I can already search only within my gallery for my photos without including anyone elses.
Why can search results not include photos that the users had made semi-private because they are not being searched from within that user's gallery? It appears that a possible solution could reside in a search results filter.
I urge Smugmug, again, to look at this with its user's privacy in mind and not to brush this off because "it's a lot of work" and "not a high priority." I'm sure with more analysis there's a solution that is meets both the requirements of the users without as much work as initially anticipated. I'm not trying to be overly anal about this but I'll bet that if enough Smugmug users were surveyed about this topic you would get a persuasive number of responses in the affirmative.
And I love smugmug and dont want to leave, but the full frontal or password being the only security option does bug the hell out of me.
If keeping the complexity down on the global search is the main concern (ie you don't want to add a "where allowSearch=true" clause), couldn't you have a separate database table for these photos' keywords?
I'm keen on smugmug because it has better privacy features than flickr, but not being able to have keyword search on photos if they aren't globally searchable is a bit of a killer!
How about adding a choice to the account settings which would insert the no index / no follow robot tags into that member's pages regardless of whether or not they have password-protected albums?
Thanks.
I just spent hours and hours and hours applying keywords to all my photos only to realize that these keywords are going to be viewable by global search engines such as google, yahoo, msn, ect. not to mention to anyone who searches for photos through the smugmug homepage.
So to keep my privacy, my only option is to make all my galleries private through password protection, right? Well, that also makes my keywords on my homepage useless and a big pain in the a** for my friends and family who will have to search through my category, sub-category, gallery, password system to find photos that could be so easy to access through keywords.
This problem is a showstopper for me.
Basically, the way I see it, the only use for smugmug would be for pros or serious amateurs to sell or showcase "professional photos" that they don't hold as sacred and private. However, for someone who is looking to share photos with thier friends and family, there is a huge risk of advertising your private life (including the private life of your family and friends who are in your photos) to the whole world.
This is a serious issue.
I will cancel my account with smugmug if I do not see some serious progress to solve this problem ASAP.
And if you read through this thread, you'll see there are many other smugmug costomers who share my concerns.
Smugmug....you have a good reputation here....do not let this issue slide and jepordize the hard work you've accomplised to date with your relationships with your costomers.
Now, what are you going to do?
www.photographyontheside.com [My blog about creating a part-time photography business]
Create A Gorgeous Photography Website with Smugmug in 90 Minutes [My free course if you need help setting up and customizing your SmugMug site]
Ouch, I'm so sorry you are not a happy camper when it comes to keywords.
I wish I had a better answer for you but I'm afraid there really isn't much smugmug can do in the near future to allow for keyword searching in private galleries without a complete redesign of the system. Making the changes to allow for the kind of functionality would be a very large and lengthy project.
I'm very sorry that there isn't more smugmug can do for you right now. We are discussing this feature and are very aware that this is an area that could use some improvement. Unfortunately, I don't know when we will be able to start and undertake a project this size.
I wish I could give you a better answer.
John
smugmug customer support
Over the course of this discussion, it has become clear that keyword functionality was designed specifically to enable global search capability. Some smugmug users however don't understand the global intent of keywords and desire to use them in a different manner (not to enable global audience access, but rather to facilitate local indexing of photos within a smugmug account). Keywords are a great piece of functionality, but Smugmug's intended use is not our desired use. This seems to be the sticking point.
Perhaps, since smugmug has determined that this feature would require too many resources for the benifit, smugmug users should be forewarned about two very important facts regarding keywords before they spend hours and hours of thier time applying them to thousands of picture (bitter...no...):
1) Keywords can ONLY be applied to public pictures and galleries and will NOT work for galleries you want to remain private.
2) Applying keywords to pictures and galleriers will make them searchable on google, yahoo, msn...and the general smugmug keyword search for anyone to see.
I would also remind smugmug users that they might want to get the permission of thier friends and family before they decide to expose them to the searchable world wide web.
Just a thought....
Yikes, I am a :cry :cry :cry man right now. I wish it could be fixed. I don't want to leave smugmug......but I might have no choice.
Thanks for listening.
www.photographyontheside.com [My blog about creating a part-time photography business]
Create A Gorgeous Photography Website with Smugmug in 90 Minutes [My free course if you need help setting up and customizing your SmugMug site]
As I post about privacy issues with smugmug, my hit count on my site goes through the roof.
Karma...
www.photographyontheside.com [My blog about creating a part-time photography business]
Create A Gorgeous Photography Website with Smugmug in 90 Minutes [My free course if you need help setting up and customizing your SmugMug site]
Previous posts in this thread provide helpful reference information about robot meta tags.
http://www.google.com/webmasters/remove.html
http://help.yahoo.com/us/ysearch/deletions/deletions-03.html
I can add meta tags to instruct the crawler to perform no indexing. However, others have pointed out that unprectable crawler behavior will result because of the conflict with smugmug's own meta tags.
I noticed that the URL references above also describe a robots.txt file as an alternate way to provide instructions to a crawler.
Question: Could a robots.txt file be used as a possible workaround solution for the meta tag conflict previously discussed?
As an experiment, I have placed the following line in my smugmug CSS customization text:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW, NOARCHIVE">
According the the Google reference, it takes 6-8 weeks for such changes to become effective (time waiting for the robot to re-crawl web pages). I will report back in November to let you know how the Google crawler behaved with the meta tag conflict.
FWIW: Google searches return a variety of backdoor links into my passworded smugmug account. However, a similar Yahoo search returns nothing. I guess that just proves how well Google does its job.
My wife and I were discussing this privacy topic last night (she is even more concerned about Google visibility to our photos than I am, and is urging me to pull the smugmug plug now - sorry, this is a very tough issue in our family). She raised an idea that at first seemed odd, but makes more sense as I think about it.
I can see why it would be difficult for smugmug to change its meta tags to "nofollow, noindex, noarchive". To deliberately cloak Google searches would be very unpopular with professional photographers and others who use smugmug to run their business. They certainly want the publicity, traffic, and maximum exposure. In addition, smugmug itself wants as many Google hits as possible to expand their user base and grow their own business.
So my wife's idea was that smugmug could spinoff a new/separate website for families and other folks who are concerned with privacy issues. It could use all the same underlying technology as the existing smugmug except that meta tags and robot.txt files would be set for maximum privacy. As I think about it more, this could also be a good business opportunity for smugmug to branch out and develop sites with functionality that is more specific to the home photographer and the professional photographer. Isn't that what other businesses do all the time? - fragment their own product line to achieve maximum market penetration.
Just an idea, but maybe worth considering.
thanks 'blotter.. i'm fwding this to the smugmug team.
Portfolio • Workshops • Facebook • Twitter
Just so you know this is an option, but rather than pulling the smugmug plug, you can immediately set up an account password and tell all your family members what the password is. Even better, yo ucan make the password self describing to anyone who knows you, by using the password hint so that any family member would know the password as soon as they saw the prompt. As an example, set up an account password. Set the "hint text" to "Type the name of our dog" or "Type the name of our yougest daughter". And, then set the password to be that name that anyone who knows you should know.
I agree this isn't the ultimate solution as we'd rather have many of the other things in this thread, but if your wife is asking you to pull the plug, this should allay her concerns. I myself have a variety of different kinds of content in my site, some of which I want open to all, some of which I need protected with an "event" password for school/children privacy reasons and some of which I want to protect with a "family" password. What's bumming me about this is that I can't use keywords and search if I want any form of privacy. Some of smugmug's best features go away if you want any privacy.
--John
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
To quote Don from an earlier post in this thread:
Given current smugmug technology, private galleries seem to be the only real solution. If it is necessary to impose restrictive access controls (i.e. private galleries) in order to shield my photos from Google's prying eyes, I am then left to answer a few fundamental questions. Does that kind of access restriction defeat the whole purpose of photo sharing? Is it worth $50 per year to host photos that cannot be easily shared? As I said before, this is a tough issue for our family. We have invested a lot of time and energy in building and customizing a nice smugmug site for our family photos. Aside from the privacy issues, we are thrilled with the entire smugmug offering. We have promoted smugmug to many friends and family members. However, when the photos and names of our very young children end up plastered all over Google searches - well, that is kind of a trump card for us. In addition to regular Google search, there is also a separate Google IMAGE search. Try out Google IMAGE search - it can be very jarring to learn that the names and associated face photos of your young children are so easily accessible to the entire world. My wife almost started crying when she saw the search results come back. Thus, you can see that we are faced with some very difficult decisions.
I would like everyone to know how I have set up my site to accommodate public visitors and protect the privacy of my family and friends. It may help some of you who are having problems with organization and security.
Note, I use PhotoshopBridge to organize my files on my computer.
1) Applied ratings to all of my pictures from 1 to 5 (this info does not get attached to the pictures permanent metadata)
2) Applied keywords to all of my pictures including the words like "friends", "family", "cats", "architecture", "1 star", "2 stars", ect. (keywords ARE embedded in the metadata)
3) Organized photos in the following folder structure on my computer:
2002
January
Cats and Dogs
Trip to Australia
February
.....
....
2003
....
....
4) Using the same file structure that I set up on my computer, in smugmug I used 2002, 2003, ect for my categories, and months for my sub-categories...and then the description of the photos for gallery names.
5) Made all of these galleries PRIVATE thus making their keywords useless in smugmug. (OUCH)
6) Created a Public Gallery and uploaded any 4 or 5 star pictures (I think about 50 of them) which I wanted to share to the world. I figured only friends and family would ever want to see my 3, 2, or 1 star photos.
7) Created links to keywords in my bio for easy access to special interest galleries - ex. 5 stars, 4 stars, black and white, architecture, ect...
Now I just need to change the colors of my links and get rid of the underline and I'll be happy with that...
8) One more thing which I haven't done yet, is caption each of my public photos with information on which gallery that photo came from and information on how to get there (ex. if you see a picture of a sunset it will say - 2002, June, Nelson Sunsets). This way friends and family can find the private gallery to see more pictures.
I think that's it for now. It is the closest balance I can find to making my photos private while also giving some tools for anyone to find the pictures I want to share with the world.
Obviously, there is a little more to it then this simplified version, so if you have any questions go to my site and click "contact" to email me with your questions.
Thanks and I have to apologize to smugmug and it's loyal customers for my uncontrolled rants yesterday. Needless to say, I will NOT be canceling my smugmug account. I am now a happy camper.
www.photographyontheside.com [My blog about creating a part-time photography business]
Create A Gorgeous Photography Website with Smugmug in 90 Minutes [My free course if you need help setting up and customizing your SmugMug site]
I'm not giving up yet.
When considering a possible solution that will allow me to cloak my photos/keywords/captions from Google crawlers, could an alternate secure path through https be the solution?
The Google reference mentions that separate robot.txt files can be used for content served up through http and https. See:
http://www.google.com/webmasters/remove.html
If this is possible, could smugmug channel content through http or https depending on a user-specified account setting, and thereby provide options for users to shield photos/keywords/captions from Google crawlers through a robot.txt file? I know nothing about https, so please pardon if this is a completely silly idea - I'm searching for any kind of solution at this point. My preference is not to abandon smugmug.
Or could you tell me why my setup doesn't work?
www.photographyontheside.com [My blog about creating a part-time photography business]
Create A Gorgeous Photography Website with Smugmug in 90 Minutes [My free course if you need help setting up and customizing your SmugMug site]
That is one of the biggest problems with using private galleries to ensure that Google doesn't crawl over your personal information - private galleries also make it virtually impossible to share your photos with visitors to your smugmug site because private galleries become invisible. The only way for you to share photos in private galleries is to send the exact cryptic URL for that gallery or photo (and the associated password if you have set one).
Let me know if I have misunderstood something about your configuration.
Are you saying my setup does not solve the privacy issue?
www.photographyontheside.com [My blog about creating a part-time photography business]
Create A Gorgeous Photography Website with Smugmug in 90 Minutes [My free course if you need help setting up and customizing your SmugMug site]
You can create a Sharegroup and put the private galleries in it. Consider this the starting point for anything sensitive that friends and family will visit. This is a bit of work, but allows you to keep photos private but easily accessible (instead of bookmarking blah.smugmug.com, they bookmark blah.smugmug.com/share/laksjdlakjsdlkj). You can even chain sharegroups by creating others to group galleries and putting links to them in the first sharegroup's description.
While this won't get around the fact that keywords will not work for the private/passworded albums, it will give you a way to hide your albums completely but share them with others.
I would like to keep my site as I've set it up with password protected galleries, but I don't want to jeapordize my family and friends privacy.
www.photographyontheside.com [My blog about creating a part-time photography business]
Create A Gorgeous Photography Website with Smugmug in 90 Minutes [My free course if you need help setting up and customizing your SmugMug site]