Sitemap and Robots.txt

pabloconrad · September 23, 2015

Yodels.

Trying to get my site higher in the rankings. Apparently, there's some line in the robots.txt that is preventing Google from searching deeper into my site.

The error message I'm getting at Google is "Sitemap contains URLs that are being blocked by robots.txt."

I have full visibility checked in my account settings and it has been that way for several years.

Also, GoDaddy says I need to upload an new "Sitemap" so they can also properly index my site. Currently, they only have 3 pages: my home page, the top of a gallery, and an individual photo in that gallery.

Any help would be greatly appreciated.

Thank you in advance.

Ben · September 23, 2015

Hey Paul,

We do restrict a few URLs in robots.txt that we don't want Google indexing, but they aren't related to content on your site. A perfect example is /cart/, which is just the shopping cart portion. We don't need or want Google indexing an empty shopping cart. So that warning, in and of itself, isn't particularly bad news.

Do you have some example queries that you are running Google that I can look at and see what is going on? On the surface, I am able to find your content in Google. I clicked through to a random gallery, and then searched for it and it came up:
https://www.google.com/search?q=paul+conrad+great+pacific+northwest&oq=paul+conrad+great+pacific+northwest

In general, the best ways to get found are by describing EVERYTHING. Descriptive gallery titles, gallery descriptions, Photo titles, captions, keywords, etc. At least in the gallery above, you are doing a solid job overall of that. The next thing that helps is getting that content linked in quality places, again with descriptive text. Blogs, forums, etc. Participating online helps your SEO. It's hard to fake though, so it does have to be quality participation on quality sites.

pabloconrad · September 29, 2015

Ben wrote: »

Hey Paul,

We do restrict a few URLs in robots.txt that we don't want Google indexing, but they aren't related to content on your site. A perfect example is /cart/, which is just the shopping cart portion. We don't need or want Google indexing an empty shopping cart. So that warning, in and of itself, isn't particularly bad news.

Do you have some example queries that you are running Google that I can look at and see what is going on? On the surface, I am able to find your content in Google. I clicked through to a random gallery, and then searched for it and it came up:
https://www.google.com/search?q=paul+conrad+great+pacific+northwest&oq=paul+conrad+great+pacific+northwest

In general, the best ways to get found are by describing EVERYTHING. Descriptive gallery titles, gallery descriptions, Photo titles, captions, keywords, etc. At least in the gallery above, you are doing a solid job overall of that. The next thing that helps is getting that content linked in quality places, again with descriptive text. Blogs, forums, etc. Participating online helps your SEO. It's hard to fake though, so it does have to be quality participation on quality sites.

Thank you for the info Ben. I was in my GoDaddy account and the pages indexed there are only three. I called them to see if I could get my site properly indexed for better SEO, but apparently they only index the pages hosted on their servers, no one elses.

And I do try to describe EVEYRTHING. It's the photojournalist in me. Gotta write accurate captions and keywords!!!

Thank you for the help.

Sitemap and Robots.txt

Comments