I'm using WP + Atahualpa 3.5.3, Not sure how to disable Google Bot to crawl non-existent URLs.
My Permalink is: "/%category%/%postname%.html"

The URL that users see is : domain.com/category/postname.html
But the URL that Google Bot crawled and error as 404 is : domain.com/postname.html

Does anyone know how to disallow Google Bot to crawl those URL ?
I saw in the Webmaster Tool : Link From is unavailable --> not sure what we should do to see the caused of problem ?


Thanks everyone for the helps.

remone the '.html' from the permalink

Hello Sir,

Thank you so very much.

Let me try to delete the .html, and will come to update in this post soon.

PS 1: But i'm afraid all working urls will be gone ?
ิ
PS 2 : i like this word"'Give a man a solution and you solve that one issue. Teach a man how to solve problems and you've freed him for a lifetime.' "

Dear Mr. Juggledad,

I've deleted the .html in the permalink on my other sites
and it doesn't work; all the current urls that arleady in google search results return 404 as those url already index in Google SERPs.

my exact current problem

google crawl both WP-domain.com/category/computer.html -- this is working for 2 years

I saw its crawl WP-domain.com/computer.html after install 4 plugins (link checking, xml sitemap, bot check, W3 Totol cache)

Do you think the url WP-domain.com/computer.html was generated by one of those plugin ?

i just disable the 3 ones :link checking, xml sitemap, bot check -- as i think is the source of problem ?


Big thanks

without knowing the URL, it is hard to tell, but with the permalink you have, itf you created a post r page titled 'computer' it would show as 'computer.html'

You might want to look at a plugin to see if you can do a redirect

Thank you for you kind update

My website is: http://tinyurl.com/3uyuanz

I just added more command on robots.txt :

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Disallow: /xmlrpc
Disallow: /tag/*
Disallow: /tag
Disallow: /rss
Disallow: /archives
Disallow: /wget/

Not sure if this correct ?
Also not sure if the current W3 Total Cache Plug-in is concern with the URL 404 error ?


After that try delete plug-in that seem to related to link generation issue like sitemap, link checking, bot checker plug-in.

I also deleted the file sitemap in the server as i'm not sure if this file link to invalid URLs ( as this file might generated from plugin that not compatible with my permalink/ word press config.)

Will see one or two days for update about crawling issue.

Thanks Mr. Juggledad and all Readers.

Other issue:

How to set Atahualpha not to let Google bot NOT to crawl RSS ?
or the urls like domain.com/postname.hml


on google webmaster tool saying that http://mydomain.com/category/feed on robots.txt restriction tab



Thanks sir.

Seem like got the the 404 url from W3 Total Cache plugin

Does anyone know how to fix the problem ?

This is also my concern. Will this be permanent? Or will a non-existent URL be eventually moved down in Google results once the other existent pages are crawled? Thank you

lifelock coupon