When web browser shows a 404 error page, it doesn’t mean the web server returning HTTP response code 404. If it doesn’t return the right response code, search engines (web spiders) will index the 404 error page or broken link.
So, how to know whether or not the web server returns the right HTTP response code to spiders? Here are two of the many ways:
Uses Google Webmaster Tools
Add your website to Google Webmaster Tools. Then, use the Fetch as Google to perform web fetching for a not existing page of your site. When fetching completes, click “fetch status” link and you should see that first line of the fetched content read as “HTTP/1.1 404 Not Found”.
Uses GNU wget program
Your Linux web server should have wget program installed. Otherwise, download wget for Windows. At command prompt window, run wget with that two option switches to retrieve and view the HTTP response code returned by web server:
Note, the option switch is case-sensitive, i.e. -s is not equal to -S.
So, how to know whether or not the web server returns the right HTTP response code to spiders? Here are two of the many ways:
Uses Google Webmaster Tools
Add your website to Google Webmaster Tools. Then, use the Fetch as Google to perform web fetching for a not existing page of your site. When fetching completes, click “fetch status” link and you should see that first line of the fetched content read as “HTTP/1.1 404 Not Found”.Uses GNU wget program
Your Linux web server should have wget program installed. Otherwise, download wget for Windows. At command prompt window, run wget with that two option switches to retrieve and view the HTTP response code returned by web server:
wget --spider -S http://www.yoursite.com/bad_url.html
Note, the option switch is case-sensitive, i.e. -s is not equal to -S.
Custom Search
2013 •