There are many issues which could affect a website's rankings on search engines which could be out of the site owners control. Below are some of the issues:

Issue 1 – Changing a Site’s IP Address:
Moving a site to a new IP (either by switching in on the server, or physically moving the site to a new server with a new IP) would affect a site’s rankings. There would be a lag time between when the DNS change would be submitted and when it would take effect and filter down to individual DNS servers. The site owner (or webmaster) would process the DNS change with a name server, and the registration would likely change within 2-3 days. However, for this change to filter down to the root name servers could take days more.

 

The structure of the internet is based upon a series of root name servers. There are 13 root name servers that provide name resolution for the internet. The DNS registration/update is applied and forwarded to these root servers who then process the information downwards. The speed of the update to the downstream servers depends on their local settings. Most are set to refresh every 15 minutes, but some can take days to do an update.

 

When a client (or search engine, or search engine spider) requests a site, it first goes to its local DNS server to request the listing. If it’s own DNS server doesn’t have the site, it forwards the request to the next upstream server, and so on until the name is resolved to an IP. If the name is found and associated to an IP, the IP address is then forwarded to the client who goes to that address. The DNS server has no idea if the address is correct or not, it only has the information since its last refresh with its upstream server. If the IP isn’t correct, the client will get an error.

 

Recommendation: To solve this problem, the ideal situation would be to have the site exist in both the old and new location for a period of 1 week. This would provide sufficient time for the main root servers and their subordinates to update and refresh their information. Since most clients pay for web hosting services on a month to month basis, and generally sign a contract for a year, it would be advisable to plan a move a few months in advance, and use the last month of the contract to host the site on two servers.

 

If this wasn’t possible, we’d need to know when search engine spiders are most active and try to move the site when they aren’t active. This would lessen the impact of the move. We also feel that sites that have paid listings would be greatly impacted by a site move, as many premium listing guarantee a regular spider visit, usually every few days. They would follow the procedure outlined above, and would likely not find the old site (if it was moved completely and not maintained at the old IP Address). The site would then likely be dropped by engines, such as Yahoo, and the site owner would probably have to re register and pay the fee to get listed.

 

Issue 2 – IP’s being banned by Engines: An issue which isn’t necessarily server related, but can be a cause for a site not being indexed is that they share an IP with a site that has been banned, or have inherited an IP from a banned site (see http://support.easystreet.com/abuse/banned_ip.html). If an engine finds a site objectionable, or feel they were being spammed by a site/IP they will ban it from their index. If this is the case the site’s chances of being picked up by that engine are virtually nil. If this site moves on to a new IP address that hasn’t been banned then they will probably get indexed until the engines catch up to the new site IP and proceed to ban them again.

 

In the mean time, the banned IP gets given to a new site. The new site has no idea the IP has been banned by the engines, but finds that they aren’t getting indexed. From the site owners perspective (and probably that of the webmaster) they don’t know that the IP has been banned and won’t know why they aren’t being indexed.

 

Recommendation: If your site hasn’t been indexed within a reasonable timeframe, and you can’t think of a reason, you could check with the webmaster to see if you have an IP that we previously given to another site, and perhaps do some searches for the other site. If it looks like a spam site to you then it may have been banned on your IP earlier by the engines. 

 

Another alternative would be to just request a new IP. Keep in mind the lag time for the DNS updates to filter down, but this wouldn’t likely affect you as much if the site has been banned because of the IP.

 

Issue 3 – Robots.txt file:
Many site owners and webmasters create a file called robots.txt to exclude or include pages from search engines. When a spider visits the site, it will ask the server for this file, if it exists, and use it as a guideline for indexing the site.

 

Some webmasters/site owners don’t realize the impact of such a document and how it will affect rankings. If they use the wrong syntax, or accidentally exclude the wrong pages they will adversely affect their rankings by excluding some, or all, of the site. If a webmaster adds this file as a standard part of the package but doesn’t add anything to it (IE: the document is empty) the site owner takes the chance of not being indexed. If the spider asks for this file and if it finds it empty it may decide not to index the site.

 

Recommendation: Unless this file is needed, it shouldn’t exist on the web server. If the site requires it to exclude certain pages (such as personal or private pages that shouldn’t be visible to the public) then the site owner/webmaster should fully understand how to properly create this file.

 

Issue 4 – Firewalls:
Many web servers use a software package called a firewall to protect themselves from intruders. There are many cases of web servers being invaded and taken over by others to fulfill their own needs. As such a firewall is now standard equipment when setting up a web server.

 

Many firewalls are extremely customizable. A person using a firewall can control how different TCP and UDP ports respond to different situations. A properly configured firewall can protect you from outside attack while allowing legitimate traffic through.

 

This is the key, however – a firewall has to be properly configured to be effective. Since a firewall can be configured to block certain types of traffic, it can also be configured to block IP addresses as well. If a web master or web site hosting company feels that an IP (or even an IP range) is causing too much traffic on the server, such as when spiders are active, they could chose to ban them. If this is done, a spider can’t visit your site to index it. This will affect rankings. This has been known to happen in the past.

 

Recommendation: The only way to solve this problem is to ask the web hosting company if they’ve banned any IP’s.  A web site owner or webmaster can view their logs to see if a spider has been visiting. If not, then check with the hosting company to see if they have banned the spider’s IP. If so you may ask if they can remove the ban, explaining why, or switch to another hosting company.

 

Issue 5 – Shared IP’s:
Although not proven, there is good evidence to suggest that similar sites which share the same IP will not get favorable traffic from search engines. 

 

Although many sites share their IP addresses with other sites without incident, there are occasions when this is not recommended. If 2 sites share an IP but have the same or similar content, the engines will likely only index 1 site and consider the other a mirror, and therefore not worth the effort to add to their database. If the content is different (IE: A legal site with a health food site) then there doesn’t appear to be an effect on rankings.

 

This is very similar to a site being banned because it inherited the IP of a banned site. Also, sharing an IP with a banned site could affect a sites performance in the engines, although this has yet to be proven.

 

Recommendation: Splitting the 2 sites to separate IP addresses should allow both sites to get indexed as they will appear to the engines as unique and therefore worthy of being indexed.

 

The five issues discussed above are all potential reasons for a particular web site to not be indexed by some or all of the search engines. Individually or together, these reasons do pose a threat to a site’s visibility. Some are in the control of the site owner, while others need to be addressed by the webmaster or web hosting service. Regardless, these are issues that are somtimes overlooked or forgotten about.  We hope that you enjoyed this Enquiro Retro Post.  Look for more in upcoming weeks.


Copyright 2008 - Enquiro Search Solutions.