What’s Up With MSN?

There’s been lots of discussion in the last few weeks about the activity of MSNbot. I’ve noticed a massive increase in the frequency this bot visits my sites, often spidering every single page every day of the week.

I have no problem with MSNbot (or Googlebot & Slurp) spidering my content. I agree with Mikel deMib Svendsen’s comments on WebmasterRadio that there is a symbiotic relationship at work where we allow search engines to spider and index our content in return for traffic and leads. The problem with MSNbot’s hyperactive activity is that it doesn’t seem to matter how often it spiders a page, they never get added to MSN Search index!

Fortunately there is a workaround. According to MSN Search’s webmaster help page you just need to add the following to your robots.txt file:

User-agent: msnbot
Crawl-delay: 864

The crawl delay is specified in seconds – this example will slow down MSNbot to 100 requests per day, which will certainly help conserve bandwidth on larger sites. However, I and others still want to know why (even after so much bot activity) pages do not get added to the index at MSN Search!

1 thought on “What’s Up With MSN?

  1. ANDREW

    i have been get a erro code number this it the one 8004888d and with msn explorer do you need to dail in to the sever becase his say dail up i have been in an d tock it off dail but it say it still so please help thanks

Leave a Reply

Your email address will not be published. Required fields are marked *