You are here: Foswiki>Tasks Web>Item530 (10 May 2009, KennethLavrsen)Edit Attach

Item530: Add Crawl-delay to robots.txt

pencil
Priority: Enhancement
Current State: Closed
Released In: 1.0.4
Target Release: patch
Applies To: Engine
Component: robots.txt
Branches:
Reported By: Foswiki:Main.KoenMartens
Waiting For:
Last Change By: KennethLavrsen
There is a non-standard robots.txt tag Crawl-delay in the non-standard robots.txt file that limits the rate at which some bots crawl a site. At least yahoo and msft honour this. We should set this in the robots.txt that ships with foswiki to prevent problems with resource usage by crawlers hitting a foswiki site.


is this completed for trunk??

-- SvenDowideit - 10 May 2009

any reason not to put this in 1.0.6?

-- WillNorris - 10 May 2009

It is in trunk and was there before we created the branch. Closing.

-- KennethLavrsen - 10 May 2009

ItemTemplate edit

Summary Add Crawl-delay to robots.txt
ReportedBy Foswiki:Main.KoenMartens
Codebase
SVN Range TWiki-4.2.3, Wed, 06 Aug 2008, build 17396
AppliesTo Engine
Component robots.txt
Priority Enhancement
CurrentState Closed
WaitingFor
Checkins distro:726e4199cdc7
TargetRelease patch
ReleasedIn 1.0.4
Topic revision: r5 - 10 May 2009, KennethLavrsen
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy