arachnode.net

RobotsDotText.IsDisallowed(CrawlRequest, ArachnodeDAO) Method

Determines whether the specified crawl request is disallowed.

public override bool IsDisallowed(
   CrawlRequest crawlRequest,
   ArachnodeDAO arachnodeDAO
);

Parameters

crawlRequest
The crawl request.
arachnodeDAO
The arachnode DAO.

Return Value

true if the specified crawl request is disallowed; otherwise, false.

See Also

RobotsDotText Class | Arachnode.SiteCrawler.Rules Namespace | RobotsDotText.IsDisallowed Overload List