arachnode.net

RobotsManager.ParseRobotsDotText Method

Parses the robots dot text.

public static List<string> ParseRobotsDotText(
Uri baseUri,
byte[] robotsDotText
);

Parameters

baseUri: The base URI.
robotsDotText: The robots dot text.

Return Value

See Also

RobotsManager Class | Arachnode.SiteCrawler.Managers Namespace