Initializes a new instance of the RobotsDotText class.
RobotsDotText Class | Arachnode.SiteCrawler.Rules Namespace