Provides robots.txt parsing functionality.

For a list of all members of this type, see RobotsManager Members .

System.Object
   RobotsManager

Thread Safety

Public static (Shared in Visual Basic) members of this type are safe for multithreaded operations. Instance members are not guaranteed to be thread-safe.

Requirements

Namespace: Arachnode.SiteCrawler.Managers

Assembly: Arachnode.SiteCrawler (in Arachnode.SiteCrawler.dll)

See Also

RobotsManager Members | Arachnode.SiteCrawler.Managers Namespace

---