arachnode.net

ApplicationSettings Properties

The properties of the ApplicationSettings class are listed below. For a complete list of ApplicationSettings class members, see the ApplicationSettings Members topic.

Public Static Properties

AssignCrawlRequestPrioritiesForFiles Gets or sets a value indicating whether [assign crawl request priorities for files]. Should File Priorities be set according to values in the Priorities table? If set to 'true', File priorities will be set according to the 'Host' of the File AbsoluteUri. If set to 'false' the File will be created as a CrawlRequest with a Priority of '0'.
AssignCrawlRequestPrioritiesForHyperLinks Gets or sets a value indicating whether [assign crawl request priorities for hyper links]. Should HyperLink Priorities be set according to values in the Priorities table? If set to 'true', HyperLink priorities will be set according to the 'Host' of the HyperLink AbsoluteUri. If set to 'false' the HyperLink will be created as a CrawlRequest with a Priority of '0'.
AssignCrawlRequestPrioritiesForImages Gets or sets a value indicating whether [assign crawl request priorities for images]. Should Image Priorities be set according to values in the Priorities table? If set to 'true', Image priorities will be set according to the 'Host' of the Image AbsoluteUri. If set to 'false' the Image will be created as a CrawlRequest with a Priority of '0'.
AssignCrawlRequestPrioritiesForWebPages Gets or sets a value indicating whether [assign crawl request priorities for web pages]. Should WebPage Priorities be set according to values in the Priorities table? If set to 'true', WebPage priorities will be set according to the 'Host' of the WebPage AbsoluteUri. If set to 'false' the WebPage will be created as a CrawlRequest with a Priority of '0'.
AssignEmailAddressDiscoveries Should EmailAddresses be assigned (parsed) from a WebPage Discovery?
AssignFileAndImageDiscoveries Should Files and Images be assigned (parsed) from a WebPage Discovery?
AssignHyperLinkDiscoveries Should HyperLinks be assigned (parsed) from a WebPage Discovery?
ClassifyAbsoluteUris Upon submission to the database, will AbsoluteUris be classified according to Scheme, Domain, Host and Extension?
ConnectionString The ConnectionString used for connecting to the arachnode.net Database.
ConsoleOutputLogsDirectory The UNC\relative path where console output logs should be written to disk.
CrawlRequestTimeoutInMinutes The duration until a CrawlRequest times out.
CreateCrawlRequestsFromDatabaseCrawlRequests Should CrawlRequests stored in the database be converted to CrawlRequests for crawling?
CreateCrawlRequestsFromDatabaseFiles Should Files stored in the database be converted to CrawlRequests for crawling?
CreateCrawlRequestsFromDatabaseHyperLinks Should HyperLinks stored in the database be converted to CrawlRequests for crawling?
CreateCrawlRequestsFromDatabaseImages Should Images stored in the database be converted to CrawlRequests for crawling?
CreateCrawlRequestsFromDatabaseWebPages Should WebPages stored in the database be converted to CrawlRequests for crawling?
DesiredMaximumMemoryUsageInMegabytes The desired number of megabytes of RAM memory that arachnode.net will use. Try to set this to at least 1/3 of the available RAM. Higher values equal increased performance. If crawling on the database server, leave room for SQL's cache, favoring SQL's cache in the number of megabytes allocated.
DownloadedFilesDirectory The UNC/relative path where downloaded Files should be saved.
DownloadedImagesDirectory The UNC/relative path where downloaded Images should be saved.
DownloadedWebPagesDirectory The UNC/relative path where downloaded WebPages should be saved.
EnableConsoleOutput Should program output be directed to the console?
ExtractFileMetaData Should the FileManager extract File MetaData?
ExtractImageMetaData Should the ImageManager extract Image MetaData?
ExtractWebPageMetaData Should the WebPageManager extract WebPage MetaData? The HtmlAgilityPack is extremely RAM hungry. Setting this property to 'true' will have an adverse effect on performance.
HttpWebRequestRetries
InsertDisallowedAbsoluteUriDiscoveries Should Crawls submit DisallowedAbsoluteUri Discoveries to the database?
InsertDisallowedAbsoluteUris Should Crawls submit DisallowedAbsoluteUris to the database?
InsertEmailAddressDiscoveries Should Crawls submit EmailAddress Discoveries to the database?
InsertEmailAddresses Should Crawls submit EmailAddresses to the database?
InsertExceptions Should Crawls, et. al. submit Exceptions to the database?
InsertFileDiscoveries Should Crawls submit File Discoveries to the database?
InsertFileMetaData Should the FileManager submit File MetaData to the database?
InsertFiles Should the FileManager submit Files to the database?
InsertFileSource Should the FileManager submit File Sources to the database?
InsertHyperLinkDiscoveries Should Crawls submit HyperLink Discoveries to the database?
InsertHyperLinks Should Crawls submit HyperLinks to the database?
InsertImageDiscoveries Should Crawls submit Image Discoveries to the database?
InsertImageMetaData Should the ImageManager submit Image MetaData to the database?
InsertImages Should the ImageManager submit Images to the database?
InsertImageSource Should the ImageManager submit Image Sources to the database?
InsertWebPageMetaData Should the WebPageManager submit WebPage MetaData to the database?
InsertWebPages Should the WebPageManager submit WebPages to the database?
InsertWebPageSource Should the WebPageManager submit WebPage Sources to the database?
MaximumNumberOfCrawlRequestsToCreatePerBatch Gets or sets the maximum number of crawl requests to create per batch. The maximum number of CrawlRequests created from the CrawlRequests, Files, HyperLinks, Images and/or WebPages tables when the Engine requests CrawlRequests to crawl.
MaximumNumberOfCrawlThreads The maximum number of crawl threads spawned by the Engine.
MaximumNumberOfHostsAndPrioritiesToSelect The maximum number of Hosts and Priorities to select for use by the PriorityQueue and the ReportingManager.
OutputConsoleToLogs Should console output be redirected to logs on disk?
OutputStatisticsObsolete. Should Crawl statistics be output to the console?
SaveDiscoveredFilesToDisk Should discovered Files be saved to disk?
SaveDiscoveredImagesToDisk Should discovered Images be saved to disk?
SaveDiscoveredWebPagesToDisk Should discovered WebPages be saved to disk?
SqlCommandTimeoutInMinutes The duration until a SqlCommand times out. This setting determines how long the application will wait for an executing SqlCommand (StoredProcedure, etc.) before timing out.
UserAgent The User-Agent HttpRequest header used to identify Crawls.
VerboseOutput Should console output be verbose? Verbose output sends the AbsoluteUri of eacu Discovery to the console.

See Also

ApplicationSettings Class | Arachnode.Configuration Namespace