Url Monitor
This feature enables you to monitor web pages for changes. It can work with multiple web sites. Each time this task runs, it updates a log file on your system, with the modified time stamp of each web page it finds, within the web site. If a change is detected, it will download the file to a selected directory. HTTPS downloads require additional SSL install files, which are available from our website.
Url to spider
Enter the full pathname of the URL's to monitor. You must include the protocol ( http://). You can specify multiple URL's, by separating then with a comma. Example - http://www.hiteksoftware.com,http://www.espn.com. This field also supports dynamic variables.
Hyperlink Depth
Select the Hyperlink depth(D) required. If D=0, only the root Url is monitored. If D=1, direct links(DL1) to the root Url are also monitored. If D=2, direct links(DL2) to each DL1 are also monitored, and so on....
WARNING: Use D=1 to test. If you have a slow connection, and/or the web site has many large files, this task could run for hours!!
Local Dir
Enter the full pathname of the local directory to place all the results. If a file is found to have been changed, then it is downloaded to this directory.
Filename Filter
Leave this field blank if you want all files to be downloaded, or enter the filename filter for the files you wish to download using the following wildcard rules . This field also supports dynamic variables. Using dynamic variables, you can select a file based on current date/time.
You can also select only specific sub folders, to be downloaded, by entering the sub folders relative path. For the task to realize, that a sub folder filter is used, the filename filter has to include at least one path separator '/'. Example: /test or /test/subfolder etc..
You can use multiple filters by separating them with ^. Example: test.htm^prod would download test.htm, products.htm, produce.htm etc.. Please note that when you use multiple filters, only one of the filters has to be satisfied, for the url to be downloaded.
Maintain Timestamp
Select this option, if you need the downloaded file, to have the same timestamp, as the file on the web server.
Download Option
There is three options available:
1) Download always - This will overwrite the existing local file, irrespective of timestamp.
2) Download based on local file timestamp - If a local file exists in the download folder, a new file will be downloaded, only if the timestamp of the file on the web server, is newer than that of the existing local file.
3) Download based on archived url timestamp - Every time a file is downloaded by this task, its timestamp on the web server is saved locally. If this option is selected, a new file is downloaded from the web server, only if its timestamp is newer than the saved value.
Place files in new subdirectory
To regularly monitor certain URL's, you can save each download to a different directory, based on current time/date. Select the "Create New Subdirectory" option, which best suits your download frequency. A new subdirectory, based on current date or time, will be created. All downloaded files and results will be placed in this subdirectory.
Proxy Parameters
If you are behind a firewall, enter the proxy settings. Your network administrator can provide you with the proxy settings. You can also obtain the proxy settings from your web browser.