Personal WebCrawler with Download Manager
The name of this program is HumanSurfer. I developed this program mostly between fall 2001 and sometime in 2004. The program has never been publicly released. I'm planning on making a shareware release of this program when I have time.
Some features of the program:
- Separation between surfing links and file links. Surfing links are mostly whole web pages, HTML, that can be surfed for more links. File links are any files that the user might want to download.
- Surfing links and file links are fully configurable by parts of URL and by MIME media type.
- Separate multi-threaded engines for surfing the surfing links (to find more links) and for downloading file links.
- Find links from many different HTML elements. Guessing algorithm to find some links from JavaScript.
- Handle large sets of links: making "dummy" links, selecting links by keyword, filtering by link type (e.g. redirect/normal, downloaded / non-downloaded).
- Possibility to add sets of links with a serial number as part of URL.
- Options to restart downloading on slow connections and continue partial downloads.
- Surfing depth configured separately as between-sites and within-site depth.
- Advanced naming schemes for downloaded files. Different options for naming downloading folders and the downloaded files.
- Export link lists as plain text or XML Sitemap files.