Quoting duncan (duncan at sodatrain.com):
> hello-
> 
> Does anyone have any experince with the htdig, mnogo or other search 
> engine software?

Both. Mnogo is better IMHO, htdig is easier to setup.

>  From what i read, i should be able to config one of them to search for 
> certain file types...
> 
> I want to spider a list of sites and grab URL's to .iso files and then 
> crunch them  and make a resource like linuxiso.org

Mnogo would be better at this and not a problems.

Server http://www.mycompany.com/foo/bar

Allow Case *.iso

-- 
Minneapolis St. Paul Twin Cities MN        | Phone : (952)943-8700
http://www.mn-linux.org Minnesota Linux    | Fax   : (952)943-8500
Key fingerprint =  6C E9 51 4F D5 3E 4C 66 62 A9 10 E5 35 85 39 D9