This program helps you scrape all the URLs available over a list of websites available. It includes URLs in clear-text as well as hyper-linked.
Pre-Requisites: bs4, tqdm, time
To install the above pre-requisites, use pip or pip3
pip3 install bs4
pip3 install tqdm
pip3 install time
Todo:
- Add all your target websites one-per-line, and name the file as 'sites.txt' and save it in the same folder as this program (DigMyDomains.py)
- Run the program
usage: python3 DigMyDomains.py