house_advertisement_crawler

A crawler that contains house advertisement data from cragslists site.

Build With

• mongoDB

• python

How To Use

• at first you should install virtualenv

apt-get install python3-virtualenv

• then create a new virtual environment

virtualenv -p python3 venv

• then we have to activate it

. venv/bin/activate

• now we got to install the requirements

pip install the requirements

• to run the projects first of all we have to find the links we want to crawl so you have to run:

python3 main.py "find_links"

• now we got the links in our db now we should crawl the links:

python3 main.py "extract_pages"

• now the crawl is done and you have pure data in case you want to download the image from links:

python3 main.py "download_images"

• Wish you have enjoyed that!Good Luck:D