trthaber-crawler

A crawler that contains news data from trthaber.

Build With

• sqlite

• python

How To Use

• at first you should install virtualenv

apt-get install python3-virtualenv

• then create a new virtual environment

virtualenv -p python3 venv

• then we have to activate it

. venv/bin/activate

• now we got to install the requirements

pip install the requirements

• first of all we have to create tables:

python3 main.py "create_tables"

• to run the projects first of all we have to find the links we want to crawl so you have to run:

python3 main.py "get_links"

• now we got the links in our db now we should crawl the links:

python3 main.py "get_articles"

• now the crawl is done and you have pure data in case you want to see status run:

python3 main.py "stats"

• Wish you have enjoyed that!Good Luck:D