Hello friends of this Mr.Rezoo
In this repository we want to explore the depths of crawling and scraping
I decided to implement several small projects in accordance with the book Practical Web Scraping and Crawling for Data Science and following the contents of it, to learn how to line up like a professional.
This book provides a complete and modern guide to web scraping, using Python as the programming language, without glossing over important details or best practices. Written with a data science audience in mind, the book explores both scraping and the larger context of web technologies in which it operates, to ensure full understanding. The authors recommend web scraping as a powerful tool for any data scientist’s arsenal, as many data science projects start by obtaining an appropriate data set.
Starting with a brief overview on scraping and real-life use cases, the authors explore the core concepts of HTTP, HTML, and CSS to provide a solid foundation. Along with a quick Python primer, they cover Selenium for JavaScript-heavy sites, and web crawling in detail. The book finishes with a recap of best practices and a collection of examples that bring together everything you've learned and illustrate various data science use cases. What You'll Learn
Leverage well-established best practices and commonly-used Python packages Handle today's web, including JavaScript, cookies, and common web scraping mitigation techniques Understand the managerial and legal concerns regarding web scrapingWho This Book is For A data science oriented audience that is probably already familiar with Python or another programming language or analytical toolkit (R, SAS, SPSS, etc). Students or instructors in university courses may also benefit. Readers unfamiliar with Python will appreciate a quick Python primer in chapter 1 to catch up with the basics and provide pointers to other guides as well.
Project is created with:
- Python: 3.9
- requests
- beautifulsoup4
- selenium
- scrapy
If you are considering a particular method, more modern technology Add to my project and send merge request, I will add you in the credits and contributors section
- first step : create virtual environment
virtualenv -p python3 venv - second step : activate virtual environment
source venv/bin/activate - third step : install package | library from requirements.txt
pip install -r requirements.txt- to run scrapy spiders
scrapy crawl <spider-name> if you want to store data in => json, jl, csv, excel
scrapy crawl <spider-name> -o <file-name.extention> - MrRezoo
- Seppe vanden Broucke | author of Practical Web Scraping and Crawling for Data Science
Distributed under the MIT License. See license for more information.