How to scrape data from a website in Python
Published in · 9 min read · Jul 15, 2020
--
We always say “Garbage in Garbage out” in data science. If you do not have good quality and quantity of data, most likely you would not get many insights out of it. Web Scraping is one of the important methods to retrieve third-party data automatically. In this article, I will be covering the basics of web scraping and use two examples to illustrate the 2 different ways to do it in Python.
What is Web Scraping
Web Scraping is an automatic way to retrieve unstructured data from a website and store them in a structured format. For example, if you want to analyze what kind of face mask can sell better in Singapore, you may want to scrape all the face mask information on an E-Commerce website like Lazada.
Can you scrape from all the websites?
Scraping makes the website traffic spike and may cause the breakdown of the website server. Thus, not all websites allow people to scrape. How do you know which websites are allowed or not? You can look at the ‘robots.txt’ file of the website. You just simply put robots.txt after the URL that you want to scrape and you will see information on whether the website host allows you to scrape the website.
Take Google.com for an example
