Businesses that don’t rely on data have a meager chance of success in a data-driven world. One of the best sources of data is the data available publicly online on various websites and to get this data you have to employ the technique called Web Scraping or Data Scraping. The purpose of this article is to walk you through some of the things you need to do and the issues you need to be cognizant of when you do large scale web scraping.
Feb 12, 2021 Mozenda is an enterprise web scraping software designed for all kinds of data extraction needs. They claim to work with 30% of the fortune 500, for use cases like large-scale price monitoring, market research, competitor monitoring. They can build and host the scraper for you Who should use this web scraping tool? With scrapestack, you don't have to worry about the tricky programming issues that arise because they are handled by the API provider. Using the scrapestack API, you can quickly and efficiently scrape web pages worldwide in milliseconds and turn them into actionable data that you can put to good use. When scraping SERPs at scale, you will come across a series of problems and challenges. You may need to make your requests from a specific location or country. Or you may want to work around CAPTCHA and other anti-bot solutions. You may simply want to make requests more frequently, to get data more frequently.
Building and maintaining a large number of web scrapers is a very complex project that like any major project requires planning, personnel, tools, budget, and skills.
You will most likely be hiring a few developers who know how to build scalable scrapers and setting up some servers and related infrastructure to run these scrapers without interruption and integrating the data you extract into your business process.
You can use full-service professionals such as ScrapeHero to do all this for you or if you feel brave enough, you can tackle this yourself.
This article will give you tips on how to build scrapers on a large scale. But to learn more about what is web scraping and how to build a web scraper using Python, you can read our guide –
How to Build Web Scrapers
The best way to build a web scraper would be using one of the many web scraping tools and frameworks. Next, its recommended to select a web scraping framework for building your scrapers – like Scrapy (Python), PySpider (Python) or Puppeteer(Javascript).
You can get started with building your own web scraping solution by following some of our web scraping tutorials.
The best reason build your own web scraper would be that you won’t run into the risk of your developer(s) disappearing one day, leaving no one to maintain your scrapers. You will not lock yourself into the ecosystem of a proprietary tool, having no way to move hundreds of scrapers into another tool if they shut down. This has already happened in our industry with a high profile company, read more about that here – Demise of Kimono Labs and the fate of their users.
The choice of tools or frameworks should depend on a few factors depending upon the website(s) that you plan to scrape. Some tools are better than others at handling the complexity of the sites. Generally speaking, the more flexible the tool, the more the learning curve, and conversely the easy-to-use tools may not handle complex sites or complex logic.
What is a complex website?
Any website that is built using some advanced JavaScript frameworks such as React or Angular is usually complex if you are extracting a lot of data from it. You will need a real web browser such as Puppeteer or Selenium to scrape the data. Alternatively, you can check and reverse engineer the REST API of the website if it exists.
What is a complex logic?
Here are some examples of complex logic:
- Scraping details from a listing website, search with the listing name in another site, combine this data and save it to a database
- Take a list of keywords, perform a search in google maps for each keyword, go to the result – extract contact details, repeat the same process on Yelp and a few other websites, then finally combine all this data.
Should you use a visual web scraping tool?
Visual web scraping tools are pretty good at extracting data from simple websites and are easy to get started with. But once you hit a wall, there isn’t much you can do. We recommend that you use visual tools for extracting data from websites that are not too complicated or if your scraping logic is complex.
We are yet to find an open-source visual web scraping tool that can handle complex logic. If the website is complex and you need to do large scale web scraping. You are better off building a scraper from scratch using a programming language like Python.
Which programming language is the best for building web scrapers?
Another common question we usually hear from our readers at ScrapeHero is: Which programming language should we use for building web scrapers?
We recommend that you use Python. The most popular web scraping framework, Scrapy is built using Python. It also has the most number of web scraping frameworks and is excellent for parsing and processing data.
You could use tools like Selenium with Python for scraping most of the modern websites built using Javascript frameworks like React, Angular, or VueJS. There is also a massive community of Python developers if you are in need of better talent.
Learn more about building web scrapers
How to run web scrapers at large scale
There is a massive difference between writing and running one scraper that scrapes 100 pages to a large scale distributed scraping infrastructure that can scrape thousands of websites or millions of pages a day.
Did you know?
ScrapeHero’s infrastructure can scrape websites at a rate of 3,000 pages per second, but we never use it all on a single website. Be nice to websites you scrape without putting a heavy load on their servers and CDNs.
Here are some tips to run web scrapers on a large scale –
Distributed Web Scraping Architecture
For scraping millions of pages daily, you will need a few servers and a way to distribute your scrapers across them and have them communicate with each other. Here are some components you would need to make this happen –
URL Queue and a Data Queue – using Message Broker like Redis, RabbitMQ or Kafka to distribute URLs and data across the scrapers that are running in different servers. You can design scrapers to read URLs from a queue in the broker. Scrape them and put the extracted data into another queue and feed newly discovered URLs into the URL Queue. Another process would read from the data queue and write it to a database while the scraper is running. You could skip this step and write directly to the database from the scraper if you are not writing a lot of data.
You would also need some process manager to make sure your scraper scripts restart automatically. If they are killed for some reason while parsing data.
Frameworks like Scrapy – Redis, and PySpider lets you skip a lot of the above tasks.
Scheduling website scrapers for periodic data collection
If you need to refresh data periodically, you can either run it manually or automate it using some tool. If you are using Scrapy, scrapyd + cron can schedule the spiders for you, and it will update the data the way you need it. PySpider also has a similar interface to do this.
Databases to store a large number of records
Once you have this massive data trove, you need a place to save it. We would suggest using a NoSQL database like MongoDB, Cassandra or HBase to store this data, depending upon the frequency and speed of data scraping.
You can then extract this data from this database/datastore and integrate it with your business process. But before you do that, you should set up some reliable Quality Assurance tests for your data (more on that later)
Use IP Rotation and Proxies to Get Past Anti Scraping Tools
Large scale scraping comes with a multitude of problems and anti-scraping tools and techniques are one of the biggest. There is a considerable number of Bot Mitigation & Screen Scraping Protection Tools aka anti-scraping tools like Akamai, Distill Networks, Shield Square, Perimeter X, etc. that block your scrapers from accessing websites usually by an IP Ban.
If any of the websites you need have any IP based blocking involved, your servers’ IP address will be blacklisted in no time. The site will not respond to requests from your servers or show you a captcha, leaving you with very few options after getting blacklisted.
As we are talking about millions of requests. You might need some of these
- Rotate the requests you make through 1000+ private proxies if you are not dealing with a captcha or anti-scraping service
- Make requests through a provider with 100,000+ geo proxies that are not completely blacklisted when dealing with most anti-scraping solutions
- Or time and resources to reverse engineer and bypass the anti-scraping solutions
Learn more about proxies and IP bans
Data Validation and Quality Control
The website data scraped is only as good as its quality. To ensure the data that you scraped inaccurate and complete, you need to run a variety of Quality assurance tests on it right after it is scraped. Especially when doing large scale web scraping, validate every field of every record before you save it or before you process it.
Having a set of Tests for the integrity of the data is essential. You can automate some of it by using Regular Expressions, to check if the data follows a predefined pattern. Else if it doesn’t then generate some alerts so that it can be manually inspected.
There are tools like Pandas, Cerebrus, Schema, etc built on Python, to help you validate the data record by record.
If the scraper is part of a data pipeline, you can set up multiple stages in the pipeline. Then verify the integrity and quality of the data using ETL Tools.
Maintenance
Data Scrapers
Every website will change its structure now and then, and so should your scrapers. Website scrapers usually need adjustments every few months or even weeks. As a minor change in target websites affecting the data fields you scrape might give you incomplete data or crash the scraper, depending on the logic of your scraper.
You need a mechanism to alert you if a lot the data you scrape is invalid or empty all of a sudden, so that you may check the scrapers and the website for changes that triggered it. You need to fix the scraper once it is broken either manually or by building some sophisticated logic to repair itself quickly to prevent disruptions in your data pipeline.
Database and Storage
If you’re doing large scale web scraping, you need lots of data storage and it’s best to plan up-front. Small amounts of data can be stored in flat files or spreadsheets. But when the data becomes too large for a spreadsheet to handle, you need to evaluate storage options including Cloud Storage and Cloud Hosted Databases (S3, Azure SQL, Azure Postgres, Redshift, Aurora, RDS, DynamoDB, Redis) or Relational Databases (MySQL, Oracle, SQL Server) or NoSQL databases (MongoDB, Cassandra etc).
Depending upon the size of data, you need to clean your database of outdated data to save space and money. You might also want to scale up your systems if you still need the old data. Sharding and Replication of databases can be of help.
You need to make sure that you do not keep anything that is not of use after the scraping is complete.
Know when to ask for help
As you might have realized by now, this whole process is expensive and time-consuming. You need to be ready to take on this challenge for large scale web scraping.
Web Scraping Applications
You also need to know when to stop and ask for help. ScrapeHero has been doing all this and more for many years now, let us know if you need help to build a scraper.
Or you can just worry about what to do with the data, we deliver to you. Interested ?
Turn the Internet into meaningful, structured and usable data
Web Scraping Selenium Python
Web Scraping At Scale Example