This is submitted to Bright Data Web Scraping Challenge: Scrap data from complex interactive websites
what i built
I developed a script to pull cell phone plans from Carphone repository, I got the data like phone, its color and available capacity, I stored this data in s3 and displayed it using my Laravel project.
I use Bright Data’s browser to avoid any blocking as telecom websites usually block after a while.
In order to extract the data, I had to interact with the page using clicks to open the next page of results, and I found Bright Data’s speed and stability to be very good.
demonstration
How I use Bright Data
I used the Bright Data scraping browser tool, which was very helpful in making my own code lightweight because I packaged the script in docker so I didn’t have to include my own browser, plus it was very Good to know I have captcha handling and a built-in proxy.
I’m running into a challenge where I can only navigate once per session. I solved this problem by requesting a new session for each page navigation. While the solution is simple, it does require some reorganization of the code.
BrightData Challenge – Monthly Phone Plan Aggregator
The project is part of the BrightData Web Scraper Challenge and aims to compile Carphone Warehouse’s monthly phone plans, including those from Vodafone and ID Mobile. The goal is to efficiently capture detailed information and present it in a user-friendly format 🚀 Challenge Tips
Use a Web Scraper API to tackle common business challenges like aggregating product prices, monitoring competitors, or collecting reviews across platforms. Use Bright Data’s Web Scraper API for efficient and scalable data collection
🌟 Project Overview Objectives
Scrape and aggregate Carphone Warehouse's monthly phone plans.
Collect details such as:
Phone models
Storage capacities
Available colors
Associated monthly plans
Solution Highlights
Scrape the entire selection of phones and plans available on Carphone Warehouse.
Schedule scraping jobs to run daily or weekly to save costs and keep data fresh.
Use Bright Data’s browser-based scraper to bypass blocking mechanisms.
🛠️ Technology used…
generalize
I thank Bright Data for giving me the motivation to finally explore Bright Data, it’s been a pleasure and I hope to continue using it in the near future!