More still needs to be done to educate organisations on web scraping, according to new Oxylabs research.
It is estimated that more than half of organisations have not implemented web scraping due to legal concerns and complexity.
According to Oxylabs' new research with Censuswide, while many organisations use web scraping to drive business decisions and competitor pricing, legal concerns have taken the top spot, showing more still needs to be done to educate businesses on the practice.
This new research, which asked over 1000 UK and US-based senior decision makers, including CTOs, VPs of Engineering, Chief Data Officers, Heads of Business Intelligence, and Heads of Data & Analytics, explains how businesses are using newer, more sophisticated technology to gain an edge over their competition.
The main concern of businesses that have not implemented web scraping has been potential legal complications. More than half (53.06%) of all organisations that haven't implemented web scraping cite legal risks as the main reason.
Lack of technical expertise is the second most cited reason (39.59%), followed closely by budget restrictions (30.20%). Web scraping has become more accessible than ever, as only 1 in 5 businesses cites complicated technology as an excuse not to implement it (20.82%).
According to Oxylabs' Head of Legal, Denas Grybauskas, web scraping is still misunderstood and retailers should be aware of the huge benefits it offers - especially during the golden quarter.
“It has been very tricky to overcome the perception that behind every data point lies a legal risk, particularly when it comes to web scraping," says Grybauskas.
"Even though organisations should always consult with legal professionals, there are best practices within the industry that, when followed, minimise most risks associated with data collection. These results indicate a need to increase our efforts and educate the industry about the legal aspects of web scraping.
“To help educate business leaders, organisations can turn to initiatives such as Ethical Web Data Collection Initiative (EWDCI), which aims to reach out to businesses and educate them on the importance of ethical data collection, as well as educate users and legislators worldwide in the interest of informed, empowered action. This will encourage industry-led stewardship of critical and widely used technology in order to address current challenges and position the industry for future success," Grybauskas says.
“Retailers can use web scraping to improve customer experience, which is extremely important during the golden quarter. Companies can maximise profit while staying competitive and without losing the customer. This can be achieved by scraping product and market information to fine-tune the strategies to increase customer satisfaction through areas such as recommendation engines. Since this data also enables you to keep a thorough check on the not-so-static market at large, it helps you create a dynamic pricing strategy to keep up with the changes and help maximise the profit.”
Lack of technical expertise is a greater barrier to US-based companies (46.77% vs 37.16%), while budget constraints affect the UK more (32.24% vs 24.19%). Risks of legal complications is the greatest factor for both regions with nearly no differences (>50% for both).
“The web scraping industry still suffers from several misconceptions, and organisations need to be educated better on this issue," Grybauskas says.
"Web scraping is a powerful tool for analysing websites, and its benefits far outweigh the drawbacks. Retailers can gain insights into how to increase traffic and sales, manifest ideal conversation rates, and, ultimately, save their businesses and give them a new and refined appearance by analysing public web data”.