Web-scraping for business information

Overview

National Statistics Offices (NSOs) have used web-scraping to collect information on the prices of goods and services for some time now. However, some work is now being done by European NSOs on using web-scraping for collecting other kinds of statistical information on businesses. The aim of this project is to investigate the feasibility of using this approach in Australia.

The initial part of the project would be to review the methods and approaches developed in Europe and to assess how applicable these would be in an Australian context. The project would then identify and develop a simple prototype to further test the feasibility of the methods. This prototype might implement only part of the web-scraping and information extraction process if the entire process would be unachievable within the timeframe of the project.

One possible use case for this work is to better identify the target population for the ABS Survey of Research and Experimental Development, Businesses (SRED). It is difficult to assess whether or not a business is engaged in research and development (R&D) from information contained in our survey frames. A frame is a dataset with a list of the units in the population of interest from which we wish to select a sample of units to participate in a survey. To create a frame for the SRED we often draw together a range of information from previous surveys, grants, tax concessions, and expert knowledge. Being able to identify whether or not a business engages in R&D from their online presences could simplify the process of developing a frame considerably.

Research engagement

Critical Literature Review, Implementation of Methods, Analytic Comparisons

Research activities

The initial part of the project would be to review the methods and approaches developed in Europe and to assess how applicable these would be in an Australian context. The project would then identify and develop a simple prototype to further test the feasibility of the methods. This prototype might implement only part of the web-scraping and information extraction process if the entire process would be unachievable within the timeframe of the project.

Research skills

Critical Thinkimg, Web and Data Science Literarcy, Written Communication

Outcomes

Written Report and Possible Presentation to the ABS

Skills and experience

Data and Computer Science Skills  Preferred

Start date

2 November, 2026

End date

19 February, 2027

Location

GP Y Block Level 8

Keywords

• Web Scraping
• Data Science

Contact

Gentry White

3138 1658

whiteg5@qut.edu.au