Web Scraping

Frances Leung

Data Science Fellow 2021-2022
School of Information

Frances Leung is a master’s student at UC Berkeley School of Information where she focuses her studies in information and data science. She has a keen interest in leveraging data-driven insights to better understand consumer behaviors and the world around us. In her professional work as a management consultant, she advises retailers and consumer businesses on digital transformation and creating web/mobile experiences that delight consumers through a human-centered approach. Frances holds a Master in Business Administration from York University, Schulich School...

The Evolving Landscape of Web Scraping on Social Media Platforms

March 11, 2025
Nanqin Ying
by Nanqin Ying. As social media platforms enforce stricter policies against unauthorized data collection, businesses and researchers must adapt to new API-based access models. This shift limits large-scale web scraping, impacting industries reliant on social media insights. The transition to paid API access and stringent compliance measures raises concerns about accessibility, cost, and ethical data collection. This article explores the evolving regulatory landscape, the enforcement of API restrictions, and how organizations can legally and ethically navigate data access in a world where scraping is becoming increasingly difficult. Understanding these changes is crucial for staying compliant while maintaining valuable insights from social media data.

Suraj Nair

Data Science Fellow 2023-2024
School of Information

I am a PhD Student at the School of Information. My research interests lie at the intersection of development economics and machine learning, with a focus on the use of large scale digital data and new computational tools to study pressing issues in global development.

Lauren Chambers

Consultant
School of Information

Lauren Chambers is a Ph.D. student at the Berkeley School of Information, where she studies the intersection of data, technology, and sociopolitical advocacy with Prof. Deirdre Mulligan. Previously Lauren was the staff technologist at the ACLU of Massachusetts, where she explored government data in order to inform citizens and lawmakers about the effects of legislation and political leadership on our civil liberties. Lauren received her Bachelor's from Yale in 2017, where she double-majored in astrophysics and African American studies, and she spent two years after graduation in...

Ini Umosen

Consultant
Economics

Ini is a PhD candidate in the Department of Economics. She studies topics in labor economics and the economics of education using applied econometrics methods. Current work in progress includes evaluating the impact of school choice systems and investigating gender and racial bias on gig platforms. She is a former Graduate Research Fellow at the California Policy Lab. She has also been a tutor for econometrics, labor economics, and macroeconomics.

Stephanie Andrews

Availability: By appointment only

Consulting Areas: Python, SQL, HTML / CSS, Javascript, APIs, Databases & SQL, Data Manipulation and Cleaning, Data Science, Data Sources, Data Visualization, Digital Humanities, Machine Learning, Natural Language Processing, Software Tools, Text Analysis, Web Scraping, Bash or Command Line, Excel, Git or Github, Tableau

Kurt Soncco Sinchi

Consultant
Civil Engineering

First generation student and looking to improve and apply Data Science core concepts into social impactful projects, as well as trying to leverage the information from previous cases for better insights of society. Focused on infrastructure and its impact under natural disasters.

Stephanie Andrews

Consultant
Info & Data Science MIDS

Stephanie Andrews is currently studying data science in the MIDS program, having previously majored in Social Welfare as an undergraduate at Cal. After graduating, she worked as an advocate for survivors of gender-based violence, as a public policy analyst focusing on anti-trafficking initiatives, and as a software engineer for progressive and social impact organizations. She is now conducting research with the Human Rights Center's Investigations Lab, using OSINT and data science methods to investigate human rights violations.

Chirag Manghani

Consultant
School of Information

Chirag is a 2nd year graduate at the I-School. Proficient in Python, Java, R, and SQL, he navigates software application development, machine learning and data science. His keen interest lies in data analysis and statistical methods, driving him to bridge theory and practice seamlessly. Chirag's dedication to excellence, adaptable mindset, and innate curiosity define him as a dynamic problem solver in the ever-evolving tech landscape.

How can we use big data from iNaturalist to address important questions in Entomology?

February 26, 2024
Leah Lee
by Leah Lee. Large-scale geographic data over time on insect diversity can be used to answer important questions in Entomology. Open-source, open-access citizen science platforms like iNaturalist generate huge amounts of data on species diversity and distribution at accelerating rates. However, unstructured citizen science data contain inherent biases and need to be used with care. One of the efforts to validate big data from iNaturalist is to cross-check with systematically collected data, such as museum specimens.