Web Scraping

The Evolving Landscape of Web Scraping on Social Media Platforms

March 11, 2025
Nanqin Ying
by Nanqin Ying. As social media platforms enforce stricter policies against unauthorized data collection, businesses and researchers must adapt to new API-based access models. This shift limits large-scale web scraping, impacting industries reliant on social media insights. The transition to paid API access and stringent compliance measures raises concerns about accessibility, cost, and ethical data collection. This article explores the evolving regulatory landscape, the enforcement of API restrictions, and how organizations can legally and ethically navigate data access in a world where scraping is becoming increasingly difficult. Understanding these changes is crucial for staying compliant while maintaining valuable insights from social media data.

Lauren Chambers

Consultant
School of Information

Lauren Chambers is a Ph.D. student at the Berkeley School of Information, where she studies the intersection of data, technology, and sociopolitical advocacy with Prof. Deirdre Mulligan. Previously Lauren was the staff technologist at the ACLU of Massachusetts, where she explored government data in order to inform citizens and lawmakers about the effects of legislation and political leadership on our civil liberties. Lauren received her Bachelor's from Yale in 2017, where she double-majored in astrophysics and African American studies, and she spent two years after graduation in...

Ini Umosen

Consultant
Economics

Ini is a PhD candidate in the Department of Economics. She studies topics in labor economics and the economics of education using applied econometrics methods. Current work in progress includes evaluating the impact of school choice systems and investigating gender and racial bias on gig platforms. She is a former Graduate Research Fellow at the California Policy Lab. She has also been a tutor for econometrics, labor economics, and macroeconomics.

Tom van Nuenen, Ph.D.

Data/Research Scientist, Senior Consultant, and Senior Instructor
D-Lab
Social Sciences
Digital Humanities

I work as a Lecturer, Data Scientist, and Senior Consultant at UC Berkeley's D-Lab. I lead the curriculum design for D-Lab’s data science workshop portfolio, as well as the Digital Humanities Summer Program at Berkeley.

Former research projects include a Research Associate position in the ‘Discovering and Attesting Digital Discrimination’ project at King’s College London (2019-2022) and a researcher-in-residence role for the UK’s National Research Centre on Privacy, Harm Reduction, and Adversarial Influence Online (2022). My research uses Natural Language Processing methods to...

Stephanie Andrews

Availability: By appointment only

Consulting Areas: Python, SQL, HTML / CSS, Javascript, APIs, Databases & SQL, Data Manipulation and Cleaning, Data Science, Data Sources, Data Visualization, Digital Humanities, Machine Learning, Natural Language Processing, Software Tools, Text Analysis, Web Scraping, Bash or Command Line, Excel, Git or Github, Tableau

Stephanie Andrews

Consultant
Info & Data Science MIDS

Stephanie Andrews is currently studying data science in the MIDS program, having previously majored in Social Welfare as an undergraduate at Cal. After graduating, she worked as an advocate for survivors of gender-based violence, as a public policy analyst focusing on anti-trafficking initiatives, and as a software engineer for progressive and social impact organizations. She is now conducting research with the Human Rights Center's Investigations Lab, using OSINT and data science methods to investigate human rights violations.

Kurt Soncco Sinchi

Consultant
Civil Engineering

First generation student and looking to improve and apply Data Science core concepts into social impactful projects, as well as trying to leverage the information from previous cases for better insights of society. Focused on infrastructure and its impact under natural disasters.

Chirag Manghani

Consultant
School of Information

Chirag is a 2nd year graduate at the I-School. Proficient in Python, Java, R, and SQL, he navigates software application development, machine learning and data science. His keen interest lies in data analysis and statistical methods, driving him to bridge theory and practice seamlessly. Chirag's dedication to excellence, adaptable mindset, and innate curiosity define him as a dynamic problem solver in the ever-evolving tech landscape.

Sahiba Chopra

Data Science Fellow 2024-2025
Haas

I'm a PhD student in the Management and Organizations (Macro) group at Berkeley Haas. I have a diverse professional background, primarily as a data scientist across numerous industries, including fintech, cleantech, and media. I hold a BA in Economics from the University of Maryland, an MS in Applied Economics from the University of San Francisco, and an MS in Business Administration from UC Berkeley.

My research focuses on the intersection of inequality, technology, and the labor market. I am particularly interested in understanding how to reduce inequality in...

How can we use big data from iNaturalist to address important questions in Entomology?

February 26, 2024
Leah Lee
by Leah Lee. Large-scale geographic data over time on insect diversity can be used to answer important questions in Entomology. Open-source, open-access citizen science platforms like iNaturalist generate huge amounts of data on species diversity and distribution at accelerating rates. However, unstructured citizen science data contain inherent biases and need to be used with care. One of the efforts to validate big data from iNaturalist is to cross-check with systematically collected data, such as museum specimens.