Data Science

What can state government do…to attract a data scientist like YOU?

March 29, 2022

What can state government do…to attract a data scientist like YOU?

By Kellie Hogue

What’s your next move? When I was in grad school, one of my professors told me that regardless of the job I am currently in, I should always be planning the next step in my career.

At the time, it made sense–academic appointments in my discipline were few and far between, and I wouldn’t get one without some major strategic networking and planning. Simply a case of too much supply, not enough demand....

Predicting Madness: This March Madness, you can be your friend group’s resident Bracketologist.

March 7, 2022

On Selection Sunday, a twelve-member NCAA committee kicks off March Madness by picking America’s best college basketball teams. Each year, millions of people build their bracket based on records, school allegiances, favorite colors, and weirdest mascots. The national college basketball event that pins the top 64 Division I teams in the country in a knockout style tournament is one of the largest betting events in sports. In the course of 68 games, over $8.5 billion across 40 million bets are estimated to be made both legally and illegally (Odds Shark, 2021). ...

Twitter data extraction with Selenium

March 1, 2022

Introduction

With online communities and social networks serving as important sites for computational social science research, Twitter has quickly become a popular data source for researchers (Frey et al. (2020), Kusen et al. (2017), Rao et al. (2010) and Ru et al. (2021)). This blog post will demonstrate one way to extract twitter data without using the Twitter API. This is especially useful for researchers who are new to exploring the use of Twitter data in their research, looking to develop a baseline corpus for a research question they are newly...

Getting Started with the NYT API

March 1, 2022

Introduction

The web is chock full of valuable troves of data that can spawn an infinite number of social science research projects. However, not all data is easily accessible! While some data can be easily downloaded, access to some sources of data are dictated by what is known as an API. Standing for application programming interface, APIs are a set of defined protocols governing the terms of access to software and servers from programs created...

Enumeration of Informal Work

March 1, 2022

The first time that I mapped out poverty statistics at a municipal scale, I was completely mind blown (figure 1). Looking at the spatial inequities from a bird’s-eye view drove my desire to find more granular data of social indicators to better understand intra-urban socioeconomic inequities. Spatial data techniques help us to find patterns and anomalies across data that improves our understanding of people’s lives in cities, raising new questions about urban infrastructure in terms of public goods provision, land-use, and access. However, finding granular socioeconomic...

Ian Castro

D-Lab Alumni
School of Information

Ian is a graduate student in the Master of Information Management and Systems program at the School of Information with a focus in applied data science. He earned his B.A. in Media Studies and B.S. in Microbial Biology from UC Berkeley, and his research interests and work experience are in STEM education. He focuses in building courses and academic programs to make data and computing accessible to historically marginalized students and those without prior exposure to the field.

PoliPy: A Python Library for Scraping and Analyzing Privacy Policies

February 8, 2022

In light of recent scandals involving the misuse and improper handling of personal data by large corporations, advocacy groups and regulators alike have given increased attention to the issue of consumer privacy [e.g., 1, 2, 3, 4, 5]. National and local governments have been enacting privacy legislation that requires companies to minimize the amount of data they collect, deters the collection of sensitive data, limits the purposes for which the data are used, and critically, gives users more transparency into data collection and use.

As part...

Where the Streets Have No Name: Spatial Data in Informal Settlements

February 1, 2022

In our era, with Google Maps on every smartphone, it may feel like spatial data is easy to come by. However, this is not the case for many communities in the world. In particular, for informal settlements, developed “outside state control over urban design, planning, and construction,” accurate maps can be hard to come by. You may open up Google Maps to find a few streets with no names, or sometimes, nothing at all. Informal settlements are...

Is your Random Sample Really Random?

January 20, 2022

One of the frequent ways people can run into random numbers is through their research. We often hear the term “random sample,” or a “randomized” assignment to control. Or, sometimes, we can randomly select a certain number of rows or columns from data to perform an analysis on a representative snapshot of the data. Additionally, for many of us from a natural science or engineering background, random numbers are often used in simulations or optimization models. Given the wide variety of uses for random numbers in Data Science, I thought it would be interesting to take an...

Working with spatial networks using NetworkX

December 7, 2021

I have always been interested in working with spatial networks. My first introduction to spatial network modeling was in Prof. John Radke’s Geographic Information Systems class when I learned about building and analyzing spatial networks using the Network Analyst extension in ArcMap. This extension provides powerful tools to solve common network problems, such as finding the best route across a city, finding the closest...