Python Web Scraping & APIs

November 2, 2022, 3:00pm to 6:00pm

To receive a Zoom link after registering above, please fill out the affiliations form if you have not done so at least once before: https://dlab.berkeley.edu/affiliations

Trying to register, but not affiliated with the UCB campus? If you are from Berkeley Lab (LBL), UCSF, CZ Biohub, or other organizations, please register via our partner portals here.

Location: Remote via Zoom. Link will be sent on the morning of the event.

Recordings: This D-Lab workshop will be recorded and made available to UC Berkeley participants for a limited time. Your registration for the event indicates your consent to having any images, comments and chat messages included as part of the video recording materials that are made available.

Date & Time: This workshop runs from
3pm-6pm on:

  • Wednesday, November 2

Start Time: D-Lab workshops start 10 minutes after the scheduled start time (“Berkeley Time”). We will admit all participants from the waiting room at that time.

Description

In this workshop, we cover how to extract data from the web using Python. We focus on two approaches to extracting data from the web: leveraging application programming interfaces (APIs) and web scraping.

APIs are often official services offered by companies and other entites, which allow one to directly query their servers in order to retrieve their data. When APIs are not available, one can turn to web scraping, which requires downloading a webpage's source code and sifting through the material to extract the desired data. This workshop demonstrates both approaches, their advantages and disadvantages, and how to use both responsibly.

Basic familiarity with Python is assumed. Understanding the material in the Python Fundamentals and Python Data Wrangling workshops highly recommended. We additionally recommend a basic understanding of HTML and CSS.

Topics Covered

  • How the web works
  • Accessing databases via RESTful APIs
  • Using a 3rd party Python package to query data from a API
  • HTML / CSS
  • Webscraping with Beautiful Soup

Requirements

We will assume a basic knowledge of Python. If you've taken the D-Lab's Python Intensive, that should be sufficient.

Prerequisites: D-Lab’s Python Fundamental introductory series or equivalent knowledge.

GitHub Repository: https://github.com/dlab-berkeley/Python-Web-Scraping

Software Requirements:Installation Instructions for Python Anaconda

Is Python Not working on your laptop? Attend the workshop anyway, we can provide you with a cloud-based solution until you figure out the problems with your local installation.

Feedback: After completing the workshop, please provide us feedback using this form

Questions? Email: dlab-frontdesk@berkeley.edu