Scraping USA Ultimate
Every summer, I play Ultimate Frisbee and get to travel across the country with my club team to compete at a high level. It’s a well run sport, with the governing body USA Ultimate managing over 600 teams and 15,000 players anually.
Unfortunately, the USA Ultimate website is an archaic ASP.NET web app that is very difficult to scrape. It’s server rendered, meaning it makes no client-side network requests to load data (e.g. teams, players, etc) and instead delivers the fully hydrated page every time you load it.
This means that to scrape usable data, you’d have to parse the raw HTML on every page. You could settle for this approach, but you would have to maintain lots of custom scraping logic per page thats both difficult to read and brittle. Plus, USA Ultimate has all sorts of different tournament formats, which means you have a lot of edge cases to consider and worry about.
Luckily for us, USA Ultimate also has a mobile app. Mobile apps will fetch data client-side, which means theres an API we can find if we observe the network requests.
We’ll be using a tool called Mitmproxy to intercept the network requests from our phone and see where the installed mobile app calls out to. Mitmproxy is free and open-source, plus it has a CA cert you can install on your device so that you can decrypt HTTPS traffic.
This gives us an API that returns beautiful JSON data, easy to parse and interpret. The only problem is we don’t know the full list of API routes, theres no openapi docs and this isn’t documented anywhere. My first thought was to just traverse different path names, given the current urls we found from the network requests
https://usau-stats.newstartmobile.com/api/usau/events/{event id}/games its reasonable to start trying different paths like /api/usau/teams or /api/usau/players.
That said, it’s 2026 and we have AI that can do this tedious task for us! I fired off the following very well thought-out and throrough prompt into Pi:
I want you to setup and run a script that searches for all endpoints on https://usau-stats.newstartmobile.com/api/usau
With that, we now have a list of endpoints we can hit to get the desired data we want. This gives us clean, structured, and most importantly: accurate data that we can use.
Using the data
Why scrape all this data? For nerds of the sport, having a comprehensive dataset of all games and results means that you can run USA Ultimate’s ranking algorithm on demand. USA Ultimate runs it with their internal data and publishes it weekly, below is an excerpt describing it.
Beginning July 29, teams are ranked weekly using an algorithm developed by the USA Ultimate Rankings Working Group to evaluate each team’s performance throughout the season. At the end of the regular season, rankings determine bids to regionals as well as the number of bids each of the eight geographical regions receive to the National Championships. Each region gets at least one automatic bid to nationals. Teams can earn additional wildcard bids for their region based on their rankings. Final rankings are released following the National Championships in October.
However, knowing your teams algorithmic ranking going into a game can impact your game plan, like knowing how many goals you need to win by to improve your rank. Now that we have an authoritative set of data that should be the same as what USA Ultimate uses, we can safely generate rankings using the algorithm. No need to worry that our jank HTML scraper missed some games.