Corey
A PySpark-native way to do recursion
Part 2: Solutions
data
spark
python
In my last post, I described an example of recursive algorithms, the Fibonacci sequence, and showed that it can’t be solved with classic SQL tools like window functions. In this post, I’ll explore possible solutions and demonstrate my preferred, PySpark-native approach.
A PySpark-native way to do recursion
Part 1: the Problem
data
spark
python
I find lots of situations in my work where recursion in Spark dataframes would be useful. Recently, I needed to define a column with values that were based on each following value, recursively. That’s to say, each row could only be computed after the row before it.
How I work remotely
tech
remote work
When I landed my job at Kin + Carta, I invested in some creature comforts for remote work. At Yale, I kept hoping the office was a few elusive weeks away, and that it wasn’t worth investing in gear that I would soon not need. That was silly at the time, but ridiculuous once I took a remote job. I began searching for a configuration that would: - save my neck the strain of looking down at a 13-inch laptop - save my back the pain of sitting in a $40 Ikea chair 8 hours a day - travel well - bring some aesthetic joy to my day - work for play as well as work
Farmington Canal Trail Bike Century
fitness
cycling
new haven
bucket list
I learned to ride a bike just before I moved to New Haven. Living in a city, abandoning my boring, dangerous, radio-less car seemed like a great statement to make about myself and the world I wanted to see. Unfortunately, Amtrak is expensive and getting to the White Mountains is pretty hard without a car.
Tip the valet!
code bits
R
Valet, the Bank of Canada’s API, did not have an R client when the YPFS was studying its COVID work, and I pretty much stick to R. So, to demonstrate that I could build a solution quickly, without reinventing the wheel, I wrote valet, which was just accepted by CRAN! If you have a tip for me, or a bug that needs fixing, let me know.
YPFS Citation Style Language
open data
YPFS
code bits
When I started at YPFS, we did all COVID all the time. I’d wake up, get paid to doomscroll, and pump out some blogs about what was happening. When we reverted to case-writing, a huge backlog of cases had to be prepped for publication. Many of these were years-old. They neither shared common formatting nor common citations, and—because the cases were so old—the original authors had moved on to bigger and better things. We needed to solve all the problems that come with a team distributed not only across space but also time.
How long do actors have left?
film
culture
political economy
predictions
My friend Avital wrote a great essay last year about an Instagram influencer. The publication she released it has, unfortunately, flopped, but I will give the gist. Lil Miquela’s 2.9 million followers pale in comparison to the platform’s biggest stars, but she has something they do not: an endless future. Her computer-generated skin will never age, her fun setting will never be burdened by kids or sickness or hardship.
Small economies face big mobile payments
Zimbabwe and Venezuela present different perspectives on forced dollarization
analysis
political economy
Bloomberg Businessweek published an article this week on the rise of mobile payments provider Zelle in Venezuela, where inflation has pushed the price of a café con leche to 890,000 bolivars. Zelle, a money transfer service owned by large US banks, is at first glance an unlikely actor in this story. Why would Venezuelans embrace a proxy of American financial power? The article argues that Zelle’s dollars offer hygienic access to a stable currency. Before covid, the Caracas-based consultancy Ecoanalítica estimated that only 15% of citizens have stable access to dollars. Zelle simply sends dollars around US banks, circumventing the need for physical currency in a dollar-strapped, pandemic-addled country.
Debt Mounts for US Retail and Lodging Mortgagors
analysis
political economy
YPFS
This post was originally published on the YPFS Systemic Risk blog
David Lynch is Toying With Us
film
analysis
I write this post while watching Rabbits, a work of exceptional mystery but unexceptional tension. Lynch casts two regulars—Laura Harring and Naomi Watts—opposite one newbie—Scott Coffey—as man-sized rabbits in what appears to be the Blue Velvet apartment set. These short films, available on YouTube, trip over several of Lynch’s signature aspects: characters sing haunting songs; mundane life is treated as a social experiment; and Angelo Baldametti’s synthesizers underscore everything. Rabbits insists that we examine our own tastes and be comfortable with those answers (and inevitably unanswered questions).
20 Miles Later
analysis
fitness
In 4 weeks, I’ll toe the line at the VCU Health 1 Richmond Marathon. This last month has seen me run consistently farther than I ever had before, so I’ve had to find new places to keep my legs and mind busy. At four weeks out, most marathon plans advocate for a very long run, of about 20 miles. I did it yesterday, and I’m still not sure if my calves will be ready to go next Saturday, let alone tomorrow!
No matching items