A recruiter once told me it’s rare to meet someone who likes building developer tools. This is hard for me to believe. In what other product space can you change code using the code you wrote? Some of you may have heard of my previous team, GitKraken. Writing code for a git client, then using that client to commit changes is what got me hooked. The GitHub developers were so excited to see us using Electron they eventually did the same with their own git client!

Fast forward a few years and I find myself in a similar position, building a developer tool a bit before GitHub. But the developer tool we’re talking about today is a step away from traditional software engineering. It was introduced to me while taking an Intro to Deep Learning course. Especially as a learning tool, Jupyter Notebooks was a delight to use. Splitting code into separate cells and keeping the heap intact between cell execution felt intuitive, and was a quick way to learn from my mistakes. But after joining a real ML company, I was surprised that notebooks were used to solve real problems.

Why Did We Build This?

In an interview with Greg Brockman, he describes jumping into his first machine learning project at OpenAI:

One thing I find very interesting is the different workflow that you end up with when doing Machine Learning. For example, if you look at a typical research workflow you'll see a lot of files named experiment1.ipynb experiment2.ipynb and they're just like slight forks of the same thing and you're like, isn’t this what version control is for? And after doing this project I learned exactly why. It’s because […] you actually do want the old thing running side by side with the new thing, and if you're constantly stashing and unstashing then you're just gonna be sad.

A lot has changed since 2017. The industry has created new and creative ways to put notebooks in production. Netflix famously debuted their framework to trigger notebooks from other notebooks. Project Jupyter released a notebook scheduler. Fast.ai wrote a library to write python libraries in notebooks. We’re not just experimenting in notebooks anymore, we’re using them to do production-y things.

With all of this important work being done by notebooks, you’d expect a commensurate level of developer tooling. Jumping into my first machine learning project at Large Fintech, I was surprised to find that this was not the case. Every git hosting service has code reviews covered, but when it comes to notebooks, you find mixed support. So why are these big players ignoring such a common problem?

What Did We Build

It turns out it’s hard to build a notebook review tool. What makes notebooks so useful and diffing them so hard is the multitude of ways you can display information. When diffing code files, you’re diffing text. When diffing notebooks, you’re diffing: code, markdown, HTML, LaTex, and images. That’s a lot of limes to hold!

We took it one piece at a time. When approaching markdown diffs, we quickly realized we also needed to diff arbitrary HTML — a tough nut to crack. Libraries will do this for you, but there’s an added constraint when diffing for a review. You have to know which line number each HTML element originated from. This way, comments will stay in place as new commits are pushed. The approach that we took was to render the markdown into an AST that includes source positions and compare elements that share unchanged text. This allows pretty cool things, like aligning bits of markdown on the left and right, making long markdown cells easier to read.

For code cells, we started with CodeMirror, the in-browser editor that Project Jupyter uses. This offered a familiar look and feel, but a fully functional editor is an overkill when you just want to view a diff. To boost performance we built our own code diffing component, making it easier to introduce features like comments, multi-line comments, and code lines that wrap. No more pesky side-scrolling!

So all of that to say, we built a diff that looks exactly like a notebook with the ability to comment anywhere. Over the past year we’ve seen GitNotebooks used in educational spaces, as well as companies in finance, computer vision, and biotech. Our customers tell us our diffs blow the competition out of the water.

Yes, Notebooks in Production

If your team relies on Jupyter Notebooks we’d love to talk. Or better yet, try it yourself!

Building a developer tool a bit before GitHub affords an opportunity to set the bar. For all of those serious about Jupyter Notebooks, we’ll continue to raise it.