The Quest for a Executable, Publishable, Git-Friendly Python Notebook#
After looking at the git differences for my first .ipynb post, I realised how ugly jupyter notebooks are in plain text. Essentially they are just large .json files with large amounts of useless information.
For example, it was keeping track of cell metadata such as how many times I ran an individual cell. Additionally, it includes entire output results in the file. Git should only consist of source code. Because of this, I set out on a quest to improve this issue.
Can execute individual cells: to make development easier.
Contains only meaningful data in files: to make it git-friendly, plus more akin to markdown.
Can be converted to a web page by Sphinx: to make publishing easier.
The first idea I had was why jupyter was a .json file at all? Because of my experience with RMarkdown, I knew a simple plain text format could achieve the same thing.
I then encountered MyST Markdown, which is stated to draw “heavy inspiration from the fantastic RMarkdown language from RStudio.” Exactly what I was looking for then, right? Well, when attempting to integrate it into VS Code, I noticed there was no way to execute cells. And from my understanding, you are only execute code when you compile the entire document. Not great.
However, on the same website, I then saw a YouTube tutorial with executable code cells. It claimed that by running JupyterLab, it was possible! So I went out and setup a dev container, only to realise that the tutorial was only demonstrating - on a regular .ipynb file - the extra formatting powers enabled by MyST Markdown. Not very useful.
We then move onto Reddit. I found a reddit thread which suggested using a tool which clears the metadata and output results from notebooks. And although this doesn’t help the hideous .json format, it was at least a step in the right direction. nb-clean automatically cleans .ipynb files in commits. To do this, you need to simply install the package, and run the following command in the project folder:
nb-clean add-filter
In my case, keeping tag metadata was important (as it dictates what cells are hidden etc.), so I ran:
nb-clean add-filter --preserve-cell-metadata tags
The caveat is that you need to commit the entire file. And to most people this is not an issue. It is only because I am an insane person who commits files line-by-line, which means that I would need to commit the whole file, and then revert lines I didn’t want to include in the commit - which is a bit backwards from simply selecting the lines I want to commit.
Another interesting package I came across was jupytext, which is exactly the plain-text notebook alternative I was looking for. And although my current Sphinx setup only supports .ipynb files, it even has a feature which pairs a .ipynb file, so that any changes you make to a jupytext file will be reflected in a proper notebook. But then I realised this isn’t very helpful considering files need to be commited to the repo before being publishable, so I’d have the exact same problem but with extra fudge on top.
But wait! A random Google result calls for the fact that the nbsphinx package (an alternative to the more commonly stated myst-nb) can in fact publish jupytext files.[1]
But I then found out that nbsphinx sucks. It first requires pandocs (some random package) to run in the first place. And after that, the conversion is less pretty and also doesn’t work with plotly. Another dead-end.