Data Plumber - unclogging broken data pipelines
he/him
Jonathan Carroll is an independent contractor and principal consultant at Irregularly Scheduled Programming. For nearly a decade he has remotely contracted with biotech companies in USA and Europe for data science and data tooling. He was awarded a PhD in theoretical physics from the University of Adelaide (programming in Fortran). He has worked in the South Australian public service in fisheries, programming in R. He has since branched out to polyglot programming in Python, Julia, Rust, Haskell, APL, and a variety of other languages. Jonathan is an editor of the RWeekly (rweekly.org) newsletter, regularly blogs about explorations in programming at jcarroll.com.au and is active on mastodon at fosstodon.org/@jonocarroll
Obsidian lets you build a personal knowledge base of your notes in plain markdown files that live on your devices with wiki-style links between notes. But those links are structurally flat; a graph view of your notes only shows unstructured connections. Turning that into a real Neo4j knowledge graph means solving a deceptively interesting data engineering problem: how do you add semantic structure to a corpus you've been building for years, without destroying the thing that made it useful in the first place? How do you then enrich that with connections you haven't found, while actually doing the learning yourself? I'll walk through some existing options for supporting such a transformation and the real-world approach I took — designing a typed relationship syntax in markdown that lives alongside the notes, aligning entities in my notes to Wikidata's ontologies, building an ingestion pipeline from notes to Neo4j, and using the resulting graph to surface connections I hadn't made yet. I'll also explain why 'just throw an LLM at it' is the wrong first instinct ("without an understanding of how one arrived at the knowledge, it is worthless"), and what you actually get when you treat this as a proper data engineering problem; something that's more than just an offloaded second brain.
