The boring, familiar problem
Sometime in the second week of November 2025, I had a problem I'd been circling for a while. Just about every song I'd played since September 2021 was logged on platforms I don't run, in formats I don't control, subject to the whims of companies that had already earned my distrust once each. Meanwhile I had an AT Protocol repository of my very own, a PDS at eurosky.social, and a growing conviction that listening history is exactly the kind of data that belongs on it: plain, queryable, timestamped, and mine. Not licensed back to me. Mine.
So I built it, out of spite. Spite is underrated as a fuel: nothing like this existed. The tool that should have existed did not, and rather than continue to be annoyed about that, I sat down and started building it anyway. The initial commit is timestamped 03:27 on 14 November 2025, which tells you everything about the schedule I was keeping. Its first name was atproto-lastfm-importer, which told you exactly what it did and nothing about why anyone should care. Its first version was a JavaScript monolith in a single file, with no rate limiting and no error handling. It worked, technically. It also took my PDS down: I blasted records at the AppView faster than it wanted to know about them, it rate limited my entire instance for a day, and I accidentally did a DoS attack on my own PDS. There is a whole other blog post about this, because it was either that or never speak of it again.
And, in the spirit of the honesty this blog runs on, the other quiet part: I built malachite during a psychotic episode. November 2025 was not a good month to be me, and I am not going to dress that up. But I keep coming back to the same detail. The thing I made at my least tethered to consensus reality has spent a year doing meticulous, boring, careful data plumbing: two million records through it, and, as far as I can tell, not one of them mangled. Whatever else that month took, it left that behind. I still do not entirely know what to make of that.
The rewrite came in TypeScript the very same evening, per a commit literally titled "REWRITE": batched writes, adaptive rate limiting, a 7,500-records-a-day safety cap so nobody can ever do to themselves what I did, and, because I'd already written an entire internal essay about why names matter, a new one. Malachite: a green copper stone, associated with preservation and transformation, that happens to sit exactly on the colour of the lexicon it publishes to. If you want the full anatomy, the January announcement post has it, at length and without apology.
It turned one on 14 November, a few days ago. This seems like the week to write it up properly.
What it grew into
A year later, "Last.fm CSV importer" has become the least of its job description. So has "command-line tool": when I announced it in January it lived in your terminal, and it has since moved into your browser, where the entire pipeline runs. The current mode list reads like the services section of a data-ownership wish list:
- lastfm, full scrobble history from a Last.fm CSV export
- spotify, extended streaming history from Spotify's JSON export
- apple, library and play history from Apple Music CSV exports
- youtube, music history from Google Takeout
- listenbrainz, listens from a ListenBrainz JSON export
- combined, all of the above merged together, with smart deduplication so plays don't get counted twice
- sync, imports only records not already in your account, safe to re-run whenever you like
- polish, migrates legacy
fm.teal.alphascrobbles into the current collection, because early adopters shouldn't be punished for being early
Nothing leaves it. You sign in with your AT Proto handle and an app password, nothing is stored, and the only place your data goes is your own PDS. No accounts, no analytics, no tracking, no server-side anything: the entire pipeline is client-side, with automatic rate-limit handling because your PDS, quite rightly, does not want two million records in one afternoon.
The numbers
The session log currently stands at 2,088,112 records imported across 151 sessions:
lastfm | ██████████████ 43
polish | █████████████ 41
spotify | ██████████ 32
sync | █████████ 30
listenbrainz | █ 2
apple | █ 2
youtube | 1Session counts by mode, bar length proportional to sessions. Make of the forty-one polish runs what you will; I tested the migration path thoroughly, and I regret nothing.
A million of that figure is me. Probably. I don't know. The session log counts every record ever processed, not unique plays, and I have run more syncs, polish passes and test imports against my own data than I could ever reconstruct without doing archaeology on my own tool.
What I can vouch for: my collection, the one I wrote up earlier this week, holds 116,000-odd records, and 102,823 of them were put there by malachite: my entire rescued Last.fm and Spotify history, resubmitted in batches across eleven client versions, from v0.3.0 to v0.19.10. Judging by the issue tracker, plenty of other people wanted their history home too. But the counter doesn't split by person, so the honest summary is this: 2,088,112 records' worth of somebody's music has passed through a tool I started at half three in the morning, out of spite. I find that slightly surreal. I allow it.
Tonight, of all nights
The universe not only has a sense of timing. It has a sense of theatre. On the very week of its birthday, Last.fm announced that from the 1st of December, holding an API key will require a Pro subscription. The walled garden malachite was built to escape chose its birthday week to raise the wall another tier. FoxxMD wrote about it. teal.fm reposted him, then posted a thread of its own: get rid of legacy tracking solutions that take control of your data, start tracking on teal.fm, and, in the same breath, "import your data from last.fm with malachite by @ewancroft.uk". My tool was named as the exit door. When I went and said thank you, they came back with something I did not expect: support both FoxxMD and me, "either financially or with code contributions", sponsors links for malachite included.
I would like to refer you back to something I wrote in January: "They can decide tomorrow that exporting your data requires a premium subscription." It is November. They have done it to their API.
I am not a measured man at the best of times, so I will simply say: thank you. That meant a lot, and not only because of the timing.
Within the hour, someone quote-posted the importer to announce it was "time to shove 170,000 scrobbles into the protocol", which is one of the nicer sentences anyone has ever assembled about something I made.
There is a nice symmetry hiding in that shoutout, too. It recommended supporting FoxxMD, of multi-scrobbler, alongside me, and FoxxMD filed malachite's very first issue, back in February: could the TID implementation be a separate package? It is one now, @ewanc26/tid. I can only assume he asks these things purely so that I end up doing them.
And then it gets better. multi-scrobbler's Teal integration now runs on @ewanc26/tid as a production dependency: the TID code that was born inside malachite is now part of the core infrastructure of the tool teal.fm told people to support tonight. A piece of malachite lives on inside its ecosystem neighbour, and its author once filed malachite's first issue. That is the kind of circular dependency graph you only get in an ecosystem small and sincere enough to keep up with itself. There's a symmetry there I enjoy.
The stone family
It has also not escaped my notice that malachite is in good company. The toolkit it belongs to is all minerals: opal converts posts to Bluesky, jasper brings your Instagram over, bismuth turns richtext into Markdown, tourmaline tells you what kind of listener you are. Someone in this ecosystem has a naming scheme, and I respect it enormously. (tourmaline is also mine, before anyone asks. The scheme-holder is also me. I contain multitudes, and they are all rocks.)
All of them now live in the same monorepo, pkgs, which also contains the TID package that fell out of malachite, the shared core that the web frontend and the CLI both build on, and, I am delighted to report, something called rock-tumbl. The naming scheme has fully taken over. I refuse to stop it.
malachite itself now lives in pkgs, my monorepo of publishing and migration tools for the decentralised web, while the original repo stands preserved as the historical record: 126 commits, from an initial commit at 03:27 in the morning to a birthday-week docs tidy. Museums keep these things for a reason.
Free, and staying that way
No ads, no accounts, no tracking, just a tool that does the job and gets out of the way. If it saved you an afternoon, a sponsorship on GitHub, a coffee on Ko-fi, or Liberapay helps keep it maintained; if you'd rather not, a star on the repo or a word to someone who'd use it helps just as much. Both are genuinely appreciated.
Thanks, as ever, to everyone who has filed an issue, reported the rate-limit edges I missed, or asked for a source until it appeared: the ListenBrainz importer exists because someone asked for it. And a special mention to whoever titled their bug report "dedupelication is broken", which remains my favourite issue title of the year.
So: happy first birthday, malachite. You were a weekend project with a mineral's name, and you ended up eating two million records. Slàinte.
malachite is free and open source: try it at malachite.croft.click, read the original announcement, "Malachite: How I Learnt to Stop Worrying and Import My Scrobbles", January 2026, and find the code in the pkgs monorepo at ewanc26/pkgs, with the original repo preserved at ewanc26/malachite. Sponsors links on my site.
Comments
Checking AT Protocol sign-in…
No comments yet
Be the first to share your thoughts on this post.