Since the first project I worked on that used a content management system (CMS), there’s been a problem I’ve wanted to solve.
Best practice for code development tells us that we should try things out in a Production-like test environment before releasing it. This is the best chance we have to catch bugs before they sneak out into the world, to ensure that our understanding of the acceptance criteria is the same as the clients, and to confirm that the way we think things will work in our head is accurate. It’s a well-worn path, with years and years of backing.
However, the same is not always true for content creation.
Often, once the code has made its way into Production, a new page, form, or image gallery is developed right there on the live site. Some CMSs, like Silverstripe, offer draft modes, variations on restricted access, and other workflows that can reduce the likelihood of a slip-up, but ideally we should be using the same flow.
So what has been the problem?
The issue is transporting something between environments. For code, it’s a package deal. You roll the whole thing out, then it’s there for the user to build with. For content it’s a little different. The most robust testing strategies usually involve setting up a piece of content on UAT, confirming the look, feel and function, and then re-creating that same piece on UAT. This is tedious, time-consuming, and prone to human error … even with the power of copy-paste.
Likewise, a typical export is usually overkill for a single piece of content; either exporting a full list of items as a CSV, a full database dump, or a web scraping tool. All of these risk bringing in more than intended, or overwriting existing good data.
Enter silverstripe-record-packer - known to their friends as Zip’n’Ship.
How did I want to solve this?
There were a couple of key drivers that I wanted to help steer the direction of this module.
- Make it CMS-friendly: it should be intuitive for a user, not a dev. It’s their content to manage.
- It should be able to transport data between environments - both up (UAT -> Prod) and down (Prod -> UAT).
- It shouldn’t try to do too much. Better not to bring surprises along.
- It should be configurable enough that a developer can make sure it’s safe for their own project, but have sensible defaults
The Pitch
Records in the CMS (with the appropriate extensions applied) will get an Export button. This button will be in the same place as the other action buttons on an item - Save, Publish etc - so that it’s understood as an action. This will launch a modal where you can give your export a name, so that important context isn’t lost. You can also choose whether or not to bundle assets in with your export. When you fire it off, two things happen:
- The record gets locked. This is a pattern I’ve used before in my silverstripe-async-publisher module to prevent changes happening while the record is being processed. The last thing we want is for the thing you are exporting to change before you export it!
- An ExportJob is added to a queue, for processing through the command line. This is because there are some _complicated_ data records in websites - and some large assets - and trying to process all of that inside a web request is asking for trouble. Better to send it elsewhere.
Once the job has completed, you’ll find your export in an Export History tab on your record. There you’ll see the description you chose, and a link to download the file. You can grab your zip file and store it away for later.
What’s in the zip?
Glad you asked.
First, there is some metadata attached to the file - number of assets contained, classnames, titles etc. If you chose to include assets, they’ll be in there. Related images, ones added through HTML text fields. They get bundled up too. Most importantly though is the manifest.json file, this is essentially the map of the data record.
It contains a node graph of the record, its important properties, and its related records along with their properties as well.
It’s generated by walking through the relationships defined in the class, with particular attention paid to the “ownership” paradigm. However, because the primary use-case is to move between environments, the IDs in the database aren’t used directly; instead, local IDs are used to define the relationships between objects, so that we can rely on writing them into the database in the new location to set the appropriate IDs. Lastly, we can exclude relations or fields from the export. This is useful for Personally Identifiable Information (PII), environment-specific fields, or anything that would send the serialiser down a rabbithole it shouldn’t go down.
What do we do with it?
On a corresponding site that contains data of the same class, you can add the “Import” button to a gridfield in the CMS. This button will launch another modal with an UploadField. Here, upload your zip.
The metadata we talked about earlier is surfaced in a preview, giving you peace of mind that you’re uploading the right file. Then, we click Import - and again, two things happen:
We are directed straight to a stub record of the class in question. This record will be locked, much like an exporting record was - we don’t want it updated before we are ready.
An ImportRecord job is added to the queue - with the same reasoning as before. We could be unpacking a lot of data - and it’s important that we do two passes at it; one to establish the records, and one to hydrate the relationships.
After the job is finished, your record will be written exactly as it was before - to the extent that it can. We load it in as draft (if the record is Versioned) so that minor adjustments can be made before first save. This is particularly useful if you are just moving it around inside the same environment.
And that’s the crux of it. A pick up and put down that makes content transfer between environments a breeze. The next module to be cut is specifically around moving SiteTree records - silverstripe-page-packer - utilising the “Add New Page” flow, and hoping to reduce the headache that is recreating User-defined forms, dense Elemental Block concoctions, and other carefully-put-together content.
How AI Contributed
I’ve been trawling about in management for a few years now. So, this project was my first real dig into how AI could complement my development workflow.
By no means was it flawless, but there were certain aspects of this I would not have been able to achieve in the timeframe I did without the use of AI. I took a bit of a roundabout route to web development, and so have essentially zero "computer science” background. My understanding of node graphs was barely above “kind of like a map, right?”. This is where AI was extremely helpful. This kind of well-researched, well-repeated pattern work found AI in its element. Additionally, I had it explain what it was doing and why back to me so that I can take that knowledge forward with me.
On the other hand, I ended up re-writing and/or re-prompting quite often when there were idiomatic Silverstripey things to implement - the Injector pattern especially - to the extent that I stopped using AI altogether for long periods.
The biggest help, though, was when I hit a point where I realised I wanted to split the core module out. I’d started on the assumption that I was focusing on SiteTree objects, but quickly realised the primary use-case was going to be simply DataObjects. Manually untangling that and managing the dependencies and repositories was a tedious-sounding chore, and thankfully Claude was happy to run that in the background of a workday.
In short - AI worked as my collaborator on this. Essentially an intern fresh out of CompSci 102 - Node Graphs; enough to plug the hole in my own understanding, and do some “manual labour”.
Other Use Cases
Giving portability to specific pieces of content has a lot of possibilities:
- Debugging gets a new lease on life when you can cherry-pick out the thing that is misbehaving and send it to your colleague.
- Onboarding a new developer to a project could skip the request for a full database-and-assets megadump and simply import the few “core” records.
- Theme designers can ship a pre-loaded, pre-formatted page to best display the intent of their work
- Agentic content-creation could become part of a workflow, updating manifest.json files and storing assets for distribution from a designer to pass to a content author
And I’m sure there are many more.
What started as a desire to reduce the amount of duplicated work I had to do has spiralled all the way back to one of my first “well that’s dumb” moments as a developer. If we can all have fewer of those in our lives, all the better.
