The Evolution of Cloud Migrations
Site Import and Export

Following Atlassian's Server EoL announcement in late 2020, the teams at Atlassian Consultancies braced themselves for what was to come.  In the years prior to this at Adaptavist, we'd seen a few Cloud Migrations popping up¹ but the process itself was anything but a solved problem.

A year earlier, in December 2019, Krisz Kovacs (co-founder here at Echidna Solutions) was invited to work with Atlassian's team at their offices in Sydney for a week to get some eyes on their in-development migration tooling JCMA and provide some feedback from the front lines.  Krisz had been leading the effort with one of our largest customers at the time on their own large-scale Cloud Migration² - a strategic customer for Atlassian that was seen by them to be something of a torch-bearer for the larger and more complex things to come.

In these early days, Cloud Migrations were an inherently bespoke affair.  With limited tooling available to teams embarking on the challenge, and processes being somewhat immature, engagements being a bit "rocky" wasn't uncommon in this period.  With the xCMA tools being newly out of beta, the reality was that when the Server EoL was announced, the primary method for Cloud Migrations was using the long-standing Site Export / Import functionality, also known as the XML backup tools.

It should be noted that throughout this section, a lot of focus will be put on Jira migrations - this is primarily because Jira is the most complex of the Atlassian suite for migrations, and represents the largest differences in the back-end data structures of behind-the-firewall and Cloud.  Similar stories to the below exist for Confluence, Bitbucket and the rest, but this section is already far too long with only Jira being covered!

¹Typically at the time, these came from keen, agile, and tech-savvy businesses that saw the writing on the wall or simply wanted to be on the cutting-edge tech.  Worth noting that, by necessity, they were on the smaller side (with one very notable exception!) due to the user count limitations on Cloud at the time.

²The notable exception mentioned in my last footnote!  They were an epic tale that lasted multiple years, and they effectively served as trailblazers for the process in large businesses, dealing with the countless challenges involved with brand-new tooling.

In behind-the-firewall environments, this tool would, in simple terms, export your entire Jira database into a single XML file (entities.xml), sprinkle around a few additional files relating to filesystem-stored information, optionally add your entire attachments folder alongside the rest of the backup¹, and return you a consolidated Zip file containing the whole kit and kaboodle. This approach had been a mainstay of the Atlassian tools since near enough the first releases of Jira in 2002².

As a Consultant, it had been something of an "occasionally useful" part of my toolbelt, but I would be lying if I said it was something I was using on a daily basis.  In particular, it was, and remains, the most consistent way to migrate a behind-the-firewall system from one database to another³, but for backups or for any platform migrations that don't require a change in database, more traditional filesystem and database migration tooling tended to be faster and far more reliable.

The main problems that the Site Export / Import process had - at least, prior to the release of Jira 8 in 2019 - stemmed from memory usage and performance.  On the memory side for export, the eager loading strategy used to speed up data retrieval for custom fields built up a cache in memory that increased for every single Issue, and didn't clear until the entire site had finished exporting.  Paired with this, the single long-term session held open by Hibernate (Jira's Object-Relational Mapping layer) during the export process built up an enormous object graph, consuming the heap steadily as the export process continued.  Paired up with the Parallel Garbage Collector (GC) and outdated Lucene Indexer, on most any large system Out of Memory issues were remarkably common during exports.

Even if this hurdle was passed, the import process had its own challenges with memory (albeit not as common as those with export) - the ID Mapping Table that Jira generated in memory to link objects' old and new IDs would scale linearly with the database size.  Although less common than with export, this did have the potential to trigger Out of Memory issues when other processes in the migration were competing for the same heap space - an occasional problem with large or under-provisioned systems.

Performance was a whole other beast, and the impact on exports was significant.  Jira was utilising the Apache OfBiz Entity Engine to manage a lot of the communications between the Application Server and the Database⁴, even for site exports.  With OfBiz on a large system, what you might expect to be a simple select * from TABLE to populate your XML actually ended up becoming tens of millions of tiny requests relating to each individual Issue, status, custom field, or even things like priority icon URL!  Compounding this inefficiency was the single-threaded nature of the processing.  As a process that had existed in the Atlassian applications since the early days, scaling of performance had (rightly) not been the highest of priorities, and single threading was used to ensure referential integrity of the process.  As time went on, instances grew, and processors started to scale horizontally rather than vertically - this limitation became more and more problematic over the years.  This is all to say nothing of the efficiency of the string manipulation used in XML encoding, or the excessive impact that the old Lucene indexing tooling would have on the process if users actually did anything on the system while the export process was running!

For imports, performance challenges were perhaps even more prevalent.  Imports had 3 main performance bottlenecks - the old Java Parallel GC, the database processes required to add new objects, and the old Lucene versions that were still in use.

With Jira 7 and below, support had not yet been added for Java 11, which came with the G1GC by default.  As such, the vast majority of customer instances were still stuck using the old Parallel GC, which really struggled when Jira was parsing the multi-gigabyte entities.xml files.  Minute-long Full Garbage Collects (FGCs) were not unheard of when importing very large instances, and without careful management, this would cause connections to be dropped, causing the migration to fail in a less-than-graceful manner, and resulting in a full restart of the import process.

It wouldn't be unfair to suggest that the process for adding objects to the database was a bit of a mess by this point in time.  Jira had been released for 17 years, and a lot of "cruft" had built up in these processes - redundant queries from long-forgotten requirements were relatively commonplace across the schema.  For normal day-to-day usage of the tool, it wasn't the biggest deal - from a timeboxing perspective, the impact was relatively insignificant to the end-user experience.  But when we're running a process like a site import that's creating thousands if not millions of unique objects in the database all at once, it can really cause problems, slowing the import process to an absolute crawl.

On the Lucene side, Jira had been stuck on Lucene 3.3 for over seven years by 2019, and this old a version didn't exactly set the world ablaze with its efficiency.  Lucene Indexes were big - often even bigger than the database itself.  In day-to-day usage, reading these indexes was broadly fine, and writing individual items to them was sufficiently responsive (even with low speed storage media).  The Site Import process was a different matter - building an entire, large-scale Lucene index from scratch, all at once was a lot to ask.  A big part of the problem was in a process called "Segment Merging" that would be performed during the mandatory full re-index that happened right at the end of the import.  This process would pull together hundreds or thousands of small segments of the index into a cohesive whole.  This process was absolute hell for both disk IO and CPU cycles, and if it lined up with an FGC from the outdated Parallel GC things would not go well.  To the user performing the import, the progress bar would finally be at 99%, with the process telling you it's performing "Startup/Reindexing".  You're feeling relief that this whole ordeal is almost over.  Unfortunately for you, this was the most common point for timeouts and crashes to occur, and managing to get through the full re-index successfully could be challenging.

As a Consultant, we had to be very aware of all these idiosyncrasies, and we did manage to construct workarounds or avoidance techniques for a lot of them.  Making sure you've got enough memory on both source and destination was fairly obvious⁵, and using the reverse proxy to forward any users wanting to hit up the source instance during export to a landing page would avoid any additional load on Lucene.  We built some polling systems to enhance connection stability in long processes, and did everything we could to reduce the size of the system prior to any Production processing taking place.  Even with all this in place, prior to Jira 8 at least, this process was really only suitable for small to medium sized instances - for anything particularly big we'd have to get more creative.

As an aside, for those that are interested, you can find a video of me from 2013 presenting on a related topic "The (not so) Dark Art of Atlassian Performance Tuning" on YouTube, and have a good laugh at my younger form...  I trust that my passion in that video goes some way to help explain the verbosity of this section!

¹The sheer scale of the attachments folder often meant that they were handled separately however.

²I had hoped to be a little more precise here, but contemporary resources from 2002 are slightly thin on the ground these days.  One little gem that did come up in my research however was a suggestion ticket for Confluence - raised by Mike Cannon-Brookes himself in 2003 (the year before the product actually launched), proposing XML Export / Import of a single page - a feature that got dropped after development apparently "when we couldn't see any purpose for it.", but that I personally would have appreciated back in the day for a strange use-case I ended up having to use the SOAP APIs to resolve, which was a bit of a nightmare!

³Word of warning to anyone that might need to change databases on their behind-the-firewall platform - be really careful of external tools to migrate database types.  Due to the history of how databases were accessed through JDBC and managed with OfBiz, data structures can be subtly different between platforms.  I have seen cases where multiple years after a migration, an Atlassian tool upgrade becomes impossible without significant manual effort to unpick and re-align indexes and incrementors with the expected structure for the new database.

⁴This was something of a legacy from Jira's early-2000s architecture.  As a decidedly database-agnostic tool, the Entity Engine would generally avoid the JOIN and UNION queries that you might expect to use to get information, instead opting to perform a lot of this logic in the Application Server itself.  In the 2000s, when databases were typically very small compared to today, this was fine - the ability to be database agnostic trumped the performance implications very easily - but by 2019, the cracks were not only starting to appear, but were becoming major structural concerns.

⁵Although not too much memory - that would cause increasingly long FGCs with the Parallel GC, which could, of course, cause a timeout and forced restart of the process.

In February 2019, Atlassian released Jira 8.0 - to a relatively muted response from users, and uproarious applause from Atlassian Consultants the world over.  Providing end-users primarily with just some basic front-end modernisation (some Atlassian Design Language implementation for boards), customer reaction to this release was pretty muted.

To Atlassian Consultants, however, this was big news.  Dubbed the "Performance" release by many, in this single Jira release, Atlassian upgraded over 40 internal libraries to more modern versions, and introduced Java 11 support.

The big news for most was the fact that Jira had finally managed to rid itself of Lucene 3.3 after 7 years.  With the implementation of Lucene 7.3, a huge amount of work was done by Atlassian to upgrade, and the completely incompatible indexes meant that any upgrade to Jira 8 would require a mandatory full re-index - a concern for many given prior experiences.  Admins were rewarded for this though, with indexes being quicker to generate, were about half the size of their predecessors, and significantly faster for day-to-day read and write operations.  This resulted in a generally "snappier" feeling system to the end-user, whilst for migrations this significantly reduced the risk of failure at 99% that had plagued Site Imports to this point.

With Lucene 7.3 taking much of the praise from the ecosystem, Java 11 support is often unfairly overlooked.  Java 11 brought with it the G1GC by default, replacing the old, slow, and troublesome Parallel GC that had previously been a staple.  In general day-to-day usage, G1GC would effectively eliminate the "stop-the-world" FGC pauses that happened on most Atlassian applications¹ - an incremental good, but not noticed by most end-users.  For anyone dealing with the extremes of performance, however, this was a godsend.  In the very-high memory allocation rates inherent in a Site Import, FGCs causing timeouts became significantly rarer, saving countless hours of debugging.

Outside of these big-value changes, there were more incremental upgrades for other internal tools - Guava (Google's core libraries for Java that provided a whole host of types, collections, libraries and utilities) was upgraded from 18.0 to 26.0, the Spring Framework (that was responsible for the Plugin ecosystem of the time) made the jump from 4 to 5, and plenty of others had similar treatment.  All these bumps helped to improve overall system performance, and gave the Atlassian Engineers many more efficient tools and technologies to continue moving the platforms forward.

Outside of these upgrades in tooling, Atlassian also spent time in 8.0 cleaning up the database schema.  Many of the redundant database queries used in object creation were cleaned up, meaning Site Imports became much quicker.  Also, the changeitem and changegroup tables² got new indexes which vastly improved Issue history loading, speeding up the process of generating our export files.

As with any large-scale infrastructure change, the 8.0 release was not without its foibles.  The next few months resulted in regular updates coming from Atlassian, and was something of an unstable few months for the cutting edge of Jira³.  This said, with the release of Jira 8.2 in May 2019 (just 3 months after 8.0 went live) a level of stability had returned to the Jira application.  For Site Exports, even with the release of 8.0, memory consumption remained exceptionally high.  Thankfully, 8.2 brought with it a fix for this issue - completely changing the way that exported Issues got stored in the Application Server's cache, removing a memory leak in the process, and so reducing the memory footprint of the Site Export feature by as much as 95%⁴.

These changes provided a much-needed boost to the Site Export / Import process, and made it a viable (although rarely fast) process to perform on most Jira instances - only the very largest would still be running into trouble at this point.

¹To the end-user on most systems, if a FGC was noticed at all, they only saw a short stutter of a few milliseconds, making the tool feel a bit sluggish at worst.  For systems with extremely large heap sizes however, these pauses could last seconds.  I recall one customer that, over a 5-day period we monitored, had almost 500 FGCs occur, averaging 20-25 seconds each - a rare case, no doubt, but it goes to show how impactful it could be..

²The mere mention of which sends shivers up my spine to this day.

³Perhaps unsurprisingly, it was only a few months later that Atlassian released their first Long-Term Support release - Jira 8.5, released in October of 2019.  The LTS releases provided a target platform for many businesses looking for stability, with an annual release schedule and two years of support.

⁴A bit of a "citation needed" moment here - I found a Jira bug ticket for this issue where, regarding the reduction in size of the custom field cache, an Atlassian Engineer noted that "Our test instance showed memory consumption reduction of 95%".  The other comments suggest different usage patterns would impact cache load, so impacts would vary.  For me at least, I don't recall improvements at the time beyond a 30-50% reduction or so with my customers, so I guess I was never one of the lucky ones!

This all brings us back around to Cloud Migrations, and the primary mechanism recommended by Atlassian in 2020 of using the Site Export / Import function to perform migrations.  We've just established in the last two sections that this feature was much more capable by 2020 than it had been in the past - so we're good, right?

Well, about that...

So, if you recall from our history lesson previously, the Atlassian tools have been running through an era of divergence since 2014 - the Server and Data Center platforms at this point stand as something of a legacy to the history of the platform, while the Cloud platform has been adorning itself with bells and whistles to attract all the hip, agile, tech-driven businesses who enjoy being on the cutting edge.  On top of this, recall that I mentioned how integral Plugins were to the end-user experience of the Atlassian tools historically, and that the Plugins v2 model allowed for direct Java-level integration into the tool, while Atlassian Connect Apps were asynchronous, communicated over web APIs, and held data on vendors' infrastructure.

It's not unfair to suggest that this was a recipe for disaster.

The Site Export / Import process was designed to be used to migrate content from one instance of a tool to another instance of the same tool.  Don't get me wrong, it's obviously possible to extract data from the XML format that's produced and carve something yourself from that data, but that's not the purpose of the tool.

Migrations between different tools (as I argue behind-the-firewall and Cloud were by this point) typically use a process called ETL, or Extract-Transform-Load.  In ETL, you extract information from a source system in the format that it produces, you then perform some level of logic to transform this data into information that's able to be readily consumed by the target¹, then you load that generated file into your target system as an import - easy as that.

With Site Export / Import, you'll notice equivalent terms to the "E" and "L" of ETL, but nothing for "T".  This is because, fundamentally, there isn't any transformation that you would perform during this process.

The Site Import feature in Jira Cloud is an interesting beast - as the platform has evolved, it's needed to fulfil two distinct functions - provide the ability to import exports generated by other Cloud environments, and provide the ability to import the export files that were generated by their behind-the-firewall siblings.  The problem was that the Cloud platform was evolving with new features and functionalities, changing fundamental parts of its infrastructure to the point where a simple "here's the contents of the database - shove it on another instance and you'll be fine" doesn't really work any more.  Atlassian Cloud tools were becoming more modular, with shared services all over, and the Site Import feature was having to be constantly patched to deal with the evolution of the platform.  Unfortunately for Atlassian, given that the Site Export / Import process was still the primary mechanism used to move teams from their behind-the-firewall implementations and into the Cloud, they still needed to keep it around.

The end result of this was something of a challenge.  In small, simple, vanilla instances, the Site Export / Import process worked absolutely fine for Cloud Migration - there was enough similarity with Issues, custom fields and the like that the vast majority of information would map reasonably well, and any oddities would be small in number, and manually recoverable.

Larger or more complex instances remained a problem here, however.  Although the process of exporting the information would work relatively smoothly in most cases since 8.2's release, the content mapping was problematic.  The most obvious issue was with the Plugin-App interaction.  In the behind-the-firewall world, Plugins would persist all their information directly in the Atlassian tool's database², meaning that all your relevant data is nice and tidy in a single location.  In Cloud, as we described previously, Apps are very different to their behind-the-firewall equivalents.  App data is not sat alongside your Jira or Confluence data, instead it's sat in an arbitrary format in the data center of whichever vendor built it.  The Site Import process in the Cloud effectively did not deal with this problem - the AO tables were as good as dropped³, and working out any kind of resolution was a problem for you and your Plugin/App vendor to handle.

From a Consultancy perspective, at this point in time, Site Export / Import was often used as a starting point for Cloud migrations - early tests would be done using this process, and analysis done on how much post-migration repair work would have to be done to get the system into a working state.  Much of the focus was on data integrity at this point in time rather than business continuity⁴ - meaning that direct 1:1 replication of all data was the key concern rather than all working practices.  Data comparison tooling was often custom-carved to ensure integrity, or to highlight key areas that would require manual intervention.  

All-in-all, the process was long, had some known issues (particularly around Plugin-App differences), but generally speaking would result in "all possible data being retained", largely seen as the goal of a successful migration.

How usable a behind-the-firewall system that has been crowbarred into the totally different Cloud platform was a different matter - this is a topic I will cover in-depth in the subsequent section covering the Echidna Solutions approach.

¹Typically focussing not only on making sure "the headings are correct" or "it's the correct file structure for consumption", but by ensuring that appropriate mapping is performed to match equivalent but different business logic to each other.

²Specifically, in the ActiveObjects tables - the Atlassian-developed Object-Relational Mapping framework that integrated into the Spring Framework, and provided the primary data persistence protocol.

³I discussed this topic with an Atlassian Engineer many years ago, and my understanding is that the import process did actually put the ActiveObjects tables into the database at point of import, but all the data in them was inaccessible to Apps, and it eventually got cleaned up by a scheduled process.

⁴Coming from a world of digging directly into Atlassian's Java code and directly fiddling with the database meant that the technical had a tendency to override the functional for many Consultants back then.

We continue our exploration of Cloud Migration tooling in our next article - Alternate Tooling.