The Evolution of Cloud Migrations
Alternate Tooling

It would be deficient of me to not include a section here on the vast array of other techniques and approaches that were used by Atlassian Consultancies in the days before JCMA to perform migrations to the Cloud.  As much as some of these tools have lost a level of relevancy to Cloud Migrations today, most remain available as part of the toolkit of people performing migrations.

The first, and most obvious in the case of Jira perhaps, is the CSV export/import process.  CSV exports have been around in one form or another since Jira 3.0 back in 2004.  For the longest time, this tool actually depended on an interesting quirk of Excel - it would correctly take an HTML-formatted table stored in an XLS file, and convert it to a spreadsheet.  With that strange feature in hand, the export to Excel function would purely take the Issue data on the screen, shove it into an HTML table¹, wrap it up with an XLS extension, and deliver it to the user.  To generate our CSV, we'd simply perform a Save As... and be done with it.  Shockingly enough, this approach to export generation continued all the way through to Jira 7.2 in 2016 - shortly after Microsoft started tightening security in Excel to prevent extension spoofing, and began warning users "The file format and extension don't match" whenever a Jira export was loaded into Excel².

In 2016, this approach was deprecated, and Atlassian implemented a much cleaner, more rational implementation in Jira 7.2.  Prior to this release, Issue exports were capped at 1,000 Issues per export - primarily due to the memory and processing overhead caused by the HTML implementation.  With Jira's new release though, file sizes reduced tenfold, and export sizes became uncapped, limited only by the server's processing capabilities and your own patience.

Interestingly enough, the CSV import functionality in Jira has actually been around for longer than the native Jira ability to export directly to a CSV format.  Dating back to the Jira 4.x era as a Plugin, before later being bundled into the core application, the CSV importer was originally designed for people migrating away from other tools and into Jira, providing a "Load" point for a traditional ETL process.

At this point, CSV became a viable path to migrate the Issues of individual Projects (or for that matter any subset of Issues for which you could write a JQL query) from a source instance to a target containing an already-configured Project.  As a point of note, this process wouldn't take any Jira configuration data across - each row of the CSV related to a single Issue in the instance - but this made sense given the original intent of this to provide a "Load" point for ETL processes coming from other non-Atlassian tools.  It's also worth noting that neither export nor import included any provision for managing change history - it was all about the current state of Issues being transferred (with the notable exceptions of the tool retaining created date, updated date, as well as the author and creation date of all comments).

All-in-all, after the implementation of true CSV exporting in 2016, CSV became a useful tool to Consultants, particularly when dealing with merges or migrations into existing platforms - a common result of mergers and acquisitions.  For Cloud migrations during this era, it was typically dismissed a little too readily - not handling the configuration of a Jira instance (which is where the complexity lies), and missing the change history were typically seen as too big a price to pay - particularly in a world focused on data integrity over most anything else.

¹A surprising note on this point - the HTML that the export produced was actually a full, rich implementation of what you'd see on the screen, including everything from the data itself to the CSS needed to render the lozenges correctly - this made XLS exports pretty massive for the amount of data actually stored in them!

²Honestly, this whole section came as a surprise to me when it came up in my research.  I recall the warning in Excel from my early days in the ecosystem, and my surprise at it not using the XLSX format, but never really put too much thought into its implementation back then.  If you want to explore the details yourself, this Issue and this article from Deviniti were the two that started me down this rabbit hole.

Another option open to Consultants of the era was using the REST API to both extract and load data from and to Jira instances.  Although not a common approach, it became an approach of last resort for some of the largest migrations that took place between the late 2010s and early 2020s.

Originally released in an unstable state alongside Jira 4.2 in late 2010, the REST API was part of Atlassian's push to modernise its tooling, and attempt to do away with the SOAP/RPC API that had existed previously.  In its unstable state it showed promise, but would not be seriously used due to regular endpoint changes.  In early 2012, Atlassian did away with the "unstable" label, and released the REST API v2 onto the world with Jira 5.0, with the SOAP/RPC API finally deprecated a few years later.

In its early days, the REST API was sufficient for many end-users, but was notably missing some key endpoints that would be required to do anything particularly complicated.  Change histories were unavailable for a good few years, forcing you to use the SOAP API or direct DB queries to get information.  Information about screens was also notably absent for a long while, resulting in "Field ABC is required" errors when creating Issues without enough care.  The biggest issue by far for migrations in the early days of the REST API was the lack of any administrative functions.  The early REST API was very much focused on using Jira rather than administering it, so most any functionality that would require Project or Jira Administrator permissions simply didn't exist.

The early days of the REST API were frustrating for power users or Consultants, but the reality was that with Java API availability covering these missing points, it was typically seen as a missed opportunity, but not really a problem.  The problems really started with the development of the Cloud platform from its On-Demand predecessor.  With Java API access in the SaaS product being phased out as part of this, and a huge uplift in REST API usage expected due to the upcoming App paradigm, the Atlassian Cloud products became a squeaky wheel to Atlassian, and many changes were made over the next few years to oil it.

Some of the necessary changes were in place prior to Cloud's release - change histories arrived through the expand=changelog flag with Jira 5.0 in early 2012, as did the issue/createmeta and issue/editmeta endpoints, which provided information about which fields were available and required on a screen¹.  Others had to wait a lot longer - administrative functionality had very limited progress made until the REST API v3 releases and integrations of the Atlassian Document Format, starting around 2019.

With all this said, by the time the Server EoL was announced in late 2020, the REST API was in a much better place than it had been historically.  The API consistency between behind-the-firewall and Cloud platforms made it an interesting proposition for exploitation.  The problem, in the context of Cloud migration, is that although almost all of the components required for migrations were there, building a platform capable of covering all migration options was a herculean task.  With Site Export / Import and other options available, the commercial viability of embarking on this particular quest just wasn't there, so end-to-end migration tooling simply wasn't developed.

This doesn't mean that this approach wasn't used however.  For some of the largest Cloud Migrations that happened worldwide, many of which I was involved in, Site Export / Import wasn't viable (both due to sheer scale and "big bang" approach), CSV wasn't rich enough², and the xCMA tooling was too unstable.  This forced compromises to be made, and custom migration pathways to be designed for the individual components that were required by the customer.  In some cases, this ended up being a mixture of manual configuration to prepare the platform, augmented with REST API calls for repetitive tasks, and REST being used alongside CSV for the majority of data injection.  In these complex cases, the process needed to be bespoke to the customer, and would typically result in long projects with big budgets and many challenges.

¹It should be noted here that these endpoints were absolute performance hogs, to the extent that on any reasonably large system, an issue/createmeta call would inevitably cause an Out of Memory error.  They were eventually deprecated in 2024, replaced by much more granular, high-performance endpoints.

²In hindsight, for some of these large migrations, I wish we'd pushed back more against the sentiment that 100% change history coverage was required.  In many cases, history data simply wasn't actively used - at most used for irregular audits - and the old database could still be retained for that purpose.  Taking a path involving CSV would have significantly improved timelines, and with greenfield Cloud design being required, had the potential to result in a much better end-user experience on the Cloud.

Perhaps this section should instead be entitled "Java APIs", but in my mind it would not be fair to not acknowledge these two particular Jira Plugins, which were fundamental to most migrations that existed wholly in the behind-the-firewall world.  Both tools were originally developed in 2013, by Awnaba Software and Botron Software respectively.  Both were originally designed as tools for migrating configuration to ensure consistency in configurations across multiple Jira instances - a real requirement for large multi-instance estates, which were common at large businesses due to a lack of horizontal scaling in Jira.

The problem that both of these tools had was that it turned out the features they had were remarkably useful for another purpose - behind-the-firewall migrations and merges.  The reality for both these businesses was that a large portion of their use came from Atlassian Consultancies or individual businesses performing one-off migrations.  At the time, tooling for merges in particular was thin on the ground - Site Import would only work on an empty instance, CSV would not bring across history or configuration, and REST endpoints went from completely unviable to an enormous investment.  With PC and CM, however, a project team could pick a Project, and carry across all its relevant configuration (including workflows, custom fields, and screens) alongside all its data, and slide it in seamlessly into an already-active instance.

The problem for both Awnaba and Botron was one of licensing.  The Atlassian Marketplace model was based on annual costs for the whole userbase, directly reflecting the core application.  They would also allow for free trials of Plugins to be used, limited to a few months per instance (although Atlassian Partners got special dispensation).  The end result of this was huge numbers of free trials being used for one-off merges, but a much more limited number of paid licenses being sold.

Over time, new licensing models were implemented which resolved some of these issues, and both companies ended up being acquired - eventually both by Appfire, although Awnaba spent a few years in Adaptavist.

In the Cloud Migration world, it's fair to say the impact of these products was limited, but non-zero.  Migration of subsets of instances to a "clean" instance for follow-up migration to the Cloud was a relatively common occurrence, either for scale or phasing reasons.  Both Configuration Manager and Project Configurator did end up getting Cloud App equivalents, but due to the limitations of the App frameworks for Cloud ended up as quite different products, and were never able to be used for Cloud Migrations in the same way that they were behind-the-firewall.

In our final article making up part 3 of our deep dive, we finally review The Rise of xCMA.