Strategic Migration in Practice
Implementations

Even with all the preparation in the world, implementation can be a tough challenge.  Back when JCMA was in its infancy and approaches were more naive this was even more true.  So when performing one of the earliest large-scale Cloud migrations, for a very technical customer, and one that Atlassian had decided would be the torchbearer for the Cloud in general, you should be prepared for a bumpy ride.

In the late 2010s, Cloud migrations were not a common occurrence.  With historic user count limitations, a constantly-updating platform, and a continued suspicion of Cloud-based services, that Atlassian Cloud platform was not often the first pick for many large businesses.  With internal development bringing more enterprise-friendly features to the platform, Atlassian needed a headline client to pave the way for other enterprise clients to follow.

The customer that Atlassian selected for this was one that we had worked with for a number of years, and it would be fair to say they were on the extremes.  As an example from our early engagements with them, we were brought in to review a performance issue that they were experiencing.  Although they were having more generalised performance problems, there was one particular spike which would happen once or twice a week on average, but not uniformly.  At times, it wouldn't happen for a couple of weeks on the trot, then suddenly it occurred 3 times in a day.  Whenever it did occur, the instance would end up nigh-on unusable for almost an hour, grinding work to a halt across their large userbase.

Without going into too much detail, we set up some appropriate logging and profiling, and waited for an event¹.  Analysing the logs after the fact, we dug into what events could be triggering the performance problems, and found something interesting - a significant spike in the creation and deletion of Issues around this point.  Creation would not have necessarily caught my interest, but deletion was strange in this instance.  The calls were also coming across the SOAP APIs, rather than being direct user calls - not particularly unusual for this customer, the users were all extremely bright and technical, and automation was the mode du jour here.  We found the IP address associated with the calls, and identified the server making the calls - one owned by one of the hardware teams - and went to have a chat.

It turns out they were in need of a database they could record hardware testing results.  With deadlines coming up, they realised that their existing Jira system was already hooked into their reporting tools, so if they could get this data into Jira, they would be able to use that connection to produce the data they needed.  How did they go about doing this?  By creating a custom Jira Project with one issue per test run, filling a custom field with the timing data from the run.  Once the 30-60 minute test was complete, they'd built an automation to pull the report, and clear out the data from the system one issue at a time.  With 2,000+ tests run in each period, this would result in a spike of issue creation for the first half of the test, then a mass deletion of issues - this was the spike in load that was the cause of the performance issues across the business.  Safe to say, this testing approach was deprecated immediately!

A silly story, no doubt, but it was indicative of the nature of the userbase, and the way that Atlassian tools had been used (and abused!) over the years.  With my personal history with the customer, it was something of a surprise that they wanted to migrate to the Cloud, but in hindsight, they had always been on the cutting edge of technology, so perhaps they were an obvious choice.

This project was enormous, and continued for multiple years - I could never cover all elements in this document for obvious reasons!  As we're jumping to a later point in the process, let me set the scene.

Early discussions had made it clear that using Site Export / Import would not be viable - due to the scale of the endeavour, performing the migration across multiple waves was imperative, and Site Import was an all-or-nothing approach.  JCMA was very much still in its infancy at this point, and although it was regularly tested during the migration process due to the guidance of Atlassian, it was still not capable of migrating all of the customer's data effectively.  CSV migrations were reviewed and discounted as an option early on - the inability to carry across issue history was seen as immediately disqualifying.

With all this said, the first migration wave would be fine (clone source, pare down the data in clone, migrate using Site Export / Import), but subsequent waves didn't have an obvious solution.  After months of waiting for JCMA to be up to the task, we began to explore the option we'd been avoiding to this point - JSON Import.

The JSON Import feature in Jira had always been something of a tricky beast - introduced in the early days of the REST API, it was notoriously picky about the content being input - every structure had to match what it was expecting throughout the import, else the whole thing would fail less than gracefully.  In general it would work for moving issues between two identical systems - since all data structures would match - but the thought of trying to map JSON accurately between BTF infrastructure and a constantly-moving Cloud target was a concern.  This said, with an increasingly frustrated customer and Atlassian breathing down our necks, needs must.

The approach that we would use was that of a standard ETL project.  Extraction of data was easy enough - REST interfaces were stable enough to pull all the data that we needed relatively quickly - and Load was through the JSON Import tool sat in the Admin screens.  The transformation was where the real work would be.

With such a complex system, the customer of course had many Plugins, and with that, many Plugin-managed custom fields.  Every one of these would need to be manually reviewed and mapped to appropriate target fields - and have their change histories updated to align to the appropriate format as well.  Developing this transformer was a long, painstaking process, made more difficult by the ever-evolving Cloud platform.  It became a regular occurrence that either Atlassian or an App vendor would make a change that would directly impact the transformer - typically with minimal or no notice given these were not strictly "supported" paths.

After many months of development effort, we finally had a working transformer, and we were able to migrate our second wave.  Work then continued for subsequent waves - each time finding new fields and dealing with changes.  Compounding the challenge was the fact that the target Cloud platform was now in more active use, and App installations, configuration schemes, and data were all changing regularly.  Each wave became a marathon effort to get to a working transformer, and even once this was achieved, there was still a risk of migration failure during a downtime window due to change occurring.  I recall at least two weekends that had migrations called off due to changes in the Cloud platform causing the JSON Importer to no longer accept the format we'd tested just a couple of days beforehand.

Eventually, the JCMA tooling matured to a point where it would be viable.  The release of Project-by-Project migrations in 2023 was the final nail in the coffin for the JSON path, and the final couple of migration waves finally utilised JCMA - the tool Atlassian had recommended they use from their initial discussions all those years before - and achieved a generally successful outcome, notwithstanding a couple of minor bugs!

There's one major lesson that comes from this story - yes, the JSON approach is possible, but I would warn everyone against it unless strictly necessary.  The amount of effort required to implement is immense, and the maintenance cost is enormous due to the constantly changing nature of the Apps and Jira itself in the Cloud.  It's a useful tool at times, but very picky in formats, and not particularly well-supported by Atlassian - here be dragons, enter at your own risk.

This all said, it would be unfair of me not to mention that if you were to try this today, you would be in a slightly better place in some respects.  Atlassian now provides the ability to lock down updates in Cloud (a feature implemented, at least in part, for this project), meaning the problems of having the ground shift between testing runs and production runs is far less likely.  Furthermore, there is much more stability in Jira's JSON formats generally these days, and with a few more years having passed, well-established Apps tend to be a lot more stable in their custom field persistence formats as well.  The only gotcha here is that the Cloud has now moved away from wiki markup formatting and to the Atlassian Document Format for rich text content (comments, descriptions and the like), so additional transformation from wiki markup to ADF will be required.

My advice today would be to understand the JSON importer and its idiosyncrasies, keep it in your toolbelt, but use it sparingly. One example of its use that I had more recently was for a single Jira Project in a customer's migration.  This Project required direct automated auditing of history that they wanted to retain in Cloud, while no other Projects had this requirement and could be migrated with CSV - a short step in the process, relatively simple data set, and big value to the customer.

As an endnote of sorts to this anecdote, I note that we've mentioned change history a few times in this story, but we've not really dived into why it was needed by the customer.  This one is a bit of an annoyance of mine, and I maintain to this day that this customer really didn't need to retain their change history.  With the source system still in place, the reality is that regardless of migration method, full and accurate audit would only ever be done on the original system anyway.  The reason it was discounted so early and, frankly, so aggressively was due to conversations that happened during the earliest of presales.  At that point, CSV was called out as an option quite dismissively, with the only mention being that "you'd lose issue history".  As a customer, I'd immediately balk at CSV given that presentation, and in this case that message was very sticky, lasting for years.  To this day, I've not been aware of any usage of Jira at this customer that really exploited the existence of that history beyond simple auditing which would be better done against the (still retained) source system.  I'll get off my soapbox now!

¹Of course, we ended up waiting 3 weeks for an event to occur, because that's how this always works!

As I've implied a number of times in this deep dive, I have a personal affinity with the CSV migration path.  It aligns with a "fresh start" mindset, and allows for a migration to drive transformative change for the better within a customer.  It all sounds wonderful, but the question remains of how this can be achieved, and what pitfalls will you need to avoid to be successful?

In one of our most recent projects, we had been tasked with the migration of a very old Jira and Confluence system to the Cloud.  With a system that had been built up over many years in quite an ad hoc manner, the impact of this organic growth had been felt in different ways on different platforms.  For Confluence, given the nature of the information being stored, it was less impactful - there were occasional concerns surrounding validity of older data in the system (which could be resolved with the application of a simple document management App against some Spaces in Cloud), but the general lack of structure or formal Information Architecture actually suited this business well.  Jira was a different matter altogether.

The wild level of scheme proliferation in the system was causing problems across the userbase.  For management, there was no way of getting any kind of consolidated understanding of larger programmes of work as Projects were fundamentally incomparable.  For teams themselves, dependency management between teams was challenging, and a lack of centralised governance and support meant they were using workarounds on a daily basis to simply use the tool.  The admin team were at a loss - the sheer scale of configuration on the platform meant that understanding of any individual setup was low, and the reality was that they were reacting to requests rather than proactively managing the system.  Something had to change, and the customer had come to the decision that the migration was the way to do this.

When designing our migration path, we ended up with 3 main threads.  Confluence would be migrated as-is using CCMA - the source instance was effectively Plugin-free, the data was simple, and the structure was deemed appropriate to continue using.  Jira had two paths - a CSV migration from the source system, using a standardised set of simple configurations for the vast majority of teams, alongside a single Cloud-to-Cloud migration that needed to occur (for the instance of a recent acquisition) using the "Copy Product Data" function.

The CCMA and Copy Product Data paths were easy enough, and not really worth commenting on in depth - vanilla Confluence is a simple enough system, and migrating data between two identical platforms is generally easy enough to perform.  The CSV migration is where the meat of this story is, as is the thousands of lines of Python script that enabled this process.  I'm going to walk through the whole process step-by-step from my own internal documentation - both to ensure I haven't missed anything, and to provide adequate detail!

Our first step in the Jira migration process was on the Cloud side - we needed to prepare the instance to receive the data from source.  In our case, this involved a few steps.  Before any migrations occurred, we needed to hook up the Azure AD connection to populate the user list, and install any Apps that were planned to be used.  Then, for each migration, we would generate Projects with all their configuration - relatively easy for this project as most were simply copied from a template project, and adjusted slightly as required.  Finally, for each Project we would also manually generate the boards associated with it - this was very much a nice-to-have for this customer, and in many previous cases the teams have simply created new boards themselves post-migration.

With the destination system prepared, we could jump into the code side of things, starting with authentication flow.  Here, the API credentials can be a little tricky - on Cloud, we have API token generation in your user account, but due to the nature of the work you would need to use the "Create API Token" link rather than the "Create API Token with Scopes" link, while DC API tokens are a little simpler.  I used the Python jira library to handle the authentication process itself - totally worth doing even if you later have to construct some manual requests.

The next step was user mapping - in order to ensure that information was carried across effectively during the migration, we would need to map users to their new user IDs in Cloud.  The CSV importer expects an email address in this field, so we needed to map from the username exported by source to the destination email address.  In our case, we built a script to pull the users from source, including username and email address, and used this as the basis for the simple mapping file.  There were a couple of changes made (email addresses that had been updated as names changed, or a strange quirk with how Azure AD handles apostrophes in email addresses), but this step was relatively simple.  The one note I will make here - ensure you keep this updated as time goes on - we had one user added during the process that got missed from this file, and some manual post-migration effort was required to re-align.

We're now onto the "Extract" of the ETL process, and again, this is not too challenging.  CSV exports are typically performed through the search panel when performing this manually, but to automate the process there is a slightly obscure URL that can be used:
/sr/jira.issueviews:searchrequest-csv-all-fields/temp/SearchRequest.csv
By passing params of jqlQuery, tempMax and pager/start, we can build a simple script to generate a CSV export from any JQL query.  Atlassian have actually produced an example script that utilises this endpoint, although it's emphasised that it's not supported, and I had to make a fair few changes to incorporate it into our use case!  One thing that was important to us for this export was actually to perform the export on a Project-by-Project basis - this enabled much of the mapping and import process that comes later.  Overall, this was relatively quick - taking less than an hour for each wave in our case, but this will obviously vary from migration to migration.

The next (and possibly most important) part of the process was the mappings.  In our case, we wanted to enable individual teams to perform their own mappings.  This could have been enabled with a web app (as I have previously done), but in this case we went with the old classic of "exploit Excel for something it really shouldn't be used for".  I used openpyxl to generate the sheets, and tried to make the whole process as user-friendly as possible.  The mapping workbooks contained 3 visible sheets each, alongside some hidden sheets that drove the logic.  The first was an instructions sheet, followed by the field mapping and value mapping sheets.

For each Project, the field mapping sheet would contain one line per custom field.  We'd generate the list based on the content in the source CSV export - only including custom fields that actually had data entered for this Project.  Informational columns in our case included the name, ID and type¹ of the field.  We then get to the primary user input column - the destination field.  Using a REST call on the destination system, we can get our list of custom fields by type and provide a limited selection of those in a drop-down list.  Finally, we have the calculated destination field ID (including things like customfield_10101) which is used later.

The value mapping sheet was specific to option fields, and would always end up containing the values for Issue Type, Status, Priority and Resolution as built-in "fields".  For each option available in each field, we had a row, containing the field name, field id, and the value on source.  The next column was then the user input - a drop-down list of option values associated with the destination field selected in the previous sheet.

I will say, the implementation of this in raw openpyxl-based Python was pretty intense, and took a fair amount of time to get working correctly - the workbook generator alone was almost 700 lines, excluding any of the data extraction logic.  In terms of advice here - if you need user-facing interfaces for doing mapping that have a decent user experience, you're probably better off carving a web app from scratch than trying to do it in Excel².

Mappings were then sent to teams to be filled in.  I will note here that we also had code to automatically highlight any problems with these same workbooks in future runs - this meant that they only needed to be completed once in full, and subsequent changes could be asked about in an ad hoc manner.  Once the spreadsheets were returned we were almost ready to begin the migration itself, only one thing remained in the way - sprints.

The Sprint field in Jira is relatively unique: its storage format is a simple array of Sprint IDs, but the IDs themselves come from the boards "Plugin" itself - a holdover from its days as GreenHopper.  The main challenge for us was that we needed to update the IDs during transformation and could not rely on an automatic mapping during Import.  Compounding this, there isn't a built-in way of migrating boards and sprints (outside of JCMA or, in the old days, Site Export / Import), so we had to build it ourselves.  As mentioned previously, we manually built the boards as part of the preparation, but Sprints at least could be automated.  I built a script to pull all board information from the source system, and manually collated the board IDs from the destination to build a mapping - in theory one could automate this process with name comparisons, but the scale of this effort was too small in this case to worry about.

With a mapping in hand, the process of generating the sprints was relatively easy - hit the get Sprints endpoint for the board ID in question, and rewrite the sprints to the destination board.  Before declaring victory, it should be noted there were two issues with this process:

1. Sprints are created through the endpoint in a "future" state - so in a board with 200 closed sprints and one active sprint, you'll end up with 201 "future" sprints - less than ideal.  To resolve this, after import we would iterate over each sprint using the update sprint endpoint, and for any that should be "active" or "closed", iterate them through the flow (note that you can't move directly from future to closed either here - step by step is required).

2. Dates are really problematic.  Although the endpoint will accept a completeDate variable, as noted in the API docs "The completeDate field cannot be updated manually".  The result of this is that all sprints appear to have been completed at the point of migration.  I maintain that this is a bug, but communications with Atlassian suggest it's unlikely to change.

These issues were annoying, but not too impactful - the state piece was automated, and the customer's teams were happy to refer to the old system to review previous sprint burndowns, and they would only be looking at the last couple typically anyway.

With the source CSV, user map, sprint map, and individual field mapping files prepared, we were finally ready to perform the transformation.  At this point, the process became significantly easier - take the CSV and use the mapping files to transform the contents of the columns appropriately - but there were still a few tricks to be aware of.

First of all, we actually created 4 files per project at this point - one containing only Epics, one containing only Standard-level Issues, one containing Subtasks, and one containing all Links.  This is because the importer runs line-by-line, and each file has dependencies on the previous - standard tasks can have Epic links, subtasks have parents, and links obviously refer to issues!  It should be noted that we excluded any link information from the original files as well here, since it would cause errors and duplicate links otherwise.

Secondly, although our first pass would produce a CSV per project (reflecting the export process), for import we would prefer to have a single, large CSV.  Consolidating the CSV files together was a relatively easy process, and we would subsequently break them up into sub-10k issue blocks to reduce the chance of import failure.  The reason we did this in two steps was actually because of the next trick.

The final thing was a sneaky approach to replicate ranking on destination.  Although the lexorank-based "Rank" field can be imported, it's not functional when you do so.  Instead, we used a quirk of the line-by-line processing to replicate the function - when an issue is created through CSV, it is created at the lowest rank in the system, meaning that the first row of the CSV, created first, will end up with the highest rank in the set.  Effectively, we would sort the CSV file by rank, with the highest rank at the top.  To ensure cross-project ranking worked correctly, we would perform this sorting on the fully consolidated CSV prior to being chunked up for import.

Overall, this generation process was quick - a matter of a few minutes in this project.

Next up (and note that I didn't say finally!), we had the import process itself.  This is always the most manual and boring part of the process.  We used the Jira CSV import wizard to import the files, in order, starting with the first chunk of the Epics file, and continuing chunk by chunk, file by file until complete.  For each import we needed to confirm the field mappings - with our mapping workbook having done much of the work, this was an easy process as names match directly, but still took a few seconds for each file.  It was long and boring work, scheduled out-of-hours for obvious reasons - we typically took around 5-8 hours for each wave in this project, which was fine for an overnight run.  No real automation hooks exist for this process, and other than building a web driver of sorts, manual effort is the only reasonable approach.

We still had one more step to complete, and an important one at that - although we had all our data in the destination system, the Issues were still missing their attachments.  In our case, we actually ended the downtime window prior to attachment migration, and informed users that attachments would migrate over the next 24 hours.  The attachment migration process would organise itself based on the attachment information in the source CSV, taking the location of the file on source, along with the original uploader, date, and Issue key, and stream the file across to the appropriate location in the Cloud.  One workaround here was that since attachments can't have their upload date or creator changed, we incorporated a comment with consolidated information about all migrated attachments on each Issue - not ideal, but entirely reasonable for this customer.

There were a few little tidbits worth mentioning here.  First, with such an old system, not all users that had created or interacted with issues were still at the company - in those cases we would default to a "migration user" in the mapping instead.  It would be easy to incorporate a comment with information about the original creator / commenter in an additional comment on the issue, but it was deemed unnecessary by this customer.  Second, although the mapping process started with the intention of having end-users update their own mapping sheets, we actually ended up automatically mapping the vast majority of fields as part of the mapping file generation.  We highlighted any files with missing mappings and reached out to those teams directly - this further reduced the load on end-users, and in hindsight probably made the pretty Excel workbooks a little redundant!  Finally, it should be noted that we also had about 900 lines of tests supporting the ~8,000 line codebase - both unit tests for the code (since it was such an extensive codebase), and validation checks for after the migration had completed.

So, I hope this walkthrough of the CSV migration process has been interesting.  Beyond just "how to do it", there are other lessons to be learned here.  First of all, don't underestimate the work here.  It is a significant technical exercise to pull off successfully, so don't bypass good practice because it's just scripting.  Second, there are a whole host of problems that can be encountered throughout the process, and for this example I tried to include everything we ran into.  This said, other systems will run into different, unique challenges, and you may find yourself having to think on your feet.  As an example, in many other migrations that I've been involved in, there has been a requirement to perform some kind of business logic on field values during transformation to account for differences between source Plugins and destination Apps - often consolidating multiple fields down into one.  Each migration is necessarily unique, and from my perspective at least, this provides me with an interesting problem solving challenge!  As a closing note - I used Python here, but there's no reason why other languages or tools couldn't be used - blimey, you could probably do most of this in Excel with enough VBA³!

¹By which I mean string, number, array, option, user, datetime or other - a relatively simplistic grouping which covered the different data structures in the CSV format very well.

²This said, the fiddly nature of the openpyxl styling might make it a good target for some AI-supported coding - this might be worth exploring.

³DO NOT DO THIS IN EXCEL.

So, we have come to the end of our deep dive into the world of Atlassian Cloud migrations, covering everything from the dawn of the SaaS products to detailed breakdowns of recent engagements.  I hope you have enjoyed reading this as much as I've enjoyed writing it - if you have any questions, please do reach out - you can catch me on my LinkedIn account, or via email at jsawyer@echidnasolutions.com.

As you have probably established from the sheer scale of this deep dive, a complete understanding of the space is hard to come by, but is fundamentally necessary for the best outcomes.  If you are embarking on a Cloud migration of your own, take a step back and think about your options - xCMA really isn't the only option, and I would argue in many cases is certainly not the best.  At Echidna Solutions we pride ourselves on being some of the most experienced Atlassian Consultants in the world, and that level of expertise can change the shape of a migration project enormously.

Cloud migrations are hard; we all know this to be true.  However, if executed correctly, they can bring real business value.  Contact Echidna Solutions for more information and to discuss your options.