Cloud - Portal Forms - Are not being attached for some customers
Incident Report for ThinkTilt
Postmortem

On 17 February 2020, an incident occurred involving the Jira Service Desk (JSD) Cloud customer portal which affected ProForma. During the incident forms submitted though the customer portal to create a new request were not attached to the request. The contents of these forms has been lost.

The cause of the incident was a code change made by Atlassian which introduced a bug that prevented multiple Atlassian Marketplace apps from interacting with the JSD portal. The code changes have now been reversed and the bug has been fixed. You can review the incident details from Atlassian here: https://jira-service-desk.status.atlassian.com/incidents/ww9x8wxl7knf.

I am now writing to update you with further information about the incident timing, it’s potential impact to your and your customers, and how you can respond to the situation. We appreciate how important form data is and we want to do our utmost to ensure that your processes aren’t unduly affected.

Impact

This incident only affected forms being used to create new requests in the JSD portal. Other form operations, such as editing the contents of an existing form, or manually adding a new form to an issue/request were not affected.

Due to the bug Atlassian introduced, when a ProForma form was submitted through the JSD portal a request was created; however, the contents of the form were not saved to the request. As a result of the bug the contents of the form were lost.

This means you may have a number of newly created request in your JSD projects, which do not have forms attached to them and may otherwise appear blank.

Timing

Atlassian has confirmed that the issue occurred between 9:00 UTC and approximately 16:00 to 17:00 UTC on Monday 17 February 2020. Each customer might have slightly different time frames because the fix rolled out over a period of time. For different timezones this incident occurred at:

  • Sydney : 10pm, 17 Feb to 4am 18 Feb
  • London : 9am to 5pm, 17 Feb
  • San Francisco : 1am to 9am, 17 Feb

Possible repair pathways:

Depending on the type of data being received, you will need to advise customers that their request was not submitted properly due to a processing error.

To find the possible issues affected, you can use the following JQL queries (adjusting the project key as appropriate). In cloud the created date is relative to the timezone set for your instance:

  • For Sydney: project = KEY AND created >= "2020-02-17 22:00" AND created <= "2020-02-18 04:00" order by created DESC
  • For London: project = KEY AND created >= "2020-02-17 09:00" AND created <= "2020-02-17 17:00" order by created DESC
  • For San Francisco: project = KEY AND created >= "2020-02-17 01:00" AND created <= "2020-02-17 09:00" order by created DESC

Once you have identified the issues you can either

  1. Ask the requestor to create a new request; or
  2. You can manually attach the required form to the existing issue and ask them to resubmit.

To attach a form to the existing issue:

  1. View the affected requests where a ticket was created but there was no form.
  2. Add the relevant form to the affected request.
  3. Mark the form as “External” so that it will be visible on the portal to the user. This will allow the user to resubmit their information with the least impact.
  4. Send a comment to the user requesting them to complete the form again.
  5. The user can log into the portal, view the request and fill out the form.
  6. Process the request as normal.

It may be useful to also label all the affected tickets for monitoring.

Next steps by ProForma:

We are also providing feedback to Atlassian and await their detailed post-mortem on how this occurred. We have a good relationship with Atlassian and will continue to work closely with them so that such breaking changes are not introduced into their cloud products. We will also examine our own response to this incident to find ways that we can better improve our own processes or products to avoid people data loss.

Again, we appreciate how important form data is, and we are genuinely sorry for the inconvenience this will cause you and your customers.

Please contact us if you have any questions or concerns.

Posted Feb 18, 2020 - 23:06 AEDT

Resolved
Atlassian have rolled back the changes around Feb 17, 2020 - 17:04 UTC. This issue has now been resolved.

For reference the Atlassian incident was https://jira-service-desk.status.atlassian.com/incidents/ww9x8wxl7knf
Posted Feb 18, 2020 - 04:15 AEDT
Update
Atlassian has escalated the issue and is continuing to investigate.
Posted Feb 18, 2020 - 04:15 AEDT
Identified
We have identified the cause of the issue. It relates to a change affecting Jira Service Desk cloud. The change has been introduced by Atlassian. We are engaging with Atlassian to have this issue resolved as quickly as possible.
Posted Feb 17, 2020 - 22:23 AEDT
Investigating
We are currently investigating an issue affecting some customers not being able to attach forms via the portal.
Posted Feb 17, 2020 - 22:08 AEDT