Portfolio Roadmaps

Incident Report for Roadmunk

Postmortem

Included below is a postmortem to the Portfolio roadmap issue we experienced. If there are any outstanding questions after reading, please feel free to reach out to our support team - support@roadmunk.com

What happened - Some users experienced Portfolio roadmaps in which source roadmaps were erroneously removed.

Why it happened - A software defect was missed during testing and was released to production. This defect introduced a logic error that unfortunately removed source roadmaps from Portfolios when a permission change happened.

What was done to fix it - When the extent of the problem became available Roadmunk’s engineering team investigated and found the root cause. The software was patched accordingly to alleviate the issue. In addition, Roadmunk ran a data restoration procedure for the affected roadmaps and restored them to their original state. All impacted roadmaps were successfully restored two days after the root cause was addressed.

How will this be prevented in the future - Moving forward, test cases have been added for this type of situation so it gets caught ahead of time. In addition, should the defect logic present itself again, we've also included safeguards to ensure roadmaps remain stable and return to a legacy state.

Posted Jan 13, 2022 - 00:05 UTC

Resolved

This incident has been resolved.

Posted Dec 07, 2021 - 18:48 UTC

Update

We are continuing to monitor for any further issues.

Posted Dec 06, 2021 - 14:19 UTC

Update

Recovery has been completed for affected users within our App server. Our team is available for any follow-up questions. You can reach us at: support@roadmunk.com.

Posted Dec 04, 2021 - 21:46 UTC

Update

Recovery has been completed for affected users within our EU server. More updates to follow.

Posted Dec 04, 2021 - 19:42 UTC

Update

Hi everyone,

Updating a previous item, the recovery process is well underway. To ensure a smooth process, our team will be restoring accounts based on their respective Roadmunk server. Corresponding updates will be made to notify when each server has been completed. If you're unsure about which server you belong to, simply sign in to Roadmunk and check the URL in your browser. Any account outside of a private deployment will see (app, eu, apac) preceding '.roadmunk.com'. To start, recovery has been completed for all users affected within our APAC server.

As always, please subscribe or check back here for any updates. For any additional questions, please don't hesitate to reach out to our support team (support@roadmunk.com).

Thanks again for your continued patience.

Posted Dec 04, 2021 - 18:16 UTC

Monitoring

We've applied a fix to help alleviate the issue. As of now, roadmap sources will not be removed from any new portfolio roadmaps you build.

With that said, our team is still monitoring the situation and a new recovery plan is underway to help restore your sources back to their original portfolio roadmap. Any new information will be updated accordingly as we receive it.

Posted Dec 02, 2021 - 16:17 UTC

Identified

We've identified an issue and are currently applying a fix that will result in 10-15 minute downtime between now and 10:15 EST. Thank you again for your continued patience.

Posted Dec 02, 2021 - 14:54 UTC

Investigating

We are currently experiencing an issue with Portfolio roadmaps. At the moment, it may not affect all users within your Roadmunk account. During this time, we ask that you refrain from building any new Portfolio roadmaps until our team has investigated the issue in full. For more details, feel free to reach out to our team: support@roadmunk.com. Thank you for your patience and understanding.

Posted Dec 01, 2021 - 17:21 UTC

This incident affected: Roadmapping.