History
GitLab has resolved the incident affecting PR scanning and integration connectivity
Resolved: Brief slowdown of the scans API (US region)
On October 13, 2026, between 07:58 and 08:28 UTC, some requests
to the scans API in the US region failed or were slow. Scan pages in the Cycode UI and the Cycode CLI may have shown temporary errors, and some pull request scans arrived a few minutes later than usual.
We increased the memory available to the service, which fully resolved the issue at 08:34 UTC. All queued scans completed shortly after, and no data was lost. We have identified the cause and a permanent improvement is in review.
(1/2)
1. Summary
On October 5, 2026, starting at 15:55 IDT, customers on our US platform saw slow responses, intermittent errors, and delays in pull request checks and CLI scans. Some customers whose repositories require Cycode pull request checks couldn't merge until scanning resumed.
The issue started when a background maintenance process in our primary database fell behind and stopped completing, and the database stopped saving new data. Restoring service took several hours because the standby database first had to process a large backlog of changes.
The database was back and the platform was available again by 19:43 IDT. Delayed processing then caught up, and the incident was resolved at 23:24 IDT. Pull requests that were missed during the incident were scanned.
2. Key Timeline (IDT)
• October 5, 2026, 12:55 IDT: A routine database maintenance process starts running much slower than normal. No customer impact yet.
• October 5, 2026, 14:53 IDT: The next maintenance cycle starts and doesn't complete.
• October 5, 2026, 15:55 IDT: The database stops saving new data, and platform performance degrades.
• October 5, 2026, 16:03 IDT: The database reaches its connection limit. Customers start seeing errors.
• October 5, 2026, 16:21 IDT: Incident declared and response team engaged.
• October 5, 2026, 16:24 IDT: First update posted to the status page.
• October 5, 2026, 16:49 IDT: Switch to the standby database started.
• October 5, 2026, 17:00 IDT: Support case opened with our cloud infrastructure provider.
• October 5, 2026, 18:24 IDT: Alternative recovery path started in parallel. Workaround published for customers blocked by required pull request checks.
• October 5, 2026, 19:43 IDT: Database available again, and the Cycode API and Cycode Platform are accessible.
• October 5, 2026, 19:54 IDT: Pull request scanning resumes.
• October 5, 2026, 20:20 IDT: Processing capacity increased to clear delayed events.
• October 5, 2026, 20:45 IDT: High-volume workloads paused to reduce database load.
• October 5, 2026, 21:56 IDT: Pull requests missed during the incident identified and scanned.
• October 5, 2026, 23:09 IDT: All services fully operational.
• October 5, 2026, 23:24 IDT: Incident resolved.
3. Root Cause
Our database runs a routine background process that regularly saves recent changes to permanent storage. Two conditions contributed to that process slowing down:
• High-volume short-lived data operations: some of our processing workloads were constantly creating and removing large numbers of short-lived data structures.
• A delayed background sync: a data-sync process had fallen far behind, which stopped the database from cleaning up after itself.
The maintenance process slowed down and eventually stopped completing. The database could then no longer save new data, and requests started to queue. Our services retried those requests and opened many new connections, which used up the database's connection capacity. Customers saw this as errors and slow responses.
We switched to our standby database. Before it could serve traffic, it had to process all changes made since the last completed maintenance cycle, about four hours of activity. That extended the outage.
4. Actions Taken
• Reduced load on the database by temporarily scaling down non-essential processing.
• Started a switch to the standby database.
• Opened a support case with our cloud infrastructure provider and worked with them on an escalated basis.
• Prepared an alternative recovery path in parallel.
• Shared a temporary workaround for customers blocked by required pull request checks.
• Once the database recovered, increased processing capacity to clear delayed events.
1. Summary
On October 5, 2026, starting at 15:55 IDT, customers on our US platform saw slow responses, intermittent errors, and delays in pull request checks and CLI scans. Some customers whose repositories require Cycode pull request checks couldn't merge until scanning resumed.
The issue started when a background maintenance process in our primary database fell behind and stopped completing, and the database stopped saving new data. Restoring service took several hours because the standby database first had to process a large backlog of changes.
The database was back and the platform was available again by 19:43 IDT. Delayed processing then caught up, and the incident was resolved at 23:24 IDT. Pull requests that were missed during the incident were scanned.
2. Key Timeline (IDT)
October 5, 2026, 12:55 IDT: A routine database maintenance process starts running much slower than normal. No customer impact yet.
October 5, 2026, 14:53 IDT: The next maintenance cycle starts and doesn't complete.
October 5, 2026, 15:55 IDT: The database stops saving new data, and platform performance degrades.
October 5, 2026, 16:03 IDT: The database reaches its connection limit. Customers start seeing errors.
October 5, 2026, 16:21 IDT: Incident declared and response team engaged.
October 5, 2026, 16:24 IDT: First update posted to the status page.
October 5, 2026, 16:49 IDT: Switch to the standby database started.
October 5, 2026, 17:00 IDT: Support case opened with our cloud infrastructure provider.
October 5, 2026, 18:24 IDT: Alternative recovery path started in parallel. Workaround published for customers blocked by required pull request checks.
October 5, 2026, 19:43 IDT: Database available again, and the Cycode API and Cycode Platform are accessible.
October 5, 2026, 19:54 IDT: Pull request scanning resumes.
October 5, 2026, 20:20 IDT: Processing capacity increased to clear delayed events.
October 5, 2026, 20:45 IDT: High-volume workloads paused to reduce database load.
October 5, 2026, 21:56 IDT: Pull requests missed during the incident identified and scanned.
October 5, 2026, 23:09 IDT: All services fully operational.
October 5, 2026, 23:24 IDT: Incident resolved.
3. Root Cause
Our database runs a routine background process that regularly saves recent changes to permanent storage. Two conditions contributed to that process slowing down:
High-volume short-lived data operations: some of our processing workloads were constantly creating and removing large numbers of short-lived data structures.
A delayed background sync: a data-sync process had fallen far behind, which stopped the database from cleaning up after itself.
The maintenance process slowed down and eventually stopped completing. The database could then no longer save new data, and requests started to queue. Our services retried those requests and opened many new connections, which used up the database's connection capacity. Customers saw this as errors and slow responses.
We switched to our standby database. Before it could serve traffic, it had to process all changes made since the last completed maintenance cycle, about four hours of activity. That extended the outage.
4. Actions Taken
Reduced load on the database by temporarily scaling down non-essential processing.
Started a switch to the standby database.
Opened a support case with our cloud infrastructure provider and worked with them on an escalated basis.
Prepared an alternative recovery path in parallel.
Shared a temporary workaround for customers blocked by required pull request checks.
Once the database recovered, increased processing capacity to clear delayed events.
Paused the workloads that were creating high volumes of short-lived data operations, to reduce database load while delayed events cleared.
Scanned the pull requests that were missed during the incident.
4b. Action Items
Change how large batches of data are written to reduce database load.
Move our highest-volume workload to its own dedicated database, so a problem there can't affect the rest of the platform.
Add alerts and deeper monitoring for database connection usage and database health.
Give each service its own dedicated database credentials.
Process incoming events more efficiently to speed up recovery after a disruption.
Improve monitoring of processing delays and of internal event alerts.
Review database capacity and performance settings to add more headroom.
MemoryDB metrics are stable, the processing lag has recovered, and the system is now fully operational.
MemoryDB metrics are stable, the processing lag has recovered, and the system is now fully operational.
The system is back to being fully operational.
The system is back to being fully operational.
1. Summary
On August 19, 2026, some customers experienced slower page loading and data display, as well as delays when starting or completing pull request scans.
The disruption was caused by a managed memory database entering a repeated restart cycle and not recovering automatically. This created a temporary slowdown in processing. Expected service behavior was restored after corrective actions, including replacement and increased capacity for the affected service. The provider is continuing to investigate why the service was unable to return to normal operation automatically.
2. Key Timeline (IDT)
• August 19, 2026, 16:10 IDT: We identified performance degradation affecting parts of the platform and pull request scanning.
• August 19, 2026, 16:30 IDT: An initial corrective change was applied and service behavior was monitored.
• August 19, 2026, 16:46 IDT: We confirmed that some delays were continuing and expanded the investigation.
• August 19, 2026, 17:14 IDT: We began working with our service provider to investigate instability in the managed memory database.
• August 19, 2026, 18:46 IDT: Recovery actions were in progress, with temporary mitigations in place to reduce customer impact.
• August 19, 2026, 19:06 IDT: The managed memory database capacity update completed, and platform responsiveness and pull request scanning returned to expected behavior.
3. Root Cause
The managed memory database entered a repeated restart cycle and was unable to recover automatically. This caused delays in the processing systems that support platform responsiveness and pull request scans.
The affected service underwent a routine replacement process, followed by a capacity increase during the recovery effort. While the capacity increase completed successfully and restored expected behavior, AWS is still researching why the service did not come back up normally after the restart cycle. Their detailed root-cause analysis is pending.
4. Actions Taken
• Applied corrective updates to reduce immediate processing demand.
• Temporarily adjusted the status update process to reduce reliance on the affected service.
• Restarted affected processing components to restore responsiveness.
• Worked with the service provider to replace affected capacity and complete a capacity increase.
• Monitored platform performance and scan processing until expected behavior was confirmed.
4b. Action Items
• Review the service provider’s detailed root-cause analysis when available.
• Improve monitoring to detect similar recovery delays earlier.
• Add safeguards to reduce the impact of temporary processing slowdowns.
• Introduce separate Redis capacity for each service to isolate workloads and reduce the impact of an issue in one service on others.
1. Summary
On August 19, 2026, some customers experienced slower page loading and data display, as well as delays when starting or completing pull request scans.
The disruption was caused by a managed memory database entering a repeated restart cycle and not recovering automatically. This created a temporary slowdown in processing. Expected service behavior was restored after corrective actions, including replacement and increased capacity for the affected service. The provider is continuing to investigate why the service was unable to return to normal operation automatically.
2. Key Timeline (IDT)
August 19, 2026, 16:10 IDT: We identified performance degradation affecting parts of the platform and pull request scanning.
August 19, 2026, 16:30 IDT: An initial corrective change was applied and service behavior was monitored.
August 19, 2026, 16:46 IDT: We confirmed that some delays were continuing and expanded the investigation.
August 19, 2026, 17:14 IDT: We began working with our service provider to investigate instability in the managed memory database.
August 19, 2026, 18:46 IDT: Recovery actions were in progress, with temporary mitigations in place to reduce customer impact.
August 19, 2026, 19:06 IDT: The managed memory database capacity update completed, and platform responsiveness and pull request scanning returned to expected behavior.
3. Root Cause
The managed memory database entered a repeated restart cycle and was unable to recover automatically. This caused delays in the processing systems that support platform responsiveness and pull request scans.
The affected service underwent a routine replacement process, followed by a capacity increase during the recovery effort. While the capacity increase completed successfully and restored expected behavior, AWS is still researching why the service did not come back up normally after the restart cycle. Their detailed root-cause analysis is pending.
4. Actions Taken
Applied corrective updates to reduce immediate processing demand.
Temporarily adjusted the status update process to reduce reliance on the affected service.
Restarted affected processing components to restore responsiveness.
Worked with the service provider to replace affected capacity and complete a capacity increase.
Monitored platform performance and scan processing until expected behavior was confirmed.
4b. Action Items
Review the service provider’s detailed root-cause analysis when available.
Improve monitoring to detect similar recovery delays earlier.
Add safeguards to reduce the impact of temporary processing slowdowns.
Introduce separate Redis capacity for each service to isolate workloads and reduce the impact of an issue in one service on others.