Cachet

This was resolved at 11:05 UTC.

git.almalinux.org exhausted its memory and suffered downtime. This has been resolved for the time being and is being addressed long term with more memory, and additional caching configurations within Gitea.

The forums just came back online. We'll continue monitoring.

Since AWS hasn't announced the all-clear yet there may be further outages.

Latest from AWS:

[10:38 AM PDT] Our mitigations to resolve launch failures for new EC2 instances are progressing and the internal subsystems of EC2 are now showing early signs of recovering in a few Availability Zones (AZs) in the US-EAST-1 Region. We are applying mitigations to the remaining AZs at which point we expect launch errors and network connectivity issues to subside.

forums.almalinux.org are hosted in us-east-1c which doesn't appear to have been fixed yet.

Latest update from AWS:

[08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations.

At this time we do believe the AWS issues with us-east-1 are the culprit.

Here's the latest update from AWS:

[07:14 AM PDT] We can confirm significant API errors and connectivity issues across multiple services in the US-EAST-1 Region. We are investigating and will provide further update in 30 minutes or soon if we have additional information.

The forums have dropped offline and we are investigating this.

There is an issue impacting AWS us-east-1 which may be a contributing factor to this outage but that has not yet been confirmed.

Updates are complete.

Discourse will be undergoing maintenance for an upgrade. Only a brief outage is expected.

Maintenance is complete.

The server has been moved to the new cabinet successfully and the new network is online. DNS has been updated to the reflect the new IPs.

We're verifying configurations and it's possible there will be a few more blips/reboots for the machine before we give the all clear.

Maintenance has begun.

https://vault.almalinux.org (and all other vault subdomains) will be undergoing maintenance today for network and hardware upgrades. This maintenance should be done within a few hours.

The system is back online.

The system exhausted its memory. The underlying cause is being investigated.

We're proactively making some adjustments before bringing the system back online.

This is being investigated.

The maintenance is complete and the updates are in production.

Mirrorlist requests (both web and API) should be significantly faster now.

The updates are nearly 50% complete and looking good so far.

We are performing scheduled maintenance on the mirrorlist service to implement a recent batch of updates as well as upgrade the back-end Valkey cluster from 7.2 to 8.1.

No downtime or loss of capacity is expected.

The issue is resolved.

The mirrorlist self-recovered after 2 minutes. We're investigating the underlying cause.

We're investigating a 2 minute outage with the mirrorlist system.