Cachet

Maintenance is complete.

chat.almalinux.org is undergoing maintenance for platform upgrades. During this time the bridge between chat platforms will also be offline.

Something unknown went wrong within the containers for Discourse.

Discourse uses a very convuluted docker setup using custom orchestration scripts making it nearly impossible to troubleshoot.

Things are back online at this time.

The forums have dropped offline. We are investigating.

This was resolved at 11:05 UTC.

git.almalinux.org exhausted its memory and suffered downtime. This has been resolved for the time being and is being addressed long term with more memory, and additional caching configurations within Gitea.

The forums just came back online. We'll continue monitoring.

Since AWS hasn't announced the all-clear yet there may be further outages.

Latest from AWS:

[10:38 AM PDT] Our mitigations to resolve launch failures for new EC2 instances are progressing and the internal subsystems of EC2 are now showing early signs of recovering in a few Availability Zones (AZs) in the US-EAST-1 Region. We are applying mitigations to the remaining AZs at which point we expect launch errors and network connectivity issues to subside.

forums.almalinux.org are hosted in us-east-1c which doesn't appear to have been fixed yet.

Latest update from AWS:

[08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations.

At this time we do believe the AWS issues with us-east-1 are the culprit.

Here's the latest update from AWS:

[07:14 AM PDT] We can confirm significant API errors and connectivity issues across multiple services in the US-EAST-1 Region. We are investigating and will provide further update in 30 minutes or soon if we have additional information.

The forums have dropped offline and we are investigating this.

There is an issue impacting AWS us-east-1 which may be a contributing factor to this outage but that has not yet been confirmed.

Updates are complete.

Discourse will be undergoing maintenance for an upgrade. Only a brief outage is expected.

Maintenance is complete.

The server has been moved to the new cabinet successfully and the new network is online. DNS has been updated to the reflect the new IPs.

We're verifying configurations and it's possible there will be a few more blips/reboots for the machine before we give the all clear.

Maintenance has begun.

https://vault.almalinux.org (and all other vault subdomains) will be undergoing maintenance today for network and hardware upgrades. This maintenance should be done within a few hours.