At 8:36 Pacific Time, TUNE began experiencing issues with report loading and delays. Application access issues quickly followed these issues, resulting in a 404 page when attempting to log in. We've identified a problem with our downstream provider, and they are working diligently to restore access.
12:36 UTC March 19, 2025
Current status: We identified an issue affecting an automatic scaling system, triggering resource exhaustion and throttling on systems that process customer requests. After implementing an initial mitigation, we have seen some customer recovery across affected deployments; however, the AWS - US East (N. Virginia) region is continuing to see the impact on warehouse operations. We are attempting to implement additional mitigations to reduce resource contention issues within the region, and we’ll provide another update within 60 minutes. Customer experience: Customers hosted in the specified regions may experience delays or timeouts while attempting to perform queries or use Snowflake services and features via Snowsight. Queries may appear to be stuck in a running state. Customers hosted in the AWS - US East (N. Virginia) region may be unable to resume warehouses.
15:32 UTC March 19, 2025
Latest Update: Within the AWS - US East (N. Virginia) region, we have identified an additional issue preventing the cloud services layer from processing warehouse requests as expected due to resource contention issues. We're continuing to implement multiple mitigations to help remediate impact, including increasing capacity and, in parallel, reverting the most recent software releases in the region.
TUNE Engineers have successfully implemented a hotfix to reduce the number of 404 errors upon log-in. However, reporting is still delayed.
16:46 UTC March 19, 2025
Latest Update: After our additional mitigation efforts, service health is recovering. We're continuing to implement optimizations and monitor the affected environments while the remaining cloud services layer clusters fully recover. As such, customer recovery times may vary across the affected infrastructure. Since resolution is being observed in the region, failing over is no longer recommended for customers with replication enabled in other regions unless they are continuing to experience impact. We'll provide another update within 60 minutes.
17:27 UTC March 19, 2025
Latest Update: TUNE services have been restored, and Reports are populated with data. Please contact support@tune.com if you are still experiencing issues loading report data.
18:14 UTC March 19, 2025
Current status: All clusters are reporting service health recovery after implementing mitigations for this issue. Please note, customer recovery times may vary across the affected infrastructure. We'll continue to monitor the environment until we're confident all services are functioning properly.
Customer experience: Customers hosted in the AWS - US East (N. Virginia) and AWS - US West (Oregon) regions experienced intermittent delays or timeouts while attempting to perform queries or use Snowsight. Starting at approximately 16:00 UTC, customers hosted in the AWS - US East (N. Virginia) region were unable to access or use Snowflake services and features.
Incident start time: 12:36 UTC March 19, 2025
Incident end time: 19:45 UTC March 19, 2025
Preliminary root cause: A code issue affecting an automatic scaling system triggered resource exhaustion and throttling on systems that process customer requests. After implementing an initial mitigation, a secondary, related issue was identified within a subset of the AWS - US East (N. Virginia) region that caused more severe impact to cloud services layer clusters that facilitate the scheduling of requests from virtual warehouses. The root cause of the secondary issue is currently under investigation.