Failure to read time series data. Heatmaps unable to load.

Incident Report for 75F

Postmortem

InfluxDB cloud, our time series database provider, experienced a prolonged outage with their query interface, which caused all hisRead operations to fail in the 75F platform. Time series writes (hisWrites) did not appear to be impacted. So far there have been no signs of customer data loss during the time of the outage.

This vendor service outage did however result in heatmaps and trend charts being rendered unusable until the incident was resolved.

Posted May 12, 2023 - 09:12 CDT

Resolved

This incident has been resolved.

Posted May 11, 2023 - 20:01 CDT

Monitoring

The outage appears to be mitigated, however they are still monitoring the issue. Heatmaps and all app/portal operations have been restored.

Posted May 11, 2023 - 19:55 CDT

Update

InfluxDB is still investigating the issue on their end.

Posted May 11, 2023 - 18:07 CDT

Update

InfluxDB is still down without an ETA on the fix

Posted May 11, 2023 - 16:12 CDT

Identified

The issue has been identified

Posted May 11, 2023 - 15:48 CDT

Update

InfluxDB Cloud is still degraded. Still no ETA on restoration.

Posted May 11, 2023 - 15:27 CDT

Update

InfluxDB Cloud is still experiencing an outage. No ETA yet on when service will be restored.

Posted May 11, 2023 - 14:58 CDT

Update

InfluxDB Cloud has updated their status and are tracking the outage on their end here: https://status.influxdata.com/

Posted May 11, 2023 - 14:43 CDT

Update

Confirmed that InfluxDB cloud is experiencing an outage and the region our installation is hosted on is impacted. Still awaiting updates from their end.

Posted May 11, 2023 - 14:28 CDT

Update

This appears to be caused by a fault with our Time Series database provider: InfluxDB Cloud.

We are working with the vendor to determine the cause and ETA on a fix

Posted May 11, 2023 - 14:22 CDT

Update

We are continuing to investigate this issue.

Posted May 11, 2023 - 14:19 CDT

Investigating

We are currently investigating this issue.

Posted May 11, 2023 - 14:18 CDT

This incident affected: Haystack API.