Error loading app.impiloplatform.com login page

Summary

On June 12, 2025 at 2:09pm Eastern the primary login page for our production application began returning an error page. This problem persisted until 3:57pm. Users who were already logged in experienced no interruption. All backend data flows (including reading processing and webhook calls) were similarly not affected. The root cause of this interruption was a large-scale disruption to the Cloudflare platform, which Impilo’s authentication provider leverages for various security-related purposes.

Detailed Analysis

Impilo’s platform was recently enhanced to support various new login types, including MFA devices and SSO. This was doing by integrating with a third-party SaaS, WorkOS. Our login process is now:

• User navigates to app.impiloplatform.com
• Without a valid JWT token, the user is redirected to app.impiloplatform.com/login
• This issues a redirect to auth.impiloplatform.com, hosted by the WorkOS platform.
• Following successful authentication through WorkOS, the user is issued a JWT and is redirected back to the app.impiloplatform.com site, where they can use the Impilo platform as expected.

WorkOS leverages Cloudflare for some protective functionality, including bot detection, DDOS protection and more. While Impilo’s platform, which does not have a public login creation functionality, has limited exposure to these risks, we do nonetheless benefit from Cloudflare on our login page through WorkOS.

During the incident, Cloudflare was no longer able to serve our auth.impiloplatform.com functionality on WorkOS' behalf due to their own outage. This led to the outage of our login prompt.

Incident Timeline

All times Eastern.

• (Background) 1:51pm: Google Cloud experienced a global configuration issue causing issues with a storage platform.
• (Background) 1:52pm: Cloudflare internally identifies issues from the upstream Google Cloud issue
• 2:09pm: Internal impilo employee reports an issue (“Error 1101”) when trying to log into Impilo.
• 2:10pm: Issue is confirmed by engineering and determined to be related to our third-party authentication provider.

• 2:10pm: Third-party authentication provider is notified via our shared Slack Connect channel.

  • Impilo engineers confirm that all reading intake is processing normally and that any existing web sessions are proceeding normally.

• 2:14pm: Impilo Status Page changed to Partial Outage.

• 2:19pm: Authentication provider updates their status page.

• 2:19pm: Cloudflare updates their status page to indicate the outage is widespread

• 3:13pm: Cloudflare indicates that recovery is beginning

• 3:57pm: Impilo sees first successful logins

• 4:01pm: Impilo determines that service is fully restored

• During this time, the Impilo Engineering team is discussing mitigation options and looking into ways to circumvent the third-party authentication page. However, due to our SSO integration being fully dependent on this provider, as well as global internet issues causing we did not deploy any changes.

Changes Made During Incident

None.

Consideration was given to creating a login prompt that would accept non-SSO logins, but two issues arose:

1. Our frontend application build pipeline depends on downloading our NPM dependencies from NPM. NPM itself was facing an outage due to the same Cloudflare incident.
2. We expected significant customer service impact from the confusion of having duplicative but non-identical login pages.

Because the incident was limited to new logins, we ultimately chose not to proceed with deployment of a build that was created outside of our trusted build process. We may prepare this functionality for usage in the future should a similar incident occur.