11:45am PST Nov 18 --- getting 524 timeout on tons of miva sites

[dreamingdigital]

11:45am PST Nov 18 --- getting 524 timeout on tons of miva sites
not just in canada but in Cali as well.
Any news?

[ILoveHostasaurus]

The sites we know of having issues have all been on Cloudflare thus far; we're attempting to investigate further to identify particular geographic locations as Cloudflare support will typically not investigate until you can present that information with IP's of people who can reproduce the issue.

[dreamingdigital]

We have lots of sites with the timeout that are not on cloudflare. I can not even get to support.miva.com

[ILoveHostasaurus]

If that site is having issues I'd appreciate if you could email me your public IP so we can look into it; it would be easier to diagnose without Cloudflare in between, as that site is not using it.

[ids]

A client, on CF, was also getting an SSL config issue...

But now, looks like the site is starting to return to normal?

Scott

[ILoveHostasaurus]

We haven't had a new report in about 35 minutes, but did notice that on Cloudflare's status page they've switched their Tampa location to a re-routed status, which would indicate problems there. Due to the Tampa market being our current primary data center location, and the networks we'd likely have in common, it's possible a fair amount of CF-sourced traffic reaches us through that data center. I've not received the source IP information needed to do debugging from anyone having issues reaching a site not going through Cloudflare; will post further updates if we can troubleshoot with that type of data. I noticed on Downdetector that a few other large CF customers had a spike in reports around the same time, such as Discord. The majority of the reports were also from west coast entities, so it's possible some regional transit provider was having issues; CenturyLink, for example, also has a spike during the same time period.

[Rick Wilson]

Final Outcome AFAWC:

We don't have definitive evidence to single out a cause.

What we do know is:

1) all but one customer who reported issues were on Cloudflare, and that customer reported problems were resolved while others were still seeing issues.

2) it seemed to mostly affect visitors whose traffic originated on, or passed through, the US west coast to midwest.

3) we added Cloudflare health checks to several customers accounts while this was occurring, and none produced a failure even while the same customers were continuing to experience issues.

4) a few customers put Cloudflare in bypass and the issues went away.

5) none of the customers reporting issues were seeing Cloudflare errors, like what you would see when you can get to Cloudflare but they can't get to us.

6) a few large websites also showed a spike in outage reports on downdetector at the same time.

7) Cloudflare's Tampa location went into 're-routed' status around the time of the issue and remained in that state until after the issue seemed to be resolved.

So, with all that being said, our suspicion is that one of the large US internet providers was having issues and it was causing slowness or outages reaching certain Cloudflare data centers, and this may have impacted Cloudflare's Tampa data center being accessible to the rest of Cloudflare's network. We do not know how Cloudflare operates internally, but if they take customer traffic in and send it out a data center closer to the destination, and in this case the destination would have been a network in Tampa, it's possible these same network issues impacted them and by taking Tampa offline, it bypassed one of the problems. If it was a regional internet issue, this would explain why customers going to Cloudflare bypass mode were able to get around the problem, and could also explain why Cloudflare health checks were not failing, since the issue was shoppers getting to a Cloudflare data center, not them getting to us.