by Dave Kellam

Gmail fixed itself

Earlier this year, Google’s Gmail service fixed itself before the engineers did. From the post-mortem blog post:

Users began seeing these errors on affected services at 11:02 a.m., and at that time our internal monitoring alerted Google’s Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time.

Found this sifting through some old bookmarks, hat-tip to Kyle Neath.

December 1, 2014 ·