What happened is that all the TCP (Transmission Control Protocol) connections of the pool got into a sort of “zombie” state: broken yet not closed. Looking at the logs and CloudWatch metrics it quickly became apparent there was a connectivity issue between our application and the database. This had little impact on the overall service, but it’s still an issue. The database failed over as it should, and most of our stack recovered swiftly.Įxcept one of our services (an application built with the Play framework). Again, this is expected and we should plan accordingly. Over the weekend our primary production database experienced a network failure. In my opinion, it’s probably the best connection pool available for the JVM. The most popular connection pool at the Guardian is called Hikari as it is the default connection pool used by the Play framework. If you use a relational database, chances are you are using a connection pool to access the database. Once the application needs to execute a request, it checks if a connection is available in the pool and uses it to send the request. A connection pool is a set of network connections kept connected to a database for the purpose of executing requests without having to create a new connection for each request. Once promoted, that copy becomes the primary database and gets to execute all the SQL requests. This process usually takes about 30s to two minutes. That copy is in fact kept warm and up to date for the sole purpose of being promoted in case of an incident (If you’re interested here’s the documentation). RDS then promotes a copy of that database as the new primary database. If you know what a fail-over or a connection pool is, you can safely skip to the next section, if you don’t let’s define them.Ī fail-over happens when RDS can’t keep the primary database alive for some reason such as a network failure, hardware failure, crash. We’ll dive into how the incident happened and what we can do to attenuate the effects of such an incident in the future, but before we do let’s go over a few concepts that we’ll need later.
0 Comments
Leave a Reply.AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |