How to Rotate Database Passwords
This guide sets out how to perform database password rotation without causing downtime for any of the GovWifi Applications.
Overview Of Process
The password must be updated first on the RDS instance, then on the app. The apps run as ECS tasks and load the database passwords into memory as an environment variable from AWS Secret Manager when they first start. If the secret is changed, they will only pick up the new value at launch time. Existing tasks have no way to detect that the secret has changed.
Instructions
Set a dual password for the RDS master user by following the instructions below 1. Log into the bastion box 1. Connect to the RDS database server by following these instructions 1. Once in the mysql console select your database for example
use govwifi_staging;
Run the following command and replace “username” with the RDS master username “newpass” with the new password you wish to use.
ALTER USER 'username' IDENTIFIED BY 'newpassword' RETAIN CURRENT PASSWORD;
The above sets a dual password, without deleting the old one. This is important because the apps running as ECS tasks will still be using the old password. If you do not retain the old password the app will fail to connect to the database and cause an incident. Ensure your new password conforms to best practice
Update the database password used by the ECS task.
- Change the database password in the Secrets Manager
The naming convention for these passwords is
rds/db/credentials. For example the name of the sessions database secret would berds/session-db/credentials:- Restart the affected applications
- You will need to restart different combinations of ECS tasks depending on which database password you are changing. Stop one ECS task and check that it restarts successfully. Run the smoke tests. If all is well, stop the remaining tasks. ECS will launch new tasks with the new password automatically:
- If you’ve changed the password for the sessions database you will need to restart:
- Logging-api tasks
- Admin tasks
- If you’ve changed the password for the admin database you will need to restart:
- Admin tasks
- If you’ve changed the password for the user database you will need to restart:
- Admin tasks
- Logging-api tasks
- User-sign-up-api tasks
Make sure you check both regions for tasks (both eu-west-2 and eu-west-1).
Delete the old password permanently by running the following command on the mysql prompt:
ALTER USER 'username' DISCARD OLD PASSWORD;