Monitoring
GovWifi uses a number of monitoring tools and scheduled jobs to collect metrics, monitor service health, and generate usage reports.
This page describes how monitoring works across Grafana, Prometheus, Google Analytics, and the related Rake tasks.
Grafana
Grafana is an open-source analytics and monitoring platform used to monitor the health of GovWifi in real time.
Every GovWifi environment has its own Grafana instance, running on AWS EC2.
This Google Document contains in-depth information on the technical setup (you must be a member of the GovWifi team to view this document).
You can access the dashboards using the links below (VPN and dashboard access required):
Where the Data Comes From / Grafana Data Sources
The data in Grafana primarily comes from Prometheus and Elasticsearch, both hosted in AWS.
- Prometheus collects data from the Radius servers (for example, authentication requests over time).
This data is more fine-grained and typically used by engineers. - Elasticsearch provides higher-level usage insights and generates monthly reports sent to GPA.
Elasticsearch
The admin and logging-api applications collect and push a range of metrics (such as active users and completion rates) to our Elasticsearch cluster in AWS.
Data is pulled from databases at regular intervals (hourly, daily, monthly, etc.) and sent to Elasticsearch via ECS scheduled jobs.
These scheduled jobs run Rake tasks that push data to Elasticsearch at specific intervals.
See this example Terraform job.
The metrics are also backed up in an S3 bucket in each GovWifi environment. This is configured by the govwifi-dashboard module in terraform. The diagram below shows the resources that Elasticsearch interacts with. A scalable version is available in our team drive:
Using Grafana Data to Generate Monthly Reports
The GovWifi team uses Grafana metrics to generate monthly reports.
Detailed instructions can be found here (GovWifi access required).
Hosted on GOV.UK PaaS (Platform as a Service)
Prior to November 2023, an additional Grafana instance was hosted on GOV.UK PaaS.
It monitored performance of the GovWifi Product Pages and Tech Docs.
This data is now collected via Google Analytics. The PaaS was scheduled for decommissioning in December 2023, and the Product Pages and Tech Docs are now hosted on GitHub Pages.
Rake Tasks
A number of Rake tasks are used to collect and publish metrics that feed into Grafana dashboards and monthly reports.
These tasks run automatically as ECS scheduled jobs in AWS, but can also be triggered manually if needed.
Logging API Tasks
The Logging API defines several tasks under the Performance::Metrics module.
Each task generates and uploads a specific set of metrics to S3 and Elasticsearch using the Performance::Metrics::MetricSender class.
Metrics collected
| Metric | Use Case Class | Destination |
|---|---|---|
| active_users | Performance::UseCase::ActiveUsers |
S3 / Elasticsearch |
| completion_rate | Performance::UseCase::CompletionRate |
S3 / Elasticsearch |
| inactive_users | Performance::UseCase::NewUsers |
S3 / Elasticsearch |
| roaming_users | Performance::UseCase::RoamingUsers |
S3 / Elasticsearch |
| volumetrics | Performance::UseCase::Volumetrics |
S3 / Elasticsearch |
| user_devices | Performance::UseCase::UserDevices |
S3 / Elasticsearch |
These tasks are invoked by scheduled jobs defined in Terraform (see logging-scheduled-tasks.tf).
The metrics are visualised in Grafana and used in monthly reports.
Admin Application Tasks
The Admin application defines a Rake namespace opensearch for publishing metrics to OpenSearch (previously Elasticsearch).
Task
rake opensearch:publish_metrics
Purpose
Collects usage data about organisations and locations, then writes the results to the govwifi-metrics index in OpenSearch.
Data collected
| Metric | Use Case Class | Description |
|---|---|---|
| organisation_usage_stats | UseCases::OrganisationUsage |
Usage per organisation |
| new_organisations | UseCases::NewOrganisations |
New organisations signing up |
| new_locations | UseCases::NewLocations |
Newly added locations |
| new_cba_organisations | UseCases::NewCbaOrganisations |
New CBA organisations added |
Google Analytics
We currently have a Google Analytics dashboard which shows a summary of visits to our Product Page and Admin site.
There is an additional dashboard which used to allow for more detailed investigations of how people used these pages. However, this dashboard is currently broken.
Prometheus
Prometheus is an
open source software application used for event monitoring and
alerting. It records real-time metrics in a time series database built
using a HTTP pull model, with flexible queries and real-time alerting.
We run a Prometheus server which scrapes metrics from Prometheus log
exporters running on the FreeRADIUS containers.
These Prometheus exporters provide a wide range of information about
the actual FreeRADIUS server state and the packages being processed.
The information is used for diagnostics and tracking service
availability.
If you have SSM access, you can run the commands below to see the
dashboard. If not, please speak to the reliability engineers on the
team about access.
The previous SSH method is in the process of being deprecated and will be removed soon, advise to setup SSM access.
ssh -L 9090:127.0.0.1:9090 prometheus.<env>.govwifi
The below code gets the instance ID and uses it to start tunnel session via SSM, update the example with the Server name and region.
INSTANCE_ID=$(gds aws govwifi-development -- aws ec2 describe-instances --filter "Name=tag:Name,Values=<ENV> Prometheus-Server" --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" --region <region> --output text)
gds aws govwifi-<env> -- aws ssm start-session --target $INSTANCE_ID --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["9090"],"localPortNumber":["9090"]}' --region <region>
eg for Dev London
INSTANCE_ID=$(gds aws govwifi-development -- aws ec2 describe-instances --filter "Name=tag:Name,Values=Alpaca Prometheus-Server" --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" --region eu-west-2 --output text)
gds aws govwifi-development -- aws ssm start-session --target $INSTANCE_ID --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["9090"],"localPortNumber":["9090"]}' --region eu-west-2
After running the command you should be able to access the Prometheus
dashboard by entering the following address in your browser: