# Production Ready Prometheus + Grafana Setup for Two Custom Metrics

# Introduction

In an [earlier article about custom metrics](https://cloudtuned.hashnode.dev/custom-micrometer-gauge-metrics) we explored the process of creating custom metrics in a `Spring Boot Application` and by the end of it we had two code examples of working metrics that could be scrapped by Prometheus via the Spring Boot actuator endpoint.

# Next Steps

This time we'll make sure:

-   Prometheus alerts you when dependencies or scheduled jobs fail

-   Grafana shows a green/red indicator with historical trends

**Prometheus Alert Rules**
--------------------------

Save this as `alerts_springboot.yml` and include it in your Prometheus `prometheus.yml` config.

```
groups:
  - name: springboot-health
    interval: 30s
    rules:
      - alert: BookingsServiceDependenciesUnhealthy
        expr: bookingsService_checkDependencies == 0
        for: 2m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "Dependencies unhealthy in the Bookings service"
          description: "The bookings service dependency check has failed for more than 2 minutes."

      - alert: BookingsServiceScheduledJobsUnhealthy
        expr: bookingsService_checkScheduledJobs == 0
        for: 2m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "Scheduled jobs unhealthy in Bookings service"
          description: "Scheduled jobs are failing or delayed for more than 2 minutes."
```

**Why this works well:**

-   `== 0` means "not OK" in your gauge system

-   `for: 2m` prevents flapping alerts for temporary issues

-   `severity` levels allow routing to different channels (PagerDuty, Slack, email, etc.)

**Grafana Panel --- Status + Trend**
----------------------------------

Here's a **panel JSON** you can import directly into Grafana.

-   **Step 1:** Go to Grafana → Dashboard → Import → Paste JSON

-   **Step 2:** Select your Prometheus datasource

-   **Step 3:** Adjust titles if needed

```
{
  "type": "stat",
  "title": "Bookings Service Health",
  "targets": [
    {
      "expr": "bookingsService_checkDependencies",
      "legendFormat": "Dependencies",
      "refId": "A"
    },
    {
      "expr": "bookingsService_checkScheduledJobs",
      "legendFormat": "Scheduled Jobs",
      "refId": "B"
    }
  ],
  "options": {
    "reduceOptions": {
      "values": false,
      "calcs": ["lastNotNull"],
      "fields": ""
    },
    "orientation": "auto",
    "colorMode": "value",
    "graphMode": "area",
    "justifyMode": "auto",
    "displayMode": "color-background"
  },
  "fieldConfig": {
    "defaults": {
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "red", "value": 0 },
          { "color": "green", "value": 1 }
        ]
      },
      "mappings": [
        {
          "type": "value",
          "options": {
            "0": { "color": "red", "text": "Unhealthy" },
            "1": { "color": "green", "text": "Healthy" }
          }
        }
      ]
    },
    "overrides": []
  }
}

```
**What you get in Grafana:**

-   **Big green "Healthy" / red "Unhealthy" indicators** for each metric

-   A quick way to see **current status at a glance**

-   Historical trend line (if you switch to time series mode)

# Conclusion

On this article we looked into a simple example of how to combine custom metrics exposed by your application, and once they are scrapped by Prometheus we were able to make a ready for production dashboard in grafana.