nationalgrid/README.md

# National Grid Real-Time Data Monitor

A Python application that fetches real-time data from the National Grid Carbon Intensity API, including:
- Load and demand information
- Fuel mixture (generation mix)
- Carbon intensity forecasts
- Regional and national statistics

## Features

- Configurable Grid Supply Point (GSP) selection
- Real-time data collection from National Grid APIs
- JSON output format (perfect for Grafana integration)
- Regional generation mix data
- National statistics and carbon intensity

## Installation

No dependencies required beyond Python 3 standard library and `requests`:

```bash
pip install requests
```

## Usage

### Default (Bishops Wood):
```bash
python3 nationalgrid_monitor.py
```

### Specify a different GSP:
```bash
python3 nationalgrid_monitor.py --gsp "London"
python3 nationalgrid_monitor.py --gsp "North Scotland"
python3 nationalgrid_monitor.py --gsp "Yorkshire"
```

### List available GSPs:
```bash
python3 nationalgrid_monitor.py --list
```

## Available GSPs

- Bishops Wood (West Midlands)
- North Scotland
- South Scotland
- North West England
- North East England
- Yorkshire
- North Wales
- South Wales
- West Midlands
- East Midlands
- East England
- South West England
- South England
- London
- South East England

## Output

Data is saved to `<GSP_Name>.json` in the current directory.

Example output structure:
```json
{
  "gsp_name": "Bishops Wood",
  "gsp_id": 11,
  "timestamp": "2025-12-11T17:39:24.668415Z",
  "collection_time": "2025-12-11T17:39:24.668436",
  "regional": {
    "intensity": {...},
    "generation_mix": [...]
  },
  "national": {
    "intensity": {...}
  },
  "generation": {
    "mix": [...]
  }
}
```

## VictoriaMetrics Integration

The tool pushes metrics to VictoriaMetrics using the Prometheus exposition format via the `/api/v1/import/prometheus` endpoint.

### Supported Endpoints

**Direct VictoriaMetrics** (single-node):
- Default: `http://localhost:8428/api/v1/import/prometheus`

**vmagent** (if using vmagent as a proxy):
- Typically: `http://localhost:8429/api/v1/import/prometheus`

**VictoriaMetrics Cluster** (vminsert):
- Format: `http://vminsert:8480/insert/0/prometheus/api/v1/import/prometheus`

### Usage

```bash
# Push to VictoriaMetrics single-node on localhost (default)
python3 nationalgrid_monitor.py --gsp "Bishops Wood"

# Push to vmagent
python3 nationalgrid_monitor.py --victoriametrics http://localhost:8429

# Push to VictoriaMetrics cluster
python3 nationalgrid_monitor.py --victoriametrics http://vminsert:8480/insert/0/prometheus

# Push to custom VictoriaMetrics instance
python3 nationalgrid_monitor.py --victoriametrics http://your-vm-host:8428

# Skip VictoriaMetrics push (JSON only)
python3 nationalgrid_monitor.py --no-push
```

### Metrics Format

All metrics are exported as `national_grid{gsp="<gsp_name>",metric="<metric_name>"} <value>`

Available metrics:
- `demand_mw` - **Actual national demand in MW** (from Elexon BMRS API - no API key required)
- `carbon_intensity` - Actual carbon intensity (gCO2/kWh)
- `carbon_intensity_forecast` - Forecast carbon intensity (gCO2/kWh)
- `load_indicator` - Load indicator based on carbon intensity
- `wind` - Wind generation percentage
- `gas` - Gas generation percentage
- `nuclear` - Nuclear generation percentage
- `biomass` - Biomass generation percentage
- `solar` - Solar generation percentage
- `hydro` - Hydro generation percentage
- `coal` - Coal generation percentage
- `imports` - Imports percentage
- `other` - Other generation percentage

### Continuous Monitoring

Run the monitoring loop to continuously push data:

```bash
# Default: Bishops Wood, 1-minute intervals
./monitor_loop.sh

# Custom GSP and interval (seconds)
./monitor_loop.sh "London" 60

# Custom interval (e.g., 5 minutes)
./monitor_loop.sh "Bishops Wood" 300

# Custom VictoriaMetrics URL
./monitor_loop.sh "London" 60 "http://10.0.0.5:8428"
```

### Systemd Service (Autostart on Boot)

To run the monitor automatically as a system service:

1. **Install the service:**
```bash
sudo cp nationalgrid-monitor.service /etc/systemd/system/
sudo systemctl daemon-reload
```

2. **Customize settings (optional):**
Edit `/etc/systemd/system/nationalgrid-monitor.service` and modify the `ExecStart` line to change:
- GSP name (default: "Bishops Wood")
- Update interval in seconds (default: 60)
- VictoriaMetrics URL (default: http://localhost:8428)

3. **Enable and start the service:**
```bash
sudo systemctl enable nationalgrid-monitor.service
sudo systemctl start nationalgrid-monitor.service
```

4. **Check service status:**
```bash
sudo systemctl status nationalgrid-monitor.service
```

5. **View logs:**
```bash
sudo journalctl -u nationalgrid-monitor.service -f
```

6. **Stop or disable:**
```bash
sudo systemctl stop nationalgrid-monitor.service
sudo systemctl disable nationalgrid-monitor.service
```

## Grafana Integration

Once metrics are in VictoriaMetrics:

1. Add VictoriaMetrics as a Prometheus datasource in Grafana
2. Query metrics like: `national_grid{gsp="Bishops_Wood",metric="wind"}`
3. Create dashboards with panels for:
   - Generation mix (stacked graph)
   - Carbon intensity (time series)
   - Individual fuel sources (gauges)
   - Load/demand trends

### Example PromQL Queries

```promql
# Wind generation for Bishops Wood
national_grid{gsp="Bishops_Wood",metric="wind"}

# Carbon intensity across all GSPs
national_grid{metric="carbon_intensity"}

# Gas generation comparison
national_grid{metric="gas"}

# Total renewable (wind + solar + hydro)
national_grid{metric="wind"} + national_grid{metric="solar"} + national_grid{metric="hydro"}
```

## API Data Sources

This tool uses multiple free APIs (no API key required):

### Carbon Intensity API
- Real-time carbon intensity data
- Generation mix by fuel type (biomass, coal, gas, nuclear, wind, solar, etc.)
- Regional and national statistics
- Half-hourly forecasts (minute resolution available)
- API: https://api.carbonintensity.org.uk/
- Resolution: minute (for most recent data)

### Elexon BMRS API
- **Actual national demand in MW** (updated every 5 minutes)
- No API key required for demand data
- API: https://data.elexon.co.uk/bmrs/api/v1/demand/outturn/summary
- Resolution: minute (for latest data)

## License

MIT