Mastering Testing and Optimization of AI Pipelines 

Below you will find some suggested materials on these topics, which we believe may help you complete the final activity in the lesson. Please note that using these resources is entirely optional, and you are welcome to explore any other sources or courses at your own discretion.

Introduction (2 Minutes)

Opening Statement:

• Welcome to this session on Monitoring and Logging in Production Systems.

• Today, we’ll explore the importance of monitoring and logging, tools like Prometheus and Grafana, and how to set up alerts to identify and address pipeline issues.

Why Monitoring and Logging Matter:

1. Reliability: Monitoring ensures systems are performing as expected and can identify anomalies before they escalate.

2. Debugging: Logs provide a detailed record of system behavior, helping engineers pinpoint the root cause of issues.

3. Performance Optimization: Monitoring metrics like latency and throughput helps improve system performance.

Part 1: Importance of Monitoring and Logging (5 Minutes)

Monitoring:

• Definition: The continuous process of collecting, analyzing, and visualizing system metrics.

• Key Metrics to Monitor:

  • System Metrics: CPU, memory, disk usage, and network traffic.

  • Application Metrics: API response times, error rates, and request counts.

  • Pipeline Metrics: Data processing latency and throughput.

Logging:

• Definition: The process of recording events, errors, and system messages.

• Benefits:

  • Provides a historical record of system events.

  • Helps identify patterns and recurring issues.

  • Supports compliance and auditing.

Real-World Example:

• Imagine a production pipeline for processing user data:

  • Without monitoring: A spike in API errors might go unnoticed, leading to downtime.

  • With monitoring: Alerts notify engineers immediately, enabling quick resolution.

Part 2: Tools for Monitoring API Performance (6 Minutes)

1. Prometheus:

• Overview: Open-source monitoring and alerting toolkit.

• Features:

  • Time-series database.

  • Flexible query language (PromQL).

  • Built-in alert manager.

• Use Case: Monitor API response times and error rates.

2. Grafana:

• Overview: Visualization tool that integrates with Prometheus.

• Features:

  • Customizable dashboards.

  • Supports multiple data sources (Prometheus, Elasticsearch, etc.).

  • Real-time alerts.

• Use Case: Create dashboards to visualize API performance metrics.

3. Integration Workflow:

• Prometheus collects metrics from your application.

• Grafana visualizes these metrics on dashboards.

• Alerts are triggered based on thresholds set in Prometheus Alertmanager.

Demo Example:

• Metric: API response time (ms).

• Dashboard Visualization: Line graph showing response times over the past hour.

Part 3: Setting Up Alerts for Pipeline Issues (6 Minutes)

Why Alerts Are Critical:

• Alerts notify engineers of potential problems before they escalate.

• Example: High latency in a data pipeline or a spike in 500 server errors.

Steps to Set Up Alerts:

1. Define Metrics to Monitor:

   • API errors (500 responses).

   • Latency exceeding a threshold (e.g., >200ms).

   • Unusual spikes in request counts.

2. Configure Prometheus Alert Rules:

Example Alert Rule:
yaml
Copy code
groups:

  - name: API Alerts

    rules:

      - alert: HighLatency

        expr: http_request_duration_seconds_avg > 0.2

        for: 2m

        labels:

          severity: warning

        annotations:

          summary: "High latency detected in API responses."

3. Set Up Alert Notifications:

   • Use Prometheus Alertmanager to configure notifications.

   • Supported channels: Slack, email, PagerDuty, etc.

   • Example: Slack Integration

     • Configure a webhook in Slack.

     • Add the webhook URL to Alertmanager configuration.

4. Test Alerts:

   • Simulate a metric spike (e.g., increase response latency).

   • Verify that alerts are triggered and notifications are sent.

Best Practices:

• Avoid excessive alerts to reduce “alert fatigue.”

• Prioritize critical alerts over informational ones.

• Regularly review and update alert rules.

Conclusion and Key Takeaways (2 Minutes)

Recap:

1. Monitoring: Use tools like Prometheus to collect and analyze key system metrics.

2. Logging: Maintain detailed logs to support debugging and compliance.

3. Alerts: Set up targeted alerts to detect and address pipeline issues proactively.

Call to Action:

• Implement monitoring and logging in your production systems today.

• Use Prometheus and Grafana to create dashboards and alerts tailored to your application.

Objective

By the end of this activity, you will:

1. 1. Integrate monitoring tools (Prometheus and Grafana) with the deployed Jarvid pipeline.

   2. Set up dashboards to visualize API performance and system health.

Step-by-Step Instructions

Step 1: Set Up Monitoring Environment

1. Install Prometheus:

   • Download Prometheus from the official website.

   • Extract the downloaded file and navigate to the Prometheus directory.

   • https://prometheus.io/download/

   • Your Mac blocks you when running Prometheus, so you need to go to Privacy & Security and grant it permissions.

   • Graphana: https://grafana.com/grafana/download?platform=mac

   • or the commands to install it directly:

   • curl -O https://dl.grafana.com/enterprise/release/grafana-enterprise-11.5.2.darwin-amd64.tar.gz

   • tar -zxvf grafana-enterprise-11.5.2.darwin-amd64.tar.gz

   • prometheus.io

   • Download | Prometheus

   • An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.

   • Grafana Labs

   • Download Grafana | Grafana Labs

   • Overview of how to download and install different versions of Grafana on different operating systems.

Start Prometheus:
bash
Copy code
./prometheus --config.file=prometheus.yml

2. Install Grafana:

   • Download Grafana from the official website.

   • Follow the installation instructions for your operating system.

Start Grafana:
bash
Copy code
grafana-server

Step 2: Configure Prometheus for the Jarvid Pipeline

Create Prometheus Configuration: File: prometheus.yml
yaml
Copy code

global:

  scrape_interval: 15s

scrape_configs:

  - job_name: 'jarvid-api'

    static_configs:

      - targets: ['localhost:8000']  # Replace with your Jarvid API host and port

1. Expose Metrics in the Jarvid Pipeline:

   • Use the prometheus_client library in your Flask/FastAPI application.

Install Prometheus Client:
bash
Copy code
pip install prometheus_client

Modify the API Code:
python
Copy code

from fastapi import FastAPI

from prometheus_client import start_http_server, Counter, Summary

import time

app = FastAPI()

# Start Prometheus metrics server

start_http_server(9100)

# Define Prometheus metrics

REQUEST_COUNT = Counter("request_count", "Total number of requests")

REQUEST_LATENCY = Summary("request_latency_seconds", "Latency of API requests")

@app.get("/")

@REQUEST_LATENCY.time()

def root():

    REQUEST_COUNT.inc()

    return {"message": "Welcome to the Jarvid API"}

3. Restart Prometheus:

   • Reload the configuration to scrape metrics from the Jarvid API:

bash
Copy code
curl -X POST http://localhost:9090/-/reload

Step 3: Integrate Grafana with Prometheus

1. Log in to Grafana:

   • Default URL: http://localhost:3000

   • Default credentials: admin / admin.

2. Add Prometheus as a Data Source:

   • Navigate to Connections > Data Sources > Add Data Source.

   • Select Prometheus and set the URL to http://localhost:9090.

3. Create a Dashboard:

   • Navigate to Create > Dashboard > Add New Panel.

   • Use Prometheus queries to visualize metrics:

Total Requests:
promql
Copy code
request_count

Request Latency:
promql
Copy code
rate(request_latency_seconds_sum[1m]) / rate(request_latency_seconds_count[1m])

Step 4: Set Up Alerts in Grafana

1. Define Alert Conditions:

   • Example: Trigger an alert if API latency exceeds 200ms.

2. Set Up Alert Rule:

   • Navigate to your panel, click Alert > Create Alert.

Define a condition:
promql
Copy code
rate(request_latency_seconds_sum[1m]) / rate(request_latency_seconds_count[1m]) > 0.2

3. Configure Notification Channels:

   • Navigate to Alerting > Notification Channels.

   • Add a new channel (e.g., Slack or email).

4. Test Alerts:

   • Simulate high latency by adding a delay in the API:

python
Copy code

import time

time.sleep(0.5)  # Simulates latency

5. Verify that the alert is triggered in Grafana.

Step 5: Validate and Test Monitoring

1. Generate Traffic for Metrics:

   • Use tools like curl or locust to simulate API requests:

bash
Copy code
locust -f load_test_script.py

2. Verify Metrics in Grafana:

   • Open the dashboard and confirm that metrics are updating in real time.

3. Document Findings:

   • Capture screenshots of the dashboard.

   • Note any anomalies or areas for improvement.

Expected Outcomes

1. Integrated Monitoring System:

   • Prometheus collects and scrapes metrics from the Jarvid pipeline.

   • Grafana visualizes these metrics on custom dashboards.

2. Operational Alerts:

   • Alerts are triggered for pipeline issues, such as high latency.

3. Enhanced Observability:

   • Real-time insights into API performance and system health.

Objective

By the end of this activity, you will:

1. Implement a logging mechanism to track errors and performance metrics in the Jarvid API.

2. Configure alerts for API downtime or performance degradation using monitoring tools.

Step-by-Step Instructions

Step 1: Set Up Logging Mechanism

Install Required Libraries:
bash
Copy code
pip install python-json-logger

1. 
   

2. Integrate Logging in the Jarvid API:

   • Modify your Flask/FastAPI application to include structured logging for errors and performance metrics.

File: app/main.py
python
Copy code

import logging

from pythonjsonlogger import jsonlogger

from fastapi import FastAPI, HTTPException

from prometheus_client import start_http_server, Counter, Summary

app = FastAPI()

# Set up Prometheus metrics

start_http_server(9100)

REQUEST_COUNT = Counter("request_count", "Total number of requests")

REQUEST_LATENCY = Summary("request_latency_seconds", "Latency of API requests")

# Set up JSON logging

log_handler = logging.StreamHandler()

log_formatter = jsonlogger.JsonFormatter()

log_handler.setFormatter(log_formatter)

logger = logging.getLogger()

logger.addHandler(log_handler)

logger.setLevel(logging.INFO)

@app.get("/")

@REQUEST_LATENCY.time()

def root():

    REQUEST_COUNT.inc()

    logger.info({"event": "root_endpoint_accessed", "status": "success"})

    return {"message": "Welcome to the Jarvid API"}

@app.get("/error")

def error():

    try:

        raise ValueError("Simulated error")

    except ValueError as e:

        logger.error({"event": "error_occurred", "error": str(e)})

        raise HTTPException(status_code=500, detail="Internal Server Error")

3. Test Logging:

Start the API and access endpoints:
bash
Copy code
uvicorn app.main:app --reload

• 
  

• Access http://localhost:8000/ to log a successful request.

• Access http://localhost:8000/error to log an error.

Step 2: Configure Alerts for API Downtime and Performance

1. Set Up Prometheus Alerts:

   • Edit the prometheus.yml configuration file to define alert rules.

File: prometheus.yml
yaml
Copy code

groups:

  - name: API Alerts

    rules:

      - alert: APIDowntime

        expr: up{job="jarvid-api"} == 0

        for: 2m

        labels:

          severity: critical

        annotations:

          summary: "The Jarvid API is down."

          description: "No heartbeat detected for the Jarvid API for the last 2 minutes."

      - alert: HighLatency

        expr: rate(request_latency_seconds_sum[1m]) / rate(request_latency_seconds_count[1m]) > 0.2

        for: 2m

        labels:

          severity: warning

        annotations:

          summary: "High latency detected in the Jarvid API."

          description: "Latency has exceeded 200ms for the last 2 minutes."

2. Set Up Prometheus Alertmanager:

   • Configure Alertmanager to send notifications.

File: alertmanager.yml
yaml
Copy code

global:

  resolve_timeout: 5m

route:

  receiver: slack_notifications

receivers:

  - name: slack_notifications

    slack_configs:

      - api_url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"

        channel: "#alerts"

4. Start Alertmanager:
   bash
   Copy code
   ./alertmanager --config.file=alertmanager.yml

5. Link Prometheus to Alertmanager:

Update prometheus.yml to include Alertmanager:
yaml
Copy code

alerting:

  alertmanagers:

    - static_configs:

        - targets: ["localhost:9093"]  # Alertmanager address

Step 3: Test Alerts

1. Simulate Downtime:

Stop the API server:
bash
Copy code
pkill uvicorn

• Verify that the APIDowntime alert is triggered in Prometheus and sent to Slack.

2. Simulate High Latency:

Introduce artificial latency in the API:
python
Copy code

import time

time.sleep(0.5)  # Add delay in response

• Access the API multiple times and verify the HighLatency alert.

Step 4: Validate and Document Findings

1. Capture Logs:

Check structured logs for recorded events:
bash
Copy code
tail -f logs.json

2. Capture Alerts:

   • Take screenshots of alerts in Slack or Prometheus.

3. Document Metrics:

   • Record metrics like request latency and downtime detection time.

Expected Outcomes

1. Logging Mechanism:

   • Structured logs capturing API events, errors, and performance metrics.

2. Alerting System:

   • Alerts configured for API downtime and high latency.

   • Notifications delivered to the configured Slack channel.

3. Improved Observability:

   • Comprehensive logging and alerting to monitor and debug API performance.