TL;DR

This guide demonstrates building a TypeScript-based web scraper that uses LLMs to parse unstructured server monitoring data from vendor dashboards, legacy admin panels, and third-party SaaS platforms. You’ll integrate OpenAI’s API or local models like Llama 3 to extract metrics, interpret alert messages, and normalize data into Prometheus-compatible formats.

The scraper uses Playwright for browser automation, Cheerio for HTML parsing, and Zod for runtime type validation. LLMs handle the messy parts – converting vendor-specific HTML tables into structured JSON, interpreting ambiguous error messages, and extracting metrics from inconsistent layouts. This approach works well when APIs are unavailable or rate-limited.

Key integration points include using GPT-4 or Claude to generate CSS selectors from natural language descriptions, parsing alert severity from unstructured text, and converting proprietary metric formats into standard units. For example, you can feed raw HTML from a hosting provider’s dashboard to an LLM with a prompt like “Extract CPU usage, memory usage, and disk I/O from this table” and receive structured JSON output.

The TypeScript implementation provides strong typing for LLM responses, making it easier to catch parsing errors before they reach your monitoring stack. You’ll use function calling to ensure the LLM returns data in your expected schema rather than free-form text.

Critical warning: Always validate LLM-generated parsing logic in a staging environment before production deployment. LLMs can hallucinate CSS selectors or misinterpret metric units, leading to false alerts or missed incidents. Implement fallback mechanisms for when LLM parsing fails, and log all raw responses for debugging.

This approach is particularly valuable for teams managing heterogeneous infrastructure where vendors provide web dashboards but no programmatic access. The scraper runs as a systemd service, collecting data every five minutes and exposing it via a local HTTP endpoint for Prometheus to scrape.

Why LLMs for Web Scraping in Server Monitoring

Traditional web scraping relies on brittle CSS selectors and XPath expressions that break whenever a vendor updates their dashboard layout. When monitoring Linux servers through third-party web interfaces – cloud provider consoles, hardware management portals, or SaaS monitoring dashboards – you need extraction logic that adapts to structural changes without constant maintenance.

LLMs excel at understanding semi-structured HTML content semantically rather than positionally. Instead of writing document.querySelector('.metric-panel > div:nth-child(3) > span.value'), you can prompt an LLM to extract “the current CPU temperature reading” from raw HTML. The model identifies relevant data based on context, labels, and surrounding text rather than fragile DOM paths.

For TypeScript-based scrapers, integrate LLMs at the extraction layer. After fetching HTML with Puppeteer or Playwright, send relevant page sections to models like GPT-4 or Claude via their APIs. Request structured JSON output containing specific metrics:

const prompt = `Extract server metrics from this HTML fragment.
Return JSON with fields: cpu_temp, memory_used_gb, disk_io_mbps.

${htmlFragment}`;

This approach handles vendor dashboard redesigns gracefully. When a cloud provider rearranges their server details page, the LLM continues extracting temperature values because it understands “CPU Temperature: 67C” regardless of its position in the DOM tree.

Validation Requirements

LLM outputs require validation before feeding data into monitoring systems. Implement schema validation with libraries like Zod to ensure extracted values match expected types and ranges:

const MetricsSchema = z.object({
  cpu_temp: z.number().min(0).max(120),
  memory_used_gb: z.number().positive()
});

Never execute LLM-suggested shell commands directly in production environments. Use LLMs for read-only data extraction and analysis, not for generating administrative commands that modify system state. Review all AI-generated parsing logic manually before deployment to catch hallucinated field names or incorrect data transformations.

Architecture: TypeScript Scraper + LLM Pipeline

The architecture splits into three distinct layers: data collection, LLM processing, and action execution. Your TypeScript scraper runs as a systemd service, pulling metrics from vendor dashboards, API endpoints, or web interfaces that lack proper programmatic access.

Use Puppeteer or Playwright for JavaScript-heavy sites. The scraper authenticates, navigates to monitoring pages, and extracts DOM elements containing server metrics. Store raw HTML snapshots in /var/log/scraper/ with timestamps for audit trails.

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://vendor-dashboard.example.com/servers');
const metrics = await page.locator('.server-status').allTextContents();
await fs.writeFile(`/var/log/scraper/${Date.now()}.json`, JSON.stringify(metrics));

LLM Processing Pipeline

Feed scraped content to Claude or GPT-4 via API with a system prompt defining your infrastructure context. The LLM parses unstructured data, identifies anomalies, and generates shell commands for remediation. Use structured output modes (JSON schema enforcement) to ensure parseable responses.

curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_KEY" \
  -d '{"model":"claude-3-5-sonnet-20241022","messages":[{"role":"user","content":"Parse this server status and suggest systemctl commands: ..."}]}'

Validation and Execution

Never execute LLM-generated commands directly. Implement a whitelist of allowed operations and parameter validation. Log all suggested commands to /var/log/llm-actions/ for review. Use a manual approval queue for destructive operations like service restarts or configuration changes.

The TypeScript service writes proposed actions to a Redis queue. A separate Python worker validates commands against your Ansible playbooks or Salt states, ensuring alignment with existing automation. Only commands matching known-good patterns proceed to execution.

Critical: Test LLM outputs extensively in staging environments. Models occasionally hallucinate package names or misinterpret metric thresholds, potentially causing outages if executed blindly.

Setting Up the TypeScript Scraping Foundation

Start by creating a new TypeScript project with the necessary scraping and AI integration libraries. Initialize your project directory and install core dependencies:

mkdir llm-scraper-monitor && cd llm-scraper-monitor
npm init -y
npm install typescript @types/node tsx
npm install cheerio axios puppeteer
npm install openai anthropic

Configure TypeScript with strict settings in tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "commonjs",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "./dist"
  },
  "include": ["src/**/*"]
}

Core Scraper Architecture

Create a base scraper class that handles HTTP requests and DOM parsing. This foundation supports both static and JavaScript-rendered pages:

import axios from 'axios';
import * as cheerio from 'cheerio';
import puppeteer from 'puppeteer';

export class ServerMonitorScraper {
  async fetchStatic(url: string): Promise<cheerio.CheerioAPI> {
    const response = await axios.get(url);
    return cheerio.load(response.data);
  }

  async fetchDynamic(url: string): Promise<string> {
    const browser = await puppeteer.launch({ headless: true });
    const page = await browser.newPage();
    await page.goto(url, { waitUntil: 'networkidle0' });
    const content = await page.content();
    await browser.close();
    return content;
  }
}

AI Integration Layer

Set up environment variables for API keys in .env:

OPENAI_API_KEY=sk-your-key-here
ANTHROPIC_API_KEY=sk-ant-your-key-here

Create a wrapper for LLM interactions that will parse scraped HTML and extract monitoring metrics:

import OpenAI from 'openai';

export class LLMParser {
  private client: OpenAI;

  constructor() {
    this.client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
  }

  async extractMetrics(html: string, schema: object): Promise<any> {
    const response = await this.client.chat.completions.create({
      model: "gpt-4-turbo",
      messages: [{ role: "user", content: `Extract metrics: ${html}` }]
    });
    return JSON.parse(response.choices[0].message.content);
  }
}

Caution: Always validate LLM-extracted data against expected schemas before using it in production monitoring pipelines. Test parsing accuracy on sample pages before deploying.

Integrating LLM Analysis for Data Extraction

Once you have raw HTML from your scraper, LLMs excel at extracting structured data from unstructured content. Rather than writing brittle CSS selectors or XPath expressions that break with every layout change, you can pass HTML chunks to an LLM with a schema definition and let it handle the parsing.

OpenAI’s function calling feature provides reliable JSON output. Define your monitoring data schema as a TypeScript interface, then convert it to a JSON Schema for the API:

import OpenAI from 'openai';

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const schema = {
  type: "object",
  properties: {
    hostname: { type: "string" },
    cpu_usage: { type: "number" },
    memory_free_gb: { type: "number" },
    disk_alerts: { type: "array", items: { type: "string" } }
  },
  required: ["hostname", "cpu_usage", "memory_free_gb"]
};

async function extractMetrics(html: string) {
  const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [
      { role: "system", content: "Extract server metrics from HTML." },
      { role: "user", content: html }
    ],
    tools: [{
      type: "function",
      function: { name: "record_metrics", parameters: schema }
    }],
    tool_choice: { type: "function", function: { name: "record_metrics" } }
  });
  
  return JSON.parse(response.choices[0].message.tool_calls[0].function.arguments);
}

Validation and Safety

Always validate LLM output before acting on it. Use libraries like zod for runtime type checking:

import { z } from 'zod';

const MetricsSchema = z.object({
  hostname: z.string(),
  cpu_usage: z.number().min(0).max(100),
  memory_free_gb: z.number().nonnegative()
});

const validated = MetricsSchema.parse(extractedData);

Never execute commands suggested by an LLM without manual review. Log all extracted data to a separate audit file before feeding it into automation pipelines. Consider running LLM-based extraction in a read-only monitoring mode for several weeks before enabling any write operations or configuration changes.

Deploying as a Systemd Service on Linux

Running your TypeScript scraper as a systemd service ensures it starts automatically after reboots and restarts on failure. This approach integrates well with standard Linux monitoring tools and allows you to leverage AI assistants for generating service configurations tailored to your environment.

Create /etc/systemd/system/llm-scraper.service with your compiled TypeScript application path:

[Unit]
Description=LLM-Enhanced Web Scraper for Server Monitoring
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=scraper
Group=scraper
WorkingDirectory=/opt/llm-scraper
ExecStart=/usr/bin/node /opt/llm-scraper/dist/index.js
Restart=on-failure
RestartSec=30
StandardOutput=journal
StandardError=journal
Environment="NODE_ENV=production"
Environment="OPENAI_API_KEY_FILE=/etc/llm-scraper/api-key"

[Install]
WantedBy=multi-user.target

AI-Assisted Configuration Generation

Use Claude or GPT-4 to generate systemd units by providing your application requirements. Example prompt: “Generate a systemd service file for a Node.js application that scrapes server metrics every 5 minutes, runs as user ‘scraper’, and logs to journald.”

Caution: Always validate AI-generated systemd configurations before deployment. Verify the ExecStart path exists, the user account has appropriate permissions, and environment variables reference actual files. Test with systemd-analyze verify llm-scraper.service before enabling.

Enabling and Managing the Service

sudo systemctl daemon-reload
sudo systemctl enable llm-scraper.service
sudo systemctl start llm-scraper.service
sudo journalctl -u llm-scraper.service -f

For scheduled scraping rather than continuous operation, consider using a systemd timer instead. AI tools can generate matching .service and .timer files when you describe your scraping interval requirements. Monitor service health with systemctl status and integrate alerts through existing monitoring infrastructure like Prometheus node_exporter or Telegraf.

Implementation Steps

Start by initializing a TypeScript project with the necessary dependencies. Create a new directory and install the core packages for web scraping, LLM integration, and server monitoring:

mkdir llm-scraper-monitor && cd llm-scraper-monitor
npm init -y
npm install puppeteer cheerio axios openai dotenv
npm install -D typescript @types/node ts-node
npx tsc --init

Configure your tsconfig.json to target ES2020 or later and enable strict mode. Store your OpenAI API key in a .env file and never commit it to version control.

Building the Scraper Core

Create a scraper module that fetches server status pages or monitoring dashboards. Use Puppeteer for JavaScript-heavy pages or Cheerio for static HTML:

import puppeteer from 'puppeteer';

async function scrapeServerStatus(url: string): Promise<string> {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: 'networkidle2' });
  const content = await page.content();
  await browser.close();
  return content;
}

Integrating LLM Analysis

Connect the scraped data to an LLM for intelligent parsing. Use the OpenAI API to extract structured information from unstructured monitoring output:

import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function analyzeMetrics(rawHtml: string): Promise<any> {
  const response = await openai.chat.completions.create({
    model: 'gpt-4',
    messages: [{
      role: 'system',
      content: 'Extract CPU, memory, and disk metrics from this HTML.'
    }, {
      role: 'user',
      content: rawHtml
    }]
  });
  return JSON.parse(response.choices[0].message.content);
}

Caution: Always validate LLM-generated commands or configurations in a test environment before deploying to production systems. LLMs can hallucinate package names, command flags, or configuration syntax that appears plausible but fails in practice. Implement schema validation on parsed output and maintain manual review processes for critical infrastructure changes.