Skip to main content

One post tagged with "sgtm"

View All Tags

Logger Patch Guide: Tag-Independent HTTP Logs in a Self-Hosted sGTM Setup

· 10 min read
DEVNT
Platform Team

Quick answer​

If you run self-hosted sGTM and conversion quality suddenly drops, start with runtime HTTP logs before touching individual tags.

In a Node.js-based sGTM container, a logger patch gives you two streams in one place: inbound requests entering the server and outbound requests sent to Meta, TikTok, GA4, CRM APIs, and other endpoints. That view lets you separate input issues from transformation issues and delivery issues without guessing.

If logging is only tag-level, you can miss real production traffic in self-hosted environments and spend hours debugging the wrong layer. Runtime-level visibility first, tag-level debugging second.

Why this is possible​

Server-side GTM is still an application process inside a container, not a sealed black box.
That means you can patch selected runtime HTTP APIs (http/https) and observe network I/O while keeping app logic in place.

Use this with discipline:

  • keep patches small and observable
  • test under load before production rollout
  • treat updates carefully, because upstream changes can affect patch behavior

How it works​

The idea is straightforward:

  1. Hook incoming HTTP requests.
  2. Hook outgoing HTTP/HTTPS requests.
  3. Write structured JSON logs with key fields:
    • method
    • URL
    • status
    • duration
    • headers
    • body (size-limited)
  4. Print logs to stdout so your log stack (Loki, ELK, Datadog, etc.) can ingest them.

In practice, this gives you a clear timeline of what came in, what went out, and how long each step took.

Core ideas behind the approach​

1) It is monkey patching, by design​

This approach works by monkey-patching Node.js runtime APIs (typically http and https methods) at startup.

In plain terms:

  • you wrap original request/response functions
  • you observe traffic before/after calling the original behavior
  • your app keeps working as usual, but now you have structured logs

Why people choose this:

  • no need to rewrite business logic
  • no need to touch every tag or integration path
  • one patch can cover HTTP traffic paths that pass through the patched runtime APIs

2) It is tag-independent in self-hosted environments​

A tag-based logger only sees what the tag implementation decides to expose.
A runtime patch sees HTTP I/O at the process level, which makes it much more useful for validating real production behavior in self-hosted environments.

That means:

  • you still get logs even if a specific tag is misconfigured
  • you can compare what entered the server vs what left to destinations
  • you are not blocked by UI preview limitations

3) It enables out-of-container observability​

Because logs go to stdout, you can inspect them from outside the container:

  • kubectl logs
  • cluster logging agents (Fluent Bit, Vector, Promtail, etc.)
  • centralized tools (Loki, ELK, Datadog)

So you do not need to exec into pods to understand failures.
You can troubleshoot from your standard ops/observability stack.

What this does not cover​

This approach does not automatically give you:

  • business-event correlation
  • root-cause classification
  • distributed tracing semantics
  • guaranteed visibility into traffic paths that bypass patched runtime APIs

Example code (Node.js)​

Below is the logger patch currently used in our GTM chart deployments:

const https = require('https');
const http = require('http');
const { URL } = require('url');

let getCurrentTraceContext;
function getTraceContext() {
    if (!getCurrentTraceContext) {
        try {
            const tracer = require('./tracer-patch.cjs');
            getCurrentTraceContext = tracer.getCurrentTraceContext || (() => ({ trace_id: null, span_id: null }));
        } catch (e) {
            getCurrentTraceContext = () => ({ trace_id: null, span_id: null });
        }
    }
    return getCurrentTraceContext();
}

const MAX_BODY_SIZE = 10 * 1024; // 10KB limit

function isBase64(str) {
    if (!str || str.length < 4) return false;
    const base64Regex = /^[A-Za-z0-9+/]+=*$/;
    return base64Regex.test(str) && str.length % 4 === 0;
}

function tryDecodeBase64(value) {
    try {
        if (isBase64(value)) {
            const decoded = Buffer.from(value, 'base64').toString('utf-8');
            if (decoded.length > 0 && /^[\x20-\x7E\s]*$/.test(decoded)) {
                return decoded;
            }
        }
    } catch (e) {
    }
    return value;
}

function redactUrl(urlString) {
    try {
        const urlObj = new URL(urlString);
        let redacted = false;
        const decodedParams = {};
        for (const [key, value] of urlObj.searchParams.entries()) {
            const decoded = tryDecodeBase64(value);
            if (decoded !== value) {
                decodedParams[key] = decoded;
            }
        }
        for (const [key, decodedValue] of Object.entries(decodedParams)) {
            urlObj.searchParams.set(key, `[base64:${decodedValue}]`);
            redacted = true;
        }
        return redacted ? urlObj.toString() : urlString;
    } catch (e) {
        return urlString;
    }
}

function tryParseBody(buffer) {
    if (!buffer || buffer.length === 0) return undefined;
    try {
        const str = buffer.toString('utf8');
        if ((str.startsWith('{') || str.startsWith('['))) {
            return JSON.parse(str);
        }
        return str;
    } catch (e) {
        return buffer.toString('utf8');
    }
}

function log(type, method, url, status, duration, reqBody, resBody, reqHeaders, resHeaders) {
    try {
        const { trace_id, span_id } = getTraceContext();
        const logEntry = {
            type,
            method,
            url: redactUrl(url),
            status,
            duration_ms: duration,
            timestamp: new Date().toISOString()
        };
        if (reqBody) logEntry.RequestBody = reqBody;
        if (resBody) logEntry.ResponseBody = resBody;
        if (reqHeaders) logEntry.RequestHeaders = reqHeaders;
        if (resHeaders) logEntry.ResponseHeaders = resHeaders;
        if (trace_id) {
            logEntry.trace_id = trace_id;
            logEntry.span_id = span_id;
        }
        console.log(JSON.stringify(logEntry));
    } catch (err) {
    }
}

function createImmediateBodyCapture() {
    let chunks = [];
    let totalSize = 0;
    let finalized = false;
    return {
        addChunk: function (chunk) {
            if (finalized || totalSize >= MAX_BODY_SIZE) return;
            try {
                const buf = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
                chunks.push(buf);
                totalSize += buf.length;
            } catch (e) {
            }
        },
        finalize: function () {
            if (finalized) return null;
            finalized = true;
            try {
                if (chunks.length === 0) return null;
                const result = Buffer.concat(chunks);
                chunks = [];
                totalSize = 0;
                return result;
            } catch (e) {
                chunks = [];
                totalSize = 0;
                return null;
            }
        }
    };
}

function spyOnReadableStream(stream, onComplete) {
    try {
        const capture = createImmediateBodyCapture();
        const originalEmit = stream.emit;
        stream.emit = function (event, ...args) {
            try {
                if (event === 'data' && args[0]) {
                    capture.addChunk(args[0]);
                } else if (event === 'end') {
                    const body = capture.finalize();
                    if (body) {
                        onComplete(body);
                    }
                }
            } catch (e) {
            }
            return originalEmit.apply(this, arguments);
        };
    } catch (e) {
    }
}

function spyOnWritableStream(stream, onComplete) {
    try {
        const capture = createImmediateBodyCapture();
        const originalWrite = stream.write;
        const originalEnd = stream.end;
        stream.write = function (chunk, ...args) {
            try {
                if (chunk) capture.addChunk(chunk);
            } catch (e) {
            }
            return originalWrite.apply(this, [chunk, ...args]);
        };
        stream.end = function (chunk, ...args) {
            try {
                if (chunk) capture.addChunk(chunk);
                const body = capture.finalize();
                if (body) {
                    onComplete(body);
                }
            } catch (e) {
            }
            return originalEnd.apply(this, [chunk, ...args]);
        };
    } catch (e) {
    }
}

function patchRequest(module, protocol) {
    const originalRequest = module.request;
    module.request = function (...args) {
        const startTime = Date.now();
        let reqBodyBuffer = null;
        try {
            const req = originalRequest.apply(this, args);
            spyOnWritableStream(req, (buffer) => {
                reqBodyBuffer = buffer;
            });
            let url = 'unknown';
            try {
                if (args[0] instanceof URL) url = args[0].toString();
                else if (typeof args[0] === 'string') url = args[0];
                else if (args[0] && typeof args[0] === 'object') {
                    const host = args[0].hostname || args[0].host || 'localhost';
                    const path = args[0].path || '/';
                    const proto = args[0].protocol || protocol;
                    url = `${proto}//${host}${path}`;
                }
            } catch (e) {
            }
            req.on('response', (res) => {
                let resBodyBuffer = null;
                try {
                    spyOnReadableStream(res, (buffer) => {
                        resBodyBuffer = buffer;
                    });
                    res.on('end', () => {
                        try {
                            const duration = Date.now() - startTime;
                            const reqHeaders = req.getHeaders ? req.getHeaders() : req._headers;
                            const resHeaders = res.headers;
                            log('outbound', req.method || 'GET', url, res.statusCode, duration, tryParseBody(reqBodyBuffer), tryParseBody(resBodyBuffer), reqHeaders, resHeaders);
                        } catch (e) {
                        }
                    });
                } catch (e) {
                }
            });
            return req;
        } catch (e) {
            return originalRequest.apply(this, args);
        }
    };
    const originalGet = module.get;
    module.get = function (...args) {
        const req = module.request(...args);
        req.end();
        return req;
    };
}

patchRequest(https, 'https:');
patchRequest(http, 'http:');

const originalCreateServer = http.createServer;
http.createServer = function (...args) {
    const server = originalCreateServer.apply(this, args);
    server.on('request', (req, res) => {
        const startTime = Date.now();
        let reqBodyBuffer = null;
        let resBodyBuffer = null;
        try {
            spyOnReadableStream(req, (buffer) => {
                reqBodyBuffer = buffer;
            });
            spyOnWritableStream(res, (buffer) => {
                resBodyBuffer = buffer;
            });
            res.on('finish', () => {
                try {
                    const duration = Date.now() - startTime;
                    const protocol = (req.socket && req.socket.encrypted) ? 'https' : 'http';
                    const host = req.headers.host || 'unknown';
                    const url = `${protocol}://${host}${req.url}`;
                    if (req.url === '/healthz') return;
                    const reqHeaders = req.headers;
                    const resHeaders = res.getHeaders();
                    log('inbound', req.method, url, res.statusCode, duration, tryParseBody(reqBodyBuffer), tryParseBody(resBodyBuffer), reqHeaders, resHeaders);
                } catch (e) {
                }
            });
        } catch (e) {
        }
    });
    return server;
};

This patch already includes inbound/outbound capture, request/response body spying with size caps, URL normalization, get() support, healthz filtering, and optional trace context wiring via tracer-patch.cjs.

Production hardening still recommended:

  • strong redaction for sensitive fields
  • endpoint-specific sampling and filters
  • schema/versioning for log payloads
  • env-based controls (body limit, header allowlist, verbosity)

How to implement it​

Step 1: Define a log contract​

At minimum:

  • type (inbound or outbound)
  • method
  • url
  • status
  • duration_ms
  • timestamp

Optional but useful:

  • request_headers
  • response_headers
  • request_body
  • response_body
  • trace_id, span_id

Step 2: Control log volume​

Do this from day one:

  • cap body size (for example, 10 KB)
  • avoid full binary payload logging
  • skip huge file and stream bodies

Step 3: Redact sensitive data​

At least mask:

  • authorization
  • cookie
  • token
  • access_token
  • refresh_token
  • password
  • secret
  • api_key

Step 4: Inject as a runtime patch​

A common pattern:

NODE_OPTIONS=--require /app/logger-patch.cjs

This keeps your app logic unchanged.

Step 5: Ship logs to a central stack​

Flow:

app stdout -> log collector -> Loki/ELK/Datadog -> dashboards + alerts

Mini-guide: apply the patch in Docker or Kubernetes​

Option A: Docker container​

  1. Put logger-patch.cjs next to your app Dockerfile.
  2. Copy it into the image.
  3. Set NODE_OPTIONS so Node loads the patch at startup.

Example Dockerfile:

FROM node:20-alpine
WORKDIR /app

COPY package*.json ./
RUN npm ci --omit=dev

COPY . .
COPY logger-patch.cjs /app/logger-patch.cjs

ENV NODE_OPTIONS="--require /app/logger-patch.cjs"
CMD ["node", "server.js"]

Quick test:

docker build -t my-app:logger .
docker run --rm -p 8080:8080 my-app:logger

Then check container logs and verify you see structured inbound / outbound entries.

Option B: Kubernetes workload​

  1. Store the patch in a ConfigMap.
  2. Mount it into the container.
  3. Set NODE_OPTIONS=--require /app/logger-patch.cjs.
  4. Roll out and verify logs via kubectl logs.

Quick apply flow:

# 1) Create/replace ConfigMap from local file
kubectl -n <namespace> create configmap logger-patch \
  --from-file=logger-patch.cjs=./logger-patch.cjs \
  -o yaml --dry-run=client | kubectl apply -f -

# 2) Apply Deployment manifest changes (volume + mount + NODE_OPTIONS)
kubectl -n <namespace> apply -f deployment.yaml

# 3) Restart rollout if needed
kubectl -n <namespace> rollout restart deployment/<app-name>

# 4) Verify runtime logs
kubectl -n <namespace> logs deployment/<app-name> --tail=200 -f

If your app has multiple containers in one pod, make sure you patch the correct Node.js container.

Kubernetes example​

Put the patch in a ConfigMap, mount it into the container, then require it via NODE_OPTIONS.

apiVersion: v1
kind: ConfigMap
metadata:
  name: logger-patch
data:
  logger-patch.cjs: |
    // paste your logger-patch.cjs here
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: app
        image: your-image:latest
        env:
        - name: NODE_OPTIONS
          value: "--require /app/logger-patch.cjs"
        volumeMounts:
        - name: logger-patch
          mountPath: /app/logger-patch.cjs
          subPath: logger-patch.cjs
          readOnly: true
      volumes:
      - name: logger-patch
        configMap:
          name: logger-patch

Risks to keep in mind​

1) Sensitive data leakage​

Risk: logs may contain emails, tokens, cookies, personal data.
Mitigation: strict redaction, allowlist approach, short retention, RBAC access.

2) Log cost explosion​

Risk: more traffic means more storage and processing cost.
Mitigation: sampling, body caps, endpoint filters, different levels for dev vs prod.

3) Performance overhead​

Risk: patching + serialization adds CPU/memory overhead.
Mitigation: keep processing lightweight, cap payloads, benchmark under load.

4) Too much noise​

Risk: a lot of logs, little insight.
Mitigation: consistent schema, clean fields, focused dashboards for latency/error rate/inbound vs outbound.

Where this approach is most useful​

  • Debugging CAPI / Conversions API pipelines
  • Validating outbound payload quality before ad platforms receive data
  • Investigating webhook failures
  • Enforcing data contract checks
  • Incident response when conversions suddenly drop or destination 4xx/5xx spikes

Final takeaway​

This is not a full observability platform, and it does not need to be. It is a small runtime layer that helps you see what came in, see what went out, and reduce guesswork when tracing failures across inbound and outbound requests. With proper redaction and log-volume controls, it becomes a practical, low-friction debugging layer for self-hosted tracking.