// SYSTEM_LOGS

Most monitoring tools (Datadog, New Relic, Prometheus node_exporter) require agents installed on every target system. For a small infrastructure, that’s overhead — more software to maintain, more attack surface, more resource consumption.

Starting Agentless

The initial approach: a central observer queries target nodes using native protocols. SSH for system metrics, SQL for database health, Unix sockets for container status. No software installed on targets.

This worked for basic monitoring. But SSH-based polling is slow — each check opens a connection, runs a command, parses output. At 30-second intervals across multiple nodes, the observer spends most of its time waiting on SSH handshakes.

The Hybrid Evolution

The solution: lightweight HTTP microservices on nodes that benefit from faster polling. A minimal Flask app (< 50 lines) exposes /health and /stats endpoints. The observer hits these endpoints instead of opening SSH sessions.

What stayed agentless: - Database health checks (native SQL protocol) - Container status (Docker socket) - Security feeds (existing HTTP APIs)

What got agents: - Edge nodes where SSH latency was noticeable - Nodes needing sub-second response times

The Lesson

“Agentless” is a design preference, not a religion. The right answer is using native protocols where they’re fast enough and adding lightweight agents where they’re not. The key constraint: agents must be stateless, single-purpose, and trivially replaceable.

The media engine was hitting the database on every request. For a content-heavy platform serving images and audio, that’s a problem.

Three Layers of Caching

Layer 1: Browser cache. Immutable assets (processed images, transcoded audio) get Cache-Control: public, max-age=31536000 — one year. Once a browser downloads an asset, it never asks for it again. This alone eliminated the majority of repeat requests.

Layer 2: Application cache. Redis sits between the API and the database. Metadata queries (file location, dimensions, format) are cached with TTLs tuned by content type: 24 hours for images, 1 hour for audio metadata, 7 days for HLS manifests. Cache invalidation happens on upload/update, not on a timer.

Layer 3: Object storage streaming. Media bytes stream directly from S3-compatible storage to the client. The API handles authentication and routing but never writes media to disk. This keeps the API server stateless and memory-efficient.

The Result

Database queries dropped by approximately 95% on frequently accessed content. The database now handles writes (uploads, metadata updates) and cold-start reads. Everything else is served from cache or streamed from storage.

The Gotcha

Cache invalidation is the hard part. We use explicit invalidation on writes — when a file is uploaded or updated, its cache keys are deleted. No stale data, but it means every write path needs to know what to invalidate. This is manageable at our scale but would need rethinking for a multi-writer system.

“Zero trust” usually comes with enterprise sales calls and six-figure contracts. It doesn’t have to.

The Core Idea

No service trusts any other service based on network location. Every connection is authenticated by identity, not IP address. This means:

• No firewall rules like “allow traffic from 10.0.0.0/8”
• No VPN concentrators that become single points of failure
• No port forwarding through NAT

The Stack

Mesh VPN handles encrypted node-to-node connectivity. Every node gets a stable identity. Traffic between nodes is encrypted with WireGuard regardless of the underlying network.

Identity provider handles SSO/OIDC for web applications. Instead of each app managing its own authentication, a central identity provider issues tokens. The reverse proxy validates these tokens before traffic reaches the application.

Reverse proxy terminates TLS and enforces identity checks via forward auth. Applications behind it never see unauthenticated traffic.

What It Costs

The mesh VPN has a generous free tier. The identity provider is open-source. The reverse proxy is open-source. Total cost: the compute to run them — which is negligible when they’re containers on existing infrastructure.

What It Doesn’t Solve

This protects the network layer. Application-level security (input validation, CSRF, content security policies) is a separate concern. Zero trust is a network architecture, not a security silver bullet.

GitHub Actions is convenient. It’s also a dependency you don’t control.

What Triggered the Switch

Three things happened in the same month: 1. A GitHub outage blocked deployments for 4 hours 2. A billing surprise from exceeding free-tier minutes 3. A workflow that worked locally but failed in Actions due to Ubuntu version differences

None of these were catastrophic. But all of them were avoidable.

What I Built Instead

A self-hosted Gitea instance with act_runner executing workflows on bare metal. The entire pipeline — source control, CI/CD, container registry, deployment — runs on infrastructure I own.

Key design decisions: - Host-mode runner — direct Docker access, no nested virtualization overhead - Dormant by design — runners stay idle until a repo includes a workflow file - Two-stage pipeline — build on push to main, deploy only on version tags - BuildKit caching — inline cache layers cut rebuild times significantly

The Trade-Off

I traded convenience for ownership. Setup took a weekend. Maintenance is minimal — the runner is a systemd service that restarts automatically. The registry is a Docker container.

What I gained: zero external dependencies, zero billing surprises, and deployments that work even when GitHub is down.

The goal was simple: production servers should hold no source code, no build artifacts, no secrets baked into images. They pull containers from a private registry and run them. That’s it.

The Problem

Traditional deployment copies code to a server, installs dependencies, and runs the application. This creates state — the server knows what it’s running. If the server dies, you need to rebuild that state. If you deploy a bad version, rollback means remembering what was there before.

The Approach

Containers as the only deployment artifact. The CI pipeline builds an image, pushes it to a private registry. The production server runs:

bash docker compose pull docker compose up -d --force-recreate --remove-orphans docker image prune -f

Three commands. The server pulls the latest image, recreates containers (zero-downtime via compose’s rolling strategy), and cleans up old images. No source code touches the production filesystem.

What This Gets You

• Rollback is trivial — point docker-compose.yml at a previous image tag
• Server replacement is fast — new server just needs Docker and compose
• No dependency drift — everything is baked into the image at build time
• Secrets stay in the pipeline — injected at build/deploy time, never stored on disk

The server is disposable. The image is the truth.

Currently leveraging the RedCup infrastructure to host 'Project Samosa', the high-interaction honeypot experiment for my Master's thesis in Cybersecurity and Network Defense. The research explores the convergence of psychological heuristics, UI design, and behavioral cybersecurity. By utilizing a "Dutch Auction" pricing algorithm, the application induces cognitive stress to test if panicking human users exhibit the same biometric telemetry—erratic keystrokes and fast mouse coordinates (X, Y)—as malicious bots. A major component of the engineering involves hardcoding "Compliance by Design." To legally collect this high-fidelity biometric data without violating the Swiss Federal Act on Data Protection (FADP), the system utilizes cryptographic one-way hashing to anonymize all sessions at the code level.

To prevent heavy tasks (like processing large photo uploads) from freezing the web interface, the system decouples web logic from compute muscle. When a user uploads a file, the Flask frontend immediately pipes the raw data into a MinIO object storage vault and drops a standardized JSON job payload into a Redis message bus. On a completely isolated virtual machine (LXC 1005), a stateless C++ compute engine polls Redis at near-zero latency. It pulls the job, downloads the file directly into a Linux RAM-disk (/dev/shm/), and processes the image using purely shared memory to prevent SSD wear. Once optimized, it fires a webhook back to the database to update the UI instantly.

With the application factory built, the network layer required enterprise-grade segmentation. The infrastructure utilizes a 5-Ring Zero Trust architecture. Public traffic hits a Cloudflare-protected DigitalOcean VPS, which acts merely as a "dumb pipe," blindly forwarding encrypted TCP packets into a Tailscale mesh network. This allows traffic to safely bypass local firewalls and exit directly into the core hypervisor. At the ingress point, a Caddy reverse proxy terminates the SSL and acts as the ultimate gatekeeper. Before any application logic is executed, Caddy forces a sub-request to Authentik (OIDC). If a user lacks the correct session identity, the connection is dropped at the edge, ensuring total blast-radius containment.

To build the RedCup architecture, I abandoned manual server configuration in favor of a strict GitOps methodology hosted on Proxmox bare-metal hypervisors. The development environment is fully ephemeral: utilizing Coder, I provision temporary, isolated Docker workspaces via Terraform. When code is pushed to the self-hosted Gitea repository, it triggers an automated CI/CD pipeline. The Gitea Runner dynamically spawns a build container, compiles the application into an immutable Docker image, and pushes it to a private registry. Deployment to the production App Node (LXC 1002) is handled with zero-downtime container swapping, ensuring the application logic remains entirely stateless and disposable.

The first iteration of the new infrastructure began with a headless Ubuntu Raspberry Pi aiming to be a secure audio streamer, but immediate DNS loops caused by a Pi-hole container forced a pivot to Tailscale MagicDNS and Split DNS. The system quickly evolved into an observability stack utilizing Grafana, Loki, and InfluxDB. However, raw log volumes from Zeek and Suricata caused 429 errors and exposed the Pi's 64GB USB drive as a critical bottleneck. The architecture was shifted to a "Distributed Monolith" by repurposing an HP laptop into a bare-metal MinIO Data Lake. Loki was stabilized to use MinIO as its backend, preserving the Pi's CPU while permanently solving the storage crisis.

My foundation in enterprise infrastructure was solidified managing large-scale VPN deployments, VLAN security enhancements, and device lifecycles via Microsoft Intune and Azure Active Directory. However, the architectural blueprint for my current cloud platform was inspired outside of traditional IT. While working in the restaurant industry, I observed the operational flow of their delivery system: kitchen prep, packaging, and delivery teams all coordinated asynchronously through a centralized terminal. It was a physical manifestation of an event-driven microservice architecture. Recognizing how optimizing this flow allowed a single restaurant to scale like a tech platform, I began designing the digital equivalent: a highly scalable, decoupled infrastructure that would eventually become RedCup Cloud.

The journey started with a strong core in the sciences, getting certified in 8 subjects including Math and Physics during high school. Transitioning into the Bachelor of Information Technology at GITI provided the platform to turn theory into practice. During this period, the focus was heavily on network solutions testing, specifically deploying VMware implementations, backup systems, and VLAN configurations. Beyond just learning the stack, hosting practical networking labs for peers solidified a deep understanding of bridging textbook concepts with real-world enterprise cloud solutions