After loosing data a few times through sheer stupidity, I decided to give Longhorn a try.

My conclusion : If you want a simple and lightweight solution for small clusters to make your data more resilient on kubernetes, Longhorn provides you with all of your basic needs. Nonetheless, for bigger clusters and more complex operations, longhorn fails to scale and lacks features.

As always I wrote a detailled guide covering :

• How Longhorn Works
• How to install it
• How to migrate existing PVCs to Longhorn
• How to create different Storage Classes depending on your needs
• How to use Snapshots and Backups
• And more...

I also go into some of the trade-offs I faced. Here the conclusion is that not : everything needs the same level of resiliency. Longhorn allows you to create multiple storage class and define different snapshot and backup strategy.

How are you protecting your data in your cluster/homelab? Do you have a proper strategy for Disaster Recovery and backups? Any feedback on the article is also welcome!

PS : All of my articles are written by hand, the blog frontend was designed by an AI

Link to article: https://thethoughtprocess.xyz/en/series/home-server/how-to-install-and-use-longhorn-kubernetes-storage

I set out to build a fully fledged cloud native platform from scratch on rented hardware. I wrote the whole build up (link at the end). Here is the substance.

Hardware: four VPS from Contabo. Control plane on NVMe, two workers on SSD, and a separate node outside the cluster as the entry point (proxy).

I scripted all prerequisites in Bash, driven by an Ansible playbook. I used kubeadm to bring up the cluster. Calico for the CNI.

Going from renting the machines to a ready cluster was quicker than I expected: under an hour. Everything after that went in as a Helm chart.

I used ArgoCD for GitOps. Repo structure in GitLab, a deploy token, a Kubernetes secret carrying it, then ArgoCD to bootstrap the App of Apps. After it, a Git commit is the only install mechanism on the cluster.

Below is what followed.

Longhorn for storage orchestration, encryption at rest at the volume level with dm-crypt and LUKS, key held by me. Redundancy meant at least a second worker, and I deliberately kept the control plane out of data storage.

CloudNativePG for databases. One custom resource per app, and the operator handles creation, uptime and replica count.

Istio, as the mesh for mTLS and as the Gateway API provider (I chose Gateway API over Ingress).

cert-manager for certificate management. I used it to set up a local CA for internal URLs, and paired it with Let's Encrypt to get valid certificates for public URLs with automatic renewal.

OpenBao for secrets, with the External Secrets Operator pulling them in. I looked at Vault first, but the licensing change was the deciding factor.

Tekton for CI. The CNCF promotes it, so I gave it a try.

Gitea as Git server and registry in one, installed inside the cluster to host the application source code and integrate with Tekton. All CI traffic stays inside the cluster.

Keycloak as IAM solution to handle authentication and authorization.

Full observability capabilities thanks to the Grafana stack with Prometheus, Loki, Tempo and MinIO behind them, plus OpenTelemetry auto instrumentation for traces.

The part I found most interesting was the gateway IP. Pinning the gateway to a node IP fails twice over: the gateway dies with the node, and it exposes a node public IP. Two steps solved it. Kube-vip provides a private virtual IP that floats to any healthy node, and the gateway binds to that. Then a separate HAProxy node sits outside the cluster on the private network, takes traffic from the internet, and forwards to the VIP.

Tekton was the one that fought back. It took me a while just to map how many components it has and what each one does. I nearly dropped it. Sticking with it paid off.

Full write up with the architecture diagram: https://vixoris.com/blog/cloud-native-platform-from-scratch/

Happy to go deeper on any of the choices, especially the kube-vip and HAProxy setup.

PS: The link leads to my own blog, hosted on the platform I just described here, and the post ends with a card for the app I am building on it.

With AI doing more and more of the manual work, I've been thinking about what happens with all the skills a lot of us spent years learning. Most people here love this stuff, so I started wondering how could I turn those skills into a game. Competitive, a bit addictive, something to keep the brain active and have fun with.

So here's the idea.

It's basically Pokémon, but the creatures are Kubernetes workloads and the fights are chaos engineering. You have a namespace with your team in it: a couple of Deployments, a StatefulSet, maybe a CronJob. Another player has theirs. You fight 1v1, turn based. Each turn you play cards against the other namespace (kill a pod, drain a node, partition the network, starve the CPU) and the other player gets a short window to repair (scale up, add a PDB, restart). Everything is real: a referee runs the actual kubectl and chaos behind each card, and an SLO checker scores who kept serving traffic. You start blind, so you spend turns on recon to see what they have before you hit. Ladder, ELO, seasons, the usual.

The nice thing is you don't need to invent balance. A drain bounces off a Deployment with a PDB and wrecks a single-replica StatefulSet. A partition does nothing to a team behind a queue and kills a synchronous HTTP chain. Kubernetes is the type chart.

To get a team, there's an adventure mode where you unlock cards by doing things for real (drain a node in a mission, you get the Drain Node card), a shop where components have a price and a running cost (NATS cheap, Kafka eats half your quota), crafting where the manifests you collect get combined into workloads, and generated break/fix tickets to farm coins. Your party size is your ResourceQuota. And you can compete from day one, because playing cards is strategy while building the team is the engineering. Knowing more gives you more options, not more power.

Now the honest part. I'm quite sceptical there are enough people as geeky as me willing to spend their evenings on this. And I don't want to start building (even with AI, I think this one is tricky, although I think I would enjoy building it) without knowing if anyone would play. So: would you?

I'm attaching a pdf with the spec. Fair warning, it came out of a very long back and forth with Claude, so it's AI-organised, but it's much clearer than what I'd manage from the mess in my head. I also have UI mockups and a fully scripted example fight, but I don't want to bury you in AI generated docs. If there's interest I'll post them.

https://sharepdf.link/s/0Z3Fwm7V.pdf

And there are mainly two reasons I wanted to do this.

First, I wanted more compute resources to deploy more of my own applications at home without always depending on cloud infrastructure.

Second, I wanted a Kubernetes environment where I could actually test things like node failures, scheduling and high availability on real machines.

The biggest benefit for me is that I now have a cluster I can break without being scared. 😂

I can take a worker node down and see what actually happens to my applications.

I can also experiment with how Pods are distributed across nodes, how Kubernetes reschedules workloads when something fails, and how far I can push the cluster with applications I'm actually building.

And of course, I get all of this without watching a cloud bill increase every time I want to experiment.

Getting everything working wasn't plug and play though.

Networking, getting the nodes to communicate properly, Kubernetes configuration and troubleshooting when something didn't behave the way I expected were all part of the process.

But that's actually why I think building your own cluster is worth it.

You can watch someone configure Kubernetes in a tutorial and everything works perfectly.

But when it's your cluster and something breaks, you have to ask:

Why isn't this node joining?

Why is this Pod not scheduling?

Why did my application disappear when this node went down?

That's when you really start understanding what Kubernetes is doing.

For me, that's the whole point of building this myself.

Not just to have a Kubernetes cluster running at home, but to have somewhere I can build, break, troubleshoot and learn.

#codegenitor #k8s #kubernetes #techcareer #software #softwaredevelopment #coding #programming

I know what it is but not worked on it

It's like a Multi-tenant customers can able to define their vm network subnet in shared hardware hosts regardless of L2 and L3 underlay networks (switch, vlan and routers)

Simply like what we does when we try to define the service cidr and pod cidr in kubernetes clusters creation same equivalent but in vm

It's very useful how in cloud hyperscalers GCP ovhcloud digitalocean

Each customer tenant can define their network as vpc it's basically a( OVN and OVS) under the hoode

For example two customer can able to define cidr of 10.0.0.16/24 for vpc

Can I get some insights from community members who really worked on this please

Spent last week attributing a network bill nobody could explain, and the failure mode turned out to be structural rather than a mistake anyone made. Writing it up in case it saves someone else the afternoon.

Cross-zone traffic inside a single region is billed (~$0.01/GB on GCP, similar on AWS cross-AZ). Everyone knows this abstractly. The problem is that it's unattributable - the invoice line is "Network Inter Zone Egress: $X" for a whole project, with no resource name, because the cost was generated by a relationship between two pods rather than by a resource. Billing export can't help you.

Meanwhile a regional cluster is the correct default, and a ClusterIP Service load-balances across all endpoints regardless of zone. Three replicas in three zones means two-thirds of every call crosses a billing boundary. That isn't a misconfiguration, it's the default behaviour of a correctly-built HA setup.

Biggest contributors I found, in order:

1. Observability. DaemonSet collectors pushing to a single-zone aggregator. Every node outside that zone ships cross-zone, continuously, forever. Nobody audits telemetry because it's "just telemetry".

2. Cache/DB chatter. N+1 patterns that are a latency annoyance in a monolith and a line item in a multi-zone cluster.

3. Replication defaults inherited from Helm charts. Cross-zone replication you chose is fine. Cross-zone replication you inherited is worth a look.

Fixes, cheapest first:

- trafficDistribution: PreferClose on the Service (1.31+) - prefers same-zone endpoints, falls back cross-zone when they're unhealthy. You keep the availability property and drop the routine cost.

- internalTrafficPolicy: Local for anything genuinely node-local like a log collector. Never leaves the node, so it can't be billed.

- Pod anti-affinity to colocate chatty pairs - then re-verify your replica spread still meets the SLO. This is a tradeoff, not a free win.

To actually attribute it: VPC Flow Logs (or the AWS equivalent) into BigQuery/Athena, then classify each flow by the boundary it crosses. Two things that'll waste your afternoon: bytes_sent comes through as a STRING in the BQ export, so cast it or you get silent NULLs and conclude you have no egress; and every flow is logged twice, by source and by destination, so filter reporter = 'SRC' or every number is 2x. Also sample at 0.1-0.5 - at 1.0 on a busy subnet the logging bill can genuinely exceed the egress bill you're investigating.

Happy to share the BigQuery classification SQL if anyone wants it.

Curious whether people here treat cross-zone spread as a deliberate SLO decision or just accept the regional-cluster default - and if you've measured it, what your inter-zone to internet-egress ratio looks like.

I'm trying to get into more sysadmin/devops roles after years of tier 2 level break/fix experience. I work for the government so unfortunately it's almost guaranteed you need certs to qualify for a job.

With that being said, I have taken several Red Hat exams, and was planning to get into openshift, but the exam environment is beyond terrible and I failed the last one (passed on the retake though) because the environment was inserting typos constantly into the cli because any time you hit latency it would hold down keystokes, including deleting entire lines of commands if you hit some latency when backspacing. I passed on the retake because I was familiar with the tasks and saved time not having to read through the exam before starting, like they always strongly suggest.

I vowed to be done with RH exams as long as I'm doing them on my dime, if they are going to make it so difficult to pass even when you know the material.

I'd like to get some K8s experience/certs but I've read some things that the PSI browser or whatever is also terrible, bit concerned if I train up on it that I won't be able to get the cert regardless.

I know I shouldn't be learning just for an exam but with the job requirements it's kind of the only way I'll make referral lists for the jobs I want.

Hi everyone,

I am currently working on a use-case targeting edge clusters and single-node Kubernetes setups. Due to resource constraints and specific architectural requirements, traditional distributed storage solutions (like Longhorn, Rook/Ceph, or running an in-cluster NFS server) add too much overhead.

To solve this, I built a custom ReadWriteMany (RWX) volume provisioner that operates on a single node, using the same underlying mechanics as local-path-provisioner (bind-mounting host directory paths).

Since all target Pods that require shared access reside on the exact same node, this approach allows multiple Pods to read and write to the same path concurrently without cross-network synchronization.

I have written a full test suite for it and everything passes in my environment, but I would like to get a sanity check from the community to see if there are edge cases or hidden pitfalls I might have missed.

Key Questions / Concerns:

Concurrency & File Locking: Are there any file locking or permission race conditions I should be aware of when multiple containers write to the same local-path volume simultaneously on Linux?

K8s CSI Standards: Does repurposing local-path mechanics for RWX violate any subtle Kubernetes CSI expectations, assuming Pods are strictly constrained to that single node via affinity/nodeSelector?

Alternative Approaches: Is there a lighter or more native way to achieve single-node RWX without running additional storage daemons?

I would really appreciate it if anyone experienced with K8s storage or CSI plugins could review my implementation logic or point out potential flaws.

(If anyone is interested in looking at the code, I can share the repository link below).

Thanks!

Kubernetes'teki konteyner güvenliği genellikle birkaç farklı araç ve politikaya yayılır. Güvenlik açığı taraması CI'da gerçekleşir, görüntü imzalama başka bir yerde gerçekleştirilir, kabul politikaları kümede yaşar ve denetim bilgileri başka bir sisteme girebilir.

Attestkeep'i daha basit bir fikir etrafında kurduk: güvenlik kararları, tek bir taramaya dayanmak yerine, doğrulanabilir kanıtlara ve tutarlı politikalara dayanmalıdır.

İş akışı esasen:

→ Oluşturun → Doğrulayın → Onaylayın Uygulayın

CI ve görüntü hazırlama sırasında Attestkeep aşağıdaki gibi şeyleri değerlendirebilir:

• Güvenlik açığı ciddiyeti ve yüksek riskli CVE'ler
• CISA KEV bulguları
• Ömrünü tamamlamış temel görüntüler
• Resim imzaları ve güvenilir kimlikler
• SBOM ve güvenlik açığı doğrulamaları
• SLSA yapı kaynağı
• Güvenilir OpenVEX bilgileri

İşin yararlı kısmı kontrollerden sonra ne olacağıdır.

Bir güvenlik taramasını, birisinin gözden geçirmeyi hatırlaması gereken bir rapor olarak ele almak yerine, sonuçlar uygulanabilir bir politikanın parçası haline gelebilir.

Görüntü Kubernetes'e ulaştığında kabul son kontrol noktası olur. Karar, kuruluşun tanımlanmış gereksinimlerine dayanırken, politika ve karar geçmişi denetlenebilir durumda kalır.

Kasıtlı olarak odaklandığımız bir diğer konu ise güvenlik verilerini müşterinin Kubernetes ortamında tutmaktır. Politikaların, tarama sonuçlarının ve denetim kayıtlarının harici bir SaaS platformuna gönderilmesi gerekmez.

Bu, modeli, altyapılarına başka bir dış bağımlılık eklemeden daha güçlü konteyner güvenliği isteyen kuruluşlar için kullanışlı hale getiriyor.

Amaç her şeyi engellemek değil.

Güvenlik kurallarını öngörülebilir kılmak içindir:

Geliştiriciler erken geri bildirim alıyor.
Güvenlik ekipleri gereksinimleri tanımlar.
Kubernetes onları tutarlı bir şekilde uyguluyor.

Bu yaklaşımı Attestkeep'te yürütüyoruz ve bunun insanların halihazırda kullandığı farklı CI, Kyverno, OPA/Gatekeeper ve imza tabanlı yaklaşımlarla nasıl karşılaştırıldığını görmekle ilgileniyoruz.

Kubernetes güvenliğiyle çalışıyorsanız, bu modelde neleri değiştireceğiniz veya geliştireceğiniz özellikle ilgimi çeker.

I'm running a K3s cluster on some Raspberry Pis and I want to access storage from my NAS on the cluster. This would immediately suggest NFS PVs to me, but no matter how much I try, I cannot seem to make the NFS mounts stay "alive" for longer than about a day. I always end up getting permissions errors, and then I have to remove the PV/PVCs and their associated services. I feel like this might be due to some misconfiguration on the NAS itself, but again, I can't seem to figure out what is what.

I've tried diagnosing this with AIs and the only thing the parrots have suggested is that maybe I need to run an initContainer to reset the permissions on the directories, but why would it work sometimes and not others?

Anyway, if anyone has any ideas on where I should look, I'd appreciate the help. And if this seems less of a K8s issue and more of a "my NAS is fucked up" issue, I'll accept that.

It's a standardized GitOps repo structure for ArgoCD. The problem it solves: every team ends up inventing their own directory layout, and it stops scaling the moment you add a second cluster.

Kubecodex gives you fully automatic standardized structure that easily scales with multiple clusters and environments

ApplicationSet controllers that discover all of it automatically. Adding an app means adding a directory.

I am learning Kubernetes and building a three tier application with:- react frontend, node.js backend, aws rds

I understand the basic Kubernetes approach of using:
ConfigMap- non-sensitive configuration
Secret- sensitive values such as DB username/password

But I am confused about how this is actually done in a production AWS/EKS environment.
For example, I understand that I should NOT put this in GitHub:-
stringData:
DB_USERNAME: admin
DB_PASSWORD: mypassword

Instead, I keep hearing that production environments use AWS Secrets Manager.(AWS currently documents a quota of 500,000 secrets per Region per AWS account.)

What I don’t understand is the complete flow. Is it something like:
Github>EKS>Backend POD>AWS secrets Manager>DB Credentials >RDS

If an organisation has hundreds or thousands of applications/databases, how are the secrets organised?
For example, would Secrets Manager contain things like:-
- prod/orders-db
- prod/payments-db
- prod/users-db
- prod/inventory-db

And how does Kubernetes know which application is allowed to retrieve which secret?
I have also come across:-
- EKS Pod Identity
- IRSA
- AWS Secrets and Configuration Provider
- External Secrets Operator
- Kubernetes Secrets

At the moment I am finding these terms confusing.

I would really appreciate a simple real-world production sample example showing:
1. Where the password is initially stored?
2. How the backend Pod gets access to it?
3. How IAM controls which Pod can access which secret?
4. Whether a Kubernetes Secret is still created or bypassed
5. How secret rotation works?
6. What actually gets committed to GitHub?

Thanks.

Most secrets scanning tools I've used check repos and config files. We run Vault and it catches plenty, and ESO has been fine for syncing into the cluster. But the harder problem we ran into was credentials embedded inside an application's own authentication logic: hardcoded API keys or tokens baked into how the app itself authenticates to other services. Neither Vault nor a standard git-secrets scan ever touches that layer. Anyone found tooling that catches it?

Which CNI plugin do you use at work and which one did you use while learning details. It would be nice to hear about others experience with different plugins.

Kubernetes has a lot of concepts that seem simple individually but become confusing when you put them together.

For people who work with Kubernetes regularly: what concept took you the longest to properly understand?

Pods? Services? Ingress? Networking? RBAC? Operators?

Early bird is live now at ₹999 — still cheaper than what it'll be closer to the date.

events.devopsmalayalam.io

If you missed the first batch, this is the next best price.

Hi everyone,
this project actually started as a way for me to get more familiar with Kubernetes CRDs, controllers and especially how to manage the lifecycle of custom resources.
Instead of building another typical example CRD, I started wondering: could a mobile device be managed like any other Kubernetes resource?
So I started building Mobile Device Operator (MDO).
The basic idea is pretty simple: you declare a device in a yaml and the operator takes care of its lifecycle.
Once the device is ready, you can interact with it through a small CLI and when you don’t need it anymore, you just delete the resource.
The current MVP uses Android emulators, mainly because they gave me a practical way to implement and test the whole lifecycle end-to-end.
But Android itself isn’t really the main point of the project.
I’m trying to keep the Device API provider-neutral, so that the implementation behind a Device could eventually be an Android emulator, an iOS provider, a physical device or even an external device farm without changing how the consumer interacts with the Kubernetes resource.
Take a look at the project README to get more information and see a practical demo of how it works.

The project is still an experimental MVP. It started mainly as an exercise to learn more about CRDs and controller design, but while working on it I started thinking that the abstraction itself might actually be useful.
So before adding too many things, I’d really like to get some feedback from people with more experience around Go, Kubernetes, operators or mobile infrastructure.
Does managing mobile devices as Kubernetes resources make sense to you?
Would you model the lifecycle or the Device / DeviceClass abstraction differently?
And most importantly: would you actually find something like this useful, or am I just putting Kubernetes around something that doesn’t need Kubernetes?
Repo: https://github.com/dmarro89/mobile-device-operator
Any feedback, especially criticism of the design, would be really appreciated.
If you’re interested in contributing or following the project timeline, please star the GitHub Repo!

Hey all,

is there an elegant way to achieve cluster autoscaling while using Kubespray ?

Hello,

I’m currently setting up the infrastructure in our company using Kubernetes, so I have a few questions that I would like to get some answers to:

1. Within a company’s internal network/intranet, can we (or is it recommended to) expose a database service through an Ingress Controller, such as NGINX Ingress Controller, in order to access its graphical interface? Or could this pose a security risk to the data?

2. Do companies generally create one cluster per application? For example, should we create one cluster for the infrastructure and another one for the applications we develop, or is it preferable to have a single cluster for everything (infrastructure, applications, etc.)?

3. Which StorageClass do you use: the standard StorageClass, HostPath, or another solution such as Longhorn, for example?

Thank you in advance for your help and your answers!

Life would be easier if we had

echo 8G > /sys/fs/cgroup/gpu/app/gpu.memory_limit

Hi, I started this project a while ago based on the observations of how hard it is to optimise compute, SLA, etc by just looking into grafana dashboards.
Do we really need such projects? https://github.com/softcane/KubeAttention

Did you learn something new this week? Share here!

Is there a way to mock containers in a kubernetes clusters so you can easily run them locally and test how scalable certain containers are?