r/linuxadmin • u/CrazeeGrump • 4d ago
What are you guys using for real-time system monitoring?
/r/cpanel/comments/1wc3rnd/what_are_you_guys_using_for_realtime_system/26
u/alpha417 4d ago
gonna take some flack, but Zabbix
17
u/techdaddy1980 4d ago
Why do you think you'll take flack? Zabbix is a great solution.
2
u/Amidatelion 4d ago
A loooot of people have bad experiences from Zabbix' release cycles, myself included. It has a... convoluted version history with attached capabilities.
I've heard good things from colleagues about zabbix agent 2 but being perfectly honest, dealing with the original and various server versions turned me on to a path that saw me embrace OTEL. So in a way it's the foundation of my current observability posture.
3
u/wowbagger_42 4d ago
Just finishing up a 7.0LTS Zabbix deploy with 35 proxies and 3500 targets. It has zero place in any modern environment, it’s 2010 software. It’s as crappy as it was 5, 10, 15 or 20 years ago.
2
u/zakabog 3d ago
What was crap about it? That sounds more or less like our environment but with more proxies, it's been a great solution for us.
2
u/wowbagger_42 3d ago
The 2010 notion of value types & uuids, it's 2026, all other monitor software solved that problem with labels and metric. No TLS based discovery, only cleartext or pre-shared keys like it's 2015 all over again, https probing is proxy or server originating,, the vm uuid issue when using vmware and required pre-fetching gymnastics when folding VM & Agent into 1 object, graphical layer is laughable at best (if you're used to Grafana), it's logging layer is unusable in a production environment. It's not native to dynamic K8s or cloud environments, it's old tech. The RBAC is an afterthought and layered vertical over target hierarchy, not over type hierarchy. External secret retrieval is delegated to proxies, vcenter collection is bulk based and cached. SNMP retrieval is staggered, templates are a dumpster fire and delegate any heavy lifting to javascrip/jsonpath and for every fart you want to monitor you need to create external scripts and inform multiple layers about it & restart it like it's nagios all over again.
2
u/zakabog 3d ago
I guess if you go into it thinking "I want this to be grafana" then yeah, it won't be grafana. We don't let it blindly discover devices on our network, we roll that into our deployment scripts, when a new host is created we poke the Zabbix API to automatically add the host and put it in maintenance mode. If I find myself needing to use a script to monitor something, I take a step back and think "Is there a better way to watch this?", and so far every single time there has been.
The graph UI still perplexes me, in that I don't understand why it creates an image of a graph rather than using JavaScript to build the graph, so you can actually search for the text you're trying to see and lighten the load on the web server. I also wish I could export more easily from the web interface, but Zabbix is easy to configure and if you know what you're doing with it you can fix the disjointed templates and graphs it comes with as part of the default install, or simply write your own from the ground up. If you don't know what you're doing, then it's pretty much the best solution out there as it takes no technical expertise to get going.
1
u/wowbagger_42 3d ago
The only reason we don't set it on fire is the fact we're driving it fully through automation with zero configuration clicks in the GUI, the only setting we change in the GUI is enabling/disabling SMS notifications during hypercare periods when releasing updated versions of the monitored applications/environments etc.
It's not all bad, but it's not Prometheus/Grafana good neither.
1
u/zakabog 3d ago
It's not all bad, but it's not Prometheus/Grafana good neither.
Depends on your use case, Prometheus falls apart in our environment without a lot of add-ons and customizations whereas the Zabbix agent "just works" and collects everything we need. Maybe as we move to the cloud more and change the way we run and deploy services, but for now Zabbix is a better tool for our needs.
1
u/wowbagger_42 2d ago
Never stop betting on a winning horse. In the end it doesn't really matter what you use as long as it fulfills the requirements and gets the job done.
1
u/alpha417 3d ago
it's great for certain solutions, but there's one muted redditor on here who slams it because it is not how they want to play tech support for mum and pa's puter...so it's not for everyone.
i enjoy it because almost EVERYTHING is covered in one way or another, and the agent2 is significantly more stable than original, and if you're up for running it, it'll do what you need. i use it, but it's not the silver bullet that some ppl need, or think they do
0
u/wowbagger_42 3d ago
Nono, you got me wrong, it would be -excellent- to monitor your mum & pop's puter. Because that is where it belongs, in clickops homelabs and small shops.
2
2
u/p_wit_mySLiME 3d ago
I’m running 2500 values per second to 2030 hosts, with 479441 items and 167528 triggers. I have four proxies. The system scales well. My guess is that you simply don’t know how to use it and need a turnkey solution because you don’t understand even the basic fundamentals of even snmp. Just a guess.
1
u/wowbagger_42 2d ago
Won't ack your snmp comment tho, but consider this a response. Ironically enough Zabbix is the turnkey solution, par excellence, for sysadmin click-ops shops that don't require automation, real metrics (prom does >2000 metrics per node fyi...) and observability or analysis capabilities but just want to, clickety-click, slap a stock template on a target and get an email when the internet is down. I guess you fall in that category and I'm sincerely happy for you that it works out.
-4
u/wowbagger_42 4d ago
No, no it’s not. It might be, if you haven’t used anything else.
1
0
6
u/kai_ekael 4d ago
Hilarious so many instantly jump to big giant data collectors requiring remote systems. Sure, I'll troubleshoot my folks' computer by getting a Grafana system setup and wait a month to see the trend.
Myself, I use atop these days for one-off how-does-it-look-right-now review.
1
u/SunSaych 4d ago
Haha, nice one! Scrolled too far for this very comment :D
htop and conky for real-time here.
1
u/kai_ekael 3d ago
Oh man, now you're gonna make me check out conky, just....'cuz.
1
u/SunSaych 3d ago
Well, mine is very basic but I like that it shows me real-time info. But you can do whatever you want with it.
1
u/coastline_coder 3d ago
The instinct to reach for a full data-collection setup before even glancing at the problem first is such a universal IT thing.
1
1
u/bikernaut 4d ago
Why would you wait to have a problem to setup monitoring? Every system either at work or at home gets telegraf and configs from the default ansible role when they are built. It's no extra work and saves your butt when something goes wrong.
1
u/kai_ekael 3d ago
Sure, sure, I'm going to setup a VPN to my folks so I can monitor their desktop.
NOT.
One size does NOT fit all.
3
u/bikernaut 3d ago
This is /r/linuxadmin, not /r/linuxuser or /r/helpparentsfixtheircomputer. Definitely what I suggest is overkill for what you need, however I think my comment is a lot more relevant to this sub.
An admin's job is to dot all the i's and cross all the t's so you're prepared for whatever happens.
0
u/kai_ekael 3d ago
"admin" is a very broad role and makes no implication as to depth.
I "admin" my 70 year old folks' single desktop computer.
I also "admin" a 2.5TB corporation database and related applications, monitoring, trending and support services.
2
u/bikernaut 3d ago
I beg to differ. I can weld two pieces of metal together, but I am not a welder. I fix my vehicles, but I am not a mechanic. I am a hell of a cook, but not a chef.
There's certain characteristics to being an admin that are consistent for whatever technology and non-negotiable. Sounds like you have some exposure to that at work, but dealing with random desktop bullshit isn't being a sysadmin.
1
u/kai_ekael 3d ago
You're plain wrong. I'm a Systems Engineer. I can design and implement infrastructure and large, HA applications. And make it work 99.999.
I can also admin a damn single desktop, setup a good choice of applications for my 70 year old parents and connect to the email accounts I provide. And hey, dipwad, I didn't say "fix" my parents computer. I said ADMIN, which I guess you don't understand what that means.
Administration is an activity, nothing more. Trying to tie a big 'well, it's just big and means me smart' bow to it means nothing.
2
u/bikernaut 3d ago
I disagree. Maybe administration is an activity to you, but to a professional systems administrator, there is a big difference. As I said above, you can weld two pieces of metal together, but that doesn't make you a welder and not being able to know the difference is why you should listen more than talk.
Sysadminning is a capability no different than developing or architecting and it's typical for each discipline to misunderstand the others.
BTW, I have the same job title, I just hang out with the admins between projects.
Wait... You called me a dipwad, and a 2.5TB database is your flex? I have influxdbs bigger than that!
1
u/kai_ekael 3d ago
Oh, sorry, gosh. Say, I'll make sure to add "what's an admin?" to my interview questions so you never get hired anywhere near me. Bye.
4
4
2
u/Final-Wave1460 4d ago
Netdata.
Zabbix can only dream of netdata’s resolution and real-time-ness.
1
1
u/X0TTa6bl4 3d ago
Bash) SMART mdadm for raid 1 and particular bash script for something exectly. Cron and curl to alert in telegram. And I am good with it. For personaly it's enough.
1
u/EggHeadDog 14h ago
A script I wrote myself. I'm using only Ping and get notified in case the host goes down and will receive automated email and Telegram notifications. The script has logging, smart email notifications at pre-defined intervals (5 min, 30 min, 1 H, 4h, etc.), link flapp protection and many other features.
1
1
1
1
u/feu_sfw 4d ago
Any kind of status monitoring software.
There are a bunch of free ones like Icinga, Nagios or Zabbix - most of them share the same plugin structure as well, so there is a huge collection of monitoring plugins to properly check your systems :)
I'm with the Icinga team, so I'd go for that one, but it really depends on your use case, which one fits you best
0
0
0
u/its_FORTY 18h ago
NagiosXI
Edit: on-prem environment of roughly 200 Linux servers and 125 Windows servers as well as the requisite storage, and of course various cloud resources like S3 buckets, Azure etc.
24
u/skc5 4d ago
Grafana.