Monitoring vs observability...And getting my brain around it

As I said in an earlier post, I sought to replace Zabbix as my monitoring solution (which I have accomplished), because it was entirely too heavy for a 20-30 host homelab environment, plus having, um, less-than-stellar support for BSD or ZFS (like reporting "pool down" when a single drive errors, which is not a good look to management).

So I looked around for other options, and found Monit and M/Monit, which looked like a winner...Until the company told me that a license for my 20 node homelab is $250/year to get updates (the license is at least currently perpetual, but you only get updates to the software for the first year). So in my mind, I was like "yeah, that's not happening." I pointed out that I had introduced other open source tools to work where we have several hundred nodes, but their response was radio silence. I will find out what happens to the demo license in 10 days.

So it was back on the hunt. I am basically looking for a monitoring tool for my homelab that doesn't look like something from 1998 that you have to configure each host by hand for every monitored service (lookin' at you, Nagios...), and is aware of unique features of FreeBSD and ZFS (and maybe at some point supporting of unique features of the other BSDs), instead of a monitor that is full of linux-isms.

I'm learning a lot on the fly...Like the fact that there is a difference between "monitoring" and "observability," with monitoring being a status of hosts, gear, and services, up/down, and other functional features of the network, whereas observability gives you a timescale second-by-second of how a host or piece of gear is performing, and graphs over time. From a monitoring sensibility, it is like drinking from a firehose, and at this point in research, it seems that alerting is a bolted-on afterthought.

So next up on my list of tools to try is grafana + influxdb + telegraf. One of the problems with this solution is that it seems to be a rapidly moving target that the documentation seems to have a hard time keeping up. In addition, the tool set seems like a monitoring erector set. Or, at the risk of showing my age, one of those Radio Shack project kits where you get a piece of breadboard and pile of components and you are supposed to build a radar detector or a crystal radio or some such...With instructions written in Taiwan in broken English.

What I have figured out so far is that you have one or more applications (called data sources) that run on the end piece of gear (host, switch, application, etc), e.g. node exporter or telegraf. These scrape data from the host and push it to, what I refer to (for lack of a better term) the middleware. This can be something like influxdb or prometheus. It appears that prometheus pulls scraped data from the endpoint (e.g. node_exporter), whereas influxdb has data pushed from the endpoint (e.g. telegraf). In addition, the grafana website has a metric crapton of plugins and application-specific end apps, like loki for logging, jaeger, etc.

The last layer of this observability game of pick-up sticks is grafana itself, which is the display engine. You can plug in one or more "data sources" that you configured above, that are getting feeds from the scrapers. On top of that are hundreds or thousands of dashboards that will display your data for you. Many of them are specific (like there are several for vmware, many, many for linux, cisco switches, etc.) It appears most of the dashboards that are BSD related are specifically for BSD appliances, and the generic server dashboards are all full of linux-isms. I haven't had time to learn how to create dashboards of my own, let alone trying to add the bits and bobs to make all of the functionality work to get the information I wish to see. It feels very much like, as my boss is fond of saying, trying to assemble the plane while you are doing your takeoff roll.

To complicate things even further, grafana has three tiers: Open Source, Cloud, and Enterprise, and I'm presuming that some things work in in only a subset of their offerings, specifically only in the paid offerings.

So a couple of years ago I had set up grafana with prometheus and node-exporter, and it worked reasonably well, until one of the data points it scraped for ZFS stats was taken away in a ZFS update. A that point, node-exporter started spamming my logs with errors, enough so that the logs were rolling over every couple of hours. However, I have read that the search for that data point was fixed in node_exporter, so I may try setting up prometheus + node exporter on the jail on which influxdb + telegraf lives, and feeding them both to grafana.

I will keep plugging along, and try to become more familiar with it. At the moment, I'm getting enough basic data from the dashboard I found to at least keep an eye on the systems. Even though a lot of the graphs say "no data," because they are looking for linux-specific data points, like kernel specific items from linux's /proc.

Hopefully, when this is all said and done, I'll be able to work it like a guy I knew in the army that could rebuild an engine, end up with a 3 gallon coffee can of leftover parts, and the engine ran better than before.