Multicloud monitoring, an example with AWS and Azure
June 12, 2019
Muutech in the Hannover Messe 2019
June 24, 2019
Monitor your HTTPS certificates with Zabbix
Nowadays it is very simple to make our service or web portal have an updated HTTPS certificate, which will have a beneficial impact on our SEO, in the confidence towards our clients and in the security of communications in general. Solutions like Let's Encrypt, powered by Internet giants like Cisco, Akamai or RedHat, allow us to have free certificates, but they expire every three months. In this case, automating their renewal is quite simple using the "certbot" developed by them. But of course, in the world of systems, everything can fail, and it is not uncommon for this certbot to fail for some reason. A few months ago, for example, they stopped supporting a certain type of authentication and this caused the certbot not to be able to renew the certificate.
What options do we have then? A good way, if we use certbot, is to redirect the output of the "certbot renew" command if we put it in cron to our email or to a log that we read with the Zabbix agent, or, being creative: to put this command as an item type "text" of Zabbix and make it run every 24 hours, returning in a variable to which we can put a trigger that looks for "error" or "fail" in the text returned. If you want to do it this way, take into account the Timeout parameters and the permissions of the Zabbix agent. The latter you can try with:
What options do we have then? A good way, if we use certbot, is to redirect the output of the "certbot renew" command if we put it in cron to our email or to a log that we read with the Zabbix agent, or, being creative: to put this command as an item type "text" of Zabbix and make it run every 24 hours, returning in a variable to which we can put a trigger that looks for "error" or "fail" in the text returned. If you want to do it this way, take into account the Timeout parameters and the permissions of the Zabbix agent. The latter you can try with:
sudo -u zabbix /usr/bin/certbot renew
This is a very good fit, if we have our certificates with Let's Encrypt and also have access to the system. In many cases this is not so, either because we have purchased the certificate from a certifier such as Thawte, Symantec, GeoTrust or Rapid SSL, or because we have it in an external hosting or with limited permissions (and we want to ensure that we have no problems due to a hosting error in the management of this certificate).
Again, Zabbix can give us a hand. If we do a quick search, we see that there are multiple solutions for this, although none integrated into Zabbix itself. Here we will use the simplest one and the one that is thought to monitor servers in which you don't have access. In this case:https://share.zabbix.com/cat-app/web-servers/ssl-certificates-check
To do this, the idea is simple: include a script in "external scripts" of Zabbix, a type of item that allows us to use a script in any language to fill values of a metric.
We enter our Zabbix and configure:
Again, Zabbix can give us a hand. If we do a quick search, we see that there are multiple solutions for this, although none integrated into Zabbix itself. Here we will use the simplest one and the one that is thought to monitor servers in which you don't have access. In this case:https://share.zabbix.com/cat-app/web-servers/ssl-certificates-check
To do this, the idea is simple: include a script in "external scripts" of Zabbix, a type of item that allows us to use a script in any language to fill values of a metric.
We enter our Zabbix and configure:
cd /usr/lib/zabbix/externalscripts wget https://www.zabbix.org/mw/images/7/71/Zext_ssl_cert.sh mv Zext_ssl_cert.sh zext_ssl_cert.sh
We can take a quick look at the script, which is always recommended as we never know where we might be downloading a backdoor or malicious code. As you can see, it's quite simple, it essentially allows you to extract the remaining time to expire, as well as the issuer or certifier (if it suddenly changes, it could be that we are being attacked or our DNS has been compromised, so it's a good check):
#! /bin/sh #------------------------------------------------------------ # zext_ssl_cert.sh # Script checks for number of days until certificate expires or the issuing authority # depending on switch passed on command line. # #Based on script from aperto.fr (http://aperto.fr/cms/en/blog/15-blog-en/15-ssl-certificate-expiration-monitoring-with-zabbix.html) #with additions by racooper@tamu.edu #------------------------------------------------------------ DEBUG=0 if [ $DEBUG -gt 0 ] then exec 2>>/tmp/my.log set -x fi f=$1 host=$2 port=$3 sni=$4 proto=$5 if [ -z "$sni" ] then servername=$host else servername=$sni fi if [ -n "$proto" ] then starttls="-starttls $proto" fi case $f in -d) end_date=`openssl s_client -servername $servername -host $host -port $port -showcerts $starttls -prexit 2>/dev/null | sed -n '/BEGIN CERTIFICATE/,/END CERT/p' | openssl x509 -text 2>/dev/null | sed -n 's/ *Not After : *//p'` if [ -n "$end_date" ] then end_date_seconds=`date '+%s' --date "$end_date"` now_seconds=`date '+%s'` echo "($end_date_seconds-$now_seconds)/24/3600" | bc fi ;; -i) issue_dn=`openssl s_client -servername $servername -host $host -port $port -showcerts $starttls -prexit 2>/dev/null | sed -n '/BEGIN CERTIFICATE/,/END CERT/p' | openssl x509 -text 2>/dev/null | sed -n 's/ *Issuer: *//p'` if [ -n "$issue_dn" ] then issuer=`echo $issue_dn | sed -n 's/.*CN=*//p'` echo $issuer fi ;; *) echo "usage: $0 [-i|-d] hostname port sni" echo " -i Show Issuer" echo " -d Show valid days remaining" ;; esac
To test it, we run it for our website (you probably have to install the tool "bc", "yum install bc" in CentOS, for example):
[root@zocker0 externalscripts]# ./zext_ssl_cert.sh -d www.muutech.com 443 69 [root@zocker0 externalscripts]# ./zext_ssl_cert.sh -i www.muutech.com 443 Let's Encrypt Authority X3
Which, it's true, if we see it in the browser:
Therefore, we only need to configure it in the Zabbix server and for it we import the following template, in Templates->Import:
https://www.zabbix.org/mw/images/7/76/Zbx_export_templates.xml
Once imported, and to avoid the problems it gives many people, you have to configure as MACRO of the template {$SNI} to empty string. This way, only in those cases in which the domain or domains to check are different from the host name, we will have to fill it.
Then we add it to our host, in our case our own web:
https://www.zabbix.org/mw/images/7/76/Zbx_export_templates.xml
Once imported, and to avoid the problems it gives many people, you have to configure as MACRO of the template {$SNI} to empty string. This way, only in those cases in which the domain or domains to check are different from the host name, we will have to fill it.
Then we add it to our host, in our case our own web:
For those of you who want to integrate this into a Docker with Alpine image, this should include openssl, bc and coreutils for this script to work.
As a note, we must configure at least one monitoring interface (even if not used) with the hostname (in our case www.muutech.com).
As a note, we must configure at least one monitoring interface (even if not used) with the hostname (in our case www.muutech.com).
Accessing Monitoring->Latest Data, we can see the result, once the first check is made (we can use the "Check now" button of the latest versions to force it):
That's great! Now the system, which death counter will warn us when there are less than 90 days left, 60, 30, 15 and 7, as well as when it expires. Our certificate is from Let's encrypt, so as we just set the template an alarm has been triggered:
No problem, we disable for this host the 90, 60 and 30 day triggers. The most important thing is that the alarms, are that, alarms and as you see, it is important to work the templates reviewing the scripts and adjusting them to our casuistry, even if we download them from the Internet. You can have a look at the result in our demo: http://demo.muutech.com . Happy monitoring!