Multicloud monitoring, an example with AWS and Azure
June 12, 2019
Muutech in the Hannover Messe 2019
June 24, 2019
Monitor your HTTPS certificates with Zabbix
Nowadays it is very simple to make our service or web portal have an updated HTTPS certificate, which will have a beneficial impact on our SEO, in the confidence towards our clients and in the security of communications in general. Solutions like Let's Encrypt, powered by Internet giants like Cisco, Akamai or RedHat, allow us to have free certificates, but they expire every three months. In this case, automating their renewal is quite simple using the "certbot" developed by them. But of course, in the world of systems, everything can fail, and it is not uncommon for this certbot to fail for some reason. A few months ago, for example, they stopped supporting a certain type of authentication and this caused the certbot not to be able to renew the certificate.
What options do we have then? A good way, if we use certbot, is to redirect the output of the "certbot renew" command if we put it in cron to our email or to a log that we read with the Zabbix agent, or, being creative: to put this command as an item type "text" of Zabbix and make it run every 24 hours, returning in a variable to which we can put a trigger that looks for "error" or "fail" in the text returned. If you want to do it this way, take into account the Timeout parameters and the permissions of the Zabbix agent. The latter you can try with:
What options do we have then? A good way, if we use certbot, is to redirect the output of the "certbot renew" command if we put it in cron to our email or to a log that we read with the Zabbix agent, or, being creative: to put this command as an item type "text" of Zabbix and make it run every 24 hours, returning in a variable to which we can put a trigger that looks for "error" or "fail" in the text returned. If you want to do it this way, take into account the Timeout parameters and the permissions of the Zabbix agent. The latter you can try with:
sudo -u zabbix /usr/bin/certbot renew
This is a very good fit, if we have our certificates with Let's Encrypt and also have access to the system. In many cases this is not so, either because we have purchased the certificate from a certifier such as Thawte, Symantec, GeoTrust or Rapid SSL, or because we have it in an external hosting or with limited permissions (and we want to ensure that we have no problems due to a hosting error in the management of this certificate).
Again, Zabbix can give us a hand. If we do a quick search, we see that there are multiple solutions for this, although none integrated into Zabbix itself. Here we will use the simplest one and the one that is thought to monitor servers in which you don't have access. In this case:https://share.zabbix.com/cat-app/web-servers/ssl-certificates-check
To do this, the idea is simple: include a script in "external scripts" of Zabbix, a type of item that allows us to use a script in any language to fill values of a metric.
We enter our Zabbix and configure:
Again, Zabbix can give us a hand. If we do a quick search, we see that there are multiple solutions for this, although none integrated into Zabbix itself. Here we will use the simplest one and the one that is thought to monitor servers in which you don't have access. In this case:https://share.zabbix.com/cat-app/web-servers/ssl-certificates-check
To do this, the idea is simple: include a script in "external scripts" of Zabbix, a type of item that allows us to use a script in any language to fill values of a metric.
We enter our Zabbix and configure:
cd /usr/lib/zabbix/externalscripts wget https://www.zabbix.org/mw/images/7/71/Zext_ssl_cert.sh mv Zext_ssl_cert.sh zext_ssl_cert.sh
We can take a quick look at the script, which is always recommended as we never know where we might be downloading a backdoor or malicious code. As you can see, it's quite simple, it essentially allows you to extract the remaining time to expire, as well as the issuer or certifier (if it suddenly changes, it could be that we are being attacked or our DNS has been compromised, so it's a good check):
#! /bin/sh #------------------------------------------------------------ # zext_ssl_cert.sh # Script checks for number of days until certificate expires or the issuing authority # depending on switch passed on command line. # #Based on script from aperto.fr (http://aperto.fr/cms/en/blog/15-blog-en/15-ssl-certificate-expiration-monitoring-with-zabbix.html) #with additions by racooper@tamu.edu #------------------------------------------------------------ DEBUG=0 if [ $DEBUG -gt 0 ] then exec 2>>/tmp/my.log set -x fi f=$1 host=$2 port=$3 sni=$4 proto=$5 if [ -z "$sni" ] then servername=$host else servername=$sni fi if [ -n "$proto" ] then starttls="-starttls $proto" fi case $f in -d) end_date=`openssl s_client -servername $servername -host $host -port $port -showcerts $starttls -prexit 2>/dev/null | sed -n '/BEGIN CERTIFICATE/,/END CERT/p' | openssl x509 -text 2>/dev/null | sed -n 's/ *Not After : *//p'` if [ -n "$end_date" ] then end_date_seconds=`date '+%s' --date "$end_date"` now_seconds=`date '+%s'` echo "($end_date_seconds-$now_seconds)/24/3600" | bc fi ;; -i) issue_dn=`openssl s_client -servername $servername -host $host -port $port -showcerts $starttls -prexit 2>/dev/null | sed -n '/BEGIN CERTIFICATE/,/END CERT/p' | openssl x509 -text 2>/dev/null | sed -n 's/ *Issuer: *//p'` if [ -n "$issue_dn" ] then issuer=`echo $issue_dn | sed -n 's/.*CN=*//p'` echo $issuer fi ;; *) echo "usage: $0 [-i|-d] hostname port sni" echo " -i Show Issuer" echo " -d Show valid days remaining" ;; esac
To test it, we run it for our website (you probably have to install the tool "bc", "yum install bc" in CentOS, for example):
[root@zocker0 externalscripts]# ./zext_ssl_cert.sh -d www.muutech.com 443 69 [root@zocker0 externalscripts]# ./zext_ssl_cert.sh -i www.muutech.com 443 Let's Encrypt Authority X3
Which, it's true, if we see it in the browser:
Therefore, we only need to configure it in the Zabbix server and for it we import the following template, in Templates->Import:
https://www.zabbix.org/mw/images/7/76/Zbx_export_templates.xml
Once imported, and to avoid the problems it gives many people, you have to configure as MACRO of the template {$SNI} to empty string. This way, only in those cases in which the domain or domains to check are different from the host name, we will have to fill it.
Then we add it to our host, in our case our own web:
https://www.zabbix.org/mw/images/7/76/Zbx_export_templates.xml
Once imported, and to avoid the problems it gives many people, you have to configure as MACRO of the template {$SNI} to empty string. This way, only in those cases in which the domain or domains to check are different from the host name, we will have to fill it.
Then we add it to our host, in our case our own web:
For those of you who want to integrate this into a Docker with Alpine image, this should include openssl, bc and coreutils for this script to work.
As a note, we must configure at least one monitoring interface (even if not used) with the hostname (in our case www.muutech.com).
As a note, we must configure at least one monitoring interface (even if not used) with the hostname (in our case www.muutech.com).
Accessing Monitoring->Latest Data, we can see the result, once the first check is made (we can use the "Check now" button of the latest versions to force it):
That's great! Now the system, which death counter will warn us when there are less than 90 days left, 60, 30, 15 and 7, as well as when it expires. Our certificate is from Let's encrypt, so as we just set the template an alarm has been triggered:
No problem, we disable for this host the 90, 60 and 30 day triggers. The most important thing is that the alarms, are that, alarms and as you see, it is important to work the templates reviewing the scripts and adjusting them to our casuistry, even if we download them from the Internet. You can have a look at the result in our demo: http://demo.muutech.com . Happy monitoring!
20 Comments
I was getting error – “zext_ssl_cert.sh[-d,{HOST.CONN},{$SSL_PORT},{$SNI}]” became not supported: Value of type “string” is not suitable for value type “Numeric (float)”. Value “sed: couldn’t write 64 items to stdout: Broken pipe” and the patch mentioned in the following link fixed it.
Fix – https://www.zabbix.com/forum/zabbix-help/45210-ssl-certificate-check-is-not-suitable-for-value-type?p=269726#post269726
Thanks for your feedback! Do you a have a https website with this error so we can test it and correct it? Kind regards!
Hi, Im not able to import the Template: https://www.zabbix.org/mw/images/7/76/Zbx_export_templates.xml
I get an error message:
Invalid tag “/zabbix_export/templates/template(1)/items/item(1)”: the tag “logtimefmt” is missing.
Coul you help me pls?
Thanks
Hello Luc, to which version of zabbix are you trying to import the template?
Hello, it is zabbix_server (Zabbix) 5.0.9.
Nice day 😉
Great! Please try this one: https://github.com/muutech/zabbix-templates/blob/master/SSL/Template%20SSL%20Cert%20Check%20External.xml
Kind regards,
Muutech Team
Thanks! Now I am able to import the template 🙂
But – it looks it works – https service is down arrived as alert to me. But when I test item {$SNI} SSL certificate validity and I write domain in {$SNI} (in my exaplme http://www.leviom.cz) I get an error:
Value of type “string” is not suitable for value type “Numeric (float)”. Value “”
Do you know what is this?
Thank you!
Luc
And I use this sh script:
#! /bin/sh
#————————————————————
# zext_ssl_cert.sh
# Script checks for number of days until certificate expires or the issuing authority
# depending on switch passed on command line.
#
# Based on script from aperto.fr (http://aperto.fr/cms/en/blog/15-blog-en/15-ssl-certificate-expiration-monitoring-with-zabbix.html)
# with additions by racooper@tamu.edu
#————————————————————
DEBUG=0
if [ $DEBUG -gt 0 ]
then
exec 2>>/tmp/sslcheckscript.log
set -x
fi
f=$1
host=$2
port=$3
proto=$4
if [ -n “$proto” ]
then
starttls=”-starttls $proto”
fi
if [ -z “$port” ]
then
port=”443″
else
port=$3
fi
case $f in
-d)
end_date=`openssl s_client -servername $host -host $host -port $port -showcerts $starttls -prexit /dev/null |
sed -n ‘/BEGIN CERTIFICATE/,/END CERT/p’ |
openssl x509 -text 2>/dev/null |
sed -n ‘s/ *Not After : *//p’`
if [ -n “$end_date” ]
then
end_date_seconds=`date ‘+%s’ –date “$end_date”`
now_seconds=`date ‘+%s’`
printf “%.0f\n” $(echo “scale=2; ( $end_date_seconds – $now_seconds )/(60*60*24)” | bc)
fi
;;
-i)
issue_dn=`openssl s_client -servername $host -host $host -port $port -showcerts $starttls -prexit /dev/null |
sed -n ‘/BEGIN CERTIFICATE/,/END CERT/p’ |
openssl x509 -text 2>/dev/null |
sed -n ‘s/ *Issuer: *//p’`
if [ -n “$issue_dn” ]
then
issuer=`echo $issue_dn | sed -n ‘s/.*CN = *//p’`
echo ${issuer}
else
echo “Certificate Issuer not found…”
fi
;;
*)
echo “usage: $0 [-i|-d] hostname (port) (proto)”
echo ” -i Show Issuer”
echo ” -d Show valid days remaining”
;;
esac
I think we need to update this article a little 🙂
Try this script with little variations over the one posted: https://github.com/muutech/zabbix-templates/blob/master/SSL/zext_ssl_cert.sh
Make sure you have the template macros properly configured for your host.
Let us know if it helps.
Kind regards,
It is necessary to correct line 53. The single quote is in the wrong position: you must remove it after “/dev/null” and put it after “CERT/p”.
Thanks! Corrected in github.
Still the same error: Value of type “string” is not suitable for value type “Numeric (float)”. Value “”
Im able to test sh script from shell: ./zexz_ssl_cert.sh -d http://www.leviom.cz (it works ok)
Try leaving $SNI macro with value = “” (it is not needed for just checking ssl web certificate) and setting your hostname withe the address of the web (“www.leviom.cz”)
The same 🙁
Here is print screen
https://ibb.co/0KLYZ9j
You need to put $SNI=”” (white, empty string) and HOST.CONN should not be 127.0.0.1; instead put “www.leviom.cz” (also configure it in the host definition): check this capture https://www.dropbox.com/t/2rEwDJTWMvFHvN9Q
Yessss 🙂 Now it works! Sorry, it was my fault…. so, now I get a value – remainig days.
But still Im waiting for Latest data – I see only https service is not runnig.
And – for every domain I will create separate host, its correct?
Yes, one host for each web. This way you can add additional checks like ICMP ping, etc.
If you change the “Interface” agent from 127.0.0.1 to http://www.leviom.cz, it should dissapear the HTTPS alert.
Everything works fine now!
Thank you so much for your patience and help at all !!!
Linus Torvalds bless you!! 🙂
Luc
if a run manualy ./zext_ssl_cert.sh -d rdp.domainname.tld 443
the response is 316
but in zabbix it is on -212 days ???
what i’ve done wrong?
Hi! Could you please share the configuration of the host and its macros?