I love data – and the importance of monitoring

I find failure causes interesting.  I enjoy fixing things, but more so I like learning why something is broke.  A large part of fixing any problem is finding the root cause.  One of the things that can aid in finding a root cause is data.  The more data you have on something, the easier it is for you to spot an anomaly and track down the root cause.

Let’s take a real world example.  We implement a monitoring tool call Nagios XI where I work and it does all sort of nifty monitoring and alerting for us.  I recently got a notification that disk space was running low for a particular mounted disk that is used to store various files.  This alert was unusual as the files on this disk are only kept for a specific time period (45 days in this case) and then they are purged.  So seeing this alert raised an eye brow.  First thing I did?  Check our monitoring system.  Let’s see what recent disk usage activity looks like for the last few months:

Look at that data! Isn’t it wonderful?  No?  Sure it is!  Let me explain… you can see things humming along nicely.  Files are getting deleted regularly keeping space usage in check… until mid-February.  Well that’s weird… what changed?  Remember what I said about the 45 day retention period?  Think what happened about 45 days before mid-February… That’s right, the New Year!  Our year changed from 2016 to 2017.  At this point, things start clicking.  Hop in to the server, and my suspicions are confirmed.  The retention job points to a year specific main folder.  It was pointed to the 2016 folder.  Simply updating the job to point to the 2017 folder and re-running the job fixed the issue.  Problem solved! At least for a year ūüôā

And thus the importance of monitoring your stuff.  Had we not been monitoring this server, we wouldn’t have known this cron job broke and that the disk was approaching full. What happens when a disk maxes out?  Well, new data can’t be written, and would have been lost. Proactive monitoring once again saves the day, allowing us to to fix a problem before it actually becomes a real problem!  Pretty neat, you know, if you’re in to that kind of stuff.

Fix SickBeard “Pushbullet Notification Failed” Error

pushbeardGetting the “Pushbullet Notification Failed” error from your SickBeard notifications? ¬†Well, the fix is simple, and here is what you have to do:

1) Find pushbullet.py in your SickBeard install directory.  For me, this file was located in:


2) Open that file in your editor of choice, for me that is vi, and make the following change:


if method == ‘POST’:
uri = ‘/api/pushes’
uri = ‘/api/devices’

And change to:

if method == ‘POST’:
uri = ‘/v2/pushes’
uri = ‘/api/devices’

Note that the devices uri doesn’t change, you just need to update the pushes uri. ¬†I could dig in to the in’s and out’s of the Pushbullet API if I really felt like figuring out why this changed, but I don’t really care that much. ¬†I just wanted to get my notifications working again.

3) Write and Quit the changes and then restart SickBeard.  This will recompile your changes in the python code and then hop in to your notifications and send a test message to make sure that worked.

That fixed it for me! Happy Pushbulleting ūüôā

UPDATE 9/14/2014: Well, Something broke again. ¬†I couldn’t figure out what (I think it has to do with the JSON body but I didn’t feel like rewriting something that’s been fixed elsewhere), so I snagged the latest pushbullet.py from the troubled SickRage project and used that instead. ¬†Note, I had to switch the devices uri back to the /api/devices value because the /v2/devices value doesn’t work for me for some reason. ¬†Hopefully this doesn’t break again :-\

Fix Synology Sickbeard Shortcut When Using https

sickbeardI have been working on getting SickBeard setup on my Synology DS1512+ NAS, and I’ve got pretty much everything worked out. ¬†One of the final things I wanted to get working properly was https support with my self signed certificate I setup for my Synology. ¬†I know, not really very important since I’ll only ever access it over my lan or via a VPN, but still… I went through the trouble of getting the self signed certificate working on my Synology, I wanted it to work here too. ¬†It was a little tricky in a couple of regards.

First, I had to get it to use my certificate and key. I tried linking straight to the existing ones the Synology uses in¬†/usr/syno/etc/ssl sub-directories but SickBeard just refused. I figured it was a permission issue since those certs were owned by root only. I decided the easiest way was to just copy over the 2 files I needed in to SickBeard’s directory and switch their owner:

Andromeda> cd /usr/syno/etc/ssl
Andromeda> cp ssl.crt/server.crt /usr/local/sickbeard-custom/var/server.crt
Andromeda> cp ssl.key/server.key /usr/local/sickbeard-custom/var/server.key
Andromeda> cd /usr/local/sickbeard-custom/var
Andromeda> chown sickbeard-custom server.crt
Andromeda> chown sickbeard-custom server.key

Then alas I was able to put in server.crt and server.key in SickBeard, restart it and it used my certs!  Woot!

I was pretty happy with myself until I clicked the SickBeard shortcut in the Synology menu and was greeted with my second issue:

The client sent a plain HTTP request, but this server only speaks HTTPS on this port.

Oh you son of a…

I’m way too anal about my things all working properly to live with that atrocity, so after a minute of poking around I quickly found the config for it the following file:¬†/usr/local/sickbeard-custom/app/config

Pop that file open in vi and change the protocol line from http to https.  Save and quit, then simply reload your Synology web interface, and bam! Your shortcut will work once again, launching Sickbeard via https! Yay!


Forcing sendmail to use a different port for outgoing mail

In the world of Linux (specifically CentOS 5 in this case) sendmail is a very useful utility. ¬†It is what allows linux to send email to regular ole mail servers. ¬†But, if you’re reading this, you likely know that.

Typically sendmail uses standard SMTP port 25 to do its business. ¬†In most cases, this works fine as is, but more and more now I am seeing ISP’s block port 25 and leave the customer with no recourse (coughAT&Tcough). ¬†This makes it particularly difficult for sendmail to do its thing since it is supposed to work on port 25.

The simplest solution is to reconfigure sendmail to send out on a port other than 25. ¬†The only catch-22 here is that the mail server must accept mail on that port as well. ¬†Now, I’ve already set that up using iptables with a simple REDIRECT function and tested to make sure my CentOS box can communicate to it on that port using telnet, so I’m not going to get in to that, as anyone can forward a port).

Read More