Inspired by this post I decided to add a notification on my phone every time an ssh session began on my servers. Seeing as I make use of Gotify for selfhosted push notifications I used that rather than signal.

First I created created the file /usr/local/bin/sshnotif. At the top you can add your own token and Gotify url

Update: I had to push the current time back a full minute in order to improve consistency. I'll defiantly want to revisit this at a later date

#!/bin/bash

exec &> /dev/null #Hide output

Gotify_URL='https://example.tld/gotify'
Gotify_Token='gotify-app-token'

notify()
{

        now=$(date -d "-60 seconds" +%s) #Get current time minus 60 seconds
        end=$((SECONDS+30)) #Set 30s Timeout for loop

        while [ $SECONDS -lt $end ]; do

                SSHdate=$(date -d "$(who |grep pts|tail -1 | awk '{print $3, $4}')" +%s) #Check for the latest SSH session

                if [ $SSHdate -ge $now ]; then #Once who is updated continue with sending Notification

                        title="SSH Login for $(/bin/hostname -f)"
                        message="$(/usr/bin/who | grep pts)"

                        /usr/bin/curl -X POST -s \
                                -F "title=${title}" \
                                -F "message=${message}" \
                                -F "priority=5" \
                                "${Gotify_URL}/message?token=${Gotify_Token}"

                        break
                fi
        done

}

notify & #Run in background to prevent holding up the login process

Run the command chmod +x /usr/local/bin/sshnotif

In the file /etc/pam.d/sshd add the following line

# note optional is set to prevent ssh login failure
session optional pam_exec.so /usr/local/bin/sshnotif

I now get a nice notification with all the open SSH sessions listed. Unlike the post on 8192.one I didn't want any IP address resolution using an online service. I plan on integrating the MaxMind GeoLite2 database at some point. However as I already have Graylog set up to do this it's not a high priority for me.

Thanks for the shoutout: https://zerosec.xyz/posts/gotify-notifications/

Running old embedded devices is a pain not to mention a major security risk. But if you are like me and are stuck with it sometimes you can take solace in software repo projects like Entware. In this case I needed to centralize all the disparate system logs on the network so I could find issues BEFORE they cause real trouble. The problem is the QNAP NAS I had could only send system logs over unencrypted UDP.

That's just not good enough, especially as I want to use client certs down the line. The simplest solution I found was to install syslog-ng to redirect the logs securely.

Note: I'm using a letsencrypt cert to make my life simpler

Setting up the NAS

Install Entware by downloading the .qpkg file, navigating to the NAS in the web browser and then selecting the install manually option in the app center.

SSH into the NAS and install syslog-ng

opkg update
opkg install syslog-ng

Configure syslog-ng by editing /opt/etc/syslog-ng.conf

# Important set the right config file version
@version: 3.20

options {
};

# Listen to local syslog connection
source localhostudp {
        udp( ip("127.0.0.1") port(1514) );
};

# Forward to remote graylog server over tls to port 1514
# To Implement Client Cert
destination graylog_loghost {
        network(
                "example.com" port(1514)
                transport("tls")
                tls( ca_dir("/opt/sbin/cadir") )
        );
};

# Enable both source and destination
log {
        source(localhostudp);
        destination(graylog_loghost);
};

Set up the Letsencrypt CA by downloading the TrustID X3 Root Certificate (formallyu known as DST Root CA X3). We then need to discover the hash of the certificate using openssl. Syslog-ng requires as simlink named with the certificate hash.

The hash should be 2e5ac55d

/opt/sbin/cadir
wget https://github.com/letsencrypt/website/raw/master/static/certs/trustid-x3-root.pem

openssl x509 -noout -hash -in trustid-x3-root.pem

ln -s /opt/sbin/cadir/trustid-x3-root.pem /opt/sbin/cadir/2e5ac55d.0

Via the web admin, set the NAS to log to 127.0.0.1 with the local port 1514. This can be found in Systems Logs in the Systems Settings category.

Ensure syslog-ng isn't running then test in the foreground for any errors

/opt/etc/init.d/S01syslog-ng stop

/opt/sbin/syslog-ng -Fvde

If no errors appear you can then start syslog-ng

/opt/etc/init.d/S01syslog-ng start

Graylog Notes

Graylog doesn't appear to directly accept the format sent via syslog-ng. While it is possible to change the format in syslog-ng I didn't figure out the best way to do it. My solution was to set the input to Raw/Plaintext TCP and then run a GROK pattern extractor when matching the conn log string

%{DATA} qlogd\[9147\]: %{DATA:facility}: Users: %{DATA:NAS_user}, Source IP: %{IP:NAS_src}, Computer name: %{DATA:NAS_id}, Connection type: %{DATA:NAS_connection}, Accessed resources: %{DATA:NAS_resource}, Action: %{GREEDYDATA:NAS_action}

I have just published the latest release for the Thunderbird plugin Lookout-fix-version.

This is a bug fix issue

Fixes are for the ics parsing

Added quick decompose_rfc822_address sanity check

Occasionally a phrase without any email addresses is used in the attendee fields and this can cause the complex regex in the decompose_rfc822_address function to freeze. As I couldn't come up with a sensible alternative (primarily due to lack of windat samples) I added a simple sanity check prior to the regex.

Added special character skip for ICS attendee data

Occasionally a special non printing character will be decoded, x20 to x7E, causing the ics file to become unreadable. Lookout now ignores these invalid fields.

Added zero( 0 ) UID check

Sometimes the UID will be decoded as 0 rather than null, if this happens lookout will now attempt one of the other UID methods as intended.

You can download this update from the Github Release page or the Thunderbird addon page

In august 2018 I took over maintenance of the Thunderbird addon [Lookout-fix-version][e7c45433]. I soon set up a the Github Organization TB-throwback so that future development can be expanded and transferred easier if I stop work on it.

It's been an interesting experience managing a small project that's over 11 years old. Especially with all the changes and rapid development Thunderbird has been going experiencing now it's separated from Mozilla.

Why did I take over?

I needed to move my office away from Outlook 2010. I had no budget to upgrade the office software, but I couldn't allow the company to keep limping along with a 9 year old product.

Thunderbird to the rescue! Except...

TNEF files, supposedly a thing of the past. Even Microsoft recommends you NOT to send such files. But we have to work with people who don't upgrade and pay the lowest bidder to configure their exchange servers.

Unfortunately the original Lookout was at this point unmaintained and severly out of date similarly Lookout+ and Lookout-fix-version hadn't seen any updates in a long time. luckily Oleksandr was still contactable via the support email and was happy to add me as a developer on the ATN page.

My first change was a simple modification to the preferences css to fix changes in Thunderbird 59. I've since been working on adding debugging, improving performance, squashing bugs and generally attempting to learn how everything is strung together.

I plan on porting the addon to a webextension in the coming months to ensure we have this useful addon for many years to come.

I often see questions on /r/selfhosted on how to secure a server. Here is a quick checklist of things you might want to look into.

Follow best practices for the basics

• Verify your install images using gpg and sha256
• Use strong passwords, defiantly think about using a password manager like KeePass
• Don't use the same password for root and the regular user.
• Don't run scripts/applications using sudo/su where possible.
• Learn linux permissions to avoid using sudo/su where possible.
• Use SSH Public Keys
• Regularly Update

Lock down the Server

• Disable root login via SSH
• Close all unused incoming ports via UFW/iptables
• Limit outgoing ports as well as incoming using UFW/iptables
• Watch for credential stuffing/brute force attacks with Fail2ban

Backup your configs/files

• Securely encrypted backup via Duplicity
• External Backup to external drive.
• Remote backup, either via a regularly swapped out external drive or via the cload

Set up monitoring services to let you know when something goes wrong

• Send email on syslog message
• Set up notifications for when updates are available
• Monitor system critical files with Tripwire
• Set up ssh login notifications with gotify
• Set up Graylog or a similar system log aggregator in order to do log analysis

Here are a few extra things you can do to bolster your ssh security

• Use a non standard port, this stops like 80% of the dumb bots
• Enable geo-blocking on your ssh port, easy to do in pfsense using the package PFBlockerNG or with iptables
• Close you ssh port based on the time, will you ever really need to ssh in at 3am? Possibly
• Add 2FA using google-authenticator-libpam this should stop 99% of brute force attacks
• Use something like fwknop for encrypted port knocking - best to avoid unencrypted port knocking using knockd

Useful resources

• Linux man pages
• Digital Ocean tutorials
• nixCraft tutorials
• crontab wizard
• Ubuntu wiki