Skip to main content

Posts for year 2019

SSH Login Notifications with Gotify

Inspired by this post I decided to add a notification on my phone every time an ssh session began on my servers. Seeing as I make use of Gotify for selfhosted push notifications I used that rather than signal.

First I created created the file /usr/local/bin/sshnotif. At the top you can add your own token and Gotify url

Update: I had to push the current time back a full minute in order to improve consistency. I'll defiantly want to revisit this at a later date

#!/bin/bash

exec &> /dev/null #Hide output

Gotify_URL='https://example.tld/gotify'
Gotify_Token='gotify-app-token'

notify()
{

        now=$(date -d "-60 seconds" +%s) #Get current time minus 60 seconds
        end=$((SECONDS+30)) #Set 30s Timeout for loop

        while [ $SECONDS -lt $end ]; do

                SSHdate=$(date -d "$(who |grep pts|tail -1 | awk '{print $3, $4}')" +%s) #Check for the latest SSH session

                if [ $SSHdate -ge $now ]; then #Once who is updated continue with sending Notification

                        title="SSH Login for $(/bin/hostname -f)"
                        message="$(/usr/bin/who | grep pts)"

                        /usr/bin/curl -X POST -s \
                                -F "title=${title}" \
                                -F "message=${message}" \
                                -F "priority=5" \
                                "${Gotify_URL}/message?token=${Gotify_Token}"

                        break
                fi
        done

}

notify & #Run in background to prevent holding up the login process

Run the command chmod +x /usr/local/bin/sshnotif

In the file /etc/pam.d/sshd add the following line

# note optional is set to prevent ssh login failure
session optional pam_exec.so /usr/local/bin/sshnotif

I now get a nice notification with all the open SSH sessions listed. Unlike the post on 8192.one I didn't want any IP address resolution using an online service. I plan on integrating the MaxMind GeoLite2 database at some point. However as I already have Graylog set up to do this it's not a high priority for me.


Getting a QNAP NAS to Log to my Graylog instance

Running old embedded devices is a pain not to mention a major security risk. But if you are like me and are stuck with it sometimes you can take solace in software repo projects like Entware. In this case I needed to centralize all the disparate system logs on the network so I could find issues BEFORE they cause real trouble. The problem is the QNAP NAS I had could only send system logs over unencrypted UDP.

That's just not good enough, especially as I want to use client certs down the line. The simplest solution I found was to install syslog-ng to redirect the logs securely.

Note: I'm using a letsencrypt cert to make my life simpler

Setting up the NAS

Install Entware by downloading the .qpkg file, navigating to the NAS in the web browser and then selecting the install manually option in the app center.

Manualy install the .qpkg file

SSH into the NAS and install syslog-ng

opkg update
opkg install syslog-ng

Configure syslog-ng by editing /opt/etc/syslog-ng.conf

# Important set the right config file version
@version: 3.20

options {
};

# Listen to local syslog connection
source localhostudp {
        udp( ip("127.0.0.1") port(1514) );
};

# Forward to remote graylog server over tls to port 1514
# To Implement Client Cert
destination graylog_loghost {
        network(
                "example.com" port(1514)
                transport("tls")
                tls( ca_dir("/opt/sbin/cadir") )
        );
};

# Enable both source and destination
log {
        source(localhostudp);
        destination(graylog_loghost);
};

Set up the Letsencrypt CA by downloading the TrustID X3 Root Certificate (formallyu known as DST Root CA X3). We then need to discover the hash of the certificate using openssl. Syslog-ng requires as simlink named with the certificate hash.

The hash should be 2e5ac55d

/opt/sbin/cadir
wget https://github.com/letsencrypt/website/blob/master/static/certs/trustid-x3-root.pem

openssl x509 -noout -hash -in trustid-x3-root.pem

ln -s /opt/sbin/cadir/trustid-x3-root.pem /opt/sbin/cadir/2e5ac55d.0

Via the web admin, set the NAS to log to 127.0.0.1 with the local port 1514. This can be found in Systems Logs in the Systems Settings category.

Control Panel -> System Logs -> Syslog Client Management

Ensure syslog-ng isn't running then test in the foreground for any errors

/opt/etc/init.d/S01syslog-ng stop

/opt/sbin/syslog-ng -Fvde

If no errors appear you can then start syslog-ng

/opt/etc/init.d/S01syslog-ng start

Graylog Notes

Graylog doesn't appear to directly accept the format sent via syslog-ng. While it is possible to change the format in syslog-ng I didn't figure out the best way to do it. My solution was to set the input to Raw/Plaintext TCP and then run a GROK pattern extractor when matching the conn log string

%{DATA} qlogd\[9147\]: %{DATA:facility}: Users: %{DATA:NAS_user}, Source IP: %{IP:NAS_src}, Computer name: %{DATA:NAS_id}, Connection type: %{DATA:NAS_connection}, Accessed resources: %{DATA:NAS_resource}, Action: %{GREEDYDATA:NAS_action}

Lookout-fix-version v1.2.25

Lookout! I have just published the latest release for the Thunderbird plugin Lookout-fix-version.

This is a bug fix issue

Fixes are for the ics parsing

Added quick decompose_rfc822_address sanity check

Occasionally a phrase without any email addresses is used in the attendee fields and this can cause the complex regex in the decompose_rfc822_address function to freeze. As I couldn't come up with a sensible alternative (primarily due to lack of windat samples) I added a simple sanity check prior to the regex.

Added special character skip for ICS attendee data

Occasionally a special non printing character will be decoded, x20 to x7E, causing the ics file to become unreadable. Lookout now ignores these invalid fields.

Added zero( 0 ) UID check

Sometimes the UID will be decoded as 0 rather than null, if this happens lookout will now attempt one of the other UID methods as intended.

You can download this update from the Github Release page or the Thunderbird addon page


LookOut fix version

Lookout! In august 2018 I took over maintenance of the Thunderbird addon Lookout-fix-version. I soon set up a the Github Organization TB-throwback so that future development can be expanded and transferred easier if I stop work on it.

It's been an interesting experience managing a small project that's over 11 years old. Especially with all the changes and rapid development Thunderbird has been going experiencing now it's separated from Mozilla.

Why did I take over?

I needed to move my office away from Outlook 2010. I had no budget to upgrade the office software, but I couldn't allow the company to keep limping along with a 9 year old product.

Thunderbird to the rescue! Except...

TNEF files, supposedly a thing of the past. Even Microsoft recommends you NOT to send such files. But we have to work with people who don't upgrade and pay the lowest bidder to configure their exchange servers.

Unfortunately the original Lookout was at this point unmaintained and severly out of date similarly Lookout+ and Lookout-fix-version hadn't seen any updates in a long time. luckily Oleksandr was still contactable via the support email and was happy to add me as a developer on the ATN page.

My first change was a simple modification to the preferences css to fix changes in Thunderbird 59. I've since been working on adding debugging, improving performance, squashing bugs and generally attempting to learn how everything is strung together.

Original TNEF file with extracted attachments

I plan on porting the addon to a webextension in the coming months to ensure we have this useful addon for many years to come.


Server set-up Checklist

Configure your linux server I often see questions on /r/selfhosted on how to secure a server. Here is a quick checklist of things you might want to look into.

Follow best practices for the basics

Lock down the Server

  • Disable root login via SSH
  • Close all unused incoming ports via UFW/iptables
  • Limit outgoing ports as well as incoming using UFW/iptables
  • Watch for credential stuffing/brute force attacks with Fail2ban

Backup your configs/files

  • Securely encrypted backup via Duplicity
  • External Backup to external drive.
  • Remote backup, either via a regularly swapped out external drive or via the cload

Set up monitoring services to let you know when something goes wrong

Here are a few extra things you can do to bolster your ssh security

Useful resources


Dugite is this video of you?

Ohhh a virus!

This morning I received a link in Facebook messenger from an old friend, someone I haven't spoken to in years. It was obviously a phishing link, the title was wrong "Dugite is this a video of you?" and the preview image was blurry, like it was stuck not loading.

Needless to say I didn't click and messaged them back saying "Probably should change your password?"

What Interests me is the reaction when this friend posted on their wall, within minuets, not to open any messages from them and they "HAVE A VIRUS ON MY PHONE". Out of 7 people 3 opened the obviously shady link.

Not one of these people who opened this link mentioned they will now need to change their password, in fact one person even said it all should be ok once it's gone through their entire address book. The non-technical people, to me at least, seem to be treating phishing and malware like you would the common cold. It'll pass, fact of life and not a real concern.

They of course should be very concerned with the majority of web browsing occurring on the mobile the amount of data that is potentially stored on a phone is astronomically large. From your banking app to your photos and everything you can gain access through your email accounts is at risk. Mobile operating systems are, thankfully, very locked down with each app operating somewhat isolated from the others, but that's not a guarantee.

I fear the next decade of tech breaches are going to get ugly and there just isn't a technical solution to user apathy.


Static Blogging

Now for the obligatory static blog post on any static blog This blog is a Nikola static blog, a python based static blog generator.

Why did I choose Nikola?

  1. It's Python based - I like python, I know python, and I can hack python. Jekyll, the most popular static blog generator, is written in Ruby. I found it to be slow and painful to work with. I couldn't be bothered learning it's particulars. Also my Desktop and Server came with python so there was no barrier to entry.
  2. Batteries are Included - I looked at the popular Pelican, it's arguably more flexible and powerful than Nikola, but you have to do more initial configuration. I had already been procrastinating on re-starting my blog for over a year, I needed to get started

Why do people like Static Blogs?

  1. You need to be savvy - Static Blogs scratch the tech tinkering itch, don't like something it's simple to change that.
  2. It's simple - Many CMS systems are over complicated messes, overkill for the typical blog. Do you need to render your blog on the fly for each visitor? PHP is useful but if you don't need it why use it?
  3. Portable - You render your site as it changes, have beef with your service provider? Change your dns and upload with next to no configuration. Done
  4. It's Fast - How fast can a server trow out a few kb of static html? Very fast! With Wordpress you need to install caching plugins to make your site essentially static, why not skip the middle man and do it from the start.
  5. You write how you like - Markdown, reStructuredText, MediaWiki Markdown or even HTML.
  6. It's secure - No database to hack, no PHP to patch.

Why do people hate Static Blogs?

  1. No web based content management system - This is something that tends to be lacking in any static blog. It's static after all! There are many ways to get around this by using something like Netlify CMS. Personally I'm using the Atom test editor and will be going over this in a future post.
  2. You need to be savvy - As a general rule you need to be able to deal with the nuts and bolts of your blog. This is a good and bad thing, depending on your disposition.

TOTP with SSH (Google Auth)

For your ssh you can use google-authenticator-libpam to add time based codes to your ssh login.

On debian/ubuntu:

    sudo apt update && sudo apt install google-authenticator-libpam

    google-authenticator

    Do you want authentication tokens to be time-based (y/n) y
    [...]
    Do you want me to update your "/home/dugite/.google_authenticator" file? (y/n) y
    [...]

You will see a QR code/secret key that you can scan with a TOTP app like andotp, authy or google authenticator (WARNING Google authenticator has no backup options). There are also your emergency scratch codes.

In /etc/ssh/sshd_config Add:

    # Use Challenge Response Auth i.e. TOTP
    ChallengeResponseAuthentication yes
    # Require both publickey and TOTP
    AuthenticationMethods publickey,keyboard-interactive

In /etc/pam.d/sshd

    # Comment out Standard Un*x authentication.
    # @include common-auth
    # Load the google TOTP Authentication module
    auth required pam_google_authenticator.so

Goodbye Chrome and other things

Google, once the tech enthusiast darling is looking more and more like Microsoft did in the mid 90's.

Google to restrict modern ad blocking Chrome extensions to enterprise users

Google is first and foremost an ad company so it should come as no surprise that now they are leading the browser market share

All hail the King

Now Microsoft is switching to chromium as a browser backend it's no surprise we see Google moving to limit Ad-blockers.

Google is eating our mail

Google really got entrenched with the tech enthusiast crowd because gmail was free, quick and had good spam filtering. Now we are all feeling the consequences of encouraging non-technical people to centralize their emails with them. A once open and vibrant standard is increasingly looking like a mono-culture with both Google and Microsoft's opaque filtering and non-standard blocking making running your own email server almost impossible. Along with the launch of google's AMP for email we see yet another example of Google pushing through their own interests over the interests of the email ecosystem.

What you can do


Locking your ssh port with fwknop

In my last post I described how I decrypt my home server remotely with ssh. Today I would like to share how I like to lock/unlock my ssh port with an encrypted port knocking implementation fwknop

The issue with port knocking

On the face of it port knocking looks like a good idea. Lock down your ssh port until you need it, avoiding any zero day issues with the ssh protocols. The problem is this, port knocking is sent in the clear over the network. Anyone looking can see your knock "code", much like if you had a secret door knock some one around the corner could heard the pattern of your knocks.

This is where fwknop comes in, it's SPA (Single Packet Authorization) cannot be re-sent it is one time only. Not to mention it's faster as you are only sending the one packet.

The main issue I had with fwknop is by default you have to specify the source IP address you want to be able to access your server. I found this to be quite painful to set-up, so I found a simple way around the issue.

Note: this only works if you are blocking ports by default. I use UFW to simplify that process. See this Digital Ocean tutorial on the basics of UFW

Server Side:

In Debian based systems fwknop is split into fwknop-client and fwknop-server. We will want both of them

sudo apt install fwknop-server fwknop-client

Next we need to set up the basic config rules on the server found in /etc/fwknop/fwknop.conf Debian and Ubuntu have changed the default interface name from eth0 to enp3s0 so we have to set that. We can also change the listening port here.

PCAP_INTF               enp3s0;

# change your port to your desired listening port.
PCAP_FILTER                 udp port 62201;

Now we use fwknop to generate our key's. We could use GpG here, but I didn't feel the extra encryption brings much to the table as we are only opening the ssh port and I have public key authentication and TOTP enabled.

fwknop -A tcp/22 -D example.tld --key-gen --use-hmac --save-rc-stanza

You can now find the KEY_BASE64 and HMAC_KEY_BASE64 in ~/.fwknoprc we will need these for the /etc/fwknop/access.conf file and the client.

In the /etc/fwknop/access.conf file. Note: I substituted the iptable commands for ufw commands. We don't have to worry about our ssh session being kicked as once it's connected the CMD_CYCLE_CLOSE (at least with ufw) won't close the existing connection.

SOURCE                          ANY

# Limit the Ports able to be opened
OPEN_PORTS                      tcp/22

# Keys from ~/.fwknoprc
KEY_BASE64                      [...]
HMAC_KEY_BASE64                 [...]

# Optionally use iptables
# CMD_CYCLE_OPEN                /sbin/iptables -A INPUT -p $PROTO --dport $PORT -j ACCEPT
# CMD_CYCLE_CLOSE               iptables -D INPUT -p $PROTO --dport $PORT -j ACCEPT

CMD_CYCLE_OPEN                  /usr/sbin/ufw allow $PORT
CMD_CYCLE_CLOSE                 /usr/sbin/ufw delete allow $PORT

# Default cycle time Mandatory for CMD_CYCLE_OPEN/CLOSE
CMD_CYCLE_TIMER                 180

A word of warning, fwknop can run arbitrary commands if ENABLE_CMD_EXEC is enabled. I don't see why you would ever really want to do that. You can also run any bash script from CMD_CYCLE_OPEN and CMD_CYCLE_CLOSE with the optional variables $PROTO, $PORT and $SRC. You can potentially get yourself in a lot of trouble if you do this so proceed with caution.

Now we need to setup the systemd file /etc/systemd/system/fwknop-server.service. Note: on a ubuntu install I had to create the folder /var/fwknop/

[Unit]
Description=Firewall Knock Operator Daemon
After=network-online.target

[Service]
Type=forking
PIDFile=/var/fwknop/fwknopd.pid
ExecStart=/usr/sbin/fwknopd
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Then we just enable and start the service

sudo systemctl enable fwknop-server.service && sudo systemctl start fwknop-server.service

Running sudo systemctl status fwknop-server.service should now show you the service is active Active: active (running). Currently if you have already allowed port 22 with ufw it will stay open until the first time you cycle fwknop with a client.

Client Side:

You have three options fwknop-client, fwknop2 on android - [F-Droid] - [Google play] or fwknop-gui available on Windows, Mac and Linux

In fwknop2 and fwknop-gui:

  • KEY_BASE64 -> Rijndael Key
  • Key Is Base 64 - Checkbox below key entry
  • HMAC_KEY_BASE64 -> HMAC Key
  • HMAC Is Base 64 - Checkbox below key entry
  • Allow IP - This can be anything as we are ignoring this setting
  • Access Ports: tcp/22

The Firewall timeout is in seconds and can be anything as long as it's long enough for you to authenticate. Remember if you have the same set-up as I do, you wont get kicked after the timeout.

And there we go a nice locked ssh port. You will now have to send a SLA to your server prior to connecting with your ssh client.