Skip to main content

Technology

As I Tinker I Learn, Somtimes I Even Write It Down.

Fixing a Patreon feed's cover artwork

Antennapod With the recent Pocketcast PR blunder I finally decided to jump back to open source Antennapod. This has been painless especially with the introduction of the Remove silence feature, the main feature that kept me with Pocketcasts for so long.

The only issue I had was a Private feed with broken cover artwork, this was frustrating but it looks to be an issue on Patreons end. Thankfully Node Red is available to rescue the situation!

The flow is really simple, the only additional node I have added is the node-red-contrib-httpauth.

  1. On a HTTP request, fetch the RSS feed.
  2. Convert from XML to and Object.
  3. Replace msg.payload.rss.channel[0].image[0].url[0] with a good url from the podcasters website.
  4. Create the txt/xml headers
  5. Return the fixed RSS Feed

Node-Red Flow

[{"id":"8d2a4ad1.4599a","type":"http request","z":"d437ad18.0999c","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"https://www.patreon.com/rss/yaddayada","tls":"","proxy":"","authType":"","x":367.5,"y":31,"wires":[["4c5cdedd.6772c8"]]},{"id":"4c5cdedd.6772c8","type":"xml","z":"d437ad18.0999c","name":"XML To Object","property":"payload","attr":"","chr":"","x":230.5,"y":78,"wires":[["adbd8c2e.7fbcd"]]},{"id":"7475a34e.9f953c","type":"http in","z":"d437ad18.0999c","name":"rss","url":"/mystupidRSS","method":"get","upload":false,"swaggerDoc":"","x":69.5,"y":31,"wires":[["26c3aede.f41d4a"]]},{"id":"26c3aede.f41d4a","type":"node-red-contrib-httpauth","z":"d437ad18.0999c","name":"","file":"","cred":"","authType":"Basic","realm":"","username":"","password":"","hashed":false,"x":213.5,"y":31,"wires":[["8d2a4ad1.4599a"]]},{"id":"56731f49.b7f77","type":"xml","z":"d437ad18.0999c","name":"Object to XML","property":"payload","attr":"","chr":"","x":647.5,"y":79,"wires":[["5aa22446.612044"]]},{"id":"5aa22446.612044","type":"change","z":"d437ad18.0999c","name":"Set Headers","rules":[{"t":"set","p":"headers","pt":"msg","to":"{}","tot":"json"},{"t":"set","p":"headers.content-type","pt":"msg","to":"text/xml","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":219,"y":131,"wires":[["8df07127.e9d0c8"]]},{"id":"8df07127.e9d0c8","type":"http response","z":"d437ad18.0999c","name":"","statusCode":"","headers":{},"x":391,"y":131,"wires":[]},{"id":"adbd8c2e.7fbcd","type":"change","z":"d437ad18.0999c","name":"Replace Cover Image","rules":[{"t":"set","p":"payload.rss.channel[0].image[0].url[0]","pt":"msg","to":"https://example.com/cover_art.png","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":440.5,"y":79,"wires":[["56731f49.b7f77"]]}]

Lookout Fix Version - v2.0.00b1 Call for Testers

Lookout!Unfortunately Microsoft Email servers still to this day send emails out in the proprietary TNEF format (We see this as a plain text email with a winmail.dat attachment). The Lookout (fix version) add-on decodes this attachment so you may open the attached files.

I've just published the V2.0.00 for the Thunderbird plugin Lookout (fix Version). This is an update for the coming versions of Thunderbird that significantly change how addons are implemented.

There are some known visual glitches and the debugging logs currently do not work, however attachments should de-code and save as expected. I would appreciate any testers to give it a shot.


Something neat I did with FitNotes and Tasker

FitNotes App for Android I use the fitness tracking app FitNotes on Android. It's a great application that I have happily used for years. The greatest issue I had with it was manually entering my body weight. Well I finally got myself into gear and fixed that issue using the fantastic staple of Android automation, Tasker

Using a Xiaomi Mi Smart Scale that I hooked up to my home server using my python gatttool wrapper I Dump it's weight data into a Google spreadsheet.

Using my Simple API (Because Google's own API is a pain) and the helper task getformatteddate, I pull a unix timestamp and the weight onto my Phone. I then run a INSERT SQL command on FitNote's database (using root of course)

I also did a bulk body weight record import via .csv using sqlitebrowser. Now I no longer have to manually enter my body weight, should have done this years ago.

Here is my Tasker task if you are interested:

    Healthapi (44)
        A1: Flash [ Text:%date %time Long:Off ]
        A2: Flash [ Text:Updating Health Report Long:Off ]
        A3: HTTP Get [ Server:Port:https://script.google.com Path:/macros/s/myprivatesheet/exec Attributes:key=myapikey Cookies: User Agent: Timeout:20 Mime Type: Output File: Trust Any Certificate:Off ]
        A4: Variable Set [ Name:%data To:%HTTPD Recurse Variables:Off Do Maths:Off Append:Off ]
        A5: Variable Set [ Name:%newline To:
     Recurse Variables:Off Do Maths:Off Append:Off ]
        A6: Variable Split [ Name:%data Splitter:%newline Delete Base:Off ]
        A7: Variable Split [ Name:%data1 Splitter:, Delete Base:Off ]
        A8: Read File [ File:Tasker/lastdate.dat To Var:%lastdate Continue Task After Error:On ]
        A9: If [ %lastdate neq %data11 ]
        A10: Write File [ File:Tasker/lastdate.dat Text:%data11 Append:Off Add Newline:Off ]
        A11: Perform Task [ Name:getFormattedDate Priority:%priority Parameter 1 (%par1):%data11 Parameter 2 (%par2):yyyy-mm-dd Return Value Variable:%date Stop:Off ]
        A12: Perform Task [ Name:getFormattedDate Priority:%priority Parameter 1 (%par1):%data11 Parameter 2 (%par2):hh:nn:ss Return Value Variable:%time Stop:Off ]
        A13: Variable Set [ Name:%measurement_id To:1 Do Maths:Off Append:On ]
        A14: Variable Set [ Name:%value To:%data12 Do Maths:Off Append:On ]
        A15: Variable Set [ Name:%query To:INSERT INTO MeasurementRecord (measurement_id, date, time, value, comment) VALUES ("%measurement_id", "%date", "%time", "%value",""); Do Maths:Off Append:Off ]
        A16: SQL Query [ Mode:Raw File:/data/data/com.github.jamesgay.fitnotes/databases/database.db Table: Columns: Query:%query Selection Parameters: Order By: Output Column Divider: Variable Array:%test Use Root:On ]

I hope you found this interesting, if only in the abstract "hey that's a thing you can totally do" kind of way. If you want me to write a complete how-to let me know in the comments down below.


Lookout Fix Version - v 1.3.00 Released

Lookout!Unfortunately Microsoft Email servers still to this day send emails out in the proprietary TNEF format (We see this as a plain text email with a winmail.dat attachment). The Lookout (fix version) add-on decodes this attachment so you may open the attached files.

I've just published the V1.3.00 for the Thunderbird plugin Lookout (fix Version). I have done a fair amount of work on quality of life changes to make it a bit more user friendly.

Have a look at my previous post on how I took up the maintenance of this very useful plugin if you want to know more about the history of the project.

New in version 1.3.00

  • Attachment drag and drop support
  • Optionally remove winmail.dat file when finished processing. This is in options and defaults to true
  • Fix attachment pane opening and closing, respects inbuilt Thunderbird preferences
  • Ignore blank VCF files to minimize clutter
  • Highlight event invitations - To be expanded to full lightning integration

Note: There are future plans to replace the plain text with the attached HTML file (body_part_0.html) that newer exchange servers include in the winmail.dat file.


SSH Login Notifications with Gotify

Inspired by this post I decided to add a notification on my phone every time an ssh session began on my servers. Seeing as I make use of Gotify for selfhosted push notifications I used that rather than signal.

First I created created the file /usr/local/bin/sshnotif. At the top you can add your own token and Gotify url

Update: I had to push the current time back a full minute in order to improve consistency. I'll defiantly want to revisit this at a later date

#!/bin/bash

exec &> /dev/null #Hide output

Gotify_URL='https://example.tld/gotify'
Gotify_Token='gotify-app-token'

notify()
{

        now=$(date -d "-60 seconds" +%s) #Get current time minus 60 seconds
        end=$((SECONDS+30)) #Set 30s Timeout for loop

        while [ $SECONDS -lt $end ]; do

                SSHdate=$(date -d "$(who |grep pts|tail -1 | awk '{print $3, $4}')" +%s) #Check for the latest SSH session

                if [ $SSHdate -ge $now ]; then #Once who is updated continue with sending Notification

                        title="SSH Login for $(/bin/hostname -f)"
                        message="$(/usr/bin/who | grep pts)"

                        /usr/bin/curl -X POST -s \
                                -F "title=${title}" \
                                -F "message=${message}" \
                                -F "priority=5" \
                                "${Gotify_URL}/message?token=${Gotify_Token}"

                        break
                fi
        done

}

notify & #Run in background to prevent holding up the login process

Run the command chmod +x /usr/local/bin/sshnotif

In the file /etc/pam.d/sshd add the following line

# note optional is set to prevent ssh login failure
session optional pam_exec.so /usr/local/bin/sshnotif

I now get a nice notification with all the open SSH sessions listed. Unlike the post on 8192.one I didn't want any IP address resolution using an online service. I plan on integrating the MaxMind GeoLite2 database at some point. However as I already have Graylog set up to do this it's not a high priority for me.


Getting a QNAP NAS to Log to my Graylog instance

Running old embedded devices is a pain not to mention a major security risk. But if you are like me and are stuck with it sometimes you can take solace in software repo projects like Entware. In this case I needed to centralize all the disparate system logs on the network so I could find issues BEFORE they cause real trouble. The problem is the QNAP NAS I had could only send system logs over unencrypted UDP.

That's just not good enough, especially as I want to use client certs down the line. The simplest solution I found was to install syslog-ng to redirect the logs securely.

Note: I'm using a letsencrypt cert to make my life simpler

Setting up the NAS

Install Entware by downloading the .qpkg file, navigating to the NAS in the web browser and then selecting the install manually option in the app center.

Manualy install the .qpkg file

SSH into the NAS and install syslog-ng

opkg update
opkg install syslog-ng

Configure syslog-ng by editing /opt/etc/syslog-ng.conf

# Important set the right config file version
@version: 3.20

options {
};

# Listen to local syslog connection
source localhostudp {
        udp( ip("127.0.0.1") port(1514) );
};

# Forward to remote graylog server over tls to port 1514
# To Implement Client Cert
destination graylog_loghost {
        network(
                "example.com" port(1514)
                transport("tls")
                tls( ca_dir("/opt/sbin/cadir") )
        );
};

# Enable both source and destination
log {
        source(localhostudp);
        destination(graylog_loghost);
};

Set up the Letsencrypt CA by downloading the TrustID X3 Root Certificate (formallyu known as DST Root CA X3). We then need to discover the hash of the certificate using openssl. Syslog-ng requires as simlink named with the certificate hash.

The hash should be 2e5ac55d

/opt/sbin/cadir
wget https://github.com/letsencrypt/website/blob/master/static/certs/trustid-x3-root.pem

openssl x509 -noout -hash -in trustid-x3-root.pem

ln -s /opt/sbin/cadir/trustid-x3-root.pem /opt/sbin/cadir/2e5ac55d.0

Via the web admin, set the NAS to log to 127.0.0.1 with the local port 1514. This can be found in Systems Logs in the Systems Settings category.

Control Panel -> System Logs -> Syslog Client Management

Ensure syslog-ng isn't running then test in the foreground for any errors

/opt/etc/init.d/S01syslog-ng stop

/opt/sbin/syslog-ng -Fvde

If no errors appear you can then start syslog-ng

/opt/etc/init.d/S01syslog-ng start

Graylog Notes

Graylog doesn't appear to directly accept the format sent via syslog-ng. While it is possible to change the format in syslog-ng I didn't figure out the best way to do it. My solution was to set the input to Raw/Plaintext TCP and then run a GROK pattern extractor when matching the conn log string

%{DATA} qlogd\[9147\]: %{DATA:facility}: Users: %{DATA:NAS_user}, Source IP: %{IP:NAS_src}, Computer name: %{DATA:NAS_id}, Connection type: %{DATA:NAS_connection}, Accessed resources: %{DATA:NAS_resource}, Action: %{GREEDYDATA:NAS_action}

LookOut fix version

Lookout! In august 2018 I took over maintenance of the Thunderbird addon Lookout-fix-version. I soon set up a the Github Organization TB-throwback so that future development can be expanded and transferred easier if I stop work on it.

It's been an interesting experience managing a small project that's over 11 years old. Especially with all the changes and rapid development Thunderbird has been going experiencing now it's separated from Mozilla.

Why did I take over?

I needed to move my office away from Outlook 2010. I had no budget to upgrade the office software, but I couldn't allow the company to keep limping along with a 9 year old product.

Thunderbird to the rescue! Except...

TNEF files, supposedly a thing of the past. Even Microsoft recommends you NOT to send such files. But we have to work with people who don't upgrade and pay the lowest bidder to configure their exchange servers.

Unfortunately the original Lookout was at this point unmaintained and severly out of date similarly Lookout+ and Lookout-fix-version hadn't seen any updates in a long time. luckily Oleksandr was still contactable via the support email and was happy to add me as a developer on the ATN page.

My first change was a simple modification to the preferences css to fix changes in Thunderbird 59. I've since been working on adding debugging, improving performance, squashing bugs and generally attempting to learn how everything is strung together.

Original TNEF file with extracted attachments

I plan on porting the addon to a webextension in the coming months to ensure we have this useful addon for many years to come.


Server set-up Checklist

Configure your linux server I often see questions on /r/selfhosted on how to secure a server. Here is a quick checklist of things you might want to look into.

Follow best practices for the basics

Lock down the Server

  • Disable root login via SSH
  • Close all unused incoming ports via UFW/iptables
  • Limit outgoing ports as well as incoming using UFW/iptables
  • Watch for credential stuffing/brute force attacks with Fail2ban

Backup your configs/files

  • Securely encrypted backup via Duplicity
  • External Backup to external drive.
  • Remote backup, either via a regularly swapped out external drive or via the cload

Set up monitoring services to let you know when something goes wrong

Here are a few extra things you can do to bolster your ssh security

Useful resources


Static Blogging

Now for the obligatory static blog post on any static blog This blog is a Nikola static blog, a python based static blog generator.

Why did I choose Nikola?

  1. It's Python based - I like python, I know python, and I can hack python. Jekyll, the most popular static blog generator, is written in Ruby. I found it to be slow and painful to work with. I couldn't be bothered learning it's particulars. Also my Desktop and Server came with python so there was no barrier to entry.
  2. Batteries are Included - I looked at the popular Pelican, it's arguably more flexible and powerful than Nikola, but you have to do more initial configuration. I had already been procrastinating on re-starting my blog for over a year, I needed to get started

Why do people like Static Blogs?

  1. You need to be savvy - Static Blogs scratch the tech tinkering itch, don't like something it's simple to change that.
  2. It's simple - Many CMS systems are over complicated messes, overkill for the typical blog. Do you need to render your blog on the fly for each visitor? PHP is useful but if you don't need it why use it?
  3. Portable - You render your site as it changes, have beef with your service provider? Change your dns and upload with next to no configuration. Done
  4. It's Fast - How fast can a server trow out a few kb of static html? Very fast! With Wordpress you need to install caching plugins to make your site essentially static, why not skip the middle man and do it from the start.
  5. You write how you like - Markdown, reStructuredText, MediaWiki Markdown or even HTML.
  6. It's secure - No database to hack, no PHP to patch.

Why do people hate Static Blogs?

  1. No web based content management system - This is something that tends to be lacking in any static blog. It's static after all! There are many ways to get around this by using something like Netlify CMS. Personally I'm using the Atom test editor and will be going over this in a future post.
  2. You need to be savvy - As a general rule you need to be able to deal with the nuts and bolts of your blog. This is a good and bad thing, depending on your disposition.

TOTP with SSH (Google Auth)

For your ssh you can use google-authenticator-libpam to add time based codes to your ssh login.

On debian/ubuntu:

    sudo apt update && sudo apt install google-authenticator-libpam

    google-authenticator

    Do you want authentication tokens to be time-based (y/n) y
    [...]
    Do you want me to update your "/home/dugite/.google_authenticator" file? (y/n) y
    [...]

You will see a QR code/secret key that you can scan with a TOTP app like andotp, authy or google authenticator (WARNING Google authenticator has no backup options). There are also your emergency scratch codes.

In /etc/ssh/sshd_config Add:

    # Use Challenge Response Auth i.e. TOTP
    ChallengeResponseAuthentication yes
    # Require both publickey and TOTP
    AuthenticationMethods publickey,keyboard-interactive

In /etc/pam.d/sshd

    # Comment out Standard Un*x authentication.
    # @include common-auth
    # Load the google TOTP Authentication module
    auth required pam_google_authenticator.so