Peek Read Info (Posts about quick post)

The Hugo Switch

Dugite-Code — Tue, 13 Feb 2024 16:00:00 GMT

I've decided to shake things up a bit, and I'm switching my blog over from Nikola to Hugo.

Page Bundles and Obsidian

Page bundles offer great convenience for storing my blog posts in Obsidian. With Hugo, they group content along with associated resources like images and files. With this portability, I can store my posts within my note structure. While Nikola could provide similar capabilities to some extent, it's worth exploring alternatives, especially as I would have to rebuild my site anyway.

Better Asset Pipelines

One of the things that really drew me to Hugo is its asset pipeline. With image scaling, format conversion (hello, webp! Everyone hates you, but Google's bots sure don't), CSS and JS minification, Hugo makes my life easier. Plus, Hugo plays nice with deployment tools like purgecss, reducing the chances of mistakes with automation.

I'm even using the module.mounts feature to import my CSS and JS from a node_modules folder, making it easier to keep them up to date.

I've Just Learned a Lot

I'll admit it—I've made my fair share of mistakes along the way with this blog. From opting for a basic Bootstrap theme with little to no dark mode support, to not optimizing my HTML as thoroughly as I could have, I've certainly had some learning moments. But it's all part of the learning experience, so I'll take this opportunity to start fresh.

Using fdupes to cleanup my file server

Dugite-Code — Wed, 05 Jul 2023 16:00:00 GMT

The overall problem:

Like many of us, I am guilty of copying files haphazardly, promising myself that I'll organize them later. This has built up to a significant problem over the years, particularly with old smartphone backups. I had a bad habit of dumping photo folder backups onto my server, with each dump containing even more dumps of old photos, resulting in multiple levels of duplication. Using the command-line tool called fdupes I've only just managed to get some of it under control.

fdupes is a command-line application designed to find and identify duplicate files within a directory or a set of directories. It employs various techniques to compare file contents and determine duplicates, enabling efficient cleanup and reclamation of storage space.

To streamline the review process and make sure I know what's about to happen before deleting any files, I created a simple bash wrapper script. This script acts as a nice safety belt, preventing accidental fat finger deletions.

Avoiding the `rm -rf` Pitfall:

As many of us have learned the hard way, the rm -rf command can have disastrous consequences if misused (goodbye email server with 2000 emails). A simple typo or a wrong path can result in irreversible data loss. To mitigate this risk, the bash wrapper script avoids using rm -rf altogether. Instead, it leverages the safer alternative of moving duplicate files to a temporary trash directory for review and then subsequent manual deletion.

#!/bin/bash

TRASH="/tmp/trash"

find_files() {
  fdupes -rn "$*" > ${file}
  echo "Duplicate files have listed in ${file}"
}

remove_files() {
  echo "Reading from ${file}"
  echo ""
  read -p "Type yes to continue" choice
  case "$choice" in
    yes )
      mkdir -p "${TRASH}"
      while IFS= read -r line; do
        mv "${line}" "${TRASH}"
      done < ${file}
      echo "Duplicate files have been moved to ${TRASH}"
      exit ;;
    * )
      echo "Exiting"
      exit ;;
  esac
}

while getopts "rf:" option; do
  case "${option}" in
    r)
      remove=true  ;;
    f)
      file="${OPTARG:-dupes.txt}"  ;;
  esac
done
shift $((OPTIND - 1))

case "$remove" in
  true )
    remove_files
  false )
    find_files $*
esac

Understanding the Script:

The script utilizes the fdupes command-line tool to identify duplicate files within a given directory or set of directories. Here's how it works:

Finding Duplicate Files:

The find_files function invokes the fdupes command with the -rn flags, instructing it to recursively search for duplicates and list the results in a specified file.
If no file name is provided as an argument, the script will use the default file name dupes.txt to store the duplicate file list.
After the duplicates are found, the script informs us that the duplicate files have been listed in the dupes.txt file.

Removing Duplicate Files:

The remove_files function allows us to decide whether to remove the duplicates. Make sure to review the dupes.txt file before running.
If no file name is provided as an argument, the script will still refer to the default dupes.txt file to read the duplicate file list.
After printing the file listing the duplicates, the script prompts us to confirm our decision by typing "yes."
If confirmed, the script creates a temporary trash directory and proceeds to move the duplicate files to it.
Finally, it provides a message confirming that the duplicate files have been successfully moved to the trash directory.

Using the Script:

To utilize the script effectively, follow these steps:

Copy the script into a text editor and save it as ddup.sh.
Open a terminal and navigate to the directory containing the script.
Make the script executable by running the command: chmod +x ddup.sh.

Execute the script with appropriate options:

To find duplicate files: ./ddup.sh <directory>
To remove duplicate files: ./ddup.sh -r

Note: If you don't specify a file name using the -f argument, it will default to using the dupes.txt file for listing duplicate files.

Readable Nginx configs

Dugite-Code — Tue, 06 Sep 2022 16:00:00 GMT

A recent project announcement on the subredit /r/selfhosted reminded me to post about a simple trick I've started using to make the configuration of the webserver Nginx a little more ergonomic.

Nginx allows you to include files inline in your configs to make re-using code simple. An example would be all your ssl proxy settings as per generated using the Mozilla ssl-config generator.

simply add this config to a file like /etc/nginx/include.d/include.ssl_sec with your cert paths modified and include it in your config:

upstream example_service {
  server 127.0.0.1:8080;
  keepalive 32;
}

server {
  server_name example.tld;

  #Mozilla modern tls config
  include /etc/nginx/include.d/include.ssl_sec;

  location / {
    #Common Proxy settings
    include /etc/nginx/include.d/include.proxy_settings;

    proxy_pass http://example_service/;
  }
}

Now you have a nice easy config file that can be easily used as a template for new services. Adding additional configurations to files really makes it quick and easy to deploy new services without needing complicated projects like Nginx Proxy Manager

The Depressing Age of the Walled Garden

Dugite-Code — Mon, 23 Nov 2020 16:00:00 GMT

I remember being excited for Android. The age of a popular Linux device was upon us! I had moved to Reddit and was seeing new and interesting things and opinions every day, The Internet and Tech was vibrant! I no longer feel this way.

Android became heavily dependent on Google

Due to the quirks of the Mobile SOC's updating was... complicated. Add to that OEM customizations and you got a recipe for Google to swoop in and "fix" the issue by making key components, that used to part of the AOSP, proprietary. Where once we had the assurance of open development we now have even more black boxes of code being downloaded onto our mobile devices.

Android got DRM

Wildvine drm snuck onto our phones. Now you can have a legitimate bit of general purpose hardware where some services don't work as well as the hardware supports (Looking at you Netflix) simply because the market share of that device wasn't great enough for them to "certify" it's use.

The Walled Gardens of the Internet

At some point the Internet seamed to shrink. Once there was new (Admittedly sometimes terrible) sites to explore every day. Now it feels like everyone is siloed into Facebook, Twitter and Reddit. Reddit introduced Anti user "optimizations" to their Mobile website in the idiotic attempt to push users into their app. In 2020 a website no longer wants to be a website built on open technologies, they want it locked down like Twitter and Facebook in an Application.

IOT locked in the cloud

I was promised a smart home when I was a child. Now we have insecure IOT devices that are, for all intents and purposes, owned by someone else. The software can randomly be killed like Google did for some Nest devices and the open API's can be locked down with little to no notice. Sure we have the hacky open ecosystem for the ESP8266 and the ESP32 is great for DIY projects. However the ESP32 introduces flash encryption and secure boot. Meaning that even in this wonderful open hardware hacking space, the future is a locked down dystopia

It's all a war against General Purpose computing

Now you see things like Apple's M1 ARM based chip replacing x86 chips for their computers. They claim it's about user experience and performance but there is no denying that by moving to ARM from x86 gives them the same locked down hardware control they enjoy on the iPhone and iPad. From the articles I have read we will also begin to see them begin to slowly port the same software controls over as well, will we soon see the death of alternative browser Engines on their computers? unless they get hit by a good anti-trust charge or too I wouldn't be surprised if it happens in just a couple of years.

Now I read that Microsoft's Pluton hardware coming to our General purpose x86 CPU's. Cryptographic technology originally employed in the XBOX for DRM

TOTP with sudo (Google Auth)

Dugite-Code — Tue, 09 Jun 2020 16:00:00 GMT

I was reading the posts over on lobste.rs and saw this post: Is sudo almost useless?. Typically I see sudo as a safety belt to protect you from doing something stupid with administrator privileges rather than a security shield. But that doesn't mean it can't be both

As with ssh, outlined in my previous post TOTP with SSH (Google Auth), you can certainly boost your sudo usefulness security wise by throwing 2FA via google-authenticator-libpam on top of it.

Install google-authenticator-libpam

On debian/ubuntu:

    sudo apt update && sudo apt install google-authenticator-libpam

Set-up your secret keys

We now need to create the secret key, this should not be kept in the user folder, after all what is the point of 2FA if the user we are authenticating can just read the secret files. In my case I keep them in the root dir

Replace the variable ${USER} if/when you create a key for a user other than the active one.

sudo google-authenticator -s /root/.sudo_totp/${USER}/.google_authenticator
sudo chmod 600 -R /root/.sudo_totp/

You will see a QR code/secret key that you can scan with a TOTP app like andotp, authy, google authenticator or in my case I added it to my yubikey. There are also your emergency scratch codes that you should record somewhere safe.

Enable in PAM

You now need to let PAM know it should be checking the codes. There are two ways to do this, Mandatory and Only if secret key exists. I have it as Mandatory any user using sudo MUST have a secret key

In /etc/pam.d/sudo add the following configuration lines to the end of the file.

# Use Google Auth -- Mandatory
auth required pam_google_authenticator.so secret=/root/.sudo_totp/${USER}/.google_authenticator user=root

# Use Google Auth -- Only if secret key exists
# auth required pam_google_authenticator.so secret=/root/.sudo_totp/${USER}/.google_authenticator user=root nullok

Bonus do this for su as well

You can do the same thing for su as well however obviously the user variable will be root rather than the user attempting to elevate their privilege's.

Setup the key as before, just for the root user

sudo google-authenticator -s /root/.google_authenticator
sudo chmod 600 -R /root/.google_authenticator

In /etc/pam.d/su add the following configuration lines to the end of the file.

# Use Google Auth -- Mandatory
auth required pam_google_authenticator.so secret=/root/.google_authenticator user=root

You should probably know about LetsEncrypt DNS challenge validation

Dugite-Code — Tue, 28 Apr 2020 16:00:00 GMT

Everyone knows the basic way to renew a LetsEncrypt cert. Open port 80 and let LetsEncrypt connect to your server. But what if you don't want to open your network or you limit access to a handful of IP addresses? Well you can just use the DNS challenge validation, no need for web servers and no need for port wrangling.

For example I use the certbot-dns-cloudflare for my work intranet allowing it to remain VPN only.

Another great option is to use acme.sh as it supports a massive list of dns providers and the ever popular duckdns out of the box.

Given in the past I found the most fragile part of my LetsEncrypt setup was making sure port 80 was accessible to LetsEncrypt I personally use this method even if I have a network accessible from the wider internet.

App Passwords for docker-mailserver

Dugite-Code — Mon, 21 Oct 2019 16:00:00 GMT

Recently I got rid of my virtual IPFire firewall and setup a Netgate SG1100 as my home firewall. I did this mainly because the NIC on the IPFire host NUC was starting to fail, also we use Pfsense at work and it's good to be able to tinker on a common platform. As my email server was virtualized on the same host NUC as my firewall I switched my virtual modoboa email server install to the docker-mailserver project. This makes my mail server more portable than the old virtual machine was.

I then setup app specific passwords for my email following this guide Below is the changes I needed to do for the docker image.

Adding this to the docker-mailserver docker-compose.yml

    volumes:
    ###################################
    #### Dovecot App Passwords Mod ####
    ###################################
    - /opt/mail/custom/dovecot/10-auth.conf:/etc/dovecot/conf.d/10-auth.conf:ro
    - /opt/mail/custom/dovecot/auth-appspecificpasswd.conf.ext:/etc/dovecot/conf.d/auth-appspecificpasswd.conf.ext:ro
    - /opt/mail/custom/dovecot/app_specific_passwd:/etc/dovecot/app_specific_passwd:ro

The /opt/mail/custom/dovecot/10-auth.conf file

auth_mechanisms = plain login
!include auth-passwdfile.inc
!include auth-appspecificpasswd.conf.ext

The /opt/mail/custom/dovecot/auth-appspecificpasswd.conf.ext file

passdb {

  driver = passwd-file

  args = scheme=SHA512-CRYPT username_format=%u /etc/dovecot/app_specific_passwd

}

The /opt/mail/custom/dovecot/app_specific_passwd file (example)

K9emaillapp:{SHA512-CRYPT}123456789...::::::user=foo

Assuming your docker-mailserver is called mail you can get the format you passwords for the app_specific_passwd file by using:

docker exec -it mail doveadm pw -s SHA512-CRYPT

You can now user the username K9emaillapp and the associated password to log in to your email account

Something neat I did with FitNotes and Tasker

Dugite-Code — Mon, 22 Jul 2019 05:00:00 GMT

I use the fitness tracking app FitNotes on Android. It's a great application that I have happily used for years. The greatest issue I had with it was manually entering my body weight. Well I finally got myself into gear and fixed that issue using the fantastic staple of Android automation, Tasker

Using a Xiaomi Mi Smart Scale that I hooked up to my home server using my python gatttool wrapper I Dump it's weight data into a Google spreadsheet.

Using my Simple API (Because Google's own API is a pain) and the helper task getformatteddate, I pull a unix timestamp and the weight onto my Phone. I then run a INSERT SQL command on FitNote's database (using root of course)

I also did a bulk body weight record import via .csv using sqlitebrowser. Now I no longer have to manually enter my body weight, should have done this years ago.

Here is my Tasker task if you are interested:

    Healthapi (44)
        A1: Flash [ Text:%date %time Long:Off ]
        A2: Flash [ Text:Updating Health Report Long:Off ]
        A3: HTTP Get [ Server:Port:https://script.google.com Path:/macros/s/myprivatesheet/exec Attributes:key=myapikey Cookies: User Agent: Timeout:20 Mime Type: Output File: Trust Any Certificate:Off ]
        A4: Variable Set [ Name:%data To:%HTTPD Recurse Variables:Off Do Maths:Off Append:Off ]
        A5: Variable Set [ Name:%newline To:
     Recurse Variables:Off Do Maths:Off Append:Off ]
        A6: Variable Split [ Name:%data Splitter:%newline Delete Base:Off ]
        A7: Variable Split [ Name:%data1 Splitter:, Delete Base:Off ]
        A8: Read File [ File:Tasker/lastdate.dat To Var:%lastdate Continue Task After Error:On ]
        A9: If [ %lastdate neq %data11 ]
        A10: Write File [ File:Tasker/lastdate.dat Text:%data11 Append:Off Add Newline:Off ]
        A11: Perform Task [ Name:getFormattedDate Priority:%priority Parameter 1 (%par1):%data11 Parameter 2 (%par2):yyyy-mm-dd Return Value Variable:%date Stop:Off ]
        A12: Perform Task [ Name:getFormattedDate Priority:%priority Parameter 1 (%par1):%data11 Parameter 2 (%par2):hh:nn:ss Return Value Variable:%time Stop:Off ]
        A13: Variable Set [ Name:%measurement_id To:1 Do Maths:Off Append:On ]
        A14: Variable Set [ Name:%value To:%data12 Do Maths:Off Append:On ]
        A15: Variable Set [ Name:%query To:INSERT INTO MeasurementRecord (measurement_id, date, time, value, comment) VALUES ("%measurement_id", "%date", "%time", "%value",""); Do Maths:Off Append:Off ]
        A16: SQL Query [ Mode:Raw File:/data/data/com.github.jamesgay.fitnotes/databases/database.db Table: Columns: Query:%query Selection Parameters: Order By: Output Column Divider: Variable Array:%test Use Root:On ]

I hope you found this interesting, if only in the abstract "hey that's a thing you can totally do" kind of way. If you want me to write a complete how-to let me know in the comments down below.

TOTP with SSH (Google Auth)

Dugite-Code — Thu, 13 Jun 2019 16:00:00 GMT

For your ssh you can use google-authenticator-libpam to add time based codes to your ssh login.

On debian/ubuntu:

    sudo apt update && sudo apt install google-authenticator-libpam

    google-authenticator

    Do you want authentication tokens to be time-based (y/n) y
    [...]
    Do you want me to update your "/home/dugite/.google_authenticator" file? (y/n) y
    [...]

You will see a QR code/secret key that you can scan with a TOTP app like andotp, authy or google authenticator (WARNING Google authenticator has no backup options). There are also your emergency scratch codes.

In /etc/ssh/sshd_config Add:

    # Use Challenge Response Auth i.e. TOTP
    ChallengeResponseAuthentication yes
    # Require both publickey and TOTP
    AuthenticationMethods publickey,keyboard-interactive

In /etc/pam.d/sshd

    # Comment out Standard Un*x authentication.
    # @include common-auth
    # Load the google TOTP Authentication module
    auth required pam_google_authenticator.so

Goodbye Chrome and other things

Dugite-Code — Wed, 29 May 2019 16:00:00 GMT

Google, once the tech enthusiast darling is looking more and more like Microsoft did in the mid 90's.

Google to restrict modern ad blocking Chrome extensions to enterprise users

Google is first and foremost an ad company so it should come as no surprise that now they are leading the browser market share

Now Microsoft is switching to chromium as a browser backend it's no surprise we see Google moving to limit Ad-blockers.

Google is eating our mail

Google really got entrenched with the tech enthusiast crowd because gmail was free, quick and had good spam filtering. Now we are all feeling the consequences of encouraging non-technical people to centralize their emails with them. A once open and vibrant standard is increasingly looking like a mono-culture with both Google and Microsoft's opaque filtering and non-standard blocking making running your own email server almost impossible. Along with the launch of google's AMP for email we see yet another example of Google pushing through their own interests over the interests of the email ecosystem.

What you can do

Switch to Firefox
Switch to alternative search engines like Duckduckgo
Use more privacy respecting tools

Peek Read Info (Posts about quick post)

The Hugo Switch

Page Bundles and Obsidian

Better Asset Pipelines

I've Just Learned a Lot

Using fdupes to cleanup my file server

The overall problem:

Avoiding the rm -rf Pitfall:

Understanding the Script:

Finding Duplicate Files:

Removing Duplicate Files:

Using the Script:

Execute the script with appropriate options:

Readable Nginx configs

The Depressing Age of the Walled Garden

Android became heavily dependent on Google

Android got DRM

The Walled Gardens of the Internet

IOT locked in the cloud

It's all a war against General Purpose computing

TOTP with sudo (Google Auth)

Install google-authenticator-libpam

Set-up your secret keys

Enable in PAM

Bonus do this for su as well

You should probably know about LetsEncrypt DNS challenge validation

App Passwords for docker-mailserver

Something neat I did with FitNotes and Tasker

TOTP with SSH (Google Auth)

Goodbye Chrome and other things

Google to restrict modern ad blocking Chrome extensions to enterprise users

Google is eating our mail

What you can do

Avoiding the `rm -rf` Pitfall: