Categories
Fedora

Anti-Spam through an IMAP Account

As we are increasingly dealing with an aging population, this adds technological problems. My elderly mother has developed dementia and has lost any technology knowledge that she had other than basic Windows operating system fundamentals. Her primary access to the Internet is via Outlook (the e-mail client) and a web browser. In her diminished capacity of memory and cognitive abilities she will open nearly every single e-mail that she gets. And does she get them. Roughly around 1,000 e-mails a day. With big ISPs (Comcast) allowing willy nilly whatever to come through their mail servers this is a disaster waiting to happen.

Well, why not just install some anti-spam for Outlook application you ask? Ah you mean the 2000’s era software that no longer works or is available? Yeah, that is why. The worthless Anti-Virus software packages ‘Anti-Spam’ has a hit rate of about 0.01%. Super effective isn’t it? It’s great for detecting viruses, not for spam e-mails. Which makes sense. It’s Anti-Virus. Entirely different thing.

Users with disabilities like dementia (again memory loss) is challenging as anything that changes can cause serious issues. I can’t just up and replace the Outlook application she has been using for years with something that may have better built in anti-spam.

Again, I cannot just switch over to a mail server I manage. She’s had her Comcast e-mail address for decades now. That is what she knows. Creating some elaborate forwarding system is just asking for trouble managing that. And yes, this would be the most ideal way of solving the problem. And I highly recommend it. Especially if you are in a position where the person with dementia onset is early and you can move over to an e-mail system that you have control over. The best place for anti-spam is before it’s acceptance and delivery to the user. And this is how most of the solutions migrated to.

Alas I do not have the ability to control what Comcast accepts as e-mail before delivery. But, I have found a middle ground solution that appears to work well. Unfortunately it has some caveats. As I mentioned, this is an old way of dealing with spam e-mail, so the software has been nearly abandoned. But there is still hope.

The solution is relatively simple in a complex system. Logging into the IMAP service, fetching and scanning the e-mail via Spamassassin anti-spam application, marking the e-mails that are detected as spam, moving them to a Spam/Junk folder (and optionally just deleting them). The solution does not require running any software on the users computer itself, although it can be. But I wouldn’t recommend it in this scenario of a dementia patient. It will need to be running on a separate (remote) computer (a VM perhaps hosted somewhere).

The solution I found is just two software applications. A python based application called ISBG (IMAP Spam Begone) and Spamassassin.

For Fedora Linux you will want to install Spamassassin.

$ sudo dnf install spamassassin

Then make sure the python installer pip is installed.

$ sudo dnf install pip

Then install isbg.

$ pip install isbg

Well, that’s it! NOT! Did you think it was going to be that easy? As I mentioned earlier, this way of anti-spam has been pushed as a thing of the past. So, the isbg software was yeeted by the developers years ago. Probably beause only 4 people in the world used it as no one really knew what to use it for. And because the isbg application is written in Python which changes every version release. And don’t get me wrong, Python is super cool and you should learn it.

When writing this article, the current version of isbg is v2.3.1. I have no idea which version of Python it was initially released for, but the current version of Python is v3.12.5. The isbg application will immediately fail with:

$ isbg
Traceback (most recent call last):
File "/home/<yourusername>/.local/bin/isbg", line 5, in <module>
from isbg.__main__ import main
File "/home/<yourusername>/.local/lib/python3.12/site-packages/isbg/__init__.py", line 10, in <module>
from .isbg import ISBG, ISBGError, __version__, __exitcodes__, __license__
File "/home/<yourusername>/.local/lib/python3.12/site-packages/isbg/isbg.py", line 23, in <module>
from isbg import imaputils
File "/home/<yourusername>/.local/lib/python3.12/site-packages/isbg/imaputils.py", line 40, in <module>
Email = TypeVar(email.message.Message)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: typevar() argument 'name' must be str, not type

Awesome! Right? Just what you wanted in your journey to hell (if you’ve gotten this far you’ve been there for awhile). But have no fear. This is actually a fairly simple solution. Mind boggling, I know. You just need to edit the file below at line 40. Use whatever text editor you are familiar with.

/home/<yourusername>/.local/lib/python3.12/site-packages/isbg/imaputils.py

Line 40:
Email = TypeVar(email.message.Message)

Change To:
Email = TypeVar(str(email.message.Message))

And now the isbg application will work. It will still have some issues with specific malformed e-mail’s and it does not appear to work with the –spamc option.

You will now need to get the username, password and the IMAP server address for the e-mail account you’ll want to scan. The e-mail account will also need a Spam or some sort of Junk folder to move the spam to already created. isbg will not create the folder. In this example, my moms account already had Junk folder, so that is what I am using. By default it is Spam. If isbg does not detect this, it will exit with an error saying so. This example also uses Comcast as the IMAP. The first run of the isbg command you will want to use the –savepw (Save Password) command, so subsequent commands will not prompt you for the password. Note that the password is saved in plain text. There’s no getting around that. isbg will by default only scan 50 e-mails per run. We’ll keep it at that for the first run.

isbg --verbose --imaphost imap.comcast.net --imapuser <yourmom>@comcast.net --spaminbox Junk --savepw

Again, if you have the default Spam folder for spam e-mails, remove the –spaminbox command. Without any errors it will login, and scan through the messages. It’s a bit on the slow side so many take a couple of minutes.

Now that it has completed you can add the better options I have found. I do not recommend running isbg wide open (meaning scanning all e-mails in a single run). It will crash on specific malformed e-mails and you will lose any sort of progress made. It only does it’s write operations after all e-mails have been scanned. I’ve found a happy medium of 250 e-mails to be acceptable. (And yes we’re talking about someone who gets several hundred to a thousand e-mails per day.) Here is the new command.

isbg --verbose --imaphost imap.comcast.net --imapuser <yourmom>@comcast.net --spaminbox Junk --maxsize 512000 --partialrun 250 --flag

The maxsize is what maximum size e-mail to fetch from the IMAP server. isbg defaults to a considerably low value. Most spam these days are much larger than it. So I put the max e-mail size to 512kbytes. The partialrun command is where you set how many e-mails are processed during this session. 0 means all of them. I do not recommend this. I set this to 250. Enough where I can cover a large group of e-mails but not lose an hour and a half of processing if isbg comes across a malformed e-mail message. (Remember we don’t have control of the delivery.) The flag command tells isbg what to do after the scan is done. Prepend the subject with [SPAM] and the score of it and move it into the Junk folder (the setting from the spaminbox).

That is really it. You can create a crontab to run this command however often. I would not recommend minutes but four times a day. Depends on the volume of e-mail coming into the account.

Anyways, that about wraps it up for this. I wrote this to mainly remember what I changed to make the isbg program to work. This was written by a real person with no AI involved. I will edit this at a later time after publication with attributions.

Welcome to Costco, I love you.

Categories
Dogs Nattai

Summer Fun

Categories
Dogs Nattai

Nattais’ 5th Birthday

May 25th Nattai turned 5 years old. I got him some cookie treats and a haul of toys from Pet Smart. A Chuck It flying Squirrel, 3 pack of Chuck It balls and a couple basic toys.

I had to read the fine print to make sure I actually bought a cookie for dogs and not people!
Nattai with his new toys. Squeaky Yellow spiked ball and the Chuck-It flying squirrel.

Categories
Dogs Nattai

Stick Is Life

Close up photo of Nattai a Blue Heeler cattle dog panting.
Close up of Nattai after fetching stick.

It’s been almost two months with Nattai now. He’s adjusting to his new life very well. I tried to set him up for success and I feel like it is working well. Might have thrown him into the fire in certain instances too soon on some things. It’s hard to not know that he hasn’t experienced things that are normal day to day for me. Or some things that you’d have expected a dog to have already experienced.

Nattai fetching a stick.
Categories
Dogs Nattai

Nattai the Cattle Dog

Meet my re-homed Australian Cattle Dog Blue Heeler, Nattai.

It was time for another chapter in my life.

Categories
Cobar Dogs Hemophilia

Cobar | 2007-2020

My beautiful, charismatic, loyal and best friend of 13 years Cobar passed away peacefully at my side on October 19th, 2020.

Cobar was an Australian Cattle Dog Red Heeler. He suffered from a a blood disease called Hemophilia. It was of the worst kind. He had 2% clotting factor (his blood didn’t clot). We were together from beginning to end. He lived a good happy life and full filled his job by making people happy and playing as much as he could.

He will be missed greatly.

photo of dog laying in grass with plush toy

Categories
Fedora Sendmail

Sendmail – Spamassassin – Configuration Errors

This is an entry to remind me of how I corrected some small issues with Spamassassin. I use Fedora, Sendmail, Spamassassin and upon upgrading to latest version I started getting some errors from Spamassassin. This is how I fixed the errors.

Can’t locate MLDBM/Sync.pm error

I got this error even though the MLDBM perl module was installed via the package system. This may be a deprecated configuration, as it appears the solution is something to do with mySQL databases. I went with the cpan, cpanm, then install the MLDBM::Sync module route. Quick and painless. As root:

; Install cpanminus.
#cpan App::cpanminus
; Using cpanm to install the MLDBM:Sync that is missing from package.
#cpanm MLDBM::Sync

FuzzyOcr: Cannot find executable for gifinter

This error is due to the giflibs v5.x no longer has many of the utilities that were in the v4.x releases and FuzzyOcr hasn’t been updated. I suspect this isn’t really used much anymore, but there’s a couple solutions. I used the method of downloading, compiling and copying over the giflib v4.x gifinter program. Just download the latest v4.x version from the sourceforge project. Do the usual ./configure; make. But not make install. Copy the gifinter program to /usr/local/bin. Edit the /etc/mail/spamassassin/fuzzyocr.cf file. Look for the #focr_bin_gifinter line and change it to /usr/local/bin/gifinter. I use /usr/local/bin to keep manually compiled programs separate from package installed programs that are usually in /usr/bin/.

Configuration file requires version 3.004002 of SpamAssassin, but this is code version 3.004003

Was surprised this even occurred. This was simple as running as root;

#sa-update
Categories
Music

FFMPEG – Converting DSF to FLAC

When converting SACD DSF audio files to FLAC (or any other PCM type format) the conversion process will most likely introduce distortion in the upper frequencies. In order to eliminate this you need to use the lowpass filter during the conversion process. This post is mainly so I won’t forget.

These are the ffmpeg commands I used to convert to regular FLAC. To convert to 24bit FLAC use s32 for the sample format.

convert one file to flac
ffmpeg -i inputfile.dsf -af "lowpass=24000,volume=6db" -sample_fmt s16 -ar 48000 outputfile.flac
Convert all dsf files in directory
for i in *.dsf; do ffmpeg -i "$i" -af "lowpass=24000, volume=6dB" -sample_fmt s16 -ar 48000 "${i%.*}.flac"; done

24 bit and 32 bit FLAC

At the time I wrote this the FLAC encoder only supported 16 and 24 bit encoding. As of ffmpeg v6 the FLAC encoder now appears to support 32 bit encoding using the -strict experimental switch. Be sure to test playback devices for the new 32 bit encoded FLAC files.

convert to 24 bit flac
ffmpeg -i inputfile.dsf -af "lowpass=24000, volume=6dB" -sample_fmt s32 -ar 48000 outputfile.flac
convert to 32 bit flac
ffmpeg -i inputfile.dsf -strict experimental -af "lowpass=24000, volume=6dB" -sample_fmt s32 -ar 48000 outputfile.flac

Audio Filters

Remember that the audio filters are specific to convert DSF files only. If you are converting other file types to FLAC they are not necessary. The values are not absolute and can be modified. The lowpass filter passes all frequencies below its setting and blocks any above it. This is required due to the high frequency distortion from converting DSD format to PCM format. The volume filter adjusts the volume. This isn’t required, but highly recommended as the volumes between the two formats are represented differently. The resulting PCM file will be lower in volume. These settings are not specific to FLAC but the conversion process from DSD to PCM.

Written by a real human. No AI involved.

Categories
Fedora Sendmail

Sendmail – LetsEncrypt and verify=OK

How To Configure LetsEncrypt and Sendmail Properly

This documentation pulls from a form post on FreeBSD from user Kuli.
https://forums.freebsd.org/threads/sendmail-and-letsencrypt.57675/

LetsEncrypt certificates aren’t listed in the main distributed ca-bundle.crt. The confCACERTand confCACERT_PATH configurations are two independent configurations that don’t really have anything to do with each other. When verifying certificates, it looks in the confCACERT_PATH for individual hashed files of root certificates. The confCACERTwill be configured with the intermediary LetsEncrypt chain.pem. Sendmail will then be happy to verify=OK the certificates. Do note that, it appears the majority of mail servers are using certificates that can’t be verified. So you will see a lot of NO. It’s better than FAIL. The script works with Fedora and probably any variant of Linux. Your experience may vary.

2021 Update!

I have discovered a much better way to generate the hashed ca files! I really struggled with the shell script. I have only tested this with Fedora 33, though I suspect it should work on other distributions.

Sendmail Configuration

define(CERT_DIR',/etc/letsencrypt/live/<your site>')
define(confCACERT_PATH',/usr/local/etc/ssl/ROOT')
define(confCACERT',CERT_DIR/chain.pem')
define(confSERVER_CERT',CERT_DIR/cert.pem')
define(confSERVER_KEY',CERT_DIR/privkey.pem')
define(confCLIENT_CERT',CERT_DIR/cert.pem')
define(confCLIENT_KEY',CERT_DIR/privkey.pem')

Create the CACERT_PATH files

2021 New Easy Way:

#p11-kit extract --format=openssl-directory --filter=ca-anchors --overwrite /usr/local/etc/ssl/ROOT/

Old Kinda Broken Shell Script Way:

#!/bin/sh
#Separate the root cert into files each with only one cert and name hashed

RCert=/etc/ssl/certs/ca-bundle.crt
DESTDIR=/usr/local/etc/ssl/ROOT
mkdir -p $DESTDIR
cd $DESTDIR
rm -f *
cat $RCert | sed -E '/^(Certificate:|SHA1 Fingerprint|#| |$)/d' | awk 'BEGIN {c=0;} /BEGIN TRUS/{c++} { print > "cert." c ".pem"}'

for a in ls $DESTDIR
do
    mv $a openssl x509 -hash -noout < $a.0
done

Categories
Other

Liquidsoap – Multi-Bitrate DASH streaming configuration

This is an example of multi-bitrate DASH streaming configuration. In this example Liquidsoap will be configured to output three different MPEGTS video streams with various frame size, bitrates and one stereo audio MPEGTS stream all via  UDP.  This configuration can be used with the same setup as my previous blog post about setting up Liquidsoap MPEGTS DASH streaming. This example is using Liquidsoap v1.3.3.

Please note that this is more of a proof of concept than practical use. A hardware encoder plugin for Gstreamer would help.

Liquidsoap Configuration: MPEGTS via UDP for Multi-Bitrate DASH Streaming
set("log.stdout",true)
set("log.level",5)
set("gstreamer.debug_level",2)

set("frame.video.width",1280)
set("frame.video.height",720)
set("frame.video.samplerate",25)
set("gstreamer.add_borders", true)

input = single("/home/user/Videos/Test/test-1280x720-25fps.ogv")

#MPEGTS Output For DASH Streaming.
#You must start Shaka Packager first before starting Liquidsoap.

#Audio Channel. Port 5000
output.gstreamer.audio(
pipeline=
"audioconvert ! fdkaacenc bitrate=128000 ! queue ! mpegtsmux alignment=7 name=muxer ! queue ! udpsink host=127.0.0.1 port=5000
sync=true",
mksafe(input))

#Video Channels.
#360p Video at 600kbps. Port 5001
output.gstreamer.video(
pipeline=
"videoscale ! video/x-raw,width=480,height=360 ! videoconvert ! x264enc bitrate=600 key-int-max=72 pass=qual quantizer=20 tune
=zerolatency ! video/x-h264,profile=baseline ! queue ! mpegtsmux alignment=7 name=muxer ! queue ! udpsink host=127.0.0.1 port=5001 s
ync=true",
mksafe(input))

#480p Video at 1000kbps. Port 5002
output.gstreamer.video(
pipeline=
"videoscale ! video/x-raw,width=640,height=480 ! videoconvert ! x264enc bitrate=1000 key-int-max=72 pass=qual quantizer=20 tun
e=zerolatency ! video/x-h264,profile=main ! queue ! mpegtsmux alignment=7 name=muxer ! queue ! udpsink host=127.0.0.1 port=5002 sync
=true",
mksafe(input))

#720p Video at 3000kbps. Port 5003
output.gstreamer.video(
pipeline=
"videoscale ! video/x-raw,width=1280,height=720 ! videoconvert ! x264enc bitrate=3000 key-int-max=72 pass=qual quantizer=20 tu
ne=zerolatency ! video/x-h264,profile=main ! queue ! mpegtsmux alignment=7 name=muxer ! queue ! udpsink host=127.0.0.1 port=5003 syn
c=true",
mksafe(input))
Shaka Packager Command

The packager will take the UDP output from Liquidsoap and encapsulate it into the DASH format for playback via the Shaka Player or any other DASH player.

$packager \
'in=udp://127.0.0.1:5000?interface=127.0.0.1,stream=audio,init_segment=audio_init.mp4,segment_template=audio_$Number$.m4s' \
'in=udp://127.0.0.1:5001?interface=127.0.0.1,stream=video,init_segment=h264_360p_init.mp4,segment_template=h264_360p_$Number$.m4s' \
'in=udp://127.0.0.1:5002?interface=127.0.0.1,stream=video,init_segment=h264_480p_init.mp4,segment_template=h264_480p_$Number$.m4s' \
'in=udp://127.0.0.1:5003?interface=127.0.0.1,stream=video,init_segment=h264_720p_init.mp4,segment_template=h264_720p_$Number$.m4s' \
--temp_dir /home/user/tmp -mpd_output test_h264.mpd

That is pretty much all there is to it. You could add higher or lower frame size and bitrates if you wanted to. However, this configuration is pushing the limits of Liquidsoap.