Categories
Fedora

Anti-Spam through an IMAP Account

As we are increasingly dealing with an aging population, this adds technological problems. My elderly mother has developed dementia and has lost any technology knowledge that she had other than basic Windows operating system fundamentals. Her primary access to the Internet is via Outlook (the e-mail client) and a web browser. In her diminished capacity of memory and cognitive abilities she will open nearly every single e-mail that she gets. And does she get them. Roughly around 1,000 e-mails a day. With big ISPs (Comcast) allowing willy nilly whatever to come through their mail servers this is a disaster waiting to happen.

Well, why not just install some anti-spam for Outlook application you ask? Ah you mean the 2000’s era software that no longer works or is available? Yeah, that is why. The worthless Anti-Virus software packages ‘Anti-Spam’ has a hit rate of about 0.01%. Super effective isn’t it? It’s great for detecting viruses, not for spam e-mails. Which makes sense. It’s Anti-Virus. Entirely different thing.

Users with disabilities like dementia (again memory loss) is challenging as anything that changes can cause serious issues. I can’t just up and replace the Outlook application she has been using for years with something that may have better built in anti-spam.

Again, I cannot just switch over to a mail server I manage. She’s had her Comcast e-mail address for decades now. That is what she knows. Creating some elaborate forwarding system is just asking for trouble managing that. And yes, this would be the most ideal way of solving the problem. And I highly recommend it. Especially if you are in a position where the person with dementia onset is early and you can move over to an e-mail system that you have control over. The best place for anti-spam is before it’s acceptance and delivery to the user. And this is how most of the solutions migrated to.

Alas I do not have the ability to control what Comcast accepts as e-mail before delivery. But, I have found a middle ground solution that appears to work well. Unfortunately it has some caveats. As I mentioned, this is an old way of dealing with spam e-mail, so the software has been nearly abandoned. But there is still hope.

The solution is relatively simple in a complex system. Logging into the IMAP service, fetching and scanning the e-mail via Spamassassin anti-spam application, marking the e-mails that are detected as spam, moving them to a Spam/Junk folder (and optionally just deleting them). The solution does not require running any software on the users computer itself, although it can be. But I wouldn’t recommend it in this scenario of a dementia patient. It will need to be running on a separate (remote) computer (a VM perhaps hosted somewhere).

The solution I found is just two software applications. A python based application called ISBG (IMAP Spam Begone) and Spamassassin.

For Fedora Linux you will want to install Spamassassin.

$ sudo dnf install spamassassin

Then make sure the python installer pip is installed.

$ sudo dnf install pip

Then install isbg.

$ pip install isbg

Well, that’s it! NOT! Did you think it was going to be that easy? As I mentioned earlier, this way of anti-spam has been pushed as a thing of the past. So, the isbg software was yeeted by the developers years ago. Probably beause only 4 people in the world used it as no one really knew what to use it for. And because the isbg application is written in Python which changes every version release. And don’t get me wrong, Python is super cool and you should learn it.

When writing this article, the current version of isbg is v2.3.1. I have no idea which version of Python it was initially released for, but the current version of Python is v3.12.5. The isbg application will immediately fail with:

$ isbg
Traceback (most recent call last):
File "/home/<yourusername>/.local/bin/isbg", line 5, in <module>
from isbg.__main__ import main
File "/home/<yourusername>/.local/lib/python3.12/site-packages/isbg/__init__.py", line 10, in <module>
from .isbg import ISBG, ISBGError, __version__, __exitcodes__, __license__
File "/home/<yourusername>/.local/lib/python3.12/site-packages/isbg/isbg.py", line 23, in <module>
from isbg import imaputils
File "/home/<yourusername>/.local/lib/python3.12/site-packages/isbg/imaputils.py", line 40, in <module>
Email = TypeVar(email.message.Message)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: typevar() argument 'name' must be str, not type

Awesome! Right? Just what you wanted in your journey to hell (if you’ve gotten this far you’ve been there for awhile). But have no fear. This is actually a fairly simple solution. Mind boggling, I know. You just need to edit the file below at line 40. Use whatever text editor you are familiar with.

/home/<yourusername>/.local/lib/python3.12/site-packages/isbg/imaputils.py

Line 40:
Email = TypeVar(email.message.Message)

Change To:
Email = TypeVar(str(email.message.Message))

And now the isbg application will work. It will still have some issues with specific malformed e-mail’s and it does not appear to work with the –spamc option.

You will now need to get the username, password and the IMAP server address for the e-mail account you’ll want to scan. The e-mail account will also need a Spam or some sort of Junk folder to move the spam to already created. isbg will not create the folder. In this example, my moms account already had Junk folder, so that is what I am using. By default it is Spam. If isbg does not detect this, it will exit with an error saying so. This example also uses Comcast as the IMAP. The first run of the isbg command you will want to use the –savepw (Save Password) command, so subsequent commands will not prompt you for the password. Note that the password is saved in plain text. There’s no getting around that. isbg will by default only scan 50 e-mails per run. We’ll keep it at that for the first run.

isbg --verbose --imaphost imap.comcast.net --imapuser <yourmom>@comcast.net --spaminbox Junk --savepw

Again, if you have the default Spam folder for spam e-mails, remove the –spaminbox command. Without any errors it will login, and scan through the messages. It’s a bit on the slow side so many take a couple of minutes.

Now that it has completed you can add the better options I have found. I do not recommend running isbg wide open (meaning scanning all e-mails in a single run). It will crash on specific malformed e-mails and you will lose any sort of progress made. It only does it’s write operations after all e-mails have been scanned. I’ve found a happy medium of 250 e-mails to be acceptable. (And yes we’re talking about someone who gets several hundred to a thousand e-mails per day.) Here is the new command.

isbg --verbose --imaphost imap.comcast.net --imapuser <yourmom>@comcast.net --spaminbox Junk --maxsize 512000 --partialrun 250 --flag

The maxsize is what maximum size e-mail to fetch from the IMAP server. isbg defaults to a considerably low value. Most spam these days are much larger than it. So I put the max e-mail size to 512kbytes. The partialrun command is where you set how many e-mails are processed during this session. 0 means all of them. I do not recommend this. I set this to 250. Enough where I can cover a large group of e-mails but not lose an hour and a half of processing if isbg comes across a malformed e-mail message. (Remember we don’t have control of the delivery.) The flag command tells isbg what to do after the scan is done. Prepend the subject with [SPAM] and the score of it and move it into the Junk folder (the setting from the spaminbox).

That is really it. You can create a crontab to run this command however often. I would not recommend minutes but four times a day. Depends on the volume of e-mail coming into the account.

Anyways, that about wraps it up for this. I wrote this to mainly remember what I changed to make the isbg program to work. This was written by a real person with no AI involved. I will edit this at a later time after publication with attributions.

Welcome to Costco, I love you.

Categories
Other

Liquidsoap – Multi-Bitrate DASH streaming configuration

This is an example of multi-bitrate DASH streaming configuration. In this example Liquidsoap will be configured to output three different MPEGTS video streams with various frame size, bitrates and one stereo audio MPEGTS stream all via  UDP.  This configuration can be used with the same setup as my previous blog post about setting up Liquidsoap MPEGTS DASH streaming. This example is using Liquidsoap v1.3.3.

Please note that this is more of a proof of concept than practical use. A hardware encoder plugin for Gstreamer would help.

Liquidsoap Configuration: MPEGTS via UDP for Multi-Bitrate DASH Streaming
set("log.stdout",true)
set("log.level",5)
set("gstreamer.debug_level",2)

set("frame.video.width",1280)
set("frame.video.height",720)
set("frame.video.samplerate",25)
set("gstreamer.add_borders", true)

input = single("/home/user/Videos/Test/test-1280x720-25fps.ogv")

#MPEGTS Output For DASH Streaming.
#You must start Shaka Packager first before starting Liquidsoap.

#Audio Channel. Port 5000
output.gstreamer.audio(
pipeline=
"audioconvert ! fdkaacenc bitrate=128000 ! queue ! mpegtsmux alignment=7 name=muxer ! queue ! udpsink host=127.0.0.1 port=5000
sync=true",
mksafe(input))

#Video Channels.
#360p Video at 600kbps. Port 5001
output.gstreamer.video(
pipeline=
"videoscale ! video/x-raw,width=480,height=360 ! videoconvert ! x264enc bitrate=600 key-int-max=72 pass=qual quantizer=20 tune
=zerolatency ! video/x-h264,profile=baseline ! queue ! mpegtsmux alignment=7 name=muxer ! queue ! udpsink host=127.0.0.1 port=5001 s
ync=true",
mksafe(input))

#480p Video at 1000kbps. Port 5002
output.gstreamer.video(
pipeline=
"videoscale ! video/x-raw,width=640,height=480 ! videoconvert ! x264enc bitrate=1000 key-int-max=72 pass=qual quantizer=20 tun
e=zerolatency ! video/x-h264,profile=main ! queue ! mpegtsmux alignment=7 name=muxer ! queue ! udpsink host=127.0.0.1 port=5002 sync
=true",
mksafe(input))

#720p Video at 3000kbps. Port 5003
output.gstreamer.video(
pipeline=
"videoscale ! video/x-raw,width=1280,height=720 ! videoconvert ! x264enc bitrate=3000 key-int-max=72 pass=qual quantizer=20 tu
ne=zerolatency ! video/x-h264,profile=main ! queue ! mpegtsmux alignment=7 name=muxer ! queue ! udpsink host=127.0.0.1 port=5003 syn
c=true",
mksafe(input))
Shaka Packager Command

The packager will take the UDP output from Liquidsoap and encapsulate it into the DASH format for playback via the Shaka Player or any other DASH player.

$packager \
'in=udp://127.0.0.1:5000?interface=127.0.0.1,stream=audio,init_segment=audio_init.mp4,segment_template=audio_$Number$.m4s' \
'in=udp://127.0.0.1:5001?interface=127.0.0.1,stream=video,init_segment=h264_360p_init.mp4,segment_template=h264_360p_$Number$.m4s' \
'in=udp://127.0.0.1:5002?interface=127.0.0.1,stream=video,init_segment=h264_480p_init.mp4,segment_template=h264_480p_$Number$.m4s' \
'in=udp://127.0.0.1:5003?interface=127.0.0.1,stream=video,init_segment=h264_720p_init.mp4,segment_template=h264_720p_$Number$.m4s' \
--temp_dir /home/user/tmp -mpd_output test_h264.mpd

That is pretty much all there is to it. You could add higher or lower frame size and bitrates if you wanted to. However, this configuration is pushing the limits of Liquidsoap.

Categories
Other

Liquidsoap – Streaming DASH via UDP

So now that you have that magical UDP stream going, what to do with it? Why  not a DASH stream using the HTML5 Shaka Player? DASH is D)umb A)ss S)treaming H)ibbtyjibbty also known as the Hard Drive Killer. Cause you ain’t gunna be doing this on a mechanical drive for very long. Who thought the idea of creating millions of tiny files constantly was a good idea? We totally needed a new wheel worse than the one before it, right? I mean, who wants to install various different media players when you now can install various different browsers. Anyways, on to the show!

This is relatively straight forward with a lot of ambiguity.  The path of the stream is as follows; Liquidsoap output MPEGTS via UDP into Shaka Packager. Shaka Packager encapsulates the MPEGTS stream into a DASH stream for playback via a web server and HTML5 clients and other media players that support DASH (such as VLC).  You will want to download the Shaka Packager and Shaka Player.  This is a bit beyond the scope of this article oh how to get the Shaka Player compiled, but just trudge through it.

Liquidsoap Configuration

First part we need to configure Liquidsoap. This will be a basic configuration with the key parts being the output settings. Most notably and very important is that the Shaka Packager does not support the RTP container so it’s MPEGTS flat out.

set("log.stdout",true)
set("log.level",5)
set("gstreamer.debug_level",2)

set("frame.video.width",512)
set("frame.video.height",384)
set("frame.video.samplerate",25)
set("gstreamer.add_borders", true)

# The input file,
# any format supported by liquidsoap
input = single("/home/user/Videos/Test/video-720x576-25fps.ogv")

#MPEGTS via UDP Streaming (Shaka Packager doesn't support RTP container).
output.gstreamer.audio_video(
 video_pipeline=
 "videoconvert ! x264enc pass=qual quantizer=20 tune=zerolatency ! video/x-h264,profile=baseline ! queue ! muxer.",
 audio_pipeline=
 "audioconvert ! fdkaacenc bitrate=128000 ! queue ! muxer.",
 pipeline=
 "mpegtsmux alignment=7 name=muxer ! queue ! udpsink host=127.0.0.1 port=5000 sync=true",
 mksafe(input))

The Shaka Packager doesn’t support MPEGTS in RTP via UDP. If you encapsulate your MPEGTS stream via RTP you will get this error from packager.

[0522/093109:INFO:demuxer.cc(89)] Demuxer::Run() on file 'udp://127.0.0.1:5000?interface=127.0.0.1'.
[0522/093109:INFO:demuxer.cc(161)] Initialize Demuxer for file 'udp://127.0.0.1:5000?interface=127.0.0.1'.
[0522/093116:ERROR:demuxer.cc(207)] Not implemented reached in shaka::Status shaka::media::Demuxer::InitializeParser()
[0522/093116:ERROR:packager_main.cc(477)] Packaging Error: 4 (UNIMPLEMENTED): Container not supported.
Shaka Packager

The Shaka Packager is just a fairly simple program. I just downloaded the binaries and put them in my /home/user/bin/.

In this scenario and since we are using UDP as the transport, the Shaka Packager will need to be started first and listening for the MPEGTS stream packets BEFORE Liquidsoap starts. This is an important and key step in this.

For this example we will start the Shaka Packager program in the root web directory of our website the DASH manifest and media files will be created in this directory.

[/var/www/html/]$packager \
 'in=udp://127.0.0.1:5000interface=127.0.0.1,stream=audio,init_segment=audio_init.mp4,segment_template=audio_$Number$.m4s'  \ 'in=udp://127.0.0.1:5000interface=127.0.0.1,stream=video,init_segment=h264_384p_init.mp4,segment_template=h264_384p_$Number$.m4s' \
--mpd_output h264.mpd

The Shaka Packager is fairly straight forward. Since the Audio and Video is in a single stream we call both streams for our audio and video.  It will create all the necessary media files and the MPD manifest file in the current directory.  Once Shaka Packager is running it will just print out that it’s initialized the demuxer for input from the UDP stream.  It is now time to start Liquidsoap.

Once you start Liquidsoap, the Shaka Packager doesn’t announce anything so it is difficult to know that it is working.  With this current setup after about a minute or two the Shaka Packager will spit an error similar to this.

[0522/095415:WARNING:representation.cc(384)] Found a gap of size 343 > kRoundingErrorGrace (5). The new segment starts at 324900702 but the previous segment ends at 324900359.

If you get this error then you know it’s working. As long as the gap size doesn’t grow the stream should be good.

Shaka Player

So now that you got the stream files all going, you could point VLC to the http://localhost/h264.mpd file and VLC should be happy to play the DASH stream.  But now onto the Shaka Player. It is beyond the scope of me to tell you how to install it. But I feel you.

Following the basic guide for the Shaka Player, just copy the shaka-player.compiled.js script into the same directory as the media files. Then create a simple index.html file.

<!DOCTYPE html>
<html>
<head>
<!-- Shaka Player compiled library: -->
<script src="shaka-player.compiled.js"></script>
<!-- Your application source: -->
<script src="myapp.js"></script>
</head>
<body>
<video id="video" width="640" poster="//shaka-player-demo.appspot.com/assets/poster.jpg" controls autoplay></video>
</body>
</html>

Then you will want to create the myapp.js file which is essentially a kind of configuration and initialization file for the Shaka Player. This is the file that points to the DASH manifest (MPD) file that the Packager has created.

// myapp.js

var manifestUri = '//localhost/h264.mpd';

function initApp() {
// Install built-in polyfills to patch browser incompatibilities.
shaka.polyfill.installAll();

// Check to see if the browser supports the basic APIs Shaka needs.
if (shaka.Player.isBrowserSupported()) {
// Everything looks good!
initPlayer();
} else {
// This browser does not have the minimum set of APIs we need.
console.error('Browser not supported!');
}
}

function initPlayer() {
// Create a Player instance.
var video = document.getElementById('video');
var player = new shaka.Player(video);

// Attach player to the window to make it easy to access in the JS console.
window.player = player;

// Listen for error events.
player.addEventListener('error', onErrorEvent);

// Try to load a manifest.
// This is an asynchronous process.
player.load(manifestUri).then(function() {
// This runs if the asynchronous load is successful.
console.log('The video has now been loaded!');
}).catch(onError); // onError is executed if the asynchronous load fails.
}

function onErrorEvent(event) {
// Extract the shaka.util.Error object from the event.
onError(event.detail);
}

function onError(error) {
// Log the error.
console.error('Error code', error.code, 'object', error);
}

document.addEventListener('DOMContentLoaded', initApp);

You should now have a functioning DASH stream with Liquidsoap.  This is a basic example and can be expanded on with enough CPU resources and x264 hardware encoding via GStreamer.  The idea of DASH is to have multiple bit rates at various frame sizes so that you can cover a wide variety of client types and connectivity.  I haven’t tested this out, but you could start with a larger frame size setting in Liquidsoap then create multiple video outputs with the various frame sizes. You will only need one audio output.

Have fun!

Categories
Fedora Other Sendmail

Sendmail – How To Deliver To IPv4 Address Per Domain

More mail servers are now accepted e-mail via IPv6.  I have had a dynamically assigned IPv6 block on my Comcast Business account for awhile and I have let Sendmail decide what to use, and about 99.9% of mail is delivered via IPv4.  Just recently it appears Comcast has assigned an IPv6 MX record for their mail server. My Sendmail picked this up and now happily attempts to deliver the mail via the IPv6 address.  Unfortunately, it is immediately rejected due to the IPv6 address does not have a PTR record.  Of course Comcast Business is far behind on assigning IPv6 blocks so there is no way to get a static IPv6 block and a PTR entry.

How do I get Sendmail to deliver to the IPv4 address instead?  It’s called the mailertable feature..  You will need this feature enabled in your sendmail.mc file. Most likely it is already enabled.

/etc/mail/sendmail.mc

FEATURE(`mailertable')

 

Now you need to make an entry into the mailertable file with the domain and IPv4 address. In order to get the IPv4 MX address for the domain you can do so by using the host command. We first look up the main domain name to get the MX records. Then lookup the IPv4 address for the MX record.  We now have the IPv4 address to where we want to deliver the mail.

[root@superstar ~]# host comcast.net
comcast.net has address 69.252.80.75
comcast.net mail is handled by 5 mx2.comcast.net.
comcast.net mail is handled by 5 mx1.comcast.net.
[root@superstar ~]# host mx1.comcast.net
mx1.comcast.net has address 96.114.157.80
mx1.comcast.net has IPv6 address 2001:558:fe16:1b::15

 

We now add these lines to our mailertable file.

/etc/mail/mailertable

.comcast.net     esmtp:[96.114.157.80]
comcast.net     esmtp:[96.114.157.80]

 

Don’t forget to issue make to update the db files for Sendmail to see the changes to the mailertable file. And then restart Sendmail.  It will now deliver to the specific IPv4 address.

[root@superstar mail]# make
[root@superstar mail]# service sendmail restart
Redirecting to /bin/systemctl restart  sendmail.service
You bet there is a catch! If the IPv4 address changes, you will need to manually make the change.
That’s it all there is to this. Sendmail is now delivering to the IPv4 address.