April is here!

So much has happened since the last update, we've migrated to a new server, we've failed to update to a new lemmy version, automated our rollouts, fought with OVH about contracts. It's been a lot.

Strap in for story time about the upgrade, or skip till you see the break for the next section.

So good news is that we are successfully on v0.19.11.

The bad news is that we had an extended downtime.

Recently I had some extra time to completely automate the rollout process so Reddthat didn't rely solely on me being on 1 specific computer which had all the variables that was needed for a deployment.
As some people know I co-manage the lemmy-ansible repository. So it wasn't that hard to end up automating the automation. Now when a new Version is announced, I update a file/files, it performs some checks to make sure everything is okay, and i approve and roll it out. Normally we are back online within 30 seconds as the lemmy "backend" containers do checks on start to make sure everything is fine and we are good to go. Unfortunately it never came back up.

So I reverted the change thinking something was wrong with the containers and the rollout proceeded to happen again. Still not up :'( Not having my morning coffee and being a little groggy after just waking up.

Digging into it our database was in a deadlock. Two connections were attempting to do the same but different which resulted in it being locked up and not processing any queries.

Just like Lemmy World, when you are "scaling" sometimes bad things can happen. re: https://reddthat.com/post/37908617.

We had the same problem. When rolling out the update two containers ended up starting at the same time and both tried to do the migrations instead of realising one was already doing them.

After quickly tearing it all down. We started the process of only having 1 container to perform the migration and then once that had finished starting everything else we were back online.

Going forward we'll probably have to have a brief downtime for every version to ensure we don't get stuck like this again. But we are back-up and everything's working.

Now for the scheduled programming.

OVH

OVH scammed me out of the Tax on our server renewal last month. When our previously 12 month contract was coming to the end we re-evaluated our finances and were found wanting. So we ended up scaling down to a more cost-effective server and ended up being able to pay in AUD instead of USD which will allow us to stay at a single known price and not fluctuate month to month.
Unfortunately I couldn't cancel the contract. The OVH system would not let me click terminate. No matter what I did, what buttons I pressed, or how many times I spun my chair around it wouldn't let me cancel. I didn't want to get billed for another month when we were already paying for the new server. So a week before the contract ended I sent a support ticket to OVH. You can guess how that went. The first 2 responses I got from them after 4 days was "use the terminate feature". They didn't even LOOK at the screenshots clearly outlining the steps I had taken and the generic error... So I get billed for another month... and then have to threaten them with legal proceedings. They then reversed the charge. Except for the Tax. So I had to pay 10% of the fee to cancel our service. Really unhappy with OVH after this ordeal.

Automated rollouts

I spent some time after our migration ensuring that we have another system setup which will be able to rollout updates. So we are not dependant on just me and my one random computer :P All was going very well until an upgrade with database migrations happened. We'll be working on that soon to make sure we don't have unforeseen downtime.

Final Forms

Now that the dust has settled and we've performed the migrations starting next month I'll probably go back to our quarterly updates unless something insane happens. (IE: Lemmy drops v1 👀 )

We also modified our "Reddthat Community and Support" community to be a Local Only community. The original idea for the community was to have a place where only reddthat could chat, but back when we started out that wasn't a thing! So now if you want to voice your opinion to other Reddthat users please feel free too knowing other instances won't come in and derail the conversation.

As a reminder we have many ways to donate if you are able and feel like it! A recurring donation of $5 is worth more to me than a once of $60 donation. Only because it allows me to forecast the year and work out when we need to do donation drives or relax knowing everything in it's current state will be fine.

• Open Collective (preferred)
• Kofi
• Litecoin Direct
• Monero Direct (preferred)

Cheers,

Tiff

Could be worse, I could be using parquet...

We just successfully upgraded to the latest Lemmy version, 0.9.10, probably the last before the v1 release.

This addresses some of the PM spam that everyone has been getting. Now when that user is banned and we remove content it also removes the PMS. So hopefully you won't see them anymore!

Over the next couple days will be planning for our migration to our new server as our current server's contract has ended. I expect the down time to last for about an hour, if not shorter. You'll be able to follow updates for the migration by our status page at https://status.reddthat.com/

Normally this update would be a week in advance and more nicely formatted that turns out the contract ends on the 25th and I don't want to get charged for another month at a higher rate when I just purchased the new server.

See you on the other side,

Tiff

EDIT:

22 Mar 2025 02:42: I'm going to start the migration in 5 mins (@ 3:00)

22 Mar 2025 03:01: that was the fastest migration I've ever done. pre-seeding the server and Infra as Code is amazing!

We've turned off our crypt donation p2pool (as no-one was using it), and two of our frontends, alexandrite and next (for the same reasons)

Time to celebrate with some highly accurate Australian content:

Hello Reddthat! We are back for another update on how we are tracking. It's been a while eh? Probably because it was such smooth sailing!

In the middle of February we updated Lemmy to v0.19.9 which contained some fixes for federating between Mastodon and Lemmy so hopefully we will see less spam and more interaction from the larger mastodon community. While that in of itself is a nice fix, the best fix is the recent thumbnail fix! Thumbnails now have extra logic around generating them and now have a higher chance of actually being created! Let us know if you think there has been a change over the past month-ish.

Budget & Upcoming Migration

Reddthat has been lucky to have such a great community that has helped us stay online for over a year and if you can believe it, in just a few more months it will be 2 years, if we can make it.

Our costs have slowly increased over the years as you can all see by our transactions on OpenCollective (https://opencollective.com/reddthat). We've managed to reduce some costs in our S3 hosting after it balooned out and bring it down to a more manageable level. Unfortunately as well, the current economic issues have resulted in the Australian dollar slipping further and as we pay everything in USD or EUR it has resulted in slightly higher costs on a month-to-month basis..

Our best opportunity to keep online for the foreseeable future is to downsize our big server from a 32GB ram instance to a 16GB ram instance which will still provide enough memory that we will be able to function as we currently do while not affecting us in a meaningful way.

This means we'll need to reassess if running all our different front ends are useful, or do we only choose a few? Currently I am looking to turn off next and alexandrite. If you are a regular user of these frontends and prefer them please let me know as from our logs these are the least used while also take up the most resources. (Next still has bugs regarding caching every single image).

We can get a vps for about ~A$60-70 per month which will allow us to still be as fast as we are now while saving 40% off our monthly costs. This will bring us to nearly 90% funded by the community. We'll still be slowly "losing" money from our open collective backlog but we'll have at least another 6 months under our belt, if not 12 months! (S3 costs and other currency conversion not withstanding).

All of this will happen in late March early April as we will need to make sure we do it before the current contract is up so we don't get billed for the next month. Probably the 29th/30th of I don't fall asleep too early on those days.
It'll probably take around 45mins to 60mins but if I get unlucky maybe 2 hours.

Age Restriction

Effective immediately everyone on Reddthat needs to be 18 years old and futher interaction on the platform confirms you are over the age of 18 and agree with these terms.

If you are under the age of 18 you will need to delete your account under Settings

This has also been outlined in our signup form that has been updated around the start of February.

Australian & UK Policy Changes

It seems the UK has also created their own Online Safety Act that makes it nearly impossible for any non-corporation to host a website with user generated content (USG). This is slightly different to the Australian version where it specifically targets Social Media websites.

Help?

I would also like your help!
To keep Reddthat online, and to help comply with these laws, if you see content or user accounts which are under the age of 18 please report the account/post/content citing that the user might be under the age of 18.
We will then investigate and take action if required.

Thanks everyone

As always keep being awesome and having constructive conversations!

Cheers,

Tiff!

PS. Like what we are doing? Keep us online for another year by donating via our OpenCollective or other methods (crypto, etc) via here

A nice in-depth article on game hacking

My talk explores the trajectory of iOS spyware from the initial discovery of Pegasus in 2016 to the latest cases in 2024.

The talk will start with an analysis how exploits, infection vectors and methods of commercial spyware on iOS have changed over time.

The second section of the talk is all about advances in detection methods and the forensic sources which are available to discover commercial spyware. This talk will also include a Case Study about the discovery and analysis of BlastPass (one of the latest NSO Exploits).

The third part will discuss technical challenges and limitations of the detections methods and data sources.

Finally, I will conclude the talk with open research topics and suggestions what Apple or we could technically do to make the detection of commercial spyware better.

The commercial spyware landscape on iOS has evolved significantly since the discovery of Pegasus in 2016. In this talk, we’ll explore that evolution through four main areas:

1. Spyware Evolution (2016-2024): By analyzing key exploits, tactics, techniques, and procedures (TTPs), infection vectors, and indicators of compromise (IOCs), we’ll trace how spyware has advanced in sophistication, highlighting changes that have led to today’s complex threats.
2. Advancements in Detection: As spyware has grown more sophisticated, so too have detection capabilities. We’ll review the main actors, public organizations and tools that have shaped spyware detection. This part will also include a case study on my discovery and analysis of a sample NSO‘s BlastPass Exploit chain.
3. Current and Future Challenges: Looking forward, we’ll examine the pressing challenges in spyware detection and speculate on how commercial spyware might evolve in response to new security measures and technologies.
4. Recommendations for Research and Detections: Finally, I’ll offer recommendations for advancing research and detection methods and capabilities to combat commercial spyware.

Attendees will gain a comprehensive view of the past, present, and future of spyware on iOS, along with actionable strategies for future research and collaboration.

Licensed to the public under http://creativecommons.org/licenses/by/4.0

We present fatal security flaws in the HALFLOOP-24 encryption algorithm, which is used by the US military and NATO. HALFLOOP-24 was meant to safeguard the automatic link establishment protocol in high frequency radio, but our research demonstrates that merely two hours of intercepted radio traffic are sufficient to recover the secret key. In the talk, we start with the fundamentals of symmetric key cryptography before going into the details of high frequency radio, HALFLOOP-24, and the foundation of our attack.

High frequency (HF) radio, also known as shortwave radio, is commonly used by the military, other government agencies and industries that need highly robust long-distance communication without any external infrastructures. HF radio uses frequencies between 3 and 30 MHz. These frequencies enable skywave propagation, where the radio signals are reflected by electrically charged particles in the upper atmosphere. While this effect enables communication across very large distances, historically, it required trained and experienced operators to establish a radio link.

This dependence on operators was reduced by the introduction of the automatic link establishment (ALE) protocol. In a nutshell, an ALE-enabled radio establishes a link to another radio by selecting a suitable frequency according to a propagation model and then transmitting a call frame. If the frequency is good, the other radio receives the frame and the two radios perform a handshake to set up a link. The encryption of these ALE frames is known as linking protection. It is primarily meant to protect unauthorized users from establishing links with radios in a network or interfering with established links. Additionally, encryption of ALE frames also protects the network from certain types of traffic analysis, which is the analysis of operating data such as network structure, frequencies, callsigns and schedules. The first ALE standard did not specify a cipher, but specified how to integrate a stream cipher with ALE. Later standards introduced the 56-bit key Lattice/SoDark cipher, which is now recommended to be replaced with HALFLOOP whenever possible.

HALFLOOP, which is standardized in US standard MIL-STD-188-14D since 2017, is essentially a downscaled version of the Advanced Encryption Standard (AES), which effectively is the most used encryption algorithm today. While this downscaling led to many strong components in HALFLOOP, a fatal flaw in the handling of the so-called tweak enables devastating attacks. In a nutshell, by applying a technique known as differential cryptanalysis, an attacker can skip large parts of the encryption process. In turn, this makes it possible to extract the used secret key and hence enables an attacker to break the confidentiality of the ALE handshake messages and also makes an efficient denial-of-service attack possible.

These attacks are described in the two research papers, Breaking HALFLOOP-24 and Destroying HALFLOOP-24. They were initiated by the presentation of the Cryptanalysis of the SoDark Cipher, the predecessor of HALFLOOP.

Licensed to the public under http://creativecommons.org/licenses/by/4.0

What's your plans for today, tomorrow, and the next year?