Fiber's shitting the bed, apparently a cut. We're on backup WAN. ETA from ISP is 5am tomorrow

Upgraded the backup connection last week, so will leave pict-rs enabled for now and see how things go.

Update: Fiber came back online around 4:30 this morning and the auto fallback seems to have worked. Yay.

This is the second time today I've had to cut over to the backup WAN (which is much slower). Fiber provider is royally screwing the pooch today, and I currently have no ETA. The only silver lining is it's a widespread issue, so hopefully they're inclined to address the problem in a timely manner.

To relieve congestion on the backup connection, I've temporarily disabled external access to pictrs. All requests (GET or POST) to /pictrs will return a 404 response until the main connection is back online.

Sorry for the inconvenience. Please direct all rage at the shit fiber provider I foolishly "upgraded" to.

Status as of 5:17PM:

Update: 4:30 PM: Tried to call and re-activate my account on my old provider since I own the equipment, and it's all still hooked up. I can even still access my old account on the web portal. Should be easy, right? Fucking wrong! They said they couldn't activate it until they sent me equipment, I returned it, and then called back to say I want to use my own. What the actual fuck?? I'm literally waving money at you, you're still charging me a $100 "install" fee (where you have to do nothing), and all you have to do is take it (and my equipment serial number). Fuck my life (and fuck those idiots, too).

Update 5:17 PM: We remain on backup WAN. ETA for fiber restoration is within 24 hours, but I'm not holding my breath. Like, how do you even fuck up this badly? (the whole goddamned state is affected including business customers)

Update 7:39 PM: Primary connection is back online (for now?). Services have successfully failed back over.

Appalachian Power is the absolute worst about giving notifications for scheduled outages. The. Fucking. Worst.

I got an email at at 8:05 PM saying there would be a planned, scheduled outage at 8:00 PM and then 45 seconds later was plunged into darkness. Didn't bother to switch on the generator :sigh: and just decided to sit on the patio and get drunk in the waning daylight.

This is not the first time they've done that, and it probably won't be the last. Guess my next purchase is an automatic transfer switch.

Did I say "fuck Appalachian Power"? Because Fuck. Appalachian. Power.

Sorry about that. lol

Let me start off by saying, in my official admin capacity, that I really don't know if I want to be here anymore (see the problem outlined below). That has implications beyond me just deleting my account and going outside since I would not allow the instance to run unattended. I'm also unsure if my backup admin would have any interest in taking over full time.

If it does come to that, there will be a subsequent announcement and either a transition plan or a sunset plan.

That is still up in the air, but for now....

The Problem

I just wanted to take a moment here to address Lemmy's extremism problem / lynch-mob mentality and how DubVee is responding to that.

Last month, a new site rule was added expressly forbidding any form of extremism with regards to violence. Every rule is there for a reason, and this was added as a direct response to the increasingly violent rhetoric I've been seeing from users.

In full:

I don't care which end of the spectrum you're on. Any post/comment calling for or glorifying violence, especially political violence, will be removed. Offenders will be dealt with on a case-by-case basis, but bans will be the norm and swift for violating this rule.

After running reports against the database, there have been a very large number of bans issued in the last few days for both explicit and implicit violations of that rule:

Explicit Extremism

• Explicitly praising or justifying acts of violence
• Explicitly justifying / advocating for violence / "revolution" / whatever
• Suggesting, justifying, or advocating political violence, vigilantism, and/or breaking with the foundational principles of this country in order to "solve" a problem.
• Hypocritical posts/comments that amount to "Fascism / authoritarianism / whatever -ism is bad, but a little bit is okay when it's my side doing it"
• Stirring the pot by jumping into otherwise civil discussions and suggesting acts of violence (I'll admit, those were mostly new accounts and likely provocateur trolls seizing on current events, but established accounts are also guilty)

Implicit Extremism

• Patterns of upvoting or boosting posts/comments that explicitly praise, endorse, or advocate for violence (or any of the 'explicit' items above)
• Patterns of downvoting posts/comments that denounce violence or those that advocate for lawful/peaceful solutions.

Note: Before you cry "thought crime!" about issuing bans based on vote patterns, it is a well-known tactic for trolls to operate bot or sockpuppet accounts to manipulate votes to serve an agenda. Owing to the federated nature of Lemmy, this becomes easier to do and harder to detect since those can be distributed among different instances that have varying moderation, signup, monitoring, and administrative policies and procedures. So, no, I have no compunction about banning accounts that express support for extremism by way of voting; there's simply no straightforward way to tell bot from person from my point of view.

Even accounting for gallows humor, I must reiterate that extremism and violence are absolutely forbidden, and there will be zero tolerance shown.

While I'm aware there's a certain "mob mentality" inherent to social media in general, "lynch-mob mentality" will not be tolerated; we're better than that, and DubVee is not here to spread hate, violence, or fan the flames.

In Conclusion

If it turns out that we end up blocking 90% of the Fediverse because of this, then I'll just shut this whole instance down. DubVee, by way of federation, will not devolve to 4chan (or 8chan or wherever the shitty little edgelords post their green text nowadays), and I will (figuratively) burn it to the ground before it gets to that point.

What Can You Do as a User?

First and foremost, make sure you report any content that violates this rule and refrain from violating it yourself.

If you're seeing less comments than you were a few days ago or are no longer seeing content from certain users, this is most definitely why. You can check the modlog to see if someone you're expecting to see has been banned.

If you (local user) wish to challenge a ban and advocate for someone who has been blocked, you may send me a DM with your request, and I will review the account further (accounts were reviewed prior to banning, but it never hurts to take a second look).

If you have a problem with this rule and our enforcement of it, there's plenty of other instances you can choose from.

This Weekend

There will be some periodic downtime this weekend as I do some physical cleanup / rearrangement to my equipment. I'll try to keep these to a minimum, but at some point, everything is going to need to go offline for a bit in order to re-cable the UPS and power connections.

Plan is Saturday afternoon between 1 and 4 pm. Hopefully any interruptions will be brief.

Next Week

I'm switching to a new fiber provider next week (FTTH vs current FTTN), so there will be a brief disruption as I cut over to that. That downtime should only be a matter of seconds since both connections will be active, but just in case it turns into a whole thing, I'm mentioning it now. The new fiber is supposed to be installed Tuesday afternoon, and I plan to do the cutover around 7PM local time.

Next Few Months

I'll be expanding the storage server in the coming weeks, but that should not have any immediate impact on DubVee's normal operation.

With these, I hope to provide a better, faster experience. I'm also looking into setting up an Invidious instance to use with DubVee as well as exploring setting up and integrating a Peertube server.

I've been working on the latest release of Tesseract for the last few weeks, and I'm happy to announce the first beta release of the upcoming 1.4.0 series.

I daily drive my dev version throughout the development process, so all the bugs I've encountered/introduced should be fixed. The beta test on the main instance is more of a formality and final shakedown. That said, please report any bugs you encounter either through Github or by describing them in a comment here.

1.4.0 is a significant update as it drops all backwards compatibility with the 0.18.x API and introduces several new features. It's also had some additional polish since 1.3.0 and added things that have been requested for quite some time (pasting images into post/comments, etc).

As always: thanks in advance for being my beta testers. Appreciate it!

Change log for 1.4.0 if you want to see what's new: https://github.com/asimons04/tesseract/blob/1.4.0/ChangeLog.md

If you've noticed content from Lemmy World lagging by a few hours the past few days, it's not just you.

Long story short, it's a problem with how Lemmy sends activities, and it's heavily impacted by latency between sending and receiving server and creates a fixed upper limit on the number of activities per second that can be sent. Lemmy World is hosted in Finland, and DubVee on the US east coast. There's only so much I can do to work around the inherent latency of a trans-Atlantic link.

We're not alone in this. Some instances, particularly in Australia, New Zealand, and US west coast, are impacted more dramatically.

While there's always some federation delay/backlog, it only escalates to noticeable levels every so often. Not sure why, but I've gone over our infrastructure top to bottom several times and cannot find any reason, on this end, for these events. The graph in the post shows the number of activities Dubvee is lagging behind Lemmy World over the last 30 days.

Normally, it's 300-500 activities which usually corresponds to a minute or less of lag between LW sending an event and DubVee processing it. Occasionally, when LW gets busy, we see spikes into the 3000-5000 range (~5-7 minutes of lag). Every so often, though, there will be huge backlog events (the spikes from 15K to 35K) which often take 8-12 hours to catch up. A month or two ago, I think the largest spike was around 180,000 (though that was a separate issue).

I believe this is being addressed in Lemmy itself, but it'll be a while before it's ready (and I'm certainly going to let other instances kick the tires before upgrading).

In the mean time, I've talked with some other admins and have deployed a Federation proxy. I won't go into the nitty-gritty details, but it does seem to be alleviating a lot, but not all, of the congestion. We went from averaging around 10,000 backlogged events to about 2500. So, definitely an improvement.

Update: Buffer has cleared, and things seem to be coming in pretty close to real time. Occasionally the buffer kicks in, but overall, it appears to be helping. Will continue to monitor. Usually 11-12 AM eastern time is when we start to see lag increasing in activities coming from Lemmy World.

Hopefully there's some more optimization I can do in the mean time, and hopefully Lemmy addresses this limitation, but for now, this should make things less bad.

Finally got around to upgrading to 0.19.3.

Upgrade did not go smoothly due to multiple DB migration step failures and less than helpful logging from Lemmy during the DB migrations. (surprised Pikachu).

After digging through to figure out what it was complaining about, the upgrade was finally able to be completed.

You will likely need to log in again (at least I did).

I'm in the process of migrating the pict-rs database to Postgres. Pict-rs has to be offline during this process, so images here will appear to be broken for a bit. You also won't be able to upload any images for the duration of this migration.

Estimated time is about 20-30 minutes.

Currently, we host 4 UIs:

• Tesseract (default UI at https://dubvee.org)
• Mlmym (old Reddit looking UI at https://old.dubvee.org)
• Alexandrite (desktop UI at https://alex.dubvee.org)
• Lemmy-UI (crappy "default" UI at https://lemmy.dubvee.org).

Due to all the bot crawler traffic that slips through, I'm having a hard time distinguishing actual usage from bot traffic. That said, it does seem like Mlmym and Alexandrite are rarely used by actual people (there's a lot of bot traffic to filter out that spoofs its user agent, so I could easily be wrong here).

Update: Mlmym does all the Lemmy API calls server-side, so all of the actual usage traffic was actually originating from my server (which I was filtering out).

I'm looking to decommission ~~Mlmym (the old Reddit style) and~~ Alexandrite.

I just updated both of those to the latest, so apologies for the brief disruption if you were using them, but ultimately I'd like to trim down the selection to just the ones being actively used.

So, poll time: Are you using old.dubvee.org or alex.dubvee.org? If so, please make your voice heard and I'll keep them around. If not, or if I don't hear back, I'll likely decommission them this weekend.

Update: Will keep Mlmym (old Reddit-style). Alexandrite is still up for decommission unless someone is using it as well.

I'm not quite done with the 1.3.0 release of Tesseract, but the last few bits are going to take longer than I expected. So I think I'm going to hold off and add those in a 1.3.x release a bit later.

I've gutted, re-implemented, and just flat-out re-wrote large portions of the application over the last month or two. I've also been daily driving the dev version, and I guess a couple other people were too. That said, it is stable enough for daily use and ready to get some more eyes on it before an official release.

So, please bear with me if you hit any bugs (I've fixed all the ones I've found so far). If you would be willing and kind enough, either submit a Github issue if you find a bug or just throw a reply to this post with as many details as you can provide.

Hope to run a few betas here before officially releasing 1.3.0.

Apologies for the downtime.

We had some severe weather come through on Tuesday that caused extensive damage to power infrastructure throughout the area. I thought my location had been spared the worst of it, but a tree decided to fall on the transmission line servicing this area Wednesday morning (according to the power company, anyway). By that point, there were about 55,000 other outages ahead of my area in line awaiting repairs (no hard feelings there).

Since power outages are rare here, at least ones lasting longer than an hour or two, I only keep about 2 gallons of fuel on hand for the generator. At typical 3/4 load, that usually last about 4-5 hours. Again, long outages are extremely rare here, so that's usually more than enough runtime. Sadly, this outage lasted much longer (even though I was able to stretch the generator runtime by slightly hacking my UPS)

Federated content should now be coming back in, but it'll take a while to catch up.

Lessons learned:

• Mother nature is a badass bitch. I'm going to start keeping at least 2 days of fuel on hand. Worst case, I'll throw it in the car before it goes stale.
• My 48v e-bike battery and stepdown converter can run my UPS and bare-minimum servers for about 6 1/2 hours (probably more; it was only about half charged and had sat untouched on a shelf for over a year). That came in handy and was why we were able to stay online as long as we did.
• My primary network provider is pretty solid. That stayed up longer than I thought it would.
• Failover to the backup WAN works but failing back to primary does not. That's a "me" problem to fix in my watchdog script.

"Why don't you just throw it in the cloud like a sane person?" you may be asking. Well, it is and it isn't already. It's a hybrid setup. The UI and front end caches/proxies are all cloud based but the DB and API are located on prem where I can throw as much resources at them as I want for free.

I've been tempted to move those up, but it would cost more money than I want to devote to Lemmy at the moment (at least if I want to maintain the same level of performance).

Most of my VPS's are at capacity, but I am going to work on setting up a standby VPS that can scale up and keep the most recent backup dump there.

Mostly, I just don't want to have to rely on donations to keep DubVee online. Right now, all of its components are secondary payloads on my existing VPS hosts or are running on-prem on my own hardware (for free, for all intents and purposes).

I'd rather deal with an outage from time to time than have to constantly wonder if I'm going to be able to pay the cloud hosting bills. It's one of the reasons I've envisioned DubVee remaining relatively small.

Weather / power related. May go offline for a bit until power is restored. Currently on UPS.

Update 1: Yep, expect downtime. UPS ran dry and moved over to aux generator. ETA from power company is tomorrow at 11 PM EST.

Update 2: Still no change in ETA. Have had to shed some load from the generator because my UPS's refuse to charge from it and every time the fridge kicks on, a random piece of my equipment would reboot :sigh:.

Hopefully my reduced power budget will stabilize that.

On the bright side, the primary network connection has remained up and haven't had to switch over to the (much slower) backup.

Update #3: (17:06) Generator running on fumes now and will probably be going offline soon to ensure a safe shutdown and save some fuel to cool the fridge tomorrow if power is still out. Apologizes in advance. I usually have very reliable power here and typically never need the generator for more than a few hours.

Update #4 (04/04/2024 14:48): Power restored 12 hours ahead of ETA (good work AEP crew!). We are now back up and running, but there is about an 18 hour backlog of federation activity that needs to be received. I've confirmed that those are starting to resume, but it will take a while. https://dubvee.org/post/dubvee.org/977774

Lemmy World

There was an issue today with content from Lemmy World not federating to us. After pulling my hair out and testing the DubVee stack top to bottom, I got in touch with the awesome admins over at LW to try to figure out the problem. After they gave their federation service workers a good, stern talking to, we're now receiving content again.

Unfortunately, due to the amount of time (about 4 hours) the federation messages were getting stuck, there is a huge backlog that needs to catch up: about 90,000 messages. As of this writing, there are still 55,024 queued ActivityPub messages in flight (that includes backlog and current messages).

So just a heads up if you're not seeing posts/votes/comments from there or if they don't show up for a while. They will eventually arrive, though they'll be in the feed based on when they were published rather than when they arrive. On the bright side, they are receiving posts/votes/comments from us; you just might not get a reply right away.

Hopefully the queue works through overnight and things are back to normal tomorrow 🤞

Update: Backlog queue down to 37,150 now.

Update #2: 6,531 and falling. Looking like we'll be back in sync shortly.

Update #3: Backlog has finished syncing as of 23:55 EST.

mander.xyz

This one was my fault. lol. About 4 months ago, I was shoring up the firewall against some malicious traffic patterns, and mander.xyz 's server IP got caught in the CIDR block crossfire. Since I see content from their users frequently, I wasn't aware there was a problem. Turns out those were just relayed to us by the home instances of the communities. After I made a firewall allowance for mander's server, I had to reach out to their admin to reset the "last seen" date for DubVee in order to "bring us back to life" as their server had marked ours as dead.

So, all of the communities on mander.xyz should now look alive again. 🎉

My sincerest thanks to @rooki@lemmy.world, @ruud@lemmy.world, and @sal@mander.xyz for their assistance today.

If you've noticed a large number of bulk post removals performed by me in the modlog recently and are wondering if I've gone full authoritarian: no, I haven't gone mad with power. Those were posts that were already removed by other community mods or posts that were self-deleted by their creators. Some were legitimate spam or otherwise violated server rules, but the bulk of them were just cleanup.

"If they're already removed/self-deleted, why remove them again?", you might ask.

Well, Lemmy treats Pictrs (the media subsystem, basically) like a black hole - images go in but it never removes them. When posts are modded or self-deleted, any media attached to them lives forever in pictrs with no clean way to remove them later*. That wastes a huge amount of disk space on my hosting stack for media that will never see the light of day again.

I'm not okay with that for so many reasons. Yeah, object storage is cheap, but why be wasteful?

The Lemmy + Pictrs integration...well, let's just say it leaves much to be desired. "Suggest a feature enhancement" or "ping the Lemmy devs about it" you may be thinking. Haha, right.

I'm building a new API to interact with Lemmy, and the admin/mod components were the first parts that I developed. I've deployed parts of the prototype API to periodically clean up removed/deleted posts along with the media that was attached to them. I could do this silently on the backend and you'd never know, but in the interest of transparency (and also testing that the API works as expected), I've let it log its activities in the modlog as it would when it moves to production.

So, in closing, no, I have not gone mad with power. I'm just trying to keep my disk usage sane and not clutter up storage with abandoned media.

*They can be removed later, but it's a clunky external process that doesn't offer any guarantees.

Apologies for the brief ~20 minute outage. Had a loss of power and one of the two UPS's failed without warning. Had to manually move equipment over to the remaining UPS until power is restored. The cold boot takes frigging forever and was responsible for the bulk of those 20 minutes, but everything should now be back up and running.

We also lost the primary internet connection and are running on the failover, so things may be a bit sluggish.

Utility company said a tree was at fault, so will likely have both electric and network service restored in an hour or two. Unless UPS-A decides to die on me too.

If you've noticed that you're landing on tesseract.dubvee.org by default now, it's because I've done some restructuring to Dubvee's infrastructure to take advantage of the new features I've written into Tesseract. It also means you will need to login to your account again. Sorry about that, but it's a small price to pay for the benefits we'll see.

If you have Dubvee installed as a PWA, you'll probably also need to "re-install" it from the tesseract. domain. It'll redirect gracefully, but unless you resinstall, it will have a title bar and not look app-like. Again, my apologies.

Why?

I've added image proxying and caching to the Tesseract server process which allows it to act as a pseudo-CDN. This takes a significant burden off of my Lemmy + Pict-rs server while also making images and other media load faster for users. It also reduces the load on other Lemmy instances by not repeatedly fetching the same media. Win/win/win!

Additional Perks

I've left this Tesseract server instance unlocked so you can login to multiple Lemmy instances at the same time and seamlessly switch between accounts.

Beta Notice

I still have this version of Tesseract (1.2.8) in beta right now, so please let me know if you encounter any bugs.

Update

Other third-party apps and frontends are unaffected by this. They'll still function as normal but won't benefit from the caching. When configuring other apps (Jerboa, Sync, etc), you'll still put dubvee.org as your instance.

They've got a massive spam problem, and their mod actions do not federate out. So, while they may be cleaning up spam on their end, none of that federates to Lemmy and we get stuck cleaning up the mess in its entirety.

Have had good experiences with users from kbin.social, so it's not them that's the problem. Looking at the magazines on the Kbin side, they are cleaning up spam, so the mods/admins are on the ball. But without those cleanup actions federating out, they're still contributing to fediverse-wide spam that everyone else has to deal with.

Also, Kbin doesn't seem to have the concept of a registration application, only a captcha, which seems to be a very low barrier to entry for the spam bots that keep popping up there.

https://codeberg.org/Kbin/kbin-core/issues/570

Once that issue with Kbin core is addressed and mod actions from Kbin start federating, we will resume federation with kbin.social.

Update: We've re-federated but have removed all communities that are hosted on kbin.social

@Antik@lemmy.world suggested removing the Kbin communities to alleviate the spam vector. That will still allow Kbin users to interact with us (they were never the problem) while preventing spam coming from their communities where their mod actions don't propagate out. If any spam from Kbin comes through to a Lemmy community, mod actions can be taken that will be federated.

Thanks, Antik!

Figured it was time for another status update. Will try to keep it brief.

System/Infrastructure Status

We are within allowances for all metered resources (bandwidth, disk, etc) and will continue to be so for the foreseeable future. Translated: it still isn't costing me any extra money to host DubVee.

Image Uploads Have Been Re-Enabled (With Limits)

Image uploads have been re-enabled but with a limit of 150kb. This is mostly to support setting your profile avatars, but I don't have enough granular control to limit it to just that purpose. As such, you can add images to posts/comments as long as the file you're uploading is less than 150KB.

Consolidating Lemmy Frontends

For quite some time, I've offered just about every alternative Lemmy front-end available. Logs have shown some to never be used, including by me. To simplify administration, I've removed Voyager and Photon from the offerings.

The remaining options are:

• Alexandrite
• Mlmym/"Old" Reddit
• Lemmy-UI

If there are any promising newcomers, I'm open to considering them. Nothing against Voyager app or its dev team. It's just that no one was using it, and I'm in housecleaning mode.

New Default UI

If you've noticed the main DubVee interface is much more sleek and elegant, it's because we've demoted Lemmy-UI from the apex domain and promoted Tesseract to be the new default. Lemmy-UI is still available at a subdomain (lemmy.dubvee.org) if you prefer to use that.

Tesseract used to be my custom build of Photon until I decided to make it an official fork so I could do larger, more comprehensive changes (backporting my customizations to each upstream release was something of a bitch).

For that reason, there's little need to continue hosting Photon which is why it was also removed from the offerings. You can learn more about Tesseract on its GitHub page or ask questions in the Tesseract Support Channel community.

Thoughts, opinions, gripes, bugs? Leave 'em below.

Now that this instance has been running for a little over two months, I thought it was a good time to zoom out a bit and do a little reflection.

Two Month Reflections

First, I cannot believe it's only been two months; it seems way longer than that (in both a good way and an "I'm so tired, you guys" way). While I do run other public-facing services, they're much more hands-off as far as day-to-day work is concerned. Lemmy, on the other hand, is very much like a living organism, almost a pet, that requires near constant supervision, interaction, and attention. The codebase is being rapidly developed, updated, and patched, new vulnerabilities and rough edges being discovered and mitigated, and bad actors constantly looking to exploit, DoS, spam, troll, or just shit all over the place.

It's not all bad, though. In the same two months, many new and exciting projects have sprung up to give Lemmy a fresh face with new UIs. DubVee is happy to offer several of those as official options (Photon, Mlmym, Alexandrite, Voyager). Watching Lemmy and these associated projects evolve, making contributions to the projects, and learning from/sharing tips & tricks with other instance admins have been exciting side-quests in my adventure of running this instance.

That said, there's hardly a dull day when it comes to running/administering the instance, but it is definitely an enjoyable experience overall.

What makes Lemmy unique, compared to the other more static services I run, is the sysadmin tasks are only half the work. The other half comes in the form of interacting with the communities I've created/joined, posting content, and moderating. While it's been rewarding and fun, it is also a lot of work sometimes. Lemmy is very much still small and growing, so there is also the pressure to "be the change you want to see in the world" which further motivates me to contribute content.

If I was just made aware of Lemmy today, would I still stand up my own instance?

Yeah, probably (ask me in again in 6 months though 😆).

Federated and decentralized ActivityPub applications are something I find very cool and hope to see gain traction as we move into the next phase of the web, so I'm happy to be a part of the early days of that.

I hail from the tail-end of the old days of the internet where people ran BBS's, forums, and sites as a hobby, when ads were rare or non-existent, and when everything, everywhere wasn't trying to monetize/inflame every human interaction you were trying to have. Lemmy seems like a modern take on the hobbyist-run forums of days past, and it's been like a breath of fresh air. Posting/commenting on Lemmy feels more like having an actual conversation/debate than the "shouting dick jokes into the void" feeling of larger platforms.

Operational Status, Funding, and Donations

Operational Status and Hosting Costs

This instance is 100% a hobby for me, so I have been prepared since day one to cover all associated costs as long as they remain reasonable. The Lemmy services are hosted from my project servers in my basement, so it costs me nothing, upfront anyway, to run them. DubVee's biggest expense to date, aside from my time/sanity, was the domain name. That was a vanity choice, though, as my first/test Lemmy instance, like many others out there, ran under my personal domain at no extra cost. At $20/yr for dubvee.org, I don't have a problem with the additional expense (I used to spend almost that much per day on fast food -- don't judge).

I use two cloud servers for the public-facing frontends and for caching/DDoS protection plus one more for status monitoring, but those are shared with my other services and have incurred no additional cost with the addition of Lemmy. Between the two servers and factoring in all of my other services, projections show my traffic egress limits and other metered resources to be sufficient for the foreseeable future and likely beyond.

With the core services being hosted on-prem, DubVee is susceptible to power and internet outages. I am fortunate that both have been largely reliable to date with minimal outages. I have approximately 40 minutes of UPS runtime which, to date, has covered 95% or more of my power interruptions. Internet uptime and stability is comparable.

I do have a backup/auxiliary internet source, but it is not of sufficient capacity to keep DubVee online for the public. However, I do have things configured to route federation traffic over the backup connection, when needed, as that traffic is more lightweight and less sensitive to latency. I do that so posts/comments from other instances will still come through.

Our application and infrastructure status page can be found here: https://status.dubvee.org

Funding/Donations

At this time, I have no plans to solicit or even accept donations. Barring a need to move fully to a cloud provider, I also do not see that changing in the future since it would, in my mind at least, turn a hobby into an obligation.

Thoughts for the Future

As stated earlier, I'm happy to be part of what I hope becomes the new model for social interactions online. I think it's in everyone's benefit to get away from engagement algorithms, constant ads, and centralized content, so I'm planning to stick with the Fediverse for the long haul.

I made an intentional choice to try to keep this instance small. The decision was made for both technical requirement reasons as well as administrative/moderation overhead. Currently, I am the sole admin of this instance, and I don't have the time/energy to manage a large number of users (at least with the current state of Lemmy's moderation toolset). Should signups/usage here increase significantly, I may need to recruit another admin as well as explore my contingency plan of moving the Lemmy backend services to a cloud provider.

If moving the instance fully to a cloud host becomes necessary, and depending on costs associated with that, I may re-evaluate soliciting/accepting donations. At this time, these are contingency plans only.

What happens if I ever decide to shut down?

While I currently have no plans to abandon DubVee or the Fediverse, things can and do change. As DubVee is still a small instance, current shutdown plans are to simply reach out to active users and let them know when I plan to sunset the instance and help them move to another. I'll also be happy to provide any data I can export for them to assist in their migration.

If by chance DubVee grows sufficiently (or is still small but someone else wants to take over operations in my stead), I would be open to transferring the domain and server-side resources to another party, assuming acceptable arrangements and guarantees can be made. I'd also be willing to co-admin or otherwise remain involved in such a scenario.

None of that should be taken as a hint I plan to shut down. My goal with the above statement is to simply codify the steps I would take if/when that scenario should arise.

What are your thoughts?

If there's anything you'd like to see implemented, spelled out in policy, or something I can do to improve the experience here, please feel free to suggest in the comments.

Did you know there's more than one way to Lemmy? Since Lemmy usage has taken off, so have the projects to develop alternate frontends. When one seems promising, we will usually adopt it as an option.

Below are the official and 3rd party Lemmy user interfaces supported by DubVee:

Lemmy UI

The default Lemmy experience through the official frontend.

• URL: https://dubvee.org
• Github: https://github.com/LemmyNet/lemmy-ui

Photon

A sleek, mobile and desktop friendly interface that can be installed as a PWA (progressive web app). This is currently my "editor's choice" for Lemmy mobile access.

• URL: https://m.dubvee.org
• Github: https://github.com/Xyphyn/photon

Voyager

A progressive web app (PWA) designed for mobile. Has both an Android and iOS (default) skin.

• URL: https://voyager.dubvee.org
• Github: https://github.com/aeharding/voyager/blob/main/Dockerfile

Mlmym

An old-Reddit style UI for Lemmy. Definitely nostalgic and scratches the "Reddit" itch, but feels a little claustrophobic sometimes. Still, it's a solid UI, and I'm sure it will improve with time.

• URL: https://old.dubvee.org
• Github: https://github.com/rystaf/mlmym

Alexandrite

A sleek Lemmy UI designed primarily for desktops. The default option is annoyingly purple, and you can only change the theme once you're logged in. The theme switcher is a slider which is weird/annoying. This UI is starting to grow on me, though.

• URL: https://alex.dubvee.org
• Github: https://github.com/sheodox/alexandrite

If you're feeling nostalgic for old Reddit, I've installed an alternate Lemmy UI called mlmym at https://old.dubvee.org

It's not my project, and is still a work in progress, but feel free to comment here with any bugs and I'll try to post them to the project's issue page.

Links

• Dubvee Old Reddit Style
• rystaf/mlmym @ Github

There seems to be some breach at lemmy.world and, at the very least, an admin account was compromised.

Out of an abundance of caution, DubVee has temporarily added lemmy.world to our blocklist while they sort out their problem.

I'm monitoring the admin chat and awaiting word from the LW admins confirming they are back in control of their site. Once that has been confirmed, federation with lemmy.world will resume.

Update: Looks like they may be back in control over at LW, but they're still sorting things out. Regardless, the bad actor that was in control had defederated all hosts besides threads [dot] net so removing lemmy.world from our blocklist right now would be pointless.

Update #2: lemmy.world seems to be back in control and the vector used for this attack was mitigated. They've posted a response here: https://lemmy.world/post/1290412

As of 8am, we are resuming federation with lemmy.world

There's an issue with the way Lemmy handles user uploads that makes me uncomfortable leaving them enabled. Without getting too deep into the technical details of the matter, the issue leaves my server open to abuse and does so in a way which is both invisible to the admins and difficult to rectify.

Until that is resolved by the upstream project, I have opted to disable user uploads entirely as a precautionary measure. While the "upload image" button is still present in post/comment options, any uploads will be rejected by the server; you will see a red error message saying "Type error: Failed to fetch" when attempting to upload. If you're using Jerboa or another native app, it may fail with a different message, but it will fail (as-intended).

Please note that this decision was not reached lightly and is the only currently-known way to ensure the safety and integrity of the platform. If you wish to embed images in your submissions, until further notice, please embed them from an outside source (imgur, giphy, getyarn.io, your own server, etc). If you're an old Redditor, this should be familiar.

This also affects setting your avatar and banner in your profile. Unfortunately, I have no way of allowing just those to remain functional while disabling comment/post images; the method I have to use to block uploads is somewhat all-or-nothing.

If you would like to set an avatar/logo, please DM me and I can make an arrangement to do it on the backend.

Welcome

Without an account, you can browse All on this instance and see posts/comments that were made to local communities or ones that are currently subscribed from other Lemmy instances.

Please note that this is not all that is available across Lemmy; just what other users have already subscribed to.

In order to search and subscribe to other Lemmy instances that aren't already in our list, you will need to be signed into an account.

Terminology

• Instance: Home Server. Your account has to live somewhere. The home server is the one you log into and shows you your feed. Somewhat like homeroom in school.
• Community: Subreddit
• Federation: It means different home servers interact and share content with each other.

Signup

If you do not have an account, you can sign up for one on this server.

To register an account, provide a unique username, email address for password resets, and set your password.

You'll also need to fill out a sign up application. This is a spam and bot prevention feature. Answer the question and submit your application.

Signup Troubleshooting

If you're trying to sign up and the submit button just spins forever, then it means there's an error. The UI for Lemmy is a bit rough around the edges right now, unfortunately, and certain errors are not shown.

Common problems with signup:

• Username and display name each must be 20 characters or less
• Password must be 60 characters or less
• Email must be unique for this instance

Email Validation

After your signup application is accepted, you should receive an email with a link. You'll need to click that link in order to validate that your email address belongs to you.

Account Approval

Once your email address is validated, the site admins will be notified of your application. Human intervention is required for this step, so please be patient as it may be up to several hours before an admin is able to review the request.

If your application is approved, you will receive an email notification. You will then be able to log in.

Unfortunately, at the moment, if your application is rejected, the system does not send any kind of notification. If you haven't gotten an approval within 8 hours, you can email lemmy_support@ptznetwork.org to see if there was a problem or for assistance signing up.

Once your account is approved (you should receive an email upon approval), log in and customize your profile and settings.

How to Find and Subscribe to Communities

There are two ways to find communities through Lemmy:

To browse communities that others in your instance are already subscribed to, tap the “Communities” tab at the top of the page and choose the “All” scope. Tapping on a community name will open it through your instance.

Finding a Community That Exists on a Different Lemmy Instance

1. Open a new tab to the Lemmy Community Browser

1A) Lemmyverse Communities is also a great resource:

2. Use the search to look up a community that interests you.

3. When you've found a community of interest, you can open its link in a new tab to preview the posts.

4. If you decide you want to subscribe to that community, copy its URL.

5. Switch back to the tab for this Lemmy instance and hit the 🔍 button in the upper toolbar.

6. Make sure that you have chosen “All” for each of the four filters: “Type”, “Scope”, “Community”, and “Creator”.

7. Paste the community’s URL into the search field and tap “Search”.

8. You will see that it says "No results". This is normal and will hopefully be addressed in a future update.

9. Wait 5 or 10 seconds and hit search again. This time you should see a result. It should be the community shown as an icon, a name, and a subscriber count. If it doesn't show up, wait a bit longer, and search again.

• • If you do not see it, or it is buried too deep in the search results, try changing “Scope” to “Local”. If that does not work, you may need to wait a bit and try again.

10. Tap on the community in the search results to open it in your instance.

11. Once a community is open in your instance, subscribe to it by tapping on the “Subscribe” button at the top of the sidebar.

12. You should then be subscribed to the community and see it in your feed.

Yes, this is very convoluted, and we are aware. The discovery process for remote instances has much room for improvement, and there are various discussions on how best to address it.

Can’t find a community you’re looking for?

If there's no community for a topic you are interested in, and if you're willing to act as a moderator, feel free to create a new community here.

Tips

• Post and Comment Formatting Syntax