3
submitted 11 months ago by jared252016@alien.top to c/main@selfhosted.forum

I love Aria2, but I'm building a web scraper / crawler and I need to download hundreds of thousands of files. Aria2 locks up around the 20,000 file mark. Is there another download manager that could possibly be able to achieve what I'm trying to do? or a more recent fork of Aria2?

I have a workaround I believe, which is to use the API to determine how many files are in queue and sleep indefinitely until there is < 1000, but I'm not sure this is the most effective. It kind of significantly slows down the download pipe.

The issue seems to lie with connections timing out in aria2, which cause them to get locked up and they have to be manually cleared. I have my timeout set at 10 seconds, but that doesn't seem to matter. I've considered running a schedule task to clean them up, but was going to give downloading with Python a try first.

Any suggestions would be appreciated.

[-] jared252016@alien.top 1 points 11 months ago

ThePirateBay, the most notorious site in the world, uses Cloudflare. This isn't China. Wiretapping is illegal in most circumstances, and that's essentially what it would be doing.

[-] jared252016@alien.top 1 points 1 year ago

Not all docker-compose files are development, although it is in this case. The documentation, if available, will tell you what each env var does.

You know the docker-compose.yml will have everything you need though, you just need to tweak it. Absolutely nothing wrong with seeking out the docker-compose as most env variables are self explanatory.

[-] jared252016@alien.top 1 points 1 year ago

Check the related comments pertaining to I believe it's defederation from an MSP with an Outlook account. Assuming you can get into the admin and GoDaddy didn't create their own.

The theory: If you can remove the MSP, you're free to sign up for the cheaper plan that's the same for cheaper. I believe it's $6/month/user with cloud based Office 365 Apps, slightly more if you want the desktop apps. Or if you prefer, Google Workspaces is $6/month/user.

With 4 users that's $24/month or $288/year, which is more than you're paying now it seems.

With Microsoft 365 you can add multiple inbox aliases to receive mail at multiple users but on one account, so the 1 user gets all the email. That would save you the most while keeping Microsoft 365.

You can however swap to one of the mentioned shared user plans that offer a set amount of shared space per user for cheaper.

Then use an email migration service to transfer the emails, or some open source tools if you run Linux and have some level of tech savvy. GoDaddy likely won't help you transfer away. Google and Microsoft have built in tools in the admin sections to transfer users. Support may help you but more than likely you're on your own with the documentation for Microsoft and Google as they expect you to be able to change DNS and everything.

Find a guide online that includes transferring email from service A to service B, or use a service like this: https://www.cloudm.io/landing-pages/migrate-to-google-workspace/

[-] jared252016@alien.top 1 points 1 year ago

Did you set the environment variables for the database and what not?

When you get stuck like this, always look at the docker-compose.yml file if provided. You can find this one here: https://github.com/mealie-recipes/mealie/blob/mealie-next/docker/docker-compose.yml

It lists all the environment variables that need to be set.

Alternatively you should be able to use docker compose in WSL, it'll still show up in Docker Desktop.

[-] jared252016@alien.top 1 points 1 year ago

I would do the ethical thing and explain the privacy implications of you hosting it but him hosting it too. Honestly I'd turn on end to end encryption.

You having their data and him having his family's data is a big responsibility. You need adequate security both online and physically.

There's also the fact that he has access to it if he's an admin. Not everyone can handle that responsibility.

As to your original question, Nextcloud breaks often, relative to how often the server will have problems. You'd def need SSH access.

It would probably be better for you to get an ASUSTOR NAS in terms of hardware. It Supports apps and Nextcloud is one of them.

It also has support other than you, which depending on a few factors (magic 8 ball), might be more time consuming than you'd think. They may not want to deal with hardware either if something did occur.

[-] jared252016@alien.top 1 points 1 year ago

From your diagram you should be able to access it already. It's what is called dual NAT where a router is behind a router.

Otherwise you could turn off DHCP on one and use static IPs / routes

Would it be possible to plug your media server into both routers?

jared252016

joined 1 year ago