Blackblaze B2 status and alternatives
1d 1h ago by lemmy.zip/u/xana in selfhosted
Hello,
I know it is strictly related to selfhosting but I don't have a reddit account so please bear with me. I've been using a Blackblaze B2 bucket as the storage for my restic backup for my whole homelab for a while and recently when reading from that bucket it is so so so so so so slow and unreliable (a lot of unexpected EOF and interruption). I've tested this on several machines (including a VPS on Hetzer) and I can access other sites just fine. Their status page says nothing and I wonder is it just me or do some of you also experience the same thing ?
Also any alternative recommendation is welcome but I can not selfhost my S3 at the moment :(
Thank you very much!
Edit: this is the log for my restic check --read-data (something similar on Hetzner VPS).
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
Load(<data/ef0e80ecc6>, 541, 6322304) returned error, retrying after 508.096097ms: unexpected EOF
Load(<data/ef0e80ecc6>, 541, 6322304) operation successful after 1 retries
Load(<data/c10459132f>, 535, 14002087) returned error, retrying after 1.352882596s: unexpected EOF
Load(<data/c10459132f>, 535, 14002087) operation successful after 1 retries
Load(<data/44089c2105>, 536, 10041504) returned error, retrying after 1.094099947s: unexpected EOF
Load(<data/44089c2105>, 536, 10041504) operation successful after 1 retries
Load(<data/c10459132f>, 481, 3262415) returned error, retrying after 1.418694544s: unexpected EOF
Load(<data/927e700367>, 467, 7776453) returned error, retrying after 634.119689ms: unexpected EOF
Load(<data/927e700367>, 473, 7798234) returned error, retrying after 849.03055ms: unexpected EOF
Load(<data/927e700367>, 467, 7776453) returned error, retrying after 1.243622488s: unexpected EOF
Load(<data/653ef7abb0>, 523, 8280899) returned error, retrying after 578.669152ms: unexpected EOF
Load(<data/fd9edf4117>, 533, 485866) returned error, retrying after 823.382046ms: unexpected EOF
Load(<data/c10459132f>, 528, 3248055) returned error, retrying after 1.402372952s: unexpected EOF
Load(<data/c10459132f>, 524, 3175555) returned error, retrying after 1.425799941s: unexpected EOF
Load(<data/9b179c749d>, 534, 13802701) returned error, retrying after 1.234603672s: unexpected EOF
Load(<data/7b827c96f6>, 531, 8412866) returned error, retrying after 1.49579899s: unexpected EOF
Load(<data/ef0e80ecc6>, 530, 5093059) returned error, retrying after 1.405885227s: unexpected EOF
Load(<data/927e700367>, 409, 7763564) returned error, retrying after 798.04783ms: unexpected EOF
Load(<data/c10459132f>, 481, 3262415) returned error, retrying after 1.362966862s: unexpected EOF
Load(<data/653ef7abb0>, 523, 8280899) returned error, retrying after 2.966804822s: unexpected EOF
Load(<data/927e700367>, 467, 7776453) operation successful after 2 retries
Load(<data/fd9edf4117>, 533, 485866) operation successful after 1 retries
Load(<data/927e700367>, 409, 7763564) operation successful after 1 retries
Load(<data/9b179c749d>, 534, 13802701) operation successful after 1 retries
Load(<data/c10459132f>, 528, 3248055) operation successful after 1 retries
...
Edit: After a day my restic check --read-data finished with no data corruption but it took a lot of retrying, even 7 retries to read the correct data :/ So I guess it is a sign for me to move to somewhere else.
I think thinking about offsite backup/object storage is very relevant to self hosting.
I've been on backblaze and using my bucket for video and it has been working well. I know it's a pain but have you tried setting up a new bucket? Maybe you got a weird server in their data center …
Thank you for your reply :D To be honest I am thinking about setting up a new bucket as well but I can not move the data out because it will takes forever
Did you reach out to support with this? If so what was the experience?
I also backup to Bb, not had issues so far.
I did but they said that no issue on their end :/ In my case backing up to B2 is good as well but checking the backup (re-read data from there) is so unreliable
I recently had a local issue where I could barely reach B2 as well. It turned out that one of their endpoints was on a DNS blocklist I use. Once I manually unblocked that everything started to work again. Not saying you're having the same issue but it could be a lot of things between you and B2, careful triage is needed to find the root cause.
I don't think it is my end tbh because I ran the exact same thing on a Hetzner VPS and had similar issue so ...
I moved from a Backblaze B2 bucket to a Hetzner storagebox and haven't looked back.
Lower cost and - for me - just easier to use, so, better for restores.
I can't help with your error messages, but I'm presuming you're just restoring a backup of data you still have? If so, just upload to Hetzner and start again.
I use Hetzner too. Never had any issues.
I am thinking about storagebox as well. How good is their bandwidth ? Because I have 2TB of backup and I want to fully read it from time to time to make sure my backup is good. Usually I can consume 1Gb/s with Blackblaze.
TBH, I haven't done a full-bore up/download.
I limit my backup bandwidth as the deltas don't take long and they can upload during the day
I also don't have a Gb internet connection, so I could only max at ~80Mb (down) anyway
My restic backups go to a bucket on scaleway. I’m not having any trouble with that.
They have a free trial, so maybe you could try and see if it works better for you.
I have experienced restic being very slow when it didn’t have a cache. Maybe something worth double checking. I believe it mentions it in the output.
Scaleway is just too expensive for the amount of data I am having but I am thinking about OVH seems like a good candidate
I set up Backblaze for my Synology NAS' and while I check to make sure it is working I've never had to touch it otherwise.
I use Jottacloud for off site restic backups. You have to use rclone, but it's not too tricky. I don't know how well it works outside of Europe. It's cheap and can store terabytes of data!
To throw in yet another opinion that does not help you right now: My friend has a backup server at his place and I have our home lab at my place.
Just a rough calculation: 2 8tb drives for a mirror is 150€ each (during a normal market, and when we bought it), so 300 combined. That is 60€per year if we assume they die quickly after 5years.
With a very light old desktop pc that is power efficient we can reach 25w
That is: 25w/1000 (for kW) * 24h * 365d * 0.35€/kW = 78€ per year of energy.
That is combined 138€/year.
The hetzner storage box is 24€/Month 10tb
So that is 288€/year.
So I would argue that its even cheaper to pay a friend to put a small old pc at their place and paying them 40ct /kwh is still cheaper than the clouds? Assuming you can find hdds at a reasonable price. But I could find cheap used offers with a runtime of 3y for 100€.
If you only need 1tb, its not worth it to run your own storage, as the power stays roughly the same:
Hetzner: 3,8€/1tb * 12m = 46.6€ So allone the power of running the backup server is not worth it financially.
For the 5tb plan: 12€*12m=144€ so that is the break even point I assume, as drives don't scale too mich with price?
Thanks for your calculation. The reason why I don't want to run S3 myself is because I don't trust myself to deal with this and I really need a off-site backup :D
Haha i get that. I feel like proxmox backup server is really great together with proxmox though. And idk but I feel like there is not too much to do wrong? Correct me if you think differently, but we use a mirrored hdd zfs pool on pbs and pve, so we have 4 copies plus snapshots ob the backup server. So harddrive failure is not really a problem. And deleting or corrupting files or databases manually on accident can happen, but they can be revived sometimes or restored from last nights backup And yes, the backup server could die, but the drives should be fine still. And yes, it can burn down but then you have a mirrored drive on the home lab. I think its rather difficult to delete stuff on the bps on accident. I would just not delete anything manually. If its too full, either increase the pruning job for old snapshots or just put in a mother pair of drives.
Maybe I am cheap but its also fun I think. Also I feel like my data is actually mine now?
When you say "something similar on Hetzner vps" do you mean the command or the log? Can you increase the verbosity to get more info on the error?
Personally, I think seven correctable errors is tolerable.
For the Hetzner VPS, I mean I run the exact same command to the same restic repository and I meet the same issue multiple times.
For the seven correctable errors, bear in mind that it is seven retries for one piece of data and there are 40k restic packs on that repository
Oh I see. Maybe they have a bad disk; I would try reuploading that part if you can.