Addressing upload errors

Some of you may have noticed "internal server error"s when attempting to upload files. These have been a combination of timeouts when communicating with Cloudflare and errors coming from Cloudflare, both happening on Heliosphere's side of the upload process.

TL;DR: I am adding/changing some retry capabilities in the backend to attempt to compensate for the errors, but no guarantees.

Just to give you a refresher on how Heliosphere works:
  1. you upload a file of some sort to Heliosphere
  2. Heliosphere unpacks that file, deduplicates it, and recompresses its constituents individually
  3. Heliosphere then uploads the processed files to its storage, hosted using Cloudflare R2
It is step 3 specifically that is causing issues. Cloudflare is not playing nice. The errors should be transient, meaning that they'll go away if retried enough, but I know that's a tedious and time- (and bandwidth-) consuming process. To hopefully help with the errors, I have tweaked internal retries and added new ones. This means that if the timeout or Cloudflare error is encountered while Heliosphere is trying to upload a file, it will automatically retry a few times before giving up. Crucially, the retry is on Heliosphere's side, so it doesn't mean you will automatically upload the file to Heliosphere again; it means Heliosphere will try to upload to Cloudflare again.

Note that this retry capability may cause the processing step to take longer, as it waits longer between each retry (with some exceptions). If Cloudflare continues to be problematic, I may investigate other file hosts or self-hosting, but there are many caveats that come with that, and it should be a last-resort solution.

I apologise for the inconvenience these intermittent errors are causing, and I hope to be able to work around them more effectively. Thank you for your patience and understanding!
 
Back
Top