I rework my content distribution

Disclaimer

I am not sponsored by any of the products mentioned here. However, I find them useful and good, and recommend them to anyone interested :)

Content distribution

The stuff I make usually relies on the storage and distribution of various files to users. Especially in the sense of webapp development, I find myself having to distribute things like images and downloads a lot. As for storage, some of my projects allow users to upload stuff to my servers, such as text and images. For example, my selfhosted Misskey instance relies on S3 storage to store files. I also host my own ShareX server, which also requires S3 storage to store the uploads.

The previous solution

Hosting webapps is probably the easiest thing to do nowadays. There are so many different PaaS services willing to host your apps and distribute them with a CDN. However, since I prefer to selfhost stuff by nature, all of my fullstack (i.e. require SSR) NextJS apps are selfhosted by me on a Coolify instance. As for a CDN, I simply proxy my domains through Cloudflare. I thank them for providing such a fast, easy to setup CDN for free.

As for static files, this is where it gets more difficult. For images and downloads, I used BunnyCDN, who provides a global CDN network with distributed object storage combined. This worked relatively well, and I was able to serve my files and downloads for my users.

Since I’m too much of a cheapskate to use AWS’s official S3 storage, I find myself with the next best thing; cheap, S3-compatible object storage. I used DreamObjects for this task, who provide S3-compatible storage at the staggeringly cheap price of $0.025/GB with no minimum fee. I was spending less than a dollar a month on my storage, even including traffic costs. Distribution, while slow because storage was only in one region could be sped up by proxying through BunnyCDN.

The problems

I was always not very satisfied with the solution I had come up with, as I found that using two different object storage providers was inefficient and finicky. However, I thought that this couldn’t really be helped, as BunnyCDN while promising to release an S3-compatible API, hadn’t (and still haven’t) released it.

As for DreamObjects, while cheap and did the job, did annoy me at times. As I am based in Japan, the uploads were rather slow, in the worst case taking over 10 seconds to upload a single JPEG image to my ShareX server. While I could ignore this because of the price, there was one other thing that really got on my nerves: the outages.

At least once a month, DreamObjects would suffer near 24-hour long outages, where not only no files could be uploaded, but previously uploaded files could also not be accessed. Again, I disregarded this at first de to the extremely low cost. However, around last month, there were three cases of day-long outages. This is when I thought that I had to change my strategy.

Step 1: Object Storage

At first, I thought I might move back to Linode’s object storage, which I had been previously using. While I hadn’t been using them, they had added an Osaka region, which I thought would be perfect for my needs. While this was a good way to go, I soon found a much cooler solution; Tigris Data. I had learned about them after they became a sponsor of Coolify, the selfhosted PaaS I use for all my webapps. (by the way, you should check them out, I highly recommend it and sponsor the project personally :)
Anyways, Tigris Data is a relatively new contender in the S3 storage space, and is a subsidiary of Fly.io, the distributed cloud provider (also something I recommend trying)
Using Fly’s distributed nature, they provide globally distributed object storage, at an insanely low price.

  • $0.02/GB distributed storage (fixed fee!)
  • No data transfer charges
  • Negligible API requests

They also provide this highly generous free tier;

  • 5GB forever free storage
  • No data transfer charges
  • 10k API requests

…which I was able to fit all my data into. I currently pay nothing for my S3 storage.

Step 2: CDN(s?!)

While I was originally planning to continue my dual-CDN approach of content distribution, I happened to have signed up for the free tier of AWS recently. I usually disregard hyperscalers, as their prices seem to be highly expensive for the same specifications you’d get from other cheaper providers. However, AWS has a very generous offer for CloudFront, their CDN network.

  • 1TB free data transfer per month

Since my traffic would never ever go over that limit, I decided to give them a try. Since CloudFront is from the makers of S3, it was very easy to setup a pullzone from the Tigris storage zone. So, my static CDN (cdn.mikn.dev) is now served CloudFront :)
I still use BunnyCDN for other things, as they have very high amounts of control over things like caching, and also have a very interesting perma-caching feature which allows cache data to be permanently stored in an object storage region.
Cloudflare still continues to be used for webapp distribution.

In conclusion: TRIPLE CDN POWER!!!!!

Conclusion

Would I recommend setting up a crazy network like this? No, not really. Cloudflare R2 and their CDN should do you good. I just like trying new things :)

Thanks for reading!