@NosibleAI is data hungry.

It collects reference, market & alt data 24/7/365 from 00's of APIs and inserts it into a version controlled linked-entity graph of >120k listings and >100k funds & ETFs 🌍

Here's a short thread on cloud storage lessons learned 😬 ...
Lesson #1 - AWS S3 and GCP GCS charge mafia prices 🕴

Their price per TB per month is high, they charge you for every API request, and they rip your eyeballs out if you ever leave (egress) 🙄

If you have lots of smallish files, just kiss your budget goodbye 💸
Lesson #2 - blockchain is not just a buzzword. It is powering the future of cloud storage RIGHT NOW 😎

There are a few decentralized storage networks available but my favourite one has to be the Sia network @SiaTechHQ 🤩

PS. is it pronounced Sigh-a or See-a?
@SiaTechHQ allows anybody anywhere on Earth to rent their excess storage space to renters like me who would like use to it 🤯

When you store a file it is encrypted, sharded, and stored on many devices on the network to ensure it is always available. It's magic 🤯
@SiaTechHQ allows anybody anywhere on Earth to rent their excess storage space to renters like me who would like use to it When you store a file it is encrypted, sharded, and stored on many devices on the network to ensure it is always available. It's magic
@SiaTechHQ allows anybody anywhere on Earth to rent their excess storage space to renters like me who would like use to it When you store a file it is encrypted, sharded, and stored on many devices on the network to ensure it is always available. It's magic
@SiaTechHQ allows anybody anywhere on Earth to rent their excess storage space to renters like me who would like use to it When you store a file it is encrypted, sharded, and stored on many devices on the network to ensure it is always available. It's magic
Lesson #3 - using Sia is 10x easier when you use @Filebase 🥳

Filebase is an S3-compliant API that sits in front of both @SiaTechHQ and @storjproject. All of the benefits with none of the hassle 🙌🏻

I trust and use @Filebase myself.
Lesson #3 - using Sia is 10x easier when you use @Filebase Filebase is an S3-compliant API that sits in front of both @SiaTechHQ and @storjproject. All of the benefits with none of the hassle I trust and use @Filebase myself.
Lesson #3 - using Sia is 10x easier when you use @Filebase Filebase is an S3-compliant API that sits in front of both @SiaTechHQ and @storjproject. All of the benefits with none of the hassle I trust and use @Filebase myself.
Lesson #3 - using Sia is 10x easier when you use @Filebase Filebase is an S3-compliant API that sits in front of both @SiaTechHQ and @storjproject. All of the benefits with none of the hassle I trust and use @Filebase myself.
Lesson 4 - there are also some great centralized alternatives too!

Wasabi is great. It doesn't charge for API requests and is ~80% cheaper than AWS and charges no egress 🤓

The only "catch" is that they charge for a minimum of 1TB + 3 months. Which is a normal use case 👍🏻
Lesson 4 - there are also some great centralized alternatives too!Wasabi is great. It doesn't charge for API requests and is ~80% cheaper than AWS and charges no egress The only "catch" is that they charge for a minimum of 1TB + 3 months. Which is a normal use case
Lesson 4 - there are also some great centralized alternatives too!Wasabi is great. It doesn't charge for API requests and is ~80% cheaper than AWS and charges no egress The only "catch" is that they charge for a minimum of 1TB + 3 months. Which is a normal use case
Lesson 4 - there are also some great centralized alternatives too!Wasabi is great. It doesn't charge for API requests and is ~80% cheaper than AWS and charges no egress The only "catch" is that they charge for a minimum of 1TB + 3 months. Which is a normal use case
Lesson 5 - you can also DIY 🔨 using @OVHcloud and @Minio.

If you need to store many 10's of TB then an even cheaper option is to self-host! OVH has cheap storage servers and MinIO is an open source S3 compliant API. Put those two together and you're winning 🏆 .
Lesson 5 - you can also DIY using @OVHcloud and @Minio.If you need to store many 10's of TB then an even cheaper option is to self-host! OVH has cheap storage servers and MinIO is an open source S3 compliant API. Put those two together and you're winning .
Lesson 5 - you can also DIY using @OVHcloud and @Minio.If you need to store many 10's of TB then an even cheaper option is to self-host! OVH has cheap storage servers and MinIO is an open source S3 compliant API. Put those two together and you're winning .
Lesson 5 - you can also DIY using @OVHcloud and @Minio.If you need to store many 10's of TB then an even cheaper option is to self-host! OVH has cheap storage servers and MinIO is an open source S3 compliant API. Put those two together and you're winning .
After a lot of testing @NosibleAI I switched from GCS to using both @Filebase AND Wasabi. That gave me 2x live copies of the graph ✌🏻

And guess what?

I ended up paying 75% LESS than for only 1x copy on GCS 🥳🤑 and I have not experiences any issues.
Bonus (negative) lesson - based on my experience I suggest you avoid @backblaze's B2 product 🤐

Not everything that glitters is gold 🙈 you're better off using FileBase, Wasabi, and/or MinIO. And if you do use MinIO just be careful because you're your own support team 🤠
You can follow @StuartReid1929.
Tip: mention @twtextapp on a Twitter thread with the keyword “unroll” to get a link to it.

Latest Threads Unrolled:
Those who think consciousness raising & movement building are disconnected, or even a distraction from electoral politics have not paid attention to the history of social change.In this thread, we
Read more
The 2020s will be known as the Remote Work decadeA few predictions of what is likely to emerge[ a thread ] Third Space: Office and Working from Home will
Read more
1/29: Have you ever had a concept explained to you that helps frame complex issues you’ve been wrestling with and opens your eyes to new possibilities? A concept that I
Read more
By continuing to use the site, you are consenting to the use of cookies as explained in our Cookie Policy to improve your experience.