One of many extra thrilling purposes of decentralized computing which have aroused a substantial quantity of curiosity prior to now yr is the idea of an incentivized decentralized on-line file storage system. At the moment, in order for you your information or knowledge securely backed up “within the cloud”, you’ve three selections – (1) add them to your personal servers, (2) use a centralized service like Google Drive or Dropbox or (3) use an present decentralized file system like Freenet. These approaches all have their very own faults; the primary has a excessive setup and upkeep value, the second depends on a single trusted celebration and infrequently entails heavy value markups, and the third is gradual and really restricted within the quantity of area that it permits every consumer as a result of it depends on customers to volunteer storage. Incentivized file storage protocols have the potential to supply a fourth manner, offering a a lot larger amount of storage and high quality of service by incentivizing actors to take part with out introducing centralization.
Plenty of platforms, together with StorJ, Maidsafe, to some extent Permacoin, and Filecoin, try to deal with this downside, and the issue appears easy within the sense that each one the instruments are both already there or en path to being constructed, and all we’d like is the implementation. Nevertheless, there’s one a part of the issue that’s notably essential: how will we correctly introduce redundancy? Redundancy is essential to safety; particularly in a decentralized community that shall be extremely populated by newbie and informal customers, we completely can’t depend on any single node to remain on-line. We might merely replicate the information, having a number of nodes every retailer a separate copy, however the query is: can we do higher? Because it seems, we completely can.
Merkle Bushes and Problem-Response Protocols
Earlier than we get into the nitty gritty of redundancy, we’ll first cowl the simpler half: how will we create at the very least a fundamental system that can incentivize at the very least one celebration to carry onto a file? With out incentivization, the issue is simple; you merely add the file, anticipate different customers to obtain it, after which while you want it once more you can also make a request querying for the file by hash. If we wish to introduce incentivization, the issue turns into considerably tougher – however, within the grand scheme of issues, nonetheless not too laborious.
Within the context of file storage, there are two sorts of actions you could incentivize. The primary is the precise act of sending the file over to you while you request it. That is simple to do; the most effective technique is a straightforward tit-for-tat recreation the place the sender sends over 32 kilobytes, you ship over 0.0001 cash, the sender sends over one other 32 kilobytes, and so on. Observe that for very giant information with out redundancy this technique is weak to extortion assaults – very often, 99.99% of a file is ineffective to you with out the final 0.01%, so the storer has the chance to extort you by asking for a really excessive payout for the final block. The cleverest repair to this downside is definitely to make the file itself redundant, utilizing a particular sort of encoding to develop the file by, say, 11.11% in order that any 90% of this prolonged file can be utilized to get better the unique, after which hiding the precise redundancy share from the storer; nonetheless, because it seems we’ll focus on an algorithm similar to this for a distinct objective later, so for now, merely settle for that this downside has been solved.
The second act that we will incentivize is the act of holding onto the file and storing it for the long run. This downside is considerably tougher – how are you going to show that you’re storing a file with out truly transferring the entire thing? Happily, there’s a answer that’s not too troublesome to implement, utilizing what has now hopefully established a well-recognized popularity because the cryptoeconomist’s finest pal: Merkle timber.
Effectively, Patricia Merkle may be higher in some instances, to be exact. Athough right here the plain outdated authentic Merkle will do.
n = 2^okayfor some
okay(the padding step is avoidable, but it surely makes the algorithm easier to code and clarify). Then, we construct the tree. Rename the
nchunks that we obtained
chunk[n]to
chunk[2n-1], after which rebuild chunks
1to
n-1with the next rule:
chunk[i] = sha3([chunk[2*i], chunk[2*i+1]]). This allows you to calculate chunks
n/2to
n-1, then
n/4to
n/2 - 1, and so forth going up the tree till there’s one “root”,
chunk[1].
Now, word that for those who retailer solely the foundation, and neglect about chunk[2] … chunk[2n-1], the entity storing these different chunks can show to you that they’ve any specific chunk with just a few hundred bytes of information. The algorithm is comparatively easy. First, we outline a perform associate(n) which provides n-1 if n is odd, in any other case n+1 – in brief, given a piece discover the chunk that it’s hashed along with as a way to produce the guardian chunk. Then, if you wish to show possession of chunk[k] with n <= okay <= 2n-1 (ie. any a part of the unique file), submit chunk[partner(k)], chunk[partner(k/2)] (division right here is assumed to spherical down, so eg. 11 / 2 = 5), chunk[partner(k/4)] and so forth all the way down to chunk[1], alongside the precise chunk[k]. Basically, we’re offering your complete “department” of the tree going up from that node all the best way to the foundation. The verifier will then take chunk[k] and chunk[partner(k)] and use that to rebuild chunk[k/2], use that and chunk[partner(k/2)] to rebuild chunk[k/4] and so forth till the verifier will get to chunk[1], the foundation of the tree. If the foundation matches, then the proof is ok; in any other case it is not.
11 = associate(10)), 4 (
4 = associate(10/2)) and three (
3 = associate(10/4)). The verification course of entails beginning off with chunk 10, utilizing every associate chunk in flip to recompute first chunk 5, then chunk 2, then chunk 1, and seeing if chunk 1 matches the worth that the verifier had already saved as the foundation of the file.
Observe that the proof implicitly contains the index – generally it is advisable to add the associate chunk on the suitable earlier than hashing and generally on the left, and if the index used to confirm the proof is completely different then the proof is not going to match. Thus, if I ask for a proof of piece 422, and also you as an alternative present even a legitimate proof of piece 587, I’ll discover that one thing is flawed. Additionally, there isn’t any manner to supply a proof with out possession of your complete related part of the Merkle tree; for those who attempt to go off faux knowledge, in some unspecified time in the future the hashes will mismatch and the ultimate root shall be completely different.
Now, let’s go over the protocol. I assemble a Merkle tree out of the file as described above, and add this to some celebration. Then, each 12 hours, I decide a random quantity in [0, 2^k-1] and submit that quantity as a problem. If the storer replies again with a Merkle tree proof, then I confirm the proof and whether it is appropriate ship 0.001 BTC (or ETH, or storjcoin, or no matter different token is used). If I obtain no proof or an invalid proof, then I don’t ship BTC. If the storer shops your complete file, they are going to succeed 100% of the time, in the event that they retailer 50% of the file they are going to succeed 50% of the time, and so on. If we wish to make it all-or-nothing, then we will merely require the storer to resolve ten consecutive proofs as a way to get a reward. The storer can nonetheless get away with storing 99%, however then we benefit from the identical redundant coding technique that I discussed above and can describe under to make 90% of the file adequate in any case.
One concern that you’ll have at this level is privateness – for those who use a cryptographic protocol to let any node receives a commission for storing your file, would that not imply that your information are unfold across the web in order that anybody can doubtlessly entry them? Happily the reply to that is easy: encrypt the file earlier than sending it out. From this level on, we’ll assume that each one knowledge is encrypted, and ignore privateness as a result of the presence of encryption resolves that subject nearly utterly (the “nearly” being that the scale of the file, and the occasions at which you entry the file, are nonetheless public).
Trying to Decentralize
So now we’ve a protocol for paying individuals to retailer your knowledge; the algorithm may even be made trust-free by placing it into an Ethereum contract, utilizing
block.prevhash as a supply of random knowledge to generate the challenges. Now let’s go to the following step: determining the right way to decentralize the storage and add redundancy. The best strategy to decentralize is straightforward replication: as an alternative of 1 node storing one copy of the file, we will have 5 nodes storing one copy every. Nevertheless, if we merely comply with the naive protocol above, we’ve an issue: one node can faux to be 5 nodes and gather a 5x return. A fast repair to that is to encrypt the file 5 occasions, utilizing 5 completely different keys; this makes the 5 equivalent copies indistinguishable from 5 completely different information, so a storer will be unable to note that the 5 information are the identical and retailer them as soon as however declare a 5x reward.
However even right here we’ve two issues. First, there isn’t any strategy to confirm that the 5 copies of the file are saved by 5 separate customers. If you wish to have your file backed up by a decentralized cloud, you might be paying for the service of decentralization; it makes the protocol have a lot much less utility if all 5 customers are literally storing every little thing via Google and Amazon. That is truly a tough downside; though encrypting the file 5 occasions and pretending that you’re storing 5 completely different information will stop a single actor from amassing a 5x reward with 1x storage, it can’t stop an actor from amassing a 5x reward with 5x storage, and economies of scale imply even that scenario shall be fascinating from the viewpoint of some storers. Second, there’s the problem that you’re taking a big overhead, and particularly taking the false-redundancy subject under consideration you might be actually not getting that a lot redundancy from it – for instance, if a single node has a 50% likelihood of being offline (fairly cheap if we’re speaking a few community of information being saved within the spare area on individuals’s laborious drives), then you’ve a 3.125% likelihood at any level that the file…
