Despite the fact that Web3 is about permissionless and verifiability there are too many projects that have a fully decentralized back end with a fully centralized front end, hosted using AWS, GCP, Vercel, Netflify etc.
Those platforms are the tools more popular among developers because of an excellent developer experience and reasonable costs. Plus deploying a fully decentralized application is not straightforward.
In this blog post, we describe how to easily deploy and host a fully decentralized frontend – for a one-time payment of just $0.003 for ~1MB of assets.
Within the Outlier Ventures Product and Engineering team, we spend a lot of time talking and thinking about the new technologies that are helping to shape our decentralized future -you can see our Web3 Tech Radar here.
As well as spending time helping our portfolio companies make the right technology choices, we also build applications ourselves – for example, the decentralized pitch deck project that Scott Canning recently built on NEAR.
We decided to build more on the Web3 stack and write about the many different design choices along the way, and to document our findings in a series of blog posts.
In this post, we look at DePIN, specifically the topic of decentralized storage.
DePIN – decentralized physical infrastructure networks – allow developers to leverage the same kind of services offered by AWS, Google Cloud, Azure, etc, but in a truly decentralized way. Participants with spare compute power are incentivized with tokens to contribute their resources to what is effectively a cloud computing platform with no intermediaries and no overall central point of control or failure.
This post explains:
- The challenges of DePIN and decentralized storage
- How we solved our hosting challenges
- The technology choices we made
- A walk through of the Arweave basics
- An intro to arweave-bundle, a library to deploy a decentralized frontend directly from your CI/CD.
The challenges of decentralized storage
DApp aren’t really decentralized if the front end is hosted centrally. What’s the benefit of having permissionless contracts if users lack confidence that the front end doesn’t gate them, or contain an exploit or even worse, that a third party hasn’t been able to compromise the website?
When both the back end and front end are fully transparent, the oft-repeated instructions “Don’t trust, verify” become increasingly meaningful.
Deciding on a hosting solution
Some acceptance criteria to keep in mind when assessing where to host a dApp:
- Ensure immutability and reachability: Allow to track the versioning of the deployed files on a public infrastructure, the dApp should be “unstoppable”.
- Ease of deployment: simple to publish new content and capable of integrating with a CI/CD.
- Cost: As cheaply as possible.
- Crypto payment: a service where we can pay with a token without being forced to fallback to FIAT
With these requirements defined, we identified a few available options:
- IPFS (using a node hosted by us)
- IPFS (using a third party provider)
- Filecoin
- Storj
- Arweave
For the purpose of our dApp, we decided to minimize reliance upon any third-party solutions such as Piñata, Web3.storage or Infura. While they bring many advantages in ease of deployment, etc, they require API keys which we see as a single point of failure and gate keeping risks. Basically as long there is an API key the immutability and reachability can’t be guaranteed.
There are inevitably tradeoffs. At Outlier, we often embrace IPFS for hosting static content. We even discussed hosting our own IPFS node.This would have offered end users the chance to validate that specific deployments matched the CID of the code package that was deployed and we could leverage IPNS and IPFS together with ENS to both allow for a mutable IPFS CID pointer when updating the website’s content along with enabling domain resolution over HTTPS.
However, because IPFS itself is not an incentivized network, pinning the dApp to our own node could create a single point of failure, something we were strictly seeking to avoid.
Next, we investigated IPFS pinning solutions such as https://web3.storage/, which offered a great developer experience, but ultimately it is a centralized solution because it is based on keys which they hold, and which allows them to revoke the service at any time. It is a useful service but in this instance we decided that it would not offer us the fully decentralized experience we were after.
Another solution https://filebase.com/ was ruled out for the same reason.
Next, we considered Filecoin, which provides a layer over IPFS and incentivizes providers to offer storage space by pinning content to IPFS nodes. Providers are paid in Filecoin, and as both IPFS and Filecoin were developed by Protocol Labs, it is a smooth and joined-up experience.
We would probably have chosen Filecoin if it was not for their business model which depends on deals . As developers, we do not need to have the ongoing need to think about who to pay and how much, so for our particular use case, we also ruled out Filecoin.
Our third option, Arweave, presented us with technical tradeoffs, but was ultimately the solution we chose. While the cost per upload was minimal – around $0.003 each time, and the network has been around for five years now, there are fewer tools and resources than within the Filecoin and IPFS ecosystems, so we had to solve some interesting challenges – including developing our own library for bundling and uploading directly from our CI/CD.
A walkthrough of Arweave tools
This is where you can follow our steps to get familiar with uploading content to Arweave, aka the Permaweb.
First, we familiarised ourselves with the basics. After we settled on building a simple React App, our first stop was https://cookbook.arweave.dev/ where we found a load of useful resources for uploading data, including deploying apps, to Arweave.
Prerequisites
You will need to have Node installed, along with npx and pnpm. You will also need an exchange account where you can buy some $AR, which you will need for deployment. We used Arweave’s mainnet rather than the testnet, as it is cheap enough to experiment.
The first step is to create an Arwave wallet and fund it with some $AR. Funny enough, the simplest way to do this is from a CEX.
Check the funds have reached your address here: https://viewblock.io/arweave
Create your wallet:
$ mkdir upload-arweave
$ pnpm install arweave
$ node -e "require('arweave').init({}).wallets.generate().then(JSON.stringify).then(console.log.bind(console)) > wallet.json
Now you have a wallet.json which contains your private and public key. Find the address by running:
npx arweave-bundler address
Arweave Hello World
First create a simple web app. The simplest today is just `pnpm create vite` and pick the best default for you.
Then have a look at https://cookbook.arweave.dev/ where you can find a load of useful resources for uploading data, including deploying apps, to Arweave or keep going through the blog post for an opinionated approach. Create a file named ar-deploy.js and paste the following:
import Arweave from "arweave";
import fs from "fs";
// load the JWK wallet key file from disk
const jwk = JSON.parse(fs.readFileSync('./wallet.json').toString());
// initialize arweave
const arweave = Arweave.init({
host: "arweave.net",
port: 443,
protocol: "https",
});
const tx = await arweave.createTransaction(
{
data: "Hello world!",
},
jwk
);
await arweave.transactions.sign(tx, jwk);
arweave.transactions.post(tx).then(console.log).catch(console.log);
console.log(`https://arweave.net/${tx.id}`);
Run it with `node ar-deploy.js` and … you just deployed your first content to the Permaweb! 🙌
This is nice but it isn’t useful yet.
Upload a file
The next step is to upload an actual file rather than a string. To do this you need to deal with tags. Basically the gateway needs to know what type of data is serving (E.g. image/png)
import Arweave from 'arweave';
import fs from "fs";
// load the JWK wallet key file from disk
let key = JSON.parse(fs.readFileSync("walletFile.txt").toString());
// initialize an arweave instance
const arweave = Arweave.init({});
// load the data from disk
const imageData = fs.readFileSync(`iamges/myImage.png`);
// create a data transaction
let transaction = await arweave.createTransaction({
data: imageData
}, key);
// add a custom tag that tells the gateway how to serve this data to a browser
transaction.addTag('Content-Type', 'image/png');
// you must sign the transaction with your key before posting
await arweave.transactions.sign(transaction, key);
// create an uploader that will seed your data to the network
let uploader = await arweave.transactions.getUploader(transaction);
// run the uploader until it completes the upload.
while (!uploader.isComplete) {
await uploader.uploadChunk();
}
Upload multiple files
Things are starting to get interesting but there is a last concept to grasp before traversing a directory and publishing all the files to the Permaweb: understanding the concept of a manifest..
When uploading files to Arweave, each file is assigned its own unique transaction ID. By default these ID’s aren’t grouped or organized in any particular manner.
So a manifest is a JSON file that contains all the IDs for a group of files. It also contains an index attribute which points to an alias pointing to any transaction id.
{
"manifest": "arweave/paths",
"version": "0.1.0",
"index": {
"path": "index.html"
},
"paths": {
"index.html": {
"id": "cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI"
},
"js/style.css": {
"id": "fZ4d7bkCAUiXSfo3zFsPiQvpLVKVtXUKB6kiLNt2XVQ"
},
"css/style.css": {
"id": "fZ4d7bkCAUiXSfo3zFsPiQvpLVKVtXUKB6kiLNt2XVQ"
},
"css/mobile.css": {
"id": "fZ4d7bkCAUiXSfo3zFsPiQvpLVKVtXUKB6kiLNt2XVQ"
},
"assets/img/logo.png": {
"id": "QYWh-QsozsYu2wor0ZygI5Zoa_fRYFc8_X1RkYmw_fU"
},
"assets/img/icon.png": {
"id": "0543SMRGYuGKTaqLzmpOyK4AxAB96Fra2guHzYxjRGo"
}
}
}
To summarize, in order to upload a Web3 front end to Arweave, you need to traverse the directory, create a transaction for each file, append the appropriate content type and then submit another transaction with the manifest crafted as above, pointing to all the transaction id of each file.
Arweave transactions are cheap but creating so many transactions is far from ideal particularly when the network is congested.
While this solution would have worked, it was sub-optimal, so we moved to our next solution: Arweave bundles (ANS-104)
Arweave bundles
A transaction bundle is a special type of Arweave transaction. It enables multiple other transactions and/or data items to be bundled inside it. Because transaction bundles contain many nested transactions, they are the key to Arweave’s ability to scale to thousands of transactions per second.
Our main requirement was to be able to bundle files and assets together so that we could upload in an atomic fashion after building the app and thus have a properly versioned dApp, rather than uploading individual files piecemeal. This also had cost advantages as it meant we were paying just once for the upload.
We considered using directly Iris (formerly known as Bundlr), which has a nice developer experience including allowing payments with many tokens, supports different chains and allow but it also add an extra layer of fees on top of Arweave and other functionalities that we didn’t need.
You can take advantage of bundling without using Irys services by taking advantage of their open-source library called arbundles and adding a bit of glue.
import { bundleAndSignData, createData } from "arbundles";
const dataItems = [createData("some data"), createData("some other data")];
const signer = new ArweaveSigner(jwk);
const bundle = await bundleAndSignData(dataItems, signer);
Our Arweave utility: arweave-bundler
We decided to create our own Arweave utility for leveraging the bundling functionality directly in our CI/CD and get the steps consistently repeatable.
In the public repo for Arweave Bundler, you will find a GitHub Action and CLI to upload static assets from a directory, which is ideal for publishing Single Page App (SPA) or other static contents to Arweave.
If you want to use the GitHub Action, config is as below:
uses: outlierventures/arweave-bundler-action@v0.3.1
with:
directory: build/
private-key: ${secret.ARWEAVE_PRIVATE_KEY}
dry-run: false
network: arweave.net
Ensure you add the private key to the GitHub Secrets for your repo. If you would prefer to use the CLI, follow the steps below to bundle and deploy your web app:
npx arweave-bundler upload build/ --private-key
${PRIVATE_KEY}
Always ensure your private key is stored as an environment variable.
Conclusion
Arweave provides a cheap and convenient way to permanently store a web application frontend, proving that decentralized storage is cheaper than its centralised equivalent. On the flip side, while centralized storage providers have a proven business model, we will need to wait to see if the Arweave model can pass the test of time. The main challenge is the incentive alignment, aka the fees. In a system without rent and with a one-time-only fee, the nodes could decide to walk away if they stop receiving enough rewards.
While the most complex part of this discovery process was finding our way without having access to extensive documentation or examples, we ultimately found a resilient solution and we hope that other developers will benefit from using arweave-bundler to simplify their deployment process.
Our next task was to look at decentralized domain management. Our next post will focus on the challenges and decision-making involved with implementing ENS and ANT (Arweave Name Tokens).
