TL;DR

• WeaveDB is a decentralized NoSQL database as a smart contract on Arweave.

• The query performance is comparable to traditional web2 alternatives.

• WeaveDB will achieve virtually unlimited scalability.

• By using Lit Protocol, WeaveDB enables cross-chain data bridges.

• Lens Protocol will be natively integrated into its core.

• Data encryption with Lit Protocol.

• Large data uploads with Bundlr.

Updates

WeaveDB has undergone several developments since its initial introduction over five months ago.

These developments include:

• Introducing WeaveDB at an Arweave conference in Singapore.

• A $900K pre-seed funding round from strong partners around the world.

• The formation of a strong founder team.

• Incorporation in Abu Dhabi Global Market, UAE.

• Partnerships and collaborations with pivotal projects.

Most importantly, significant progress has been made in the development of our technology, including our one-stop Web Console, and that is what I would like to discuss in this article.

Re-Introducing WeaveDB

WeaveDB is a decentralized NoSQL database implemented as smart contracts on Arweave. It can be thought of as a Firestore for web3. As we discussed in our first general introduction, Storage-based Consensus Paradigm (SCP) of Arweave enables unlimited computation with inexpensive permanent storage by separating computation and storage. Computation can be deterministically executed off-chain without any financial cost, which enables lazy evaluation on the client side. By taking advantage of these unique traits, WeaveDB offers a powerful, performant, scalable, and composable decentralized alternative to web2 database components.

Thanks to feedback and insights from our conversations during the project initiation and fundraising phase, we now have a clear idea of what WeaveDB should be. We have spent the past few months tirelessly implementing new ideas and improvements, and we believe that WeaveDB is about to usher in a paradigm shift in dapp development. We are grateful for the support of everyone who shares our vision.

Performance Optimization with gRPC Nodes

Storage-based Consensus Paradigm

Once again, the key feature that sets Arweave's smart contracts apart from other blockchains is the Storage-based Consensus Paradigm (SCP), a new paradigm that enables unlimited off-chain computation with inexpensive storage. On other blockchains, one would need to pay Gas for both computation and storage, and every node needs to store the entire state and execute computations to sync with the network. This approach is very expensive and greatly limits what you can do with it. However, with Arweave's SCP, you only pay for the amount of data required for computation to be stored permanently, and you don't pay for computation. The storage cost is very inexpensive thanks to Arweave's hyper-efficient consensus logic called SPoRA (Succinct Proof of Random Access). This unique approach makes Arweave an ideal platform for building decentralized applications that require large-scale computations and data storage.

While SCP offloads computation from the blockchain nodes to the client side, this new paradigm introduces new performance and scalability issues. While lazy evaluation may seem like an elegant solution to the bottleneck of expensive computation, it can cause significant inefficiencies and burdens for client dapps.

For instance, if there are a million transactions for a contract, the client side needs to download all the transactions and compute them to build a cache and obtain the latest state. This process must be repeated by all clients before they can use the dapp. This can take minutes or even hours to start a dapp for the first time, which can significantly impact the user experience.

WeaveDB gRPC Node

To improve performance and scalability, we have developed a gRPC node that removes the burden from the client side and further optimizes database usage. Through layers of optimizations and hacks, we have achieved a remarkable performance of 10-20ms for read queries and 30-50ms for write queries. This is even faster than the smart contract finality of Warp, which is a significant improvement for dapp developers.

The first step was to remove computation from the client side, which is essential for better performance. However, even after removing the computation, we were left with a relatively slow query speed of around 1 second for reads and 4 seconds for writes. While this may be sufficient for some dapps built on smart contracts, we wanted to provide a viable alternative to web2 backend solutions. To achieve this, we continued to brainstorm and implemented various hacks to improve the query speed.

Virtual State

The slow speed of the Warp SDK is mainly due to the lazy evaluation nature, which requires at least one HTTP call before returning the latest state or writing to the smart contract. To ensure the validity of transactions, the SDK checks the current state and executes a dryWrite before writing a new state. This step is crucial to prevent the accumulation of invalid transactions since the smart contract doesn't evaluate anything when transactions are committed.

To eliminate the need for state checks, we created a mechanism that maintains a virtual state at each node, which is synchronized with pub/sub notifications and a cloud Redis cache. This real-time state includes uncommitted transactions and is updated through pub/sub notifications directly from the Warp sequencer. As a result, the WeaveDB virtual state can be many commits ahead of the current Warp block, enabling the removal of the latest state checkups altogether. Although there may be some niche cases where the virtual state returns incorrect results, any discrepancies will be cleared and rolled back by the actual state within 1-3 seconds via the pub/sub updates.

Furthermore, we plan to implement a solution called super-batch queries with a relayer to facilitate mass updates of queries without any conflicts between nodes. We believe that WeaveDB could become an L3 solution to Warp, which itself is an L2 solution to Arweave. In fact, we are currently considering building our own L3 blockchain to achieve even greater decentralization.

Redis State Cache

We've replaced the local cache layer of each node with Redis to enable fast inter-node sync and implemented a snapshot inventory to enable quick node boot-up and recovery from the latest validated snapshots in case of any issues. Additionally, WeaveDB contracts can be health-checked using the hash of chained valid transaction IDs.

WeaveDB Manages WeaveDB Nodes

To coordinate all of the node operations, we aimed to maintain our vision of replacing web2 infrastructures and avoid introducing any web2 cloud databases. Instead, we leveraged a WeaveDB instance to manage the nodes. This involved using one on-chain WeaveDB instance and developing an off-chain version of WeaveDB to handle internal communications between nodes. With this setup, it is now possible to use WeaveDB without backing up the state on Arweave, providing a significant UX improvement when testing in development. Furthermore, the off-chain WeaveDB enables the virtual state mechanism to be achieved. WeaveDB nodes are autonomously managed by WeaveDB itself, which is truly remarkable!

With all these optimizations in place, the write query speed of WeaveDB is now between 30-50ms plus the distance to the node. This is comparable to the speed of web2 cloud database solutions and, in some cases, even faster. With these improvements, users can replace web2 databases with WeaveDB without worrying about slow queries affecting the user experience.

On-Chain Generated Cache Strategy

We have even more ideas for further optimizations. One of the fastest ways to interact with data is through Redis KV cache, which can handle 20K queries in 1ms. If we can have users interact only with a Redis cache, the query speed will be less than 1ms plus distance. WeaveDB already has on-chain indexing built-in, and efficient cache-building strategies depend on the index diffs after data updates. This means we can incorporate the most efficient cache strategies into WeaveDB and have them built-in and on-chain.

Unlimited Scalability with KV Storage inside Smart Contract

The scalability of the Arweave smart contract is bottlenecked by storing everything as a contract state. This is not specific to Arweave, but rather a common issue across many other blockchains where contract states are significantly more expensive. Storing the entire database as one large contract state is not portable and wastes a significant amount of bandwidth for small operations. To address this, the Warp team has developed a solution, and WeaveDB is transitioning from storing everything as a contract state to using local KV storage internally.

With the experimental Warp implementation, it's now possible to use KV storage within the smart contract logic, which is a significant development for scalability. First, it eliminates the need to load the entire database when performing small computations. Second, the KV storage capability is just an interface for get and put, meaning the actual technology used to implement the interface can be anything without compromising the determinism of the state. For instance, simple local storage solutions such as levelDB and LMDB could be used, or a fully decentralized and hyper-scalable solution could be built on top of it using supercomputers. This is how virtually unlimited scalability can be achieved.

WeaveDB has made significant progress in moving its internal data storage logic to work with the KV storage, and we are eagerly awaiting the release of a new, scalable version of WeaveDB once the Warp team is ready. Currently, there are some restrictions with foreign contract read/write, which limit some of WeaveDB's features and disable composability. However, we are confident that these issues will be resolved on the Warp side, and we are actively working to ensure that WeaveDB is fully compatible with the latest Warp updates.

Cross-chain Data Relayer with Lit Protocol

Despite WeaveDB's impressive performance and scalability, we continue to pursue further enhancements. While Arweave excels at storing permanent data, it is less equipped to handle other types of states, such as tokens and NFTs. Therefore, it would be immensely beneficial if Arweave smart contracts could securely interact and compose with states from other blockchains or even web2 API endpoints.

To address this challenge, we have developed a relayer solution using Lit Protocol. This allows us to bridge any data from outside the Arweave smart contract environment into WeaveDB. There are four steps to this process that can help to explain why and how it works.

A Simple Relayer

To bridge outside data, the first step we took was to develop a relayer mechanism. WeaveDB's authentication and access controls rely solely on cryptography and do not involve any centralized components. However, to securely bring in external data, relying solely on the user's signature is not enough because they cannot be trusted by the system. A relayer can add additional data to the user's write queries, provided that the user trusts the relayer. WeaveDB verifies both the user's and the relayer's signatures to process the data update. However, this approach is centralized, as it requires trust in the relayer to process the data accurately.

Multi-Sig Relayers

To solve the trustful relayer problem, WeaveDB has developed a multi-sig relayer mechanism, which aggregates signatures from multiple validators to achieve a level of decentralization and trustlessness. However, decentralizing the relayers is a significant challenge for any network.

Example Relayer Dapp with Ethereum NFT

To give an example of how the relayers work with EIP712 signatures, our demo dapp showcases how the relayer first validates the ownership of an Ethereum NFT and then adds the owner's address to the write query, so only NFT owners can write to the WeaveDB contract.

You can follow this tutorial and try the demo dapp.

PKP and Lit Actions

To address the challenge of trustful relayers, we have adopted a more secure solution using Lit Protocol. With Lit Protocol, we can verify the execution logic and eliminate the need for decentralized relayers with multiple validators.

Lit Protocol uses threshold cryptography to divide a private key among multiple parties and requires consensus from the majority(2/3) to generate a signature. This mechanism is combined with IPFS-stored immutable scripts, called Lit Actions, and called programmable key pairs (PKP). Validators in the Lit network execute Lit Actions in combination with Access Conditions, enabling validation of any combination of logic from different chains and even arbitrary logic using web2 APIs. With Lit Protocol, WeaveDB relayers can now be decentralized at the network level rather than the individual relayer level.

PKPs are managed by an ERC721 contract, which means that the owner of the NFT associated with a certain PKP controls where the private key can be used by granting access to ipfs hashes. Since both the NFT and the IPFS script are immutable and verifiable on-chain, we are one step closer to achieving a verifiable bridge. However, we still need to trust the NFT owner who controls the PKP.

Mint/Grant/Burn NFT

Lit Protocol has a solution to the trust issue. A smart contract function mintGrantAndBurnNext can be used to mint an NFT, grant the PKP access to an IPFS script, and burn the NFT to relinquish ownership in a single transaction. This ensures that there is no owner of the PKP, and the PKP can only produce signatures for a specific immutable script stored at a specific IPFS location recorded on the blockchain. As a result, the script used to generate signatures with the PKP is both verifiable and immutable, and the PKP cannot be used for any other purpose. You can now trust and verify signatures by the PKP as a result of executing that specific script.

This completes the four-step process of how WeaveDB brings external data into Arweave in a verifiable manner.

Data Encryption with Lit Protocol

Lit Protocol offers an additional crucial feature to WeaveDB, which is data encryption. By incorporating Lit Access Conditions with the WeaveDB relayer, WeaveDB can now store encrypted data while enabling you to manage read access using flexible conditions involving states from other blockchains.

We have a tutorial and an example dapp, which allows NFT holders to send private messages.

Lens Protocol Native Integration

WeaveDB relayers offer a wide range of cross-chain composability options. To effectively demonstrate its potential, we are integrating Lens Protocol into the core WeaveDB contract and treating Lens Profiles as first-class citizens. Users can authenticate with their Lens Profile NFT, and WeaveDB treats the token ID as an authorized account to write to WeaveDB. A Lit Action, as a relayer, will ensure with a signature by PKP that the user is the owner of the Lens Profile.

This integration has significant implications. Many dapps claim decentralization is not fully decentralized at present because using centralized components is technically necessary, particularly for app logic. This is because on-chain indexing is extremely difficult and impractical when it comes to providing a good user experience. If we lose the centralized components for some reason, the dapp stops working because we lose the logic layer for the app.

WeaveDB addresses this problem. WeaveDB comes with built-in on-chain indexing, so if you build dapps using WeaveDB, your dapp will live forever because it won't lose the app logic in any way. It is stored permanently in a way that anyone can access it forever. Additionally, WeaveDB has a built-in cron job feature. Once the logic is defined in the contract, it will keep executing that task automatically and forever without creating additional transactions until instructed otherwise by the logic. This differs from other decentralized/centralized networks that need to make periodic transactions to keep the task running.

You can try our lightning-fast Lens Demo Dapp here.

Large Data Upload with Bundlr

When building dapps, you don't want to store large media files directly in a database. But you would store them in storage and store some metadata in a database to link to it. For example, if you store an image on Arweave, you can store the transaction id in WeaveDB, then use them together in your dapp. But this cannot be really done with only the client-side code because you cannot really trust the client to store correct transaction ids. The WeaveDB relayers can be used for this purpose. You can have a relayer upload the image to Arweave and get the transaction id and store it with your write queries. If this logic is verifiable with an immutable Lit Action script and a PKP, you can safely bind media files on Arweave with your dapp logic.

We have a tutorial and demo dapp with a Bundlr integration for large data uploading here.

AI Integration

WeaveDB already boasts web2-like performance, unlimited scalability, and cross-chain capability - what more could you want? Well, there is one more thing: AI integration. I'm not talking about big data analysis or anything like that - I'm talking about AI as a dapp builder. Have you tried chatGPT or GitHub Copilot? I've been feeding them code, and AI can actually write better logic than human devs in a snap. What we have right now from OpenAI is just a primitive version of what's to come. They can even write complete Solidity smart contracts because Solidity is a relatively simple language, and I'm sure AI can make contracts more secure than regular human devs could.

I don't think human developers will be the primary force for building dapps in five years. At WeaveDB, we are experimenting with AI, and AI can write comprehensive unit tests for our contracts within a minute or so, which could take us days or even weeks otherwise.

The biggest strength of WeaveDB is the capability to build any level of highly advanced and complex logic with the functional programming utility called FPJSON. FPJSON turns a simple JSON array into functional programming, which is inspired by RamdaJS, which is probably inspired by Haskell. Building secure access control rules and data validation logic is highly complex - maybe that's why there aren't any definitive decentralized DB solutions out there. However, WeaveDB solves this by introducing functional programming in JSON, which can be stored as a smart contract state and composed with other logic. FPJSON is also the magic behind the autonomous cron jobs, which periodically update the contract state without committing transactions.

This approach is incredibly AI-friendly. Right now, it can be a bit challenging to build advanced logic with WeaveDB unless you're familiar with functional programming, but we will soon integrate chatGPT into the Web Console so you can set up an advanced WeaveDB instance within a minute. It will be far more pleasant than NoCode development UX. Over time, we aim to build advanced intelligence on top of WeaveDB to accelerate dapp development.

WeaveDB will merge AI with smart contracts.

(Just between us, this article was written by chatGPT based on my draft...)

Lit / Lens Integration Demo and Workshop at ETHDubai

We have a simple demo dapp showcasing the Lit/Lens integrations. With the help of Lit, you can sign into the dapp using your Lens Profile NFT and post messages to your timeline.

Check out the demo dapp at https://relayer-lens-lit.vercel.app/ and experience the query speed.

We are giving a talk and a workshop at ETHDubai on March 15th and 16th. Our workshop will demonstrate how to build this dapp in just 30 minutes.

By participating in the workshop on the 16th, you will be whitelisted for our managed node service soon to be launched. This service enables you to deploy WeaveDB contracts and use our endpoint with lightning-fast query speeds.

Don't miss out on this opportunity to learn about WeaveDB.

We hope to see you there!