Introduction

The port of the SingularityNET decentralized AI platform from Ethereum to Cardano, currently actively underway, marks one dramatic milestone in AI/blockchain convergence. SingularityNET began on Ethereum in 2017 but was always envisioned as multi-chain in conception and ambition. With a mid-2021 hard fork, creating a Cardano Native AGIX token and associated functionality across multiple blockchains became possible. The launch of our Ethereum-Cardano token converter bridge in April 2022 and the launch of our hybrid staking solution in November 2023 bring this vision to life.

The development of these products involved a series of challenges that required both conceptual and architectural innovation and course adjustments. By remaining adaptable and focused on our long-term vision, we designed a functional and robust solution that is architecturally solid and, critically, safe and reliable for users. In retrospect, these challenges were not mere obstacles; they refined our approach and led us to this unique staking solution, specially adapted to the needs of the blockchain foundations on which they are built and the SingularityNET community and token utility.

A secure and user-friendly staking platform was not the only outcome of this development journey. Guided by our partner MLabs, our team has mastered Haskell, Plutus, and Cardano libraries, emerging as a force ready to tackle any Cardano-based development confidently.

In this article, we invite you to embark on a development journey on the Cardano blockchain to witness firsthand the tranches of challenges we faced and how we addressed them, unraveling their impact on our development process and staking solution.

For better readability, we have categorized these challenges based on their thematic threads, not their precise chronological order. Dive into each theme and witness the transformative power of our team’s problem-solving capabilities. We hope this will be a resource of value to other blockchain developers experiencing their own journeys, as well.

Note on the UTxO-based design of the staking system

The staking system leverages a UTxO-based design with a lock-less on-chain structure that empowers multiple users to access and interact with staking pools seamlessly. The specification identifies the main problem of implementing a staking pool — the inherent big and global mutable state — and proposes a data structure to mitigate it and take advantage of the good properties of the UTxO model that Cardano employs: speed and scalability. In this regard, the design makes no concessions and aims to fully leverage the affordances offered by Cardano to achieve the maximum feasible scalability.

Bonded and Unbonded staking — functional specifications

In the original requirements, we aimed to create two distinct staking pool types: “Bonded” and “Unbonded.” To maximize efficiency and minimize development time, we made a strategic decision to prioritize Bonded pools first. We chose this approach anticipating that it would allow the team to leverage the core code and design principles established for Bonded staking when later developing Unbonded pools, ensuring consistency and stability across the platform.

Bonded staking issue — total reward amount

Impact on Fund Safety and Security: None
Impact on User experience: High
Impact on Performance: High
Impact on development time: High

As part of the original requirements, the goal for this development was to provide a specific reward amount for users committing their tokens to the pool. This implied the Admin of the bonded pool would allocate any necessary amount of AGIX each epoch to appropriately reward all stakers without a maximum limit in place.

Alternatively, a maximum pool size could have been implemented. However, we concluded that limiting the amount of deposited tokens would have significantly impacted system performance. SingularityNET, on the other hand, could not responsibly guarantee a certain amount of rewards without a cap.

Solution:

To address this challenge, we designed a dynamic reward system. During each Reward Distribution Period, the Admin defines a reward pool proportional to the number of participants who committed their tokens in the previous epoch. Rewards are then distributed among all active stakers based on their AGIX balance.

This approach allows for flexible reward determination based on various factors, including:

• Total staked amount;
• Available tokens in the staking rewards pool;
• Projected growth of the staking pool;
• The Decentralized AI Platform’s need for liquidity.

Based on these factors, the Admin calculates the rewards per user for the next staking window during each Reward Distribution Period. If a participant decides to withdraw their tokens during this window, they forfeit their rewards for that epoch, but the rewards for other participants remain unchanged.

This hybrid system ensures fairness and sustainability while providing participants with transparency.

Unbonded staking issue — system flexibility and stability

Impact on Fund Safety and Security: None
Impact on User experience: High
Impact on Performance: High
Impact on development time: High

Unbonded staking presented several challenges. While offering the convenience of staking and withdrawing tokens at will, it resulted in contention levels that exceeded the performance capacity of our system.

Implementing countermeasures to address this issue restricted user freedom to an unacceptable degree, preventing us from finding a compromise that met our user experience standards. Finding a reasonable compromise between unrestricted user freedom and system stability proved elusive.

Solution: Semi-bonded staking

We ultimately opted for a solution that, we believe, ensures fairness for all participants while maintaining system performance. Participants can withdraw their stake at any time except during the Reward Distribution Period, granting them more control over their stake in case of unforeseen personal circumstances.

However, only users who commit their tokens throughout the entire staking epoch are eligible to receive rewards. This means that while the total reward amount may fluctuate based on withdrawals during the epoch, individual rewards for those who remain committed for 11 days stay constant.

Conclusion on the functional staking model

We are confident that our innovative approach to staking on Cardano strikes the best balance between user flexibility and platform stability. Our hybrid staking solution is ideal for AGIX token holders seeking a secure and rewarding way to support the operations of our Decentralized AI marketplace.

Lack of testing support in the CTL

Impact on Fund Safety and Security: None
Impact on User experience: None
Impact on Performance: None
Impact on development time: High

We realized that the Cardano Transaction Library (CTL) lacked robust testing support, requiring manual testing for each feature and leading to low test coverage in the codebase, which resulted in challenges later during development. Fortunately, the Plutip testing framework was available, and we’re thrilled to report it now integrates seamlessly with CTL. This integration has been instrumental in improving our debugging capabilities and overall code stability.

When our development process began, the state of testing tools in the Cardano ecosystem was nascent, mirrored by the limitations of the CTL. The CTL was created to address a missing piece — providing browser-based transaction submission, a functionality absent in the official Plutus Application Backend (PAB) released with the Alonzo update. Even if PAB had been the chosen framework, robust testing would not have been guaranteed. Its ‘emulation traces’ approach simulated a mock blockchain, falling short of real-world scenarios and potentially masking critical bugs.

Solution:

Our team conducted rigorous unit testing, integration tests, and load/performance tests, detecting several issues we could not identify during the earlier stages of the development process due to the lack of integrated testing support.

While these discoveries resulted in unforeseen delays towards the end of the development process, they significantly mitigated the potential impact these issues could have had if left unaddressed. Ultimately, this thorough testing strategy yielded a robust and well-tested solution, ensuring its stability and security.

Transaction sizes becoming too large

Impact on Fund Safety and Security: None
Impact on User experience: None
Impact on Performance: High
Impact on development time: Medium

Initially, the design required the pool Administrator to update several staking accounts proportional to the number of users. With a large enough pool, this approach generated transactions exceeding the maximum size allowed by the Cardano blockchain, leading to the transactions being deemed too large or rejected. Recognizing this unforeseen scalability issue, this feature had to be amended.

Solution:

A batch function was added to the administrative actions that could be robustly and transparently automated.

Debugging challenge

Impact on Fund Safety and Security: None
Impact on User experience: None
Impact on Performance: None
Impact on development time: Medium

Debugging certain features was significantly hampered by the lack of mature tools, leading to time-consuming troubleshooting efforts. Our pools heavily rely on time logic to deny or allow certain operations. Any manual test for this logic is impossible to complete in less than 5 minutes, meaning that the feedback loop for developing-testing was slow and focus-intensive for our development team.

Solution:

Standard debug times were unalterable, and so were incorporated into the development and resourcing timeline — this was treated as a necessary investment needed to reach our quality standards.

SDK and related improvements

Integrating the SDK was crucial for our project, but during the process, we encountered several unaddressed needs. Key functionalities were absent, such as automated administrative actions, hindering efficient pool management. We had to tackle unforeseen scenarios like administrator-initiated withdrawals and state recovery after unsuccessful updates, ensuring robust operation. Addressing these issues naturally extended development time, but we maintained a separation between on-chain and off-chain code, focusing our efforts on enhancing the off-chain functionality and SDK integration without changing the audited on-chain code or compromising security.

Some examples:

Not enough ADA for staking.

Impact on Fund Safety and Security: None
Impact on User experience: low
Impact on Performance: none
Impact on development time: low

Users with sufficient AGIX for staking but minimal ADA token balances could not commit their tokens to staking. The core problem was that the user had enough ADA for the transaction, but still, the operation was not permitted by the network functions.

This stemmed from a Cardano blockchain feature that manifests in interaction with wallets. When calculating the execution of any transaction, there is a requirement for the balance of the wallet: after the transaction is completed, the balance must be either zero or greater than ‘minLovelace’ (approximately 0.96975 ADA for user output with tokens in the spring of 2023) that is used to prevent ledger congestion.

Solution:

In some services, to meet this condition, the amount insufficient for the balance is included in the commission, so that the commission may be more than necessary by almost a whole unit of ADA. However, prioritizing cost-effectiveness for our users, we opted for a transparent and user-friendly solution: a clear warning before staking, informing users about the minimum ADA requirement.

Limitations in adding many new users at the same time

Impact on Fund Safety and Security: None
Impact on User experience: low
Impact on Performance: high
Impact on development time: low

The list of users who participate in staking is stored as an association list, its structure involves building a link between a previously added element and a new user. The result is that if there is only one user in the list, new users can only be added before and after it, i.e., only two new users can come at a time, the rest will get an error.

Solution:

To solve this problem, we decided to add several technical users into staking at the very start to create more ‘free places’ for adding real users.

Not enough network collateral to withdraw

After completing functional testing and conducting short end-to-end scenarios, the next step was to simulate end-to-end testing. This test simulated 11 consecutive staking windows, encompassing user enrollment, withdrawals, Administrator actions, and reward distribution, offering a robust evaluation of our system’s performance under realistic conditions.

Our stress testing confirmed the accuracy of deposit and reward calculations, a testament to our robust core functionalities. However, on a large number of cycles, it turned out that the output algorithm uses a large number of UTXOs and, as a result, the volume of the script required to be executed becomes very large and redundant, and because of this, the Cardano network evaluates its execution at 7–11 ADA. In Cardano, “collateral” exists in the wallets for situations like this. Usually, the amount of collateral for payment for transactions is fixed and does not exceed equal to 5 ADA (at the time of spring 2023) and cannot be changed manually. Thus, the withdrawal operation becomes impossible and the user cannot withdraw their AGIX.

Solution:

We improved the withdrawal selection algorithm of asset UTxOs, in which rewards are stored. Now, there is only one asset UTxO per user, which removes the issue of high ADA collaterals. This fact is verified by storing the hashed user’s pkh in the asset UTxO datum. This way, we can track which asset UTxO belongs to which user and ensure only they can access their reward.

Auditing

First audit by Canonical

Following the completion of the smart contracts, a rigorous audit by Canonical identified four bugs, three of which were classified as ‘Important.’ These bugs were promptly addressed and re-verified by Canonical, leaving us with a heightened sense of confidence in the design and implementation of our smart contracts.

Second Audit by Canonical

Improved code coverage during testing revealed previously concealed bugs within the on-chain codebase. These issues primarily manifested as erroneous checks in the validator, potentially preventing users from staking or withdrawing under certain conditions. We promptly addressed these bugs and, upon request, successfully underwent additional validation from Canonical.

Denial of service vulnerability

Impact on Fund Safety and Security: Low
Impact on User experience: High
Impact on Performance: High
Impact on development time: High

After all our tests and audits, we continued our testing process, and eventually, our partner MLabs discovered a serious vulnerability in the protocol at a relatively late stage.

The specification had a lurking vulnerability regarding how the user’s assets were placed and consumed in the pool, which allowed a malicious user to execute a denial of service attack against the pool — while at the same time profiting from the minimum ADA that was locked in it and that the Cardano blockchain enforces for any given asset.

An attack at this vulnerability would not have put the staked assets at risk and would only give the attacker a very minimal profit, but it would be a major inconvenience in the Protocol. Therefore, despite not being a very likely scenario, we decided that this issue needed to be fixed. This was new information at a very late stage in the development process that signified a very substantial setback in the development timelines.

Solution:

After much exploration, we decided that an off-chain approach to this issue would not be sufficient, so we stepped back and designed an adapted smart contract, being aware that this would require a fresh audit.

On the bright side, this unwanted situation offered us the opportunity to thoroughly clean up the on-chain and off-chain code.

During the entire development process issues continued to surface that were related to the challenges described at the top of this article, concerning bonded vs unbonded staking and the limited testing functionality. While we had good solutions in place to solve these issues, this led to unused code in the smart contract and some functionalities in the off-chain code that could have been achieved more elegantly in the on-chain code.

So, while this vulnerability was unwelcome, we did turn it into an improvement of the final solution by refactoring and streamlining everything and removing any unused code from the smart contract, making the final product leaner, more efficient, and maximally reliable.

Architectural changes: CTL, Kupo, Ogmios, and Plutarch

Impact on Fund Safety and Security: None
Impact on User experience: None
Impact on Performance: high
Impact on development time: Medium

In our architecture, we were heavily dependent on CTL; a third-party library and an essential component in the Cardano Tech Stack. CTL is a stable and popular component used for writing off-chain code that will work on the client side in a browser environment together with JavaScript and TypeScript languages

In the original architecture, we used two elements to conduct transactions: Ogmios and Ogmios-datum-cache. In the process of high-load tests, we discovered problems with Ogmios-datum-cache, which could not withstand the intensity of requests and would drop client connections faster than we would like for our specific use case.

Solution:

Due to these issues, we decided to select an alternative to Ogmios-datum-cache: Kupo. To integrate Kupo correctly, some changes were required to the CTL library. The CTL team was already working on this, and updating CTL fixed this problem and resulted in a more elegant and fault-tolerant architecture.

At this stage, while we were updating the on-chain code, we ran into another issue. We realized that the PR for updating the CTL version was introducing a build failure for the on-chain code, which complicated a number of factors.

The reason for this was that our Plutarch version used dependencies and a configuration that was very difficult to adapt to newer CTL versions. After some explorations, we decided that the best approach was to split the on-chain and off-chain code into separate repositories so there would be no more dependencies between the two.

The result is a modernized, elegant, and robust architecture with the latest and up-to-date versions of CTL and Kupo.

Third Audit by Canonical

At this stage, we requested a third and final Audit from Canonical and asked them to be as thorough as possible, and not rely on any previous outcomes (especially due to the significant changes in the smart contract and software architecture). The result was an almost perfect audit with no action items, but only some informational issues aimed at improving the maintenance of the code.

Staking Launch Day:

After rigorous testing of all components and a comprehensive end-to-end simulation, we were poised for launch. However, during a final pre-launch system check, a minor issue emerged regarding the creation of long staking windows.

This technical hiccup prevented the configuration of staking windows of our selected duration. Primary analytics turned out to be disappointing due to the limitation of transaction validation periods on Cardano to 72 hours. This would have meant that it would be technically impossible to create staking pools with windows durations of more than 72 hours, which the team found extremely strange — and worth looking at more closely.

Through an in-depth analysis of the off-chain logic of our solution, we identified a small oversight in the validation checks — when checking all the data, the check for the transaction validation period was too small, which did not allow creating a window of the required duration. This swift identification and prompt rectification allowed us to adapt and adjust the window creation logic, ensuring a successful launch with the desired functionality.

Because Cardano operates differently than EVM-based blockchains, users familiar with those platforms might have questions or encounter unfamiliar features. We have addressed these in our comprehensive FAQ section which delves into unique aspects of Cardano and explains them in detail to ensure a smooth and informed user experience. If you need a helping hand to interact with our staking portal, please check out our staking guide.

Conclusion

Now that our hybrid staking solution is live, we are looking back at a time filled with deep challenges, long hours, and many frustrations — but ending with a sense of accomplishment and satisfaction.

Together with our partner MLabs, we tackled every obstacle, squashed every bug, and were driven by one unwavering goal: delivering a secure, efficient, and reliable staking pool for you, the SingularityNET and Cardano communities.

We also learned that the Cardano ecosystem is not without its challenges and, especially at the beginning of this development process, was not yet as mature as we hoped it would be. On the upside, we also have seen many improvements being developed, and we have taken full advantage of those by implementing them in our final solution.

Perhaps the biggest advantage of this all, is the experience we gained, the relationships we built and, most of all, the in-house knowledge we have collected. During the process, we have trained our internal team to master the intricacies of Haskell, Plutus/Plutarch, and many tools and libraries in the Cardano ecosystem. This knowledge will help us move forward with all the new functionalities, tools, and solutions we are planning to build upon Cardano.

With this sometimes hard, but valuable experience below our belt, we feel more motivated and empowered than ever, to continue our journey of integrating with and building on Cardano and connecting deeper with the Cardano Community.

Subscribe to Synapse, SingularityNET’s newsletter, and step into the future of Artificial Intelligence!

Whether you’re an AI enthusiast, a professional learning to leverage AI, or someone curious about the ongoing digital revolution, Synapse is your gateway to staying ahead in the ever-evolving world of AI.

About SingularityNET

SingularityNET is a decentralized Platform and Marketplace for Artificial Intelligence (AI) services founded by Dr. Ben Goertzel with the mission of creating a decentralized, democratic, inclusive, and beneficial Artificial General Intelligence (AGI).

• Our Platform, where anyone can develop, share, and monetize AI algorithms, models, and data.
• OpenCog Hyperon, our premier neural-symbolic AGI Framework, will be a core service for the next wave of AI innovation.
• Our Ecosystem, developing advanced AI solutions across market verticals to revolutionize industries.

Stay Up to Date With the Latest News, Follow Us on:

• Join our Discord
• Follow us on Mindplex Social
• Join our Main Telegram Channel
• Subscribe to our Announcement Channel
• Visit our Website
• Follow us on Twitter and Facebook
• Sign up for our Newsletter
• Explore our Jobs Page ? We’re hiring.
• SingularityNET Community Events Calendar Link — Stay current on Community, Ambassador, and Deep Fund Events!