Dear Singularitarians,

Recently, an insightful fireside chat sparked between Dr. Ben Goertzel, CEO of SingularityNET, and Dr. Steve Omohundro, Founder and CEO of Beneficial AI Research, in which the two visionary leaders explored the depths of artificial general intelligence (AGI) and dove deep into the importance of provable AI safety — the development and implementation of formal methods to ensure that AGI operates under a predictable, reliable, and safe development path.

With their combined experience spanning decades, their conversation provided a vibrant, in-depth look into the current state (and future potential) of AI, emphasizing the need for safety and formal verification.

Dr. Steve Omohundro’s journey into AI began in the early 1980s while he was pursuing his PhD in Physics at the University of California, Berkeley. Post-PhD, he transitioned into AI, working with Thinking Machines, the University of Illinois, and other renowned institutions. It’s this extensive background in machine learning, neural networks, and formal methods that positioned him as one of the leading voices in AI safety today. He emphasized the necessity of formal verification to ensure that safety.

One way he discussed was through mathematical proofs that guarantee that AI systems operate predictably (and therefore with a higher degree of security). He also highlighted that advancements in automated theorem proving such as Meta’s HyperTree Proof Search (HTPS) have resulted in major progress when it comes to verifying AI actions.

With that said, and with acknowledging the vast amount of progress made, it’s still clear that applying automated theorem proving to AGI safety remains a complex challenge.

The two also discussed various other approaches and their potential to improve AI’s reliability and security, among which were provable contracts, secure infrastructure, cybersecurity, blockchain, and security measures that prevent rogue AGI behavior.

But ultimately, are they enough to navigate the risks and challenges that come with AGI?

Steve and Ben delved into the potential risks associated with AGI, highlighting the need for systems that can autonomously model and improve their behavior.

Steve spoke about his development of a programming language called Saver, which facilitates parallel programming and minimizes bugs through formal verification. He stressed that ensuring AI’s actions are safe is fundamental as these systems become more integrated into society.

The concept of “provable contracts” emerged as a key solution. These advanced smart contracts would restrict dangerous actions unless specific safety guidelines are met.

He explained that such contracts could prevent rogue AGIs from performing harmful activities, like manipulating military systems or releasing biohazards. This method aims to create a safeguard, ensuring that AI actions are within a set of safe boundaries.

Building a Global Infrastructure for AI Safety

Building a global infrastructure for provably safe AGI is a monumental task, requiring significant resources and (ideally) global coordination.

Dr. Omohundro suggested that rapid advancements in AI theorem proving could make verification processes more efficient, potentially making secure infrastructure both feasible and cost-effective. He argued that as AI technology advances, building secure systems could become cheaper than maintaining insecure ones due to fewer bugs and errors.

Ben expressed his concerns about the practical challenges of implementing such an infrastructure, especially in the context of a decentralized tech ecosystem. They discussed the need for custom hardware optimized for formal verification and the potential role of AGI in refactoring existing systems to enhance security. The idea of AGI-driven cybersecurity battles also came up, highlighting the dynamic and evolving nature of these technologies.

In their conversation, Steve and Ben also explored the potential for AGIs to operate under principles of super rationality and cooperation. They envisioned a future where AGIs could influence human culture towards more pro-social behavior, creating a more collaborative and harmonious society.

Steve emphasized the importance of creating an ecosystem where transparency and cooperation are the norms. He proposed that with the right safeguards, AGIs could be designed to promote and enforce pro-social behaviors, preventing entities with harmful goals from causing disruption. This approach could shift the direction of technological development towards greater cooperation and mutual benefit.

Addressing Practical Challenges and Ethical Considerations

As they delved deeper into the practical challenges, Steve and Ben discussed the significant investment required to achieve provably safe AGI.

Ben noted that such initiatives would need substantial funding, potentially in the hundreds of billions of dollars, to develop the necessary hardware and software infrastructure.

Steve highlighted the progress in AI theorem proving as a positive sign, suggesting that with further advancements, the financial and technical barriers could be overcome.

They also touched upon the ethical considerations of AGI development.

Ben raised the issue of large corporations pushing towards AGI for profit, potentially at the expense of safety. He emphasized the need for a balanced approach, combining innovation with robust safety measures. Naturally, the two experts agreed, noting that while corporations are driven by profit, they also have a vested interest in ensuring that their technologies are safe and reliable.

The Role of Global Cooperation

As the discussion highlights, certain themes naturally emerge when exploring the complexities of AI development.

One of these key themes is global cooperation in developing beneficial AGI.

Steve and Ben acknowledged that building a secure AI infrastructure requires collaboration across nations and industries. They discussed the potential for international agreements and standards to ensure that AGI development is conducted safely and ethically.

In this fireside chat, these two visionaries, experts, and leading voices in the world of AI came together to emphasize the importance of ongoing dialogue and collaboration among AI researchers, policymakers, and industry leaders. And with that, position the BGI conference as an opportunity to build alliances and deepen discussions on AGI safety and ethics.

Their dialogue underscores the complexities and opportunities in ensuring that the future of AI is both secure and beneficial for humanity. By nurturing more cooperation in the field, advancing a safe and predictable path forward for the development of AGI, and addressing ethical considerations, the vision of a safe and harmonious AI-driven future is, they believe, within humanity’s reach.