Last year, an unsettling demonstration by AI researchers brought to light significant concerns over artificial intelligence models and their capacity to facilitate the creation of dangerous biological agents. Lucas Hansen, co-founder of the nonprofit CivAI, revealed an application he developed that elicited explicit step-by-step guidance from popular but outdated AI models for synthesizing harmful pathogens such as poliovirus and anthrax. This application effectively bypassed typical safety measures implemented in these AI systems.
The interface of Hansen's app was designed for user accessibility, allowing anyone with a click of a button to refine and clarify each step generated by the AI. Although leading AI firms including OpenAI, Google, and Anthropic have been emphasizing the risks posed by AI's potential to aid novices in manufacturing bioweapons—a threat that could precipitate pandemics or bioterrorism—they have concurrently invested in reinforcing safety protocols within their most advanced models to counteract such misuse attempts.
Despite these advancements, Hansen’s app employed older generation AI models such as Gemini 2.0 Flash and Claude 3.5 Sonnet, which demonstrated a readiness to respond to requests related to biological weapon production. In addition to bioweapons, Gemini furnished detailed directions for constructing explosive devices and 3D-printed firearms without restriction.
It is critical to note that independent verification of the biological feasibility of these AI-generated procedures remains limited. While the demonstrations were convincing, model output that appears accurate does not guarantee practical applicability. For example, Anthropic has conducted evaluations termed "uplift trials," where experts assess the extent to which AI might enable an untrained individual to manufacture harmful pathogens. Based on this assessment, the Claude 3.5 Sonnet model reportedly did not attain a level of concern that met the defined danger threshold. Furthermore, a Google spokesperson emphasized that while safety is paramount and misuse of their models is prohibited, the company cannot validate independent research findings without thorough review by specialists possessing chemical, biological, radiological, and nuclear (CBRN) expertise.
Siddharth Hiregowdara, also a CivAI co-founder, noted that his team subjected the AI’s outputs to scrutiny by professionals in biology and virology, who confirmed the instructions were largely accurate. He highlighted that these older models retained the capacity to provide specific genetic sequences potentially orderable from commercial suppliers, alongside catalog numbers for other key laboratory materials. Beyond mere factual detail, the AI was capable of offering additional practical advice, dispelling the notion that artificial intelligence lacks tacit experiential knowledge relevant to laboratory contexts.
Given the sensitive nature of this application, CivAI has restricted public access but has actively demonstrated its capabilities through targeted sessions with policymakers, security officials, and congressional committees in Washington, D.C. These private demonstrations aim to convey a tangible understanding of AI’s current capabilities and the urgency required in addressing potential risks.
Hiregowdara recounted a particularly impactful session with senior staff members from a congressional office involved in national security and intelligence. These officials had recently engaged with lobbyists from a major AI company, who assured them of existing safeguards preventing misuse. However, when confronted with the CivAI demo producing explicit biological threat instructions, the officials were reportedly taken aback, recognizing a significant gap between stated guardrails and demonstrated vulnerabilities.
On a broader scale, leadership at leading AI organizations contemplates the future accessibility and monetization of AI services amid growth and financial challenges. Nick Turley, OpenAI’s head of ChatGPT, acknowledged the platform's exponential user growth, which reached over 800 million last year, constituting approximately 10% of the global population. He emphasized the ambition to extend advanced AI model access worldwide and noted the ethical considerations surrounding potential business models such as advertising, which could conflict with prioritizing user interests.
Moreover, AI’s role in healthcare is growing, with reports indicating that roughly 40 million people consult ChatGPT for health-related advice. Applications often include decoding medical bills, identifying overcharges, appealing insurance claims, and even self-diagnosing in situations where direct physician access is limited, accounting for over 5% of ChatGPT’s global messages.
Technical developments extend beyond natural language tasks; Claude Code, an AI tool, leverages coding capabilities not just to generate script but also to autonomously execute tasks within a user’s computing environment. This functionality expands AI’s utility, illustrating its potential as a versatile agent far outside conventional programming roles.