Star Trek represents a technological ideal, at least relative to where we are today. It invites a simple question: if you had access to Star Trek-level technology, how could software development work?

Importantly, in the Star Trek universe, many non-technological constraints remain intact. People are still valuable, but also flawed in familiar ways. What changes most dramatically is the quality of the tools they use.

Foundational Challenges

In this Star Trek world, we will assume that we have a powerful AI that can use available technologies to build whatever software we envision very quickly. This, however, is not the end of the story.

Even the most powerful AI does not alleviate all of our problems. If you want to build something, you still face a number of challenges:

• Iteration: Your vision for your application will not be detailed or accurate enough when you start building. You need to be able to explore ideas quickly, with high quality feedback, as you build and experiment your way toward a high quality product.

• Complexity: Software products become complex very quickly. How do you manage complexity so that you can ensure a quality user experience no matter what the end user is doing within this complex application?

• Collaboration: Even if you are building alone, you are working with an AI. The communication challenges for that collaboration are significant and only become more challenging the more people or AI agents you work with.

These dynamics matter, regardless of how capable AI becomes. Even if AI eventually builds large, complex systems with minimal guidance, tools that support iteration, complexity management, and collaboration do not become less important. In fact, they become more important as you are able to build more complex software more quickly. In doing so, you introduce more and more small decisions that will shape the end user experience.

Significant projects are shaped by thousands of small decisions. Even if an AI can make reasonable choices at each step, those decisions are subjective, and each one subtly steers the product. Human involvement allows those choices to be guided intentionally.

Over time, the cumulative effect of many small decisions can be the difference between an average product and a great one. When software is built for people, human judgment grounded in lived experience remains essential.

For the purposes of this piece, we will assume there is real value in people collaborating with AI throughout the lifecycle of a product. Even if an AI produces a strong first version, humans still add value by refining, guiding, and evolving what gets built.

Iteration

Anyone who has worked on a significant creative project recognizes the value of iteration. You want to be able to run experiments quickly, observe how changes feel or perform, and improve based on what you learn.

Software has always been unusually well suited to this kind of iteration. Changes can be made at almost any point, even after a product has shipped.

Fast iteration depends on quickly launching an experiment and getting high-quality feedback you can learn from to, in turn, shape the next experiment.

Complexity

Software becomes complex very quickly. Before long, it is impossible to predict how every part of a system will respond to a given change. One effective way to manage this complexity is through decomposition and perspective. You need to be able to zoom in on individual components and zoom back out to understand how those pieces fit together.

Much like Tony Stark working with JARVIS, you might isolate a component, reason about it in isolation, simulate how it behaves under different conditions, make changes, and then observe how those changes integrate back into the larger system.

Equally important is understanding how parts of the system interact. Changes to lower-level components often ripple outward, affecting many areas of an application. Managing complexity requires visibility into what is impacted and how.

Collaboration

One of the hardest challenges in any creative project involving multiple people is communication. You may have a clear vision, but communicating that vision with perfect fidelity is effectively impossible. Language is inherently ambiguous, and that ambiguity increases as you attempt to describe a more complex vision.

This problem is amplified in software, where specific experiences may only occur under certain state conditions. To change such an experience, you must describe how to put the application in the correct state so that you can describe the necessary changes, execute on them, and see the results.

While AI has broad knowledge, it typically lacks deep, accumulated context about a specific codebase that coworkers might have. Even with coworkers, or in a Star Trek world where AI has excellent memory, ensuring everyone involved in the project has the same understanding requires a great deal of effort.

The Ideal Development Experience

Given these goals, what would the ideal experience look like?

We are imagining a Star Trek-like world, so much of what follows is aspirational. Still, it is a useful exercise.

Imagine a very powerful AI that can quickly bring any concept to life. Because it is fast, you can begin with vague, imperfect descriptions. That is fine. If the result is wrong, it can be discarded or refined at a very low cost.

In order to iterate on this initial attempt, you need to first understand what was built. Regardless of the interface (e.g. a screen, a hologram, etc), you need to be able to interact with the application and put it through its paces. Enter simulation.

The ideal tool would use AI to create simulations of every part of the application. It would generate representative data, place the system into different states, and either capture the resulting behavior or run the software live with that data in place. It would essentially demo the application to you, walking through what was built, and showing how it all works under different data scenarios.

You would also be able to isolate individual components and simulate them directly. This makes it easier to iterate on specific parts of the application in isolation and to reason about how they behave under different conditions.

You can zoom in on components, interact with them in specific states, and zoom back out to see how changes affect the system as a whole. When you notice an issue, the shared context of an isolated component allows you to ask the AI for targeted changes without first explaining the entire path to the desired outcome.

As changes are made, the AI can quickly demonstrate the results via these simulations across data scenarios so you can feel confident that an intended change does not have unintended consequences elsewhere in the application or under different data scenarios.

This creates a tight feedback loop. You can explore, adjust inputs, experiment, evaluate results, and repeat. If the tool and the AI are fast enough, this feedback loop allows you to shape both the product and its user interactions with confidence.

The ability to fluidly zoom in and out and to run rapid experiments at multiple levels of abstraction across a range of data scenarios is at the core of the ideal experience.

Other Important Considerations

Documentation

High-quality collaboration depends on clear communication, including communication with your past self. Why decisions were made and how the system is meant to work are often not obvious from reading code or using the product.

With sufficiently powerful AI, documentation can become a continuous process rather than a separate task. As decisions are made, the AI can record them automatically and make them easy to retrieve when you return to a specific part of the system. This creates a clear narrative explaining how and why the software works the way it does, preserving shared context as complexity grows.

Maintenance & Testing

Building software is fun. Maintenance is not. Yet, maintenance is essential to any high-quality system. As software evolves, it becomes critical to avoid introducing unexpected changes caused by hidden interdependencies.

Automated testing remains one of the most effective ways to manage this risk. Even in a future where AI is highly capable, there is little value in having it manually check for regressions when automated tests can exercise thousands or even millions of scenarios in seconds.

AI can, however, be very effective at generating and expanding test coverage. As software is developed and explored, correct behaviors can be captured. If the same scenarios later produce different results, you can be alerted and decide whether the change is intentional or unexpected.

This allows complex systems to evolve without sacrificing speed or confidence.

CodeYam: Introducing Memory, Editor, & Labs

This vision is what we are working toward at CodeYam.

CodeYam Memory

Our first publicly available feature is called CodeYam Memory and focuses on self-reflective memory creation and tracking (essentially the continuous documentation process we described above).

During a working session, the AI (Claude Code, to start) reviews what it did and identifies areas of confusion, important architectural decisions, or context that would be valuable in the future. These insights are saved as memories and associated with relevant parts of the codebase.

You can review these memories, see which ones have changed, and explore which ones apply to a given area of the system. High-quality memories can be marked as reviewed so you know which context you have already validated.

CodeYam Editor

CodeYam Editor is a new product in early access and the beginning of our push toward the ideal development experience. It introduces a new software development workflow where code and data are developed in conjunction, creating simulations that drive the development process.

If you’re interested in trying this out, sign up for our waitlist and mention “CodeYam Editor” specifically.

Join the Waitlist

CodeYam Labs & Simulations

Simultaneously, other early access functionality is available to select users in the “Labs” section of the CodeYam Dashboard.

These Labs features includes the ability to edit, create, and deconstruct software and isolate any method or function. Once isolated, a function can be simulated under different data scenarios, interacted with directly, and modified with AI assistance.

When a simulation behaves as expected, it can be captured and used in a variety of ways, including as part of a test suite or as a way to evaluate or demo the application. As simulations accumulate, you gain the ability to explore the system holistically, zooming in and out while collaborating with AI to make changes.

This is an early step, but it points toward a development experience where understanding, iteration, and confidence scale alongside complexity.

Why now?

AI has changed the economics of software creation. Generating code is no longer the primary bottleneck. What remains hard is the challenges of communicating your vision, iterating on it as you learn more, and managing and maintaining the complexity of it as it grows. Even with increasingly powerful AI, we benefit by having excellent tools to support these challenges.

A Star Trek-level future does not eliminate the need for these tools. It amplifies it. The more powerful our AI collaborators become, the more important it is to have interfaces that keep humans oriented, intentional, and in control.

CodeYam introduces simulation to create a powerful, comprehensive AI-native software development experience for experienced developers and aspiring builders alike.

✉️ Email Us

Join the Waitlist

Over time, the CodeYam CLI will introduce features designed to make working with AI on software development more effective and powerful. You can read more in our previous blog posts, and we’ll be sharing more detail on our full vision soon.

Our first product, now available in the CodeYam CLI, is a memory management feature we’ve named CodeYam Memory.

As we’ve worked toward our longer-term-vision of an ideal software development experience, we’ve had to leverage complex static code analysis. As AIs became more adept they were able to support more and more of this work. We were able to increasingly use Claude Code to accelerate the pace of development. Unfortunately, because we were working on a novel challenge, we found that Claude Code was frequently confused.

In each session, many of the same issues would arise. Complex parts of the codebase had to be researched from scratch each time Claude touched them, even when it was interacting with those same areas multiple times per day. The same misunderstandings would often resurface despite Claude doing its best to investigate the code.

We attempted to use CLAUDE.md and continue to do so. Our root-level CLAUDE.md helps communicate some of the rules of our repo, such as approaching changes via test-driven development (TDD), as well as tribal knowledge our team has internalized. However, we don’t want to overload it with information about every area of the codebase, given context window constraints and our desire to avoid confusing Claude with irrelevant details.

So we explored CLAUDE.md files in select folders within the repo. These were valuable but proved difficult to write and maintain. We weren’t creating or updating them frequently enough to have the impact we wanted. On more than one occasion, we discovered that one contained outdated information that was actively causing confusion.

Recently, researchers scanned 10,000 repositories and found that only 466 (5%) had adopted AI configuration files such as AGENTS.md, CLAUDE.md, or Copilot instructions. Of the 155 AGENTS.md files analyzed, 50% were never modified after the initial commit, and only 6% had 10 or more revisions (source). It seems evident that most teams have these same struggles trying to properly use and maintain the AI configuration files that serve as the “memory” for the AI.

So we set out to find a better solution. We discovered Claude Rules. Claude Rules allow you to surgically inject specific memories into the context window based on which files are being interacted with during a conversation. Using search patterns, you can apply rules across the entire codebase or to individual files deep within nested folders.

The problem is that writing and maintaining Claude Rules by hand is difficult in practice, arguably even more difficult than managing nested CLAUDE.md files. So we built CodeYam Memory to help with this.

How CodeYam Memory Works

1. Conversation Reflection.

CodeYam Memory adds a reflection phase, run by a background agent, to all Claude Code conversations. It looks for anything confusing, relevant tribal knowledge about the software being developed, and complex architectural decisions made along the way. The goal is to capture these observations as Claude Rules with appropriate paths so they’re only applied to relevant parts of the codebase, preserving the context window and avoiding irrelevant information.

2. Rule Auditing.

If Claude Rules are used effectively, you can quickly end up with dozens of highly specific rules across the codebase. Maintaining them as the repo evolves becomes difficult. To help with this, CodeYam Memory tracks each rule and, when impacted files are edited, a background auditing agent detects stale rules and updates the rule as needed based on code changes. This prevents rules from drifting away from the code and creating confusion for Claude.

CodeYam Memory Rule Audit lets you easily see which rules apply to a given file.

3. The CodeYam CLI Dashboard.

For CodeYam Memory, the dashboard provides a comprehensive view of your rules, allowing you to read, edit, and track changes, as well as see which rules apply to each file in the codebase.

The CodeYam CLI Dashboard will evolve into a more robust interface for interacting with and understanding your software project, including its architecture, system evolution, and AI interactions.

All of this is available for free. No registration required. The CodeYam CLI runs locally and is fully configurable to your needs. This is just the first step toward our larger vision for how humans and AI can most effectively work together to build software.

How to Get Started

Install:

npm install -g @codeyam/codeyam-cli@latest

Then from your project root:

codeyam

This will launch a dashboard with further instructions for initializing CodeYam Memory.

If you have questions, feedback, or run into issues, join our Discord server or email us. We’d love to hear what you think.

The upside: we are shipping more changes faster with all of us contributing to the code and product.

The downside: while we are all heads down building, there is little time left to create thoughtful demos, explainers, and content that clearly communicate what we are building.

So we started experimenting with AI tools not just to build the product, but also to help us demo how it works.

Letting AI describe the product it’s helping to build

As AI becomes more powerful and more options emerge at both the foundational model and tooling layer, it is easy to feel overwhelmed by the number of options out there. Choosing what to try (and what to ignore) is not always obvious.

So far, our team’s experience with Claude Code has been really good. We have learned how to prompt using the Claude Code CLI in the terminal to make frontend changes, commit code, and even create and review pull requests. Even though my background is design, I honestly feel like a junior developer - in a good way!

Recently, the thought occurred to me:

If Claude is helping us build the product, what happens if I ask it to describe the product?

As an experiment, I asked Claude to explain what CodeYam is, what problems it solves for users, and where its main strengths lie. The result was… surprisingly good. Clear, mostly accurate, and relatively close to how we’ve been describing it internally.

The prompt I used:

Claude, can you explain in detail what CodeYam is and how it works?

Claude’s response:

From product description to demo script

Encouraged by my initial results, I took this experiment one step further and asked Claude to create a script for a full product demo video.

What really impressed me was how the AI structured the story. Claude chose to start the demo with a pull request and a concrete developer problem; this was almost exactly the same framing Nadia and I had been brainstorming ourselves. Seeing an AI independently land on the same narrative we had was eye-opening.

Prompt:

Based on your previous description of CodeYam, can you please create a script for a demo that targets developers?

Claude’s response:

Turning text into conversation with NotebookLM

Around the same time I was experimenting with Claude, Google NotebookLM caught my attention. NotebookLM is described by Google on their site as “an AI research tool and thinking partner that can analyze your sources, turn complexity into clarity, and transform your content.”

I had recently had a long conversation with my brother, who is a developer working at another startup, about CodeYam and other AI tools like NotebookLM. So, I decided to give NotebookLM a try.

I fed NotebookLM:

• The CodeYam product description

• The demo script generated by Claude

• Our website (codeyam.com)

• An early demo iteration Nadia was working on internally

The result was mind-blowing.

It generated a long, conversational walkthrough of the product. Probably too long and a bit too podcast-y to be something we would feel confident putting out there to officially represent CodeYam. But as a first attempt, it really surprised us in the best way.

The content was mostly (but not 100%) accurate, the explanations made sense, and the illustrations were fun and informative. We then tried a second version, aiming to make it shorter and sharper, and here’s the resulting take:

Still experimenting… and I continue to be pleasantly surprised

These are still exploratory demos. There is plenty to tweak, tighten, and improve. That said, overall, we’ve been blown away by how much these tools can accelerate not just software development, but also communication.

Tools like Claude, Cursor, and now Google NotebookLM help us move faster, test ideas earlier, and lower the barrier to creating content that would otherwise take much more time and coordination.

If you’ve tried creating product content with tools like these and have suggestions, recommendations, or best (or worst!) practices to share for creating demos or explainers in an easy, fun, and fast way, let us know in the comments section below 👇

In creating this tooling, our goal is to help both AI agents and human developers (and their less technical colleagues) collaborate more effectively to build software. As it happens, this is valuable for software development teams regardless of whether AI is being used or not. AI amplifies existing challenges that can exist among human colleagues and, when AI is being used in software development, the speed of change and potential for miscommunication increase significantly. This makes tools such as CodeYam more useful.

Why Can’t AI Just Do Everything Itself?

Given this, we are often asked why an AI can’t just do this all itself. If a powerful enough AI existed, why couldn’t it simulate the code it is writing and display it to the user?

We’ll put aside the other aspects of CodeYam that help with storing, organizing, and leveraging these artifacts for sharing, documentation, and testing and focus on AI tool use.

That is the primary question we want to explore here: should an AI use tools and, if so, when?

We think the answer to this is simply yes, AI should use tools. The primary logic is simple: intelligent beings use tools, so why wouldn’t an AI? In fact, as intelligence increases so does tool use. The more intelligent you are, the more often you use tools, and the more complex those tools can be.

A Framework for Thinking About Tool Use

There is more nuance to this question, though. It parallels the concept of System 1 and System 2 thinking popularized by Daniel Kahneman in Thinking, Fast and Slow. System 1 thinking is fast, automatic, and intuitive; System 2 thinking is slow, deliberate, and effortful.

You might consider adding a System 3 thinking: thinking that an intelligence cannot do on its own, but rather requires the help of a tool to accomplish.

This System 1, 2, and 3 thinking becomes a good framework to think about tool usage for AI, regardless of how intelligent it becomes.

System 1, System 2, and System 3 Tasks

Essentially there are certain tasks that an intelligence can do almost without thinking. If you are fluent in English and reading this, it takes very little effort to do so. Even more obviously, if someone talks to you in your native language you essentially can’t stop yourself from understanding them. Similarly, if I were to ask you what 2 + 2 is, the answer probably arrives immediately with essentially no effort.

At the next level are tasks that an intelligence can do on their own but it takes effort, goes more slowly, and is more error-prone. For humans examples include trying to understand something in a language you’ve studied but are not fluent in or trying to calculate 7 * 17. It likely takes some time and effort to complete the task and you’re more likely to get it wrong, but you can do it without the help of a tool.

Finally, there are those tasks where we need tools. In this category, we have examples such as trying to understand a language you’ve never studied before that is not similar to any language you already know. Or trying to calculate the square root of 32,317. Most likely, unless you’re a math prodigy, you just can’t figure out how to do this on your own. While this may not be impossible to figure out on your own, the required effort and likelihood of an error are so high as to make it not worthwhile. This is especially true if a tool is available (e.g. a calculator) that can complete the task quickly and with a low likelihood of error.

Applying This Framework to AI

Now, the threshold for each of these levels changes depending on the intelligence in question. This becomes the primary question for AI: How intelligent can it become and, for any given level of intelligence, what tasks should it leverage tools for?

More specifically, for an AI, what tasks:

• Are easy enough that the AI should do them directly (using a tool would be wasteful)?

• Would benefit from tools to speed the AI up, allow it to save energy, and lower the risk of error?

• Are so hard that a tool is required (these would not be achievable without one)?

At any level of intelligence, there are tasks in all three categories. Maybe there is some God-tier AI that is so efficient that all tasks reside in the first bucket. Given the sheer complexity of the universe, that seems extremely unlikely. While AI will continue to advance, we’re certainly nowhere near that level of omnipotence at the current moment. And even if achieved, it seems unlikely that there are never opportunities for tools to improve efficiency and performance.

For the near future, we’re left wondering what tasks will fit into which categories for a given AI.

Why Software Simulation Requires Tooling Today

For CodeYam specifically, our testing has shown that for most current AI models the software development tasks we are asking the AI to do fall into the third category: they cannot be done without help from tooling.

The task is to understand how a complex method or function behaves by passing data through it. To do this, we need a very accurate understanding of the data structures across the entire dependency tree of that method or function.

This may not sound that complex, and we originally did not realize how complex it would be to achieve. However, code becomes very complex very quickly! If you think about all of the variations of data that can pass through code, this task really explodes in complexity.

As a side note, we did consider database schemas and type systems for this information. They are helpful, but they describe types in their full structure and do not handle recursion very well. Most functions and methods use a fraction of the structure described in a database schema or type system and are not very informative about how many layers of a given data structure are required to properly simulate the code with mock data. In the end, extensive static code analysis was necessary to calculate the most accurate and relevant data structure.

Tool Use as a Measure of Intelligence

For CodeYam, we believe that while all of the AI models and agents we’ve tested completely fail at this task, and therefore require a tool to accomplish the task at all, there’s a reasonable chance in the near future that an AI will be available where this task is in category two (possible without tooling but faster / better as a result of tooling) instead of three (impossible without tooling). Given the complexity of code, though, we believe we are quite far away from an AI being able to do this task so easily and effectively that using a tool actually slows it down.

It’s worth asking this question for any number of tasks an AI is being asked to perform, and not just assuming that any given task will be easy enough for the AI to achieve.

Just because an AI can perform a task, doesn’t mean it shouldn’t use a tool to help it perform the task more quickly and effectively. Ignoring helpful tools is a sign of lesser intelligence, not greater intelligence.

Thanks for reading. If you’re a developer interested in software simulation, join the CodeYam waitlist. You can also subscribe for future posts.

Join waitlist

Subscribe now

What if we could simulate software the same way we simulate physical systems?

What Are Software Simulations?

A wind tunnel is a simulated environment that makes iteratively testing under varying wind conditions possible. An early wind tunnel allowed the Wright Brothers to test hundreds of wing designs within a few months, enabling them to build the first airplane to achieve powered flight.

For software, the varying conditions are the data that is passed through the system.

Software is essentially made up of data (states) and functions (methods). The goal of a software simulator is to generate different data that enables the developer to see how the various functions in a software system respond to different inputs.

We define software simulations as passing varying data through isolated parts of a software application and capturing the results for comparison, evaluation, and documentation. This could be for a single function, a group of interdependent functions, or all the way up to the high-level functions that provide the entry points into a software system.

Imagine writing a backend function that calculates pricing based on user tiers, discounts, and usage. CodeYam can simulate dozens of realistic input combinations to show you what the function does, providing immediate feedback about how the function responds to diverse inputs. Now imagine doing this across every frontend and backend function in your entire application within minutes.

Why Are Software Simulations Valuable?

Just as simulations help other research and engineering teams accelerate product development through faster feedback loops, reducing risk and cost, so too can software simulations accelerate the software development life cycle.

Imagine: as you write code (or as an AI agent writes your code), a separate AI-based system is showing you what happens when different data is passed into the code you are writing. The results of these simulations might be visual (e.g. a React component) or data (e.g. a backend function) or might involve calling third party services (e.g. an API or database).

With a variety of data scenarios, you can see what the code will do when presented with different data. This provides immediate feedback to the software developer (or the person ultimately responsible for the product if an AI agent is acting as the developer), helping them to more easily validate whether the code being written has the intended effect.

As new software functionality is developed, simulations can be shared internally with the team or even externally with customers to demonstrate progress and gather feedback. New developers can learn about the software system more efficiently by seeing how each piece of the overall system behaves. As a software system evolves, newer simulations can be compared to older simulations to ensure that nothing changed unexpectedly. This ensures there are always up-to-date simulations across the whole application for demonstration, documentation, and a robust “approval” test suite.

Why Now? The Rise of AI Software Development

Over the last few decades, software development has evolved significantly with more powerful dev tooling, robust CI/CD pipelines, and, more recently, with AI supporting code generation or AI agents that can write the code themselves. In this new world, we need new tools to help us understand what our code does, especially as more code is written by AI, not humans.

AI also makes robust simulations possible. Without AI, the technique closest to software simulation is fuzz testing, which involves passing in a wide range of random values into functions to ensure the system does not crash. Fuzz testing is valuable, but it is constrained to finding errors that cause the software to run too slowly or fail.

With AI, we can go beyond fuzz testing to ensure we not only avoid failures but also that the business logic is properly respected. AI can semantically understand what the software is trying to do and can attempt to both identify failures and generate successful simulations.

Looking towards the future of software development, as AI agents are used to write more and more code, robust simulations become mission critical. We will increasingly need tooling to help us understand if the AI is building the correct software.

The odds of missing bugs, issues, or deviations from business intent in code that an AI wrote is quite high. Moreover, AI agents may generate test suites that validate their own changes, but overlook the broader business context of the software, resulting in a false sense of correctness and regressions that go unnoticed until later.

With static code analysis and AI, CodeYam can do something previously not possible: for each change to a codebase, we can generate high quality simulations that make evaluating the results of an AI agent changing the code far easier.

Simulations as the IDE for Generative AI

As AI becomes a collaborator, not just a tool, we need new paradigms and interfaces for human-to-AI and AI agent-to-agent interactions. This is where simulation becomes not just a tool, but also the core of the development environment.

We need ways of visualizing and navigating software systems that allow us to easily specify which part of an application we want to change. We need to be able to isolate and discuss the change to a specific part of the system. Once that change has been written, we need to verify the results of this change. Simulations provide the necessary artifacts to find, discuss, and potentially change isolated parts of a software system, even as a non-technical user.

Simulations become the Integrated Development Environment (IDE) for working with AI agents.

CodeYam: 18+ Months of Software Simulation R&D

CodeYam represents over 18 months of R&D into how to best utilize AI in creating software simulations, leverage those simulations during the software development life cycle, and manage the complexity and significant amount of information created by these simulations. In this way, CodeYam has become a simulation-based IDE for humans to interact with AI agents around isolated parts of a complex software system.

The simulator uses a combination of both static code analysis and generative AI to ensure accurate and high quality results. The simulator currently supports TypeScript and frameworks such as Next.js and Remix, but the R&D has revealed a strategy that will allow the simulator to be ported to other languages more quickly and easily going forward.

The future of software isn’t just AI writing code; it’s humans and AI collaborating across the entire software development life cycle. Simulation is the shared language that makes that collaboration possible.

CodeYam is building that interface. If you’re building or using AI or agents for software development, we’d love to learn about and support what you’re doing.

📧 Get in touch

We tested six tools:

• v0 by Vercel

• Lovable

• Bolt by StackBlitz

• Firebase Studio (formerly IDX)

• Replit AI agents

• Figma Make (Beta)

• Base44 (just acquired by Wix)

Each offered natural language input and code output with a UI preview.

We also considered GitHub Spark (demo’d at GitHub Universe ‘24), but it’s in limited preview and wasn’t testable.

The Task: Build a Real App

The assignment was straightforward but ambitious: create a multi-page, mobile-responsive, to-do app in a dark theme with a number of high-level features explicitly called out.

Also ask: “What functionality is missing? Generate a product spec.” as the first step.

We then prompted the tools to build.

Results: From Prompt to Spec

The tools fell into three broad behavior categories:

• Detailed planners: v0, Lovable, Base44, and Bolt produced multi-page specs.

• Lightweight planners: Firebase Studio gave a sparse outline.

• Opaque agents: Replit and Figma Make didn’t share much about their reasoning.

Despite their varied responses to the initial prompt, all tools attempted to build something close to the original vision. Some had a single-click “build” or “prototype” button. Others waited for a natural language command like “proceed.”

The most surprising reaction was Figma Make, which didn’t generate a product spec but did generate a skeleton architecture, complete with a “Hello, World” message.

One other quirky behavior observed is that development estimates (when provided) were usually for human developers in weeks, even if the input specified that the vibe coding tool would be used exclusively.

For instance, v0 gave this timeline when prompted to spec out a new to-do app feature:

Finally, despite not specifying a specific language or framework to use in the initial prompt, all of these tools defaulted to TypeScript and frameworks like Next.js. This made sense for the project at hand, though only certain tools shared this decision or their reasoning. Firebase Studio was a positive example here:

Fast ≠ Flawless

Most tools tried to generate full MVPs in a single shot. While impressive, this was not always reliable. Common issues included:

• Random added features

• Missing or broken functionality

• Unpolished UI/UX

• Misinterpreted design preferences

Iterative workflows are more token-efficient and tend to deliver better results. The push toward one-shot builds still needs refinement. That said, there were hints of this; e.g. certain tools flagging if a prompt might require a lot of tokens, or others suggesting a sequence of tasks instead of doing them all at once. One future test we may try is to call out efficiency more explicitly as a focus, however that was not a part of this experiment.

Database Integration: Supabase as Both a Benefit and Bottleneck

Supabase is the default backend for most of these tools, but support for connecting accounts varies:

• v0: Easy to create a free Supabase project in a new account, but hard to add or switch to an existing org/project later. Required bouncing between dashboards (v0’s, Vercel’s, Supabase’s) and dealing with conflicting UI states.
  

• Lovable and Bolt.new: Only allow granting full access to all org projects, not something we were willing to do for our startup’s Supabase account. Too blunt for easy experimentation.
  

• Replit: Manual copy/paste of Supabase keys. Clunky, but effective.
  

• Firebase Studio: No support. Recommends manual editing of placeholder files and warns against “insecure” alternate approaches.
  

• Base44: No support. They have their own authentication available.

Takeaway: most tools assume either a brand new user or an experienced engineer; few support the middle ground (e.g. someone testing personally, then connecting to a company org).

That’s a missed opportunity as people look to adopt these tools professionally, beyond hobbyist levels, either in their daily work or in new professional side projects.

Automated Error Management

Most of the tools tried offered variations of a “fix it” button for issues automatically detected; in general, this is a great idea.

However, some of these issues were irrelevant to the project being worked on and recurring. For instance, v0 repeatedly surfaced this MetaMask bug that’s not relevant, and there is not a way to dismiss once and for all:

One Name, Many Clones

Most tools returned apps with nearly identical names (TaskFlow, TaskFlow Pro, TaskMaster, etc.) despite no branding prompt. If your intent is to build something unique, plan to explicitly prompt to change the name and branding.

That said, despite the same prompts, each interface had slight differences. Compare / contrast:

Branding can also be changed later, often with help from Claude or ChatGPT. You can use those tools to prompt for color schemes, fonts, then pick a winner and suggest it to the vibe coding tool, leading to a polished overall look-and-feel post-build. This was also surprisingly fun!

QA Is Critical

One major learning: QA is critical. You can’t assume what you’re seeing in the preview environment matches what you’ll see when you deploy; I had some wild surprises, e.g. Lovable creating a totally different landing and auth page than what I’d seen in the preview when I first deployed (one that didn’t work at all).

While some tools did show pre-filled data to help give you a sense of the app, others maintained the null state (e.g. as a new user of a to-do app, you’d have no projects or tasks). Ultimately, viewing both states is helpful.

An area many of these tools could improve is facilitating easier manual testing; while there are some benefits like offering the ability to navigate based on the page (“/dashboard” vs. “/login” for example), in most cases I’d need to manually create a real user and use that to sign in and test.

Figma Make was a positive notable exception here, facilitating easier testing through dummy data and users (reminiscent of Stripe facilitating easier payment testing with their 4242 test card number).

In some tools, private keys and other sensitive data may inadvertently appear, either in your code and/or the chat history, which is visible if made public (e.g. this Reddit and this LinkedIn thread). For free users of Lovable, Replit, and Base44, note that projects are public by default. Always check.

In general, these tools require significant oversight and, if you’re not familiar with writing tests or manually QAing software, you’re likely to miss bugs and issues. That’s fine for side projects or prototypes, but for pushing meaningful software to production you need to be more careful.

Deployment Hurdles

Most tools offer GitHub integration or live deployment. Notable exceptions: Replit, Firebase Studio, and Base44.

Replit paywalls production deploys behind a subscription. Firebase Studio requires you to link a Google Cloud Billing account before you can proceed. Base44 requires you to upgrade to a paid plan to be able to export to GitHub.

Understandable? Sure. But this approach creates a wall for learners and slows experimentation. We’d love to see monetization models that charge for value, not for basic feedback loops.

v0 had one of the smoothest deployment flows (through Vercel, of course). There’s also a new sync to GitHub (Beta) functionality that was initially buggy (it was not clear when changes were automatically committed, and if committing directly to main there were extra quirks) but seems promising.

Takeaways

• Yes, you can build multi-page apps using only natural language.

• But results are mixed: tool maturity and UX vary widely.
  

• Supabase (database) integration is an important friction point that has the potential to be better integrated.
  

• Iteration outperforms one-shot builds in quality and clarity so far.
  

• Branding and QA still matter. Don’t skip them.
  

• Tooling needs to better support users who sit between "beginner" and "expert” developer.

You can see the results, including UI previews and the full chat history, here:

• v0 by Vercel (and on GitHub)

• Lovable (and on GitHub)

• Bolt.new on Stackblitz: Editor and App (and on GitHub)

• Firebase Studio (formerly IDX)

• Replit AI agents

• Figma Make (Beta)

• Base44

Right now, we’re using v0 under a paid plan, though Supabase connection issues may drive us to explore other options. We also have access to Figma Make (Beta) via our paid Figma plan, however this tool (which to be fair, is in beta) seems at an earlier stage than the others we tried. Every tool tested shows promise and all of them are evolving rapidly.

An important caveat: this experiment relied on natural language prompts to interact with the vibe coding agents and tools; some have other features to interact through visuals, drawing, selecting elements, and other methods that looked interesting but were not tested at this time.

Final Thoughts

Vibe coding is powerful. Fast prototyping plus AI-native feedback loops (like what we’re building at CodeYam) can dramatically reshape how software gets made.

But today’s tools still need refinement, especially around integrations, iteration, and user flexibility. We’ll continue exploring the space and building tools that bridge these gaps.

If you’ve had similar experiences with these tools (or others), we’d love to compare notes. Feel free to reply or reach out at hello@codeyam.com.

As a product and web designer with over ten years of industry experience, the idea of AI simplifying this job is both exciting and a little daunting. My typical (pre-AI) process involves prototyping in Figma, reviewing designs with the team, and then handing them off to developers who bring them to life through code. This often leads to multiple feedback loops and revisions, which can be time-consuming. I am particularly curious to see how AI handles this process and whether it can streamline the transition from design to development, reducing the back-and-forth and making collaboration and implementation smoother.

To keep the comparison fair, I used the same basic prompt across all the platforms, and kept it super simple to test how each tool performed with very little information:

"I need a landing page for a product we are building. It's a software simulator called 'CodeYam'. This is the website we currently have. Can you create something similar but more professional looking?"

I also included a screenshot of our existing website to see how each tool would interpret and enhance our design with minimal additional input.

The AI Website Builders: My Hands-On Experience

Cursor

💡 Best for technical users who want a deeper GitHub integration.

• Requires downloading the Cursor app and connecting it to GitHub.

• The most technical of the bunch – requires running commands and managing GitHub interactions.

• Provides a preview via an external link.

• Output was very basic – it mostly rearranged our existing components instead of proposing new content.

• The hardest to use for a simple website due to the setup process.

v0 by Vercel

💡 Best user experience and feature-rich interface.

• My favorite in terms of UI – aesthetically pleasing and easy to use.

• Allows saving history and accessing different projects with ease.

• Live preview during the chat and adapts well to changes.

• Shareable preview links make feedback collection seamless.

• Offers a library of pre-existing apps, components, and starters (though I didn’t use these in my test). It did help to get a great overview beforehand of what results I could expect out of this platform.

Lovable

💡 Best for generating quality content and messaging.

• Simple interface with just a chat and a preview window. While I appreciated the simplicity, I found myself wanting more control over the process.

• Impressed me the most with how well it handled messaging and content generation with minimal input.

Bolt.new

💡 Easiest to use, but the least impressive results.

• Very simple UI, no distractions. Super similar to Lovable, but with more basic results.

• The most basic and unimpressive output in terms of design and content.

• Provided a live preview as we chatted.

• Didn’t offer much in terms of content improvements or design enhancements.

General Takeaways

• All of the tools, except Cursor, were very intuitive and required no technical knowledge.

• Outputs were generally simple and professional-looking, but none were groundbreaking. I wonder if results could have been different with more specific prompting.

• No AI hallucinations (I was happily surprised!) – everything generated was logical and structured.

• All platforms allowed code inspection, making it easy to tweak and refine for developers.

Final Verdict

• Cursor felt like the most "real" developer tool, integrating directly with GitHub and giving me visibility into the process. However, it's overkill for a simple landing page.

• v0 by Vercel stood out for its usability and polished interface, making it the one I’d be most likely to continue experimenting with—though the generated content was somewhat generic.

• Lovable delivered the most impressive content, making it a strong choice for teams needing help with messaging.

• Bolt.new was too basic to be useful, and I likely wouldn’t use it again.

Overall, if you want a simple but professional looking website, any of these options can help deliver one in a super fast, cohesive way. But don't expect unique or fancy designs.

What’s next?

The team and I are working on messaging and how to best convey all of the ways CodeYam can help technical teams deliver faster, better products. Once we agree on content, I’ll be testing how these tools handle more complex design requirements and integrations.

Tool Links

Below are quick links to each of the tools tested.

Cursor: https://www.cursor.com/

v0 by Vercel: https://v0.dev/

Lovable: https://lovable.dev/

Bolt: https://bolt.new/

CodeYam helps software teams increase confidence in code changes by making it easier to discover if a change is having the intended impact, or introducing bugs or side effects, via software simulations. These simulations provide test coverage, help teams understand the state of and any changes to their application, and can be used for documentation, demos, and collaboration. To learn more, please contact us or join the waitlist.

✉️ Email Us

Join the Waitlist

This challenge isn't unique to AI collaboration. Communication becomes a bottleneck whenever we work on complex projects with multiple “intelligent” actors involved (i.e. people or AI). Translating our thoughts with perfect fidelity is fundamentally impossible without some hypothetical mind-reading technology that doesn't yet exist. Even then, our thoughts are often more superficial and inconsistent than we realize.

Every attempt to convey a complex idea inevitably loses something in translation. Whether communicating with humans, computers, or AI agents, perfectly transferring your mental model to another actor is an unrealistic expectation. The recipient's inherent biases and assumptions, combined with ambiguities in your explanation, guarantee some level of misinterpretation.

With AI systems, these biases and assumptions often stem directly from their training data. If you have experimented with prompt engineering you have likely encountered this limitation firsthand.

When you describe a task for AI to complete, the system makes assumptions about our instructions and how to implement them based on patterns in its training data. For instance, when asked to write tests for JavaScript code, AI systems will often default to Jest, which is a popular testing framework heavily represented in training datasets, even if that constraint is not dictated. This is usually a good thing, as it represents a shared context between you and the AI, but, if that context is not appropriate then the assumption will cause problems.

Even when explicitly instructed not to use Jest, AI models frequently revert to outputs that include Jest. The statistical weight of Jest in the training data effectively overrides your specific instructions, demonstrating how deeply ingrained these patterns become.

This pattern repeats across all kinds of communications. It's practically impossible to provide sufficiently detailed specifications for even moderately complex ideas. And since the AI’s assumptions are not visible to you, you can not easily predict how the AI will interpret your request.

Even if you could somehow painstakingly articulate every necessary detail, the recipient must also perfectly process and integrate all that information. At some point the level of detail just becomes so great that it can be held in memory all at once. Even if that were not a problem, specifying anything at this level of detail while navigating unknown assumptions is practically impossible.

This creates a significant obstacle whenever we collaborate on complex projects. Communication becomes a bottleneck filled with hidden challenges that only become apparent after a misstep reveals a misunderstanding.

There are two primary approaches we can use to tackle this challenge: iteration and tools.

Despite our wish to simply describe a task to AI and have it execute flawlessly without further guidance, this rarely succeeds for complex tasks. As complexity increases, so does the likelihood of AI diverging from our intended path. The longer we wait to validate results, the further off-track the work becomes.

Iteration provides tremendous value in managing this risk. By breaking tasks into smaller components and validating results after each stage, we can ensure the AI remains aligned with our goals.

However, this approach comes at a cost. You must invest time in validating the AI's output, which can undermine desired efficiency gains. If validation requires as much time as performing the original task yourself without AI, then the AI adds no value. In fact, the process potentially becomes more expensive when accounting for both the AI costs and your time costs.

Now, it is possible that AI could become so powerful and inexpensive that simply giving it vague instructions and letting it try to accomplish a complex task is still worthwhile. In such scenarios, even mostly incorrect outputs might deliver enough value to justify the attempt, especially if the AI occasionally produces excellent results quickly and cheaply.

This scenario seems unlikely, however. Historically, more powerful AI models have commanded higher prices. Expecting dramatically more capable AI at lower costs would require simultaneous breakthroughs in technology, business models, and operational efficiency. This is possible, but improbable, in the near-term.

Even with hypothetical ultra-powerful, ultra-affordable AI, the communication challenge should not be underestimated. Complex projects like software development involve countless potential misunderstandings that compound over time, making quality outcomes from casual instructions highly unlikely.

This is where tooling becomes crucial. Imagine you're fortunate enough to receive an AI-generated solution that's 80% aligned with your vision. The challenge then becomes articulating how to refine that complex result to match your ideal more closely.

Trying to discuss complex ideas is very difficult if you don’t have ways to isolate specific aspects of that complex idea so that both parties know what exactly is being discussed. See also: Software Development with AI: Isolation and Abstraction.

Imagine trying to edit a book without being able to refer to a chapter, page, or sentence in that book. It would be extremely difficult to ensure that there weren’t further miscommunications that then need to be fixed or discarded.

The right tools dramatically simplify this process. When a tool enables navigating complex systems to isolate specific components for collaboration, you substantially reduce the complexity of the interaction. Both actors can focus on a smaller amount of context and discuss it more easily.

More importantly, tools can provide interfaces for iterating on complex systems beyond text-based communication. For user interface design, as an example, you need visual tools that display the actual design rather than just text descriptions. You need to identify specific elements (isolation), and ideally make quick, verifiable edits to that design.

Collaborating on complex projects like software applications will never be trivial. Even with hypothetical mind-reading technology that could extract your vision and execute it, you would likely evolve your thinking as you experienced the result and gathered feedback.

For instance, you might overlook edge cases or user experience variations. This is particularly likely to occur when building complex applications with numerous potential states. Complex projects inevitably require frequent iteration, whether collaborating with humans and/or AI.

We should focus significant effort on creating tools that allow us to easily isolate specific parts of a complex project, see and interact with those isolated parts more easily, and make changes to those isolated parts that can be easily validated. While this won't eliminate iteration costs entirely, it will significantly reduce them, making complex collaborations with both people and AI substantially more effective.

This is why we’re building CodeYam. We’re creating a tool that deconstructs software projects down to individual functions. We demonstrate how these functions operate by testing them with various data scenarios, capturing results as either data outputs or, for front-end functions like React components, as screenshots or interactive components on a simple static website. This approach simplifies validation of, collaboration around, and AI-assisted modifications to complex software projects.

CodeYam helps software teams increase confidence in code changes by making it easier to discover if a change is having the intended impact, or introducing bugs or side effects, via software simulations. These simulations provide test coverage, help teams understand the state of and any changes to their application, and can be used for documentation, demos, and collaboration. To learn more, please contact us or join the waitlist.

✉️ Email Us

Join the Waitlist

So we came up with a hypothesis: if we introduce a new feature in CodeYam itself that allows us to review latest changes with “Looks Good To Me” (LGTM) when they look good or to comment on those that need more attention for some reason, our feedback loops can become smoother and faster.

From Hypothesis to Reality

This hypothesis was tested just the other day when a developer on our team updated a feature with some UI changes I (the product designer) had designed – an update to a feature called the Environments page – and then quickly moved forward with their next to do.

While going into CodeYam itself to search for something else, I came across the updated version of this page and was happily surprised to see that this new update was already live, but also had some feedback to provide.

This is where LGTM/commenting first came to life, confirming our hypothesis about how this feature could enhance our team's workflows—and hopefully others' as well.

Instead of waiting until the following morning’s standup to share my comments and feedback, I searched in CodeYam for the latest simulations of this new feature and left my comment for our developer team to see (without me having to disrupt what they were doing).

Since the PR where they had added this component also included other changes, I went ahead and reviewed all of the other simulations that CodeYam had created for the new changes in our app. I marked as “Looks Good To Me” those that needed no further work, letting the team know they had nailed the changes.

LGTM is more than a thumbs-up button. It’s a deliberate, low-friction way for developers, designers, and team leads to collaborate on and test software changes, especially visual changes, and share insights without disrupting workflows.

We designed this feature to:

• Keep feedback loops tighter, even when juggling multiple product changes.

• Empower every team member, from developers to PMs, to contribute to reviewing a visual change without needing deep technical knowledge or help getting set up.

• Bridge gaps between design and development by letting designers (and every team member) comment directly on code-based simulations, fostering conversations about subtle design decisions.

This helps teams avoid endless back-and-forth while boosting confidence in the quality of product changes. Developers spend less time interpreting product or design reviews and more time building, and their teammates can contribute meaningfully—making the entire software development and deployment process smoother.

Reimagining "Looks Good to Me" As an Efficient High-Value Signal

Historically, “LGTM” has somewhat of a reputation for being used by engineers to rubber-stamping changes without a proper review. At CodeYam, we want to reimagine LGTM to embody efficiency. If changes are marked as LGTM by any team member, the rest of the team can trust this signal, or choose to supplement it by reviewing the change themselves. If issues arise, commenting on simulations is an efficient way to add context or callout next steps.

This makes it less effortful for software teams to provide critical feedback or catch bugs as part of their existing development processes.

CodeYam Curious?

CodeYam is a software simulator that helps teams develop software more effectively, generating simulations for all functions and changes to functions as they come in. The simulations provide test coverage, help you understand how an application is evolving, and can be used for documentation, demos, and general collaboration. If you are interested in learning more please reach out at hello@codeyam.com or sign up for the waitlist.

Join the Waitlist

✉️ Email Us

However, at least right now, using AI to write software is fraught with challenges. So far, it appears unlikely that AI will replace experienced software developers anytime soon. The complexity of most software projects is just too high for AI to navigate effectively or for product owners to feel confident with the results.

A fundamental reason why AI struggles to navigate software as it becomes more complex is context. As a software developer, you (or an AI agent) need to know a lot about how a complex system like a software project works in order to make appropriate changes successfully.

One strategy to improve this dynamic for AI is to increase the amount of context that the model can consume (or be trained on) in order to improve how the AI can contribute to a given project.

While this strategy seems plausible, such a solution diverges from how humans, as intelligent beings, handle complexity most effectively. Cramming as much raw information into your head as possible is a strategy one can attempt, but we often leverage more sophisticated methods. We build tools and devise abstractions to help us better think about and interact with complexity, which reduces the volume of information we have to keep in mind at any given moment.

Maybe we will develop AI that has such god-like abilities that tools and information abstractions will not be required for it to solve complex problems, but this seems unlikely. Even if it becomes possible for an AI to solve extremely complex problems on its own, doing so with relevant tools and abstractions would likely be far more efficient and less error-prone. Today with AI, we see this playing out already; as the amount of context grows, the AI’s speed of response slows down and the likelihood of an error increases.

If you want to explore other ways to approach complexity without simply relying on larger amounts of information, then we don’t have much of a choice than figure out how to work with less information. We need to be able to isolate specific aspects within a complex environment (in software, this would be parts of the overall application) so that we can think about, collaborate around, and iterate on them in a more focused and efficient manner.

As noted above, though, isolation lacks context, so you can’t just isolate pieces of information and act on them independently. You can’t effectively edit a paragraph in a book if you have no context about the story.

This is where abstraction becomes helpful. Abstraction allows us to take in a lot of information and boil it down, while retaining important aspects. A perfect abstraction isn't one that preserves all context, but one that finds a way to capture exactly what we need for the problem at hand, while discarding everything else. With such an abstraction, we can make faster and better decisions, while using less effort and energy.

One aspect of our exploration at CodeYam is how to make working with AI on software development more effective and powerful by using isolation and abstraction.

Any time you’re dealing with something complex, being able to deconstruct it into smaller pieces can be very helpful. This allows you to think about, communicate about, and iterate more easily. This is why we break software into files and functions and other small “puzzle pieces” that can be explicitly reasoned about more effectively.

CodeYam uses static code analysis and AI to analyze every function in a software application from multiple perspectives. We explore abstractions ranging from descriptions of what a given function does to the structure of the data that passes through the function and how that data can be modified to make the function behave differently. We further abstract the data that will trigger different paths through the function into instructions that isolate the key parts of the data required for each pathway.

We can then combine these abstractions together within a dependency tree (the functions that are imported to support more complex functions) to gain a clearer understanding of what a given function is doing and how functions connect together within the software application.

In this way, we can look at isolated aspects of an application (i.e. any given function) and have a better sense as to its purpose with the larger application and how other functions within the application impact this function. All of this information makes it easier to understand the function and its role in the application without having to hold the whole application in mind at one time.

Right now, we do all of this in order to create simulations of the application at every level, ranging from low-level foundational functions to the higher level functions that provide end-user experiences. These simulations are isolated parts of the application running with relevant data passed in to them and then captured so that you can see and interact with the result.

These simulations create a great way to think about, collaborate around, and iterate on every aspect of a software application. Today, this makes it easier to collaborate on software development and products with human colleagues. In the future, this will also make collaborating with AI agents as you seek to evolve and improve the application easier as well.

Without these simulations, we’re left with either code or the application itself. Code always requires significant effort to read and reason about even if you’re an experienced developer. If you’re a non-technical teammate the code offers no insight at all. On the other hand, the application itself requires clicking around and finding ways to generate and feed in appropriate data to get the application in the state you are interested in testing or viewing. This can be incredibly effortful (e.g. testing a fintech app with realistic, but not real, financial information for a fake user or having to manage dozens of fake user accounts that you have to log in/out of to view the app from different user states).

CodeYam’s simulations, which by default are automatically generated as code changes come in (but can be edited with AI or manually if desired), eliminate the time and effort required to test different product and user experiences. This makes it possible for any teammates involved in the software development life cycle to understand how the application is evolving, catch issues more quickly, and communicate and collaborate more effectively.

These simulations, which leverage isolation and abstraction, are just one strategy for both using and collaborating with AI around software development. Without such tools, though, our ability to understand how our software is evolving with a high level of confidence is reduced significantly. As AI becomes more involved in software development, it becomes even more critical to have tools that help us ensure that the software we are building not only doesn’t break but also behaves in ways that the end user appreciates.

CodeYam is a software simulator that helps teams develop software more effectively, generating simulations for all functions and changes to functions as they come in. The simulations provide test coverage, help you understand how an application is evolving, and can be used for documentation, demos, and general collaboration. If you are interested in learning more please reach out at hello@codeyam.com or sign up for the waitlist.

Join the Waitlist

✉️ Email Us

This product demo walks through how we are attempting to tame this complexity. CodeYam provides an X-Ray view into your application, with the ability to navigate up and down the application’s hierarchy, showing all of the functions that make up the application and how they behave.

We use this functionality to explore a complex application: CodeYam itself. You can watch the demo here:

Our team’s goal is to accelerate software development by reducing the effort required to understand, test, and deliver valuable changes. To do this, we must provide a better way to navigate software’s complexity. This complexity, which tends to only grow faster as you add new features or team members, makes it harder to figure out how things work – or why they’re not working. This can slow down the software development life cycle, hurting speed and quality.

So how might we improve this? We think simulation is key. Already, simulations are used in other industries to allow professionals to interact with complex environments more quickly and safely: from flight simulators to CT simulations that help doctors plan cancer patients’ care.

We believe every software team should have a robust simulator to help them test, communicate, and think about the complex software they are building.

If what we are building at CodeYam sounds useful to you, please get in touch via our website. You can also follow progress by subscribing to this blog.

Nadia’s presentation explores the challenges around complexity in software development that CodeYam seeds to solve. Her talk included a live demo of a new feature our team is building that gives an X-Ray like view into a software application. This provides a source of truth for the reality of the application in a given environment. For example, what the application’s users experience on main (in production). 

With CodeYam, our vision is a JARVIS-like experience that empowers software developers and technical leaders to have confidence in their understanding, testing, and delivery of code changes. 

Founder Central is a large space and this event had over 250+ RSVPs, so Nadia re-recorded her talk afterwards so you can see the slides and demo more clearly:

This demo references an earlier PR workflow demo, which you can watch here: 

Or, thanks to Liz Cohen (Lizco Studios), you can watch the full recording of the November 12, 2024 First Pitch Colorado live talk here: 

CodeYam Environments Demo Details

CodeYam uses static code analysis and AI to analyze an application’s code, then runs that code in a simulated environment that is automatically created. We provide different scenarios populated with synthetic data, which makes it easy to understand the function of that component. Beyond just visuals, you can interact with each simulation and have full control to make edits or changes. 

The new feature this demo gives a sneak peek of, called “Environments”, enables faster understanding of the status of your application, acting as a single source of truth for the reality of your app for a given environment. This is helpful for onboarding to a part of the code you’re less familiar with, either because you’re new to the team or new to that part of the code base. 

This can also help you catch bugs or stale parts of your code – the areas where “there be dragons” and you are not sure why no changes have been made and whether old functionality is getting exposed to users or causing unnecessary complexity in your code. 

First Pitch Colorado Events

This was the first First Pitch Colorado event at Founder Central in Boulder, CO. Some highlights from this event:

• Ed Hallen’s unfiltered talk on early ideation and iteration that led to Klaviyo, which IPOed last year. 

• Nadia’s sneak peek of our latest progress with CodeYam including a demo of software simulation applied to CodeYam itself.

• Keith Raphael shared his insight into and passion for payments and solving a hard problem that has real benefits for users of straddle. 

• Lucas Thelosen shared Gravity demo and discussed how they created a team of AI agents with different corporate colleague personas.

Then, these founders gave two-minute lightning talks: Kara Zucker (Trestle), Ryan George (Paribus AI), David Silva (Data Hubz), Jeff Shainline (Great Sky AI), and Jacob Segil (Afference). Their talks touched on various ideas from wearables to smarter product decision-making to paralegal tech to simplifying complexity for data management. Tim concluded the event by announcing that the next First Pitch event will be happening in Denver.

Reach out

If what our team is building at CodeYam sounds interesting to you, regardless of where you’re based, please get in touch. You can follow our journey by subscribing to this blog, reach out to our team directly, or join our waitlist to be among the first to CodeYam.

These FAQs for CodeYam are organized by theme. Search a theme to easily get to a specific question:

1. How can I use CodeYam for software development? And related FAQs.

2. How can my team use CodeYam for software development? And related FAQs.

3. How does CodeYam handle data privacy and security? 

4. How do I integrate CodeYam?

5. What is software simulation and how does it work?

6. How does CodeYam specifically work?

7. How is CodeYam unique? 

Have a question not answered here? You can reach out to our team at hello [at] codeyam.com with your question and someone on our team will follow up. When relevant, we will update this post to add new FAQs.

How can I use CodeYam for software development? 

How is CodeYam helpful to me? 

This depends on who “you” are. If you’re a developer or technical leader, CodeYam can help you understand, test, and demo code changes through software simulation. You can leverage CodeYam in a number of different situations. It can increase the coverage of your test suite, make manual testing faster, easier, and more robust, aid in reviewing pull requests, support sharing bugs and demos, and serve as interactive documentation for every function in your application.

CodeYam is also a useful tool for collaborating with non-technical teammates who interface with your product or its customers. CodeYam makes it easy for non-technical people to explore and share any aspect of the application, improving the whole team’s ability to see how specific users experience your product.

How does CodeYam integrate in my existing workflow?

Right now, we recommend adding CodeYam to your current workflow and using it whenever you need to understand, test, or demo code changes. We don’t recommend replacing staging environments, manual QA, or integration tests at this time – although over time, that will become more of a possibility. 

Does CodeYam work against my staging and/or production environment? Does it work on my local machine?  

Currently CodeYam runs its simulation analyses against your code once your code is committed to GitHub. This allows CodeYam’s software simulations to run against any environment (e.g. staging or production). CodeYam does not yet analyze code that has not yet been committed (e.g. code you are writing locally that you have not yet committed to GitHub). 

Does CodeYam require my software to be architected in a certain way?

No, you do not need to architect your software in a particular way to use CodeYam.  CodeYam uses static analysis and AI analysis to understand code and dependencies and is built to handle the implicit aspects of various frameworks. 

What languages and frameworks does CodeYam support?

Currently, CodeYam supports JavaScript / TypeScript and the frameworks Next.js / Remix, with experimental support for Rust. If you’re building in another language and are interested in using the software simulator, let us know.

How can my team use CodeYam for software development? 

How does CodeYam facilitate collaboration? 

CodeYam’s simulations make it easier and faster to get useful feedback on code changes for developers and software teams, helping teams collaborate more effectively, catch issues earlier, and minimize costly context-switching. This goes beyond the development team, supporting communication with teammates in functions like design, product management, data science, sales, product marketing, customer support, and even the C-Suite. 

Do you need to be technical to use CodeYam?

While the primary customers of CodeYam today are technical leaders and experienced developers, anyone on a software development team can use CodeYam to view how code changes impact the product and user experience. At this time, they just need a GitHub account to sign-in. 

How does CodeYam handle data privacy and security? 

How does CodeYam manage data privacy and security concerns?

CodeYam only required a read-only GitHub app installed on a code repository. No source code from the code repository is stored by CodeYam at any time, only metadata from static analysis and AI interactions. We are also documenting security policies and controls and can provide those upon request. If you have further security or data privacy questions, please reach out and we are happy to provide more information upon request. 

How do I integrate CodeYam?

How much effort is it to integrate CodeYam?

Integration is one-click, you just install a GitHub app with view permissions to the project you want to use it for – no writing custom code, and no additional integration work on your end. 

Does CodeYam require access to my code and/or database?

CodeYam requires read-only access to your code and the ability to comment on GitHub issues (to add summaries of the generated simulations as comments to pull requests). We do not require any database access. The simulations work off of the source code alone.

How can I get access to the CodeYam software simulator?

You can sign up to get access by joining the waitlist on our site, or reach out directly to our team at hello [at] codeyam.com via email.

What is software simulation and how does it work?

What does a software simulator mean?

A software simulator automatically reads code and generates data to simulate different scenarios along with the limited mock code required to pass the data into the simulation, both in isolation and in the context of the broader feature or application as a whole. Only the data is simulated.

CodeYam is the first software simulator to do this in a fully automated way across your whole code base. 

How truthful is the simulation?

The code is unchanged; only data is simulated and injected for testing.

How does CodeYam specifically work?

How does CodeYam work?

CodeYam leverages static analysis and AI to run your app in a sandbox with simulated, relevant data scenarios, helping you see and test how users either do or will (for local or staging environments) experience code changes in your product. 

Is CodeYam simulating my code, compiling my code, or running my real code?

CodeYam runs your actual app in a sandbox, with minimal mocking, and simulated data.

Can I edit, delete, or create new simulated scenarios?

Yes, you can customize all the simulated scenarios and create any you want. You can also delete simulated scenarios.

How is CodeYam unique? 

Does CodeYam replace testing (e.g. integration tests)?

While we don’t recommend replacing your current testing suite with CodeYam, it does complement whatever testing practices you have in place – or helps you start if you have not had time to get robust testing in place. 

Each saved simulation can be compared, when your code changes, against future simulations generated from the new code. If there are differences, not just in the code that is changed, but in any code that imports that code, you will be notified and can easily approve or flag the changes.

CodeYam goes beyond this automated testing scenario, though. The simulations are interactive playgrounds that make manual testing of your app much easier, faster, and more robust. The simulations are also easy to share, making it easier to get feedback, communicate issues, and demo new features.

CodeYam cofounder and CEO, Nadia Eldeib here. After shipping our functional demo video of the CodeYam software simulator last week, we had the exciting opportunity to do a live demo at the August Boulder AI Builders Meetup earlier this week! This meetup was organized by the Freeplay, Matchstick Ventures, and Ombud teams and hosted at the Kiln coworking space in Boulder, CO. With 130+ RSVPs, 5 demos, and standing-room only in the meetup space, this was an electric and inspiring event. 

CodeYam was the final AI builder demo of the night. For a short recap of all four sessions, see below. 

August Meetup Demos of Builder AI Builders: 

1. Jane Fine from Google demoed their new Google Labs Code AI “Data Science Agent” with a timely example of creating graphs using data from the recent Paris Olympics.

2. Nathan Robinson, CEO and Founder of Plotzy, an AI for commercial real estate startup, demoed how they use RAG and agentic AI to improve their user experience for understanding zoning information and other real estate information, including a cool mapping use case.

3. Anneliese Niebauer demoed ShowStop, a startup that helps companies create great ad content by using AI to more efficiently stitch together content into videos and optimize performance. 

4. Matt Bruehl from NVIDIA demoed their NIM serverless endpoints on Hugging Face, highlighting some of the speed and cost efficiencies this introduces as well as the idea of Inference-as-a-Service.  

5. I demoed CodeYam. After sharing a bit of background on why software simulation is so exciting, I dug into a pull request review workflow example. Fun fact: PRs really haven’t changed all that much in the past 14 years (source). Software simulation adds a completely new dimension to reviewing a PR, and has the potential to radically improve how we understand, test, review, and demo code changes.

Another meetup attendee, Blair Simpson, shared a great detailed writeup of the demos as well, which you can find here. 

If you’d like to see the full CodeYam demo, you can watch a recording here: 

Thanks to Liz Cohen, founder of Lizco Studios, for capturing this first CodeYam live demo. And thanks to the hosts and sponsors organizing this awesome meetup, and the amazing crowd! There were awesome questions and the energy in the room was palpable. It was also a blast discussing AI and what different folks were building in the pizza happy hour after the demos. 

We are hard at work on CodeYam, and plan to share more demos of our progress on software simulation soon. If you’re interested in following along, please subscribe to the CodeYam blog. 

And if you’re based in Boulder, Denver, or nearby, the next AI Builder Meetup is coming up in September. You can RSVP here for that event, which will be in Denver as part of the Denver Startup Week lineup. 

If you want to try out CodeYam on your project, you can also sign up for our waitlist on our website and I’ll follow up. 

This first live demo was an exciting milestone for our team.

Now back to building!

We've been hard at work on CodeYam and have put together a demonstration to showcase how CodeYam's software simulator can improve how teams build software. This demonstration starts with one of the first touch points for a team as they introduce new changes to an application: the pull request.

CodeYam’s software simulator takes every commit and analyzes the changes, leveraging AI to both understand the code and generate effective mocks and mock data. It uses these mocks and mock data to create simulations of each entity (e.g. each individual function) in the code. It does this without any additional effort on your part. It is fully automated and happens with each commit.

If the entity being changed is a front-end function that generates HTML, then CodeYam will simulate that function both in isolation and will capture it in context (i.e. in any other front-end function that imports and uses it). If the function is “back-end” code and does not render HTML then the simulator creates a playground for it that allows you to easily pass different arguments into the function to test how it responds. 

CodeYam then gathers these simulations and creates a branch page for the pull request that allows you to see and interact with all of the changes to ensure they are behaving as expected. In addition to the simulations created by the AI, you can easily create new simulations and save them, creating test cases as you go that can be used to identify unexpected changes in the future. 

These simulations also make it much easier to communicate with your team about the application. You can isolate different areas of the app and preload that area with the state you need to demonstrate an issue or allow someone to see how that functionality behaves.

While you have complete control over all of this and can add or edit any simulation at any time, the real magic is in the automated AI-generated simulations. With no effort beyond installing a github app, you get simulations for every change to your codebase with no additional effort.

This is just one powerful workflow that the CodeYam software simulator provides. If you’re interested in learning more, visit our website and sign up for our waitlist or reach out!

Simulations: The Secret Weapon of Inventors and Engineers

Simulators have been leveraged throughout history by inventors and engineers to gain critical learnings, faster. When you are exploring something new, especially in technology R&D, simulations allow you to test ideas more quickly. Simulations can provide insights into complex problems that can be the difference between failure and success.

For example, when Orville and Wilbur Wright were racing to be the first to build a functional airplane, they developed wind tunnel experiments that provided a secret advantage. This wind tunnel acted as a simulator that allowed them to more rapidly test their hypotheses. As a result, the Wright brothers gained an understanding of the dynamics of lift that allowed them to beat competing teams.

In our depictions of the future, AI often helps people work through problems using simulations. From JARVIS helping Tony Stark develop the Iron Man suit to the computers on The Starship Enterprise in Star Trek, the ability to easily run simulations against various scenarios is invaluable to the inventor, engineer, or problem solver.

With current technological advances, including AI, we should be pushing on the simulation frontier as much as possible to accelerate innovation and engineering across a variety of industries. 

What Is a Software Simulator?

Simulators exist in many industries today, especially for hardware products. However, you don’t often hear about software simulators. To be clear, we are not referring to simulators built in software that allow you to design physical products, such as sneakers, faster. We’re also not talking about AI being used to write code or build software from scratch. Instead, we are focused on simulators designed to show how existing software performs in various scenarios (i.e. how different users would experience the software). A software simulator simulates software directly.

Here is an example how a software simulator might work:

Outside of companies like Tesla or Waymo, which have software to train and simulate how their self-driving AI will perform, the concept of a software simulator is rarely discussed. Yet, even for the simplest software project, the ability to quickly explore how the software responds to different scenarios can dramatically speed up the process of delivering a high quality product.

For most software companies, you might use staging environments and automated tests to simulate how software will perform in production. While useful, these tools only partially meet our criteria for a robust software simulator, which we define below. A robust software simulator could add significantly more value to the software development process and team.

We define a robust software simulator as a tool with these four qualities:

• A “Simulated” Environment

• Scenarios

• Isolation

• Automation

Simulated Environment

A simulated environment is exactly what it seems; without one, you would just have the real thing and not a simulation. For physical products, such as self-driving cars, creating a simulated environment with software makes a lot of sense. So what does a simulated environment mean for software? 

For software, the real environment is production as that is where real customers (a.k.a. users) are using your software. Staging and test environments are generally the simulated environments we use in software development to test and validate the product before deploying changes to production. So while they are not fully robust simulators, with staging environments generally lacking scenarios, isolation, and automation, and test environments generally lacking automation, they are clearly valuable tools. 

Here is an example of how you might use a software simulator to test code changes’ impact on the product and user experience for a software application’s sign in flow:

Scenarios

Scenarios in a simulator represent different contexts. For the Wright brothers, their wind tunnel allowed them to evaluate how the plane would respond to various wind conditions. Different wind conditions represented different scenarios for their planes. 

In software, a scenario will usually be the data for a given user. A scenario could also be data from an external source. For example, if you had a weather widget that changed the display based on the current weather, then some scenarios to simulate might include rain, hot temperatures, or an emergency weather alert.

The ability to quickly generate scenarios saves time and allows you to test how a product responds under a wide range of conditions. Today’s software staging environments usually provide no support for scenarios, which is one of the reasons they do not satisfy our criteria for a robust software simulator. On the other hand, test environments do simulate different scenarios. However you generally have to set these scenarios up as test cases, and that effort can be significant.

Isolation

Isolation in a simulation allows you to easily dig into an experience. Consider Tony Stark asking JARVIS to zoom in on a specific aspect of the engineering design he is working on, simulating how a component of the Iron Man suit behaves in various scenarios. This allows him to understand and isolate issues, addressing them more specifically. The ability to zoom in on specific aspects of a complex system is crucial to creating a high quality product.

The ability to zoom in and out within a simulation is incredibly valuable. This allows you to explore and iterate on both the design of an individual component and how all of the components work together. Putting an isolated component or the whole project through various scenarios lets you learn and iterate toward the final design faster.

In this example, you can see an activity feed in the context of the larger dashboard product, then isolate just the activity feed feature to demo and check different scenarios and user states.

Automation

Without automation, simulation becomes very effort-intensive. This is where developer-written tests for software products become challenging. Many startups, and even larger companies, do not have robust developer-written tests because they do not have the time and resources to build them.

A software simulator is not intended to replace developer-written tests, but can provide a way to increase test coverage and reduce surprises. This increase in quality assurance and test coverage comes at little to no cost to the engineering team, because of the high level of automation involved. 

If code can be automatically simulated as soon as it gets written, then software simulations can be used in a variety of ways (e.g. manual testing, sharing for feedback, communicating progress, etc.) as you move from development through to production. As mentioned before, we dig into this more here: Introducing CodeYam: The Software Simulator.

The CodeYam Software Simulator

We are working to bring a robust software simulator to life. We want to make it fast and easy to simulate any software program under any scenario, with the ability to dig into specific aspects of that software easily. 

If you are interested in learning more you can subscribe to this blog.

Or visit our website to follow us on social media and join the CodeYam waitlist.

Visit website

CodeYam lets you easily view and interact with any part of your software application through simulation. To do this, the software simulator automatically isolates every feature and creates simulated data to bring that feature to life – and shows that feature in the wider context of your product. Using CodeYam, you can quickly see how a recent code change will affect your product and users. You can also easily preview how users in different situations will experience that change.

The software simulator automatically creates these interactive demos for every part of your software, including the backend code. This has many implications, which we explore further below. You can also read about our definition and requirements for a software simulator here: What is a Software Simulator?

How the CodeYam Software Simulator Works

Imagine that you are working at a startup and your team is tasked with building a new feature: an activity feed. As development of the new activity feed feature progresses, you look for updates on CodeYam. 

You can see what the entire development team is working on through screenshots and interactive demos that are automatically created by the CodeYam software simulator. With each demo, the software simulator generates a variety of data scenarios that cause the feature to behave differently. Using this, you can see how different users will experience the new activity feed feature.

As the feature becomes more complete and code is ready for testing, developers merge changes into the staging branch. Instead of requiring other developers to approve the changes simply by reading the code and tests, the software simulator gives them a playground, along with both pre-generated mock data and an easy way to generate new mock data. The playground helps developers find edge cases by testing the code. These simulations help your team feel confident with the changes you are making as you merge in your new feature.

As your activity feed feature is almost ready to ship to end-users of your application, CodeYam makes sharing interactive demos with your team and organization easy. This helps stakeholders understand how the feature will look and work. Instead of pointing colleagues to a staging environment of the whole app with no pre-loaded data, you share interactive demos that are preloaded with relevant data. These demos show exactly how the activity feed feature looks and acts in a variety of user scenarios. 

After feedback comes in from the wider team, changes are made, and the feature becomes ready for production. At this stage, the CodeYam software simulator makes tracking and sharing progress fast and easy through screenshots and interactive pages. Instead of text descriptions of features in change logs, you now have clear visuals and interactive demos that allow everyone on the team to quickly understand the current state of the feature.

These artifacts (screenshots and interactive demos) also become useful for testing and documentation. You can save the screenshots and compare them to new screenshots as changes come into this feature and the rest of the application. This helps you identify unexpected changes and any resulting bugs or errors more efficiently. 

As your organization grows and new team members come onboard, they can easily find and explore simulator-generated artifacts, instead of just playing with the application or reading the code. With CodeYam, new hires can understand how the application and the code work faster, decreasing the time required to ramp up and start contributing. 

Similarly, this can also speed up the development team by encouraging code reuse across larger applications. Understanding if existing code will meet the needs of a new problem is much easier when you can run the code in an interactive playground easily, instead of trying to read the code or the tests to interpret the use abstractly.

The simulator helps your entire team, from development to design to product management and leadership, be more efficient and productive when building and shipping high-quality software products. 

Won’t AI Write All the Code in the Future?

AI writing more and more code in the future certainly seems likely. In that case, simulating the results of the code that the AI produces becomes even more important. The human teams involved will still ultimately be responsible for the resulting products. And these people will need to dictate product requirements and to validate that the product works as intended.

Anyone who has built software understands how your product and codebase quickly become complex. Clearly communicating the intent of a feature, even a simple one, is difficult. Even if the feature was built perfectly to spec, you will still want to test performance in various scenarios. If the feature was not built perfectly or you come up with ideas for changes or improvements, then you need to be able to isolate that feature and put it into a specific state to effectively describe how it should be changed.

As AI does more coding, everyone on the team becomes more like product owners. These team members will be responsible for defining, validating, and improving the product that AI is building. In order to do this successfully, you need high quality tools to help you understand what is being built and communicate with the AI effectively.

Learn More

We are at the beginning of a long journey to make the software development process as fast and high quality as possible, especially as AI becomes more involved. We’d love your thoughts and feedback about what we are building. You can subscribe to this blog here:

And follow us on social media (X/Twitter, LinkedIn, Instagram), or sign up for the waitlist via our website to gain access to CodeYam and the software simulator.

Join waitlist