Agentic AI has made building applications faster and more affordable than ever. In an era of vibe coding, non-IT professionals have joined software engineers in a rush to create the latest and greatest features.

But when almost any idea can become a feature, deciding what deserves to reach customers becomes a significant challenge and a potential risk. At a panel session at the recent Harness Unscripted event in London, engineering leaders discussed how organizations can embrace experimentation and ship effective products.

The panel recognized that AI makes software easier to build; the hard part is ensuring humans of suitable standing remain very much in the loop. As James Ford, Principal Engineer at price comparison specialist Compare the Market, said, AI increases the speed of development, but quality remains a human consideration:

The scarce resource is human attention and proper judgment over things. We can build stuff faster, but the scarcity is getting human eyes on this 10x that we're doing.

Matt Murphy, Principal Engineer at financial services giant Barclays, suggested human eyes should be firmly fixed on the pipeline. Yes, an AI-enabled development process generates much more code. However, failing to maintain tight control over the order of releases will mean organizations struggle to exploit potential increases in coding speed:

The business has got to decide which customers to prioritize, and which feature provides the biggest bang for the buck. You can deliver more software using AI, but you still must get that pipeline to say what we are going to deliver in which order.

Agreeing development priorities is tough, agreed Aditya Bhardwaj, Head of SaaS Development at software firm Temenos. In addition to focusing on the tool that will generate the greatest value, organizations must consider other costs. For example, applications that are pushed to production will require long-term support.

In these circumstances, the panel recognized that traditional metrics, such as costs and outputs, could help software development teams confirm that what they're building will boost business outcomes. Ford said managers must understand that there’s a big difference between increasing the speed of delivery and generating value:

What I'm worried about for the future is if everything's cheap to ship, do we lose the discipline? Do we stop caring so much about the KPI because we're so focused on building and getting stuff out fast at 10x speed?

The panel also considered the impact of AI on experimentation. Barclays’ Murphy said there are clear potential benefits from using emerging technology to expose and test new ideas with your audience:

AI gives us the ability to fail fast. You can rapidly prototype. You can get it in front of your customer base a lot faster. So, you can do multiple iterations and keep on iterating and taking feedback and changing. I think that's a good thing.

Temenos’ Bhardwaj also recognized the opportunity to experiment. However, while AI makes it easier to test potential routes to market, developers should avoid running too many tests simultaneously. Running multiple tests can increase costs and make it difficult for developers to home in on the routes to market that will positively impact customers.

Compare the Market’s Ford said most developers would probably like to exploit AI to experiment and then ship as many tools as possible. Unfortunately, this broad approach to testing and deployment creates a bandwidth problem. Building and shipping everything would lead to a significant management headache for the humans that remain in the loop:

You've got the complicated math of, ‘Well, how do I push 20 things live at once, and how do I divvy up my traffic between all of them, so we don't get overlapping groups and the wrong signals out of the measurements as to whether this experiment is working or not?’ Measuring the impact of what you've done is the challenge to solve.

Bhardwaj agreed, suggesting it’s hard for development teams to break down agentic costs, particularly when AI-enabled projects can lead to multiple business outcomes. He gave the example of decoupling AI investments to show that an experiment proved a hypothesis, such as an increase in net promoter scores

If you're running multiple simultaneous experiments, how do you associate costs? How do you split them? I think this is something that we found to be very challenging. Yes, we will be experimenting, but we need to think more about how we associate costs with experiments.

Crucially, the panel also stressed that agentic AI allows non-developers to create software products. While these explorations cost money, expanding development beyond the IT department produces benefits. Barclays’ Murphy said professionals can have great ideas, and he gave an example from his own business:

We had an offsite, and it was interesting because the best person at creating the rapid prototypes was a project manager. He wasn't bound by the concept of, ‘We're developers, we know how to do it right.’

Temenos’ Bhardwaj said non-IT professionals must understand it still takes weeks to get tools into production because the whole software development lifecycle is not agentic. To this end, Compare the Market’s Ford suggested the key difference between vibe coding and AI engineering is an understanding that the prototype is just the beginning:

You've got to have all the security checks, all the other elements of quality, all the metrics, and all the robustness and the engineering underneath something before you can put it live to anyone. It needs to go through the system to make it ready to put out with some confidence.

The panel suggested the key to success is tight boundaries. Humans must establish the rules for automated systems, such as not leaving irreversible decisions to AI. Broadly, the amount of automation depends on the use case. While Murphy believed a software house might be open to pushing features live regularly, business leaders in other sectors, including his own, will be more reticent to rely on automation:

Working in a heavily regulated environment, I think we might always have to have a human in the loop. The nirvana state might be, ‘Well, let's keep continuous deployment,’ but I don't think we'll ever get there in a bank.

AI might make it easier for developers and non-IT staff to push products live, but that doesn’t mean they should. Everyone who touches the software development lifecycle must recognize that prototyping is the first stage in a longer process – and the human in the loop is key to ensuring outputs are safe and secure.