I haven’t published anything for a while here, because I’ve been digging deeper into the problem of measuring productivity of engineering teams. Okay, let me share what I found.

Software teams have never lacked metrics. We count tickets, commits, pull requests, story points, review time, deployment frequency, defects, incidents, and almost everything else that leaves a digital trace. Now that AI can collect and interpret this data automatically, engineering performance dashboards are becoming even easier to build.

That sounds like progress, but it also creates a new risk: we may become very efficient at measuring the wrong things.

I recently took part in a discussion about an AI-powered reporting tool connected to Jira and GitHub. The tool gathered activity data for each engineer and generated performance assessments based on completed tasks, pull requests, comments, cycle time, review activity, and stalled work.

The logic was understandable. More completed work, faster movement, and more visible communication should indicate stronger performance. Yet engineering contribution is rarely that simple. One developer may close ten small tickets in a day. Another may spend the same day investigating a scaling issue, identifying a weak architectural dependency, and preventing a future production failure. The first person generates more visible activity. The second may create much greater long-term value.

A dashboard can easily see the ten tickets. It may completely miss the disaster that never happened. Well, activity is useful, of course, but it is not the same as value.

Let me explain what I mean. Traditional engineering metrics help teams understand flow, workload, bottlenecks, and delivery speed. All this is good, but the problem begins when activity indicators are treated as direct measures of contribution. Commits show that code changed, but they do not show whether the change improved the product; pull requests show that work moved through a repository, but they do not prove that the architecture became stronger; and story points show estimated effort, yet they can’t tell us whether that effort created business value.

The SPACE framework, developed by researchers from GitHub, Microsoft Research, and the University of Victoria, makes this point clearly: developer productivity cannot be reduced to one metric. It must be examined through several dimensions, including performance, activity, collaboration, satisfaction, and flow.

That was already true before generative AI. Today, it is even more important. AI can generate large amounts of code, tests, documentation, and infrastructure logic in minutes. As a result, a team can dramatically increase its engineering output without producing the same increase in engineering value.

More code may simply mean more code to review. More pull requests may mean more fragmented changes, and more generated tests may mean broader coverage, or merely more scripts that nobody fully trusts. AI makes production easier, but it does not automatically make engineering decisions better.

The New Bottleneck Is Judgment

In AI-native teams, the most valuable work is gradually shifting. Engineers still write code, but a growing part of their contribution comes from deciding what should be built, selecting the right architecture, validating AI-generated output, identifying hidden risks, rejecting weak suggestions, documenting decisions, and preserving knowledge for the future.

These activities are less visible than commits, but often more important:

• A senior engineer who rejects a poor AI-generated solution may create more value than a developer who accepts and merges hundreds of generated lines.

• A QA engineer who redesigns a testing strategy may prevent dozens of defects without creating much visible delivery activity.

• A DevOps specialist who reduces deployment risk may contribute more to business continuity than someone who closes a long list of infrastructure tickets.

This is why I believe engineering organizations need to separate two ideas:

Engineering activity measures what people and systems did. Engineering effectiveness measures the sustainable value created by those actions and decisions.

A Practical Definition of Engineering Effectiveness

I define Engineering Effectiveness as the ability of a team to maximize sustainable business value while minimizing delivery time, engineering cost, technical risk, and future effort.

This definition shifts the conversation.

Instead of asking only:

• How many stories were completed?
• How many commits were made?
• How quickly were pull requests merged?

We also ask:

• Did the work improve revenue, customer experience, or operational efficiency?
• Is the solution secure, reliable, and maintainable?
• Did the team reduce future engineering effort?
• Was technical debt controlled?
• Was knowledge documented and shared?
• Were AI-generated outputs properly validated?
• Will the system still be understandable and manageable in two years?

To organize these dimensions, I proposed a conceptual Engineering Effectiveness Index:

EEI = (BV + QV + KV + AIQ) ÷ (T + EC + R)

Where:

• BV — Business Value: customer impact, revenue potential, cost reduction, or strategic importance.
• QV — Quality Value: architecture, reliability, maintainability, security, testing, and defect reduction.
• KV — Knowledge Value: documentation, reusable assets, mentoring, knowledge transfer, and recorded design decisions.
• AIQ — AI Utilization Quality: responsible AI use, human supervision, validation, and rejection of poor AI output.
• T — Time: lead time, cycle time, waiting, and review delays.
• EC — Engineering Cost: human effort, infrastructure cost, AI-tool cost, and rework.
• R — Risk: technical debt, compliance exposure, security risk, maintainability problems, and rollback probability.

This formula is not intended to produce an unquestionable scientific truth. It is a management framework that helps teams evaluate value and cost together instead of ranking people by visible activity alone.

An Illustrative Example

Consider a team building an AI-powered customer support platform over a four-month period.

The team consists of six specialists: a solution architect, two backend engineers, a frontend engineer, a QA automation engineer, and a DevOps engineer.

During the project, the team produces:

• 1,180 commits
• 310 pull requests
• 94 completed stories
• 860 automated tests
• 48 technical documentation pages
• 3 production incidents
• 0 rollbacks
• 12 reusable software components
• 21,000 interactions with AI development tools

The activity numbers look healthy. But the EEI model examines what those numbers produced.

After analyzing business results, software quality, documentation, code reviews, incidents, AI usage, and delivery data, the following normalized scores are assigned:

• BV = 86
• QV = 89
• KV = 78
• AIQ = 91
• T = 28
• EC = 23
• R = 16

The calculation is:

EEI = (86 + 89 + 78 + 91) ÷ (28 + 23 + 16)

EEI = 344 ÷ 67

EEI = 5.13

The number itself is less important than the explanation behind it.

The project scores strongly because it created clear business value, maintained good software quality, and demonstrated disciplined human supervision of AI-generated work. The relatively lower Knowledge Value score shows that documentation and reusable assets could still be improved. The Time score suggests that review delays slowed delivery, while the Risk score remained moderate because the team had several incidents but no rollbacks.

A useful report would not simply announce that the team scored 5.13.

It would explain:

• The architecture is strong and maintainable.
• AI-generated code was reviewed carefully.
• Business objectives were achieved.
• Documentation is adequate but not yet excellent.
• Code-review waiting time should be reduced.
• More reusable components could improve future delivery.

This turns the index into a coaching tool rather than a surveillance mechanism.

What an Engineering Effectiveness Agent Could Do

An AI-assisted Engineering Effectiveness Agent could collect signals from Jira, GitHub, CI/CD pipelines, testing systems, documentation platforms, incident reports, and AI development tools.

It could then produce a multidimensional assessment that shows:

• where engineering value was created,
• where cost or risk is growing,
• whether AI is being used responsibly,
• what knowledge may be lost,
• and which improvements would have the greatest long-term effect.

The goal should be to understand whether the team is creating durable business and technical value. That matters especially in long-term engineering partnerships and embedded-team models. Customers need more than speed and lower cost. They need continuity, ownership, preserved knowledge, predictable quality, manageable technical debt, and systems that remain understandable after the original developers have moved on. These qualities are often included in sales presentations; they are much less often measured.

AI is making it easier to generate more software activity. That does not necessarily mean it is making software organizations more effective. The next generation of engineering measurement should therefore focus less on how much work is visible and more on what that work changes for the business, the product, and the future engineering team. Commits, tickets, pull requests – all these things matter… but only as evidence. The real question is whether the decisions behind them created value that lasts.