Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
Last year people were asking "where are the apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
For that other stuff you mentioned like the right click menu. Those huge corporate projects suffer more from layers of institutional dysfunction and will be very very slow to show any improvement. Their dysfunction can't be solved with just faster coding.
A good chunk of what my company has been doing with AI falls into either burning down our known tech-debt and "easy wins" that no one ever had the bandwidth to approach... And improving / automating our processes. The former is having a direct and meaningful impact on the quality and availability of our services.
Our QA, formerly a fairly frequent blocker of all our releases, are doing more in-depth reviews and catching issues earlier in our release process. They have become unblocked to the point they are actively chasing down work that starts to slip.
We have cleaned up and tuned both our security alerts and operations logs and improved our tenant isolation in our service in a way that makes customer and formal audits SIGNIFICANTLY easier.
We're setting ourselves up for faster human development of the hard-things. Our development environment and infrastructure are faster, cleaner, more auditable processes, and cheaper overall to operate.
These fixes mostly don't show up in our product change logs, and definitely don't fall into "new features". It would largely be invisible to the outside world, but our costs are going down (though to be fair, not offsetting the spend on AI to date), internal productivity has improved, operational incidents are down, and customer satisfaction is up.
The same reason it took many years for corperate america to get a real productivity increase from computers and the internet, all the old ways of doing things had to be redone. I think the largest companies are least equipped to take advantage of AI productivity gains. Agile no longer makes sense, Org charts as no longer make sense, etc...
The follow on question is if it's making us all so much more productive, where is the increased revenue? As far as I can tell, it's mostly the AI labs seeing that, not everyone using them (modulo small founders building new things and doing okay, I think)
AI doesn’t make a responsive right-click menu a higher business priority. But we are seeing a ton of small custom projects that are as easy to dismiss as they are to abandon.
Surely it makes the backlog clear faster to the point where you reach the non-priority stuff? But I'll bite, what business priorities are being created at 100x?
In my experience everyone is just rebuilding the same wheel over and over.
A lot of people may be more empowered to create things now with less up front effort but it doesn’t lead to having better ideas or more actual system architects.
In my case it's not the CI that's the bottleneck. It's the human testing side. Does it work, sure. But does it actually do the thing we want (and more importantly) does it do it in a way our customers will understand and actually like?
I feel like if anything LLMs have reduced coding/software to "throw as much as possible at the wall and see what sticks". It feels quite shortsighted and wasteful, especially considering that humanity needs to get better about how it produces and consumes energy. It's sort of bleak.
Think of it as a layered problem. If the bottom layer (CI) cannot keep up with the output of agents, then solving problems at a higher layer - like user experience checks - will be exponentially slower and less reliable. Kind of like how optimizing tight inner loops makes your whole program faster.
Clearly you have to replace obsolete human testers with agentic AI testers, duh.
At some point, with all this velocity, human users become the bottleneck, unable to keep up with and learn all the changes and new features. Luckily, there's a simple solution: just replace the human users with agentic AI users.
> The forceful executive, Henry Ford II, and the leader of the automobile workers union, Walter Reuther, both saw many examples of advanced machinery operating at the plant. The words they exchanged brilliantly encapsulated the paradox of automation:
> Henry Ford II: Walter, how are you going to get those robots to pay your union dues?
> Walter Reuther: Henry, how are you going to get them to buy your cars?
I think you're being sarcastic, but there is actual truth behind what you're saying.
Because every developer is now a slop cannon by default, by default users will experience churn and whiplash, and things will break all over the place. As you point out, this is bad. It's also impossible to fix without deploying agents on the QA side. Like it or hate it, agentic testing is inevitable to protect users from the churn and noise caused by the slop cannon. I don't think that replaces test engineers at all - if anything it makes the job more fun. If you've ever had to keep playwright tests in sync with the target manually, and kept the CI environment up to speed with toolchain changes, you know what I mean.
Whether the "slop cannon by default" situation could have been avoided in the first place, is another question... But we're here now and there's no going back. Might as well deal with it as best as we can.
> Moving our workloads off GitHub Actions to third-party runners with faster CPUs, higher-performance storage, and better cache infrastructure gave us faster machines to run the same pipeline on
Yeah, was not surprised to read this. Actions is convenient if you already use GitHub, but it can also be pretty slow. Given reliability is also a major issue with GitHub these days I expect to see more orgs moving to different pipelines
We switched our Actions workload to blacksmith.sh (not affiliated) and have been pretty happy with how fast and inexpensive they are. I wouldn't be surprised to see this trend continue.
I’m conflicted about Linear’s progression. I dislike some of the features but on the whole they’ve managed to keep the software pleasant to use, it doesn’t feel to me that it is drifting towards Jira territory, rather, it feels like it is losing the carefully considered product design because now code is cheap to generate. I’m not worried about it turning into Jira but it has lost its soul. Still a great product.
I haven't used Linear in any serious capacity, but I feel like there's plenty of market in "Jira that doesn't feel like it hates its users" and "Jira but we care about performance".
Not saying that they're shooting for either of those segments, but someone should. Plenty of enterprise orgs that need (or are convinced they need) Jira's featureset.
I lead a bazel conversion for a pretty complex piece of software written in 5+ programming languages and shipping native binaries to all 3 major OSes a few years ago, and it took multiple years to get it done.
For a less complex project (1 programming language, still shipping to all 3 major OSes), with my knowledge and agents I got the bazel conversion done in 2 weeks.
The setup cost for bazel just went down by a lot, and I don't think the industry as a whole is aware of that yet.
The entire industry, including its outputs that LLMs are trained on, hasn’t reconsidered what’s easy vs. hard or fast vs. slow. LLMs consistently recommend against code changes because they will take “a weekend”. No, Claude. You will do the work and it will take 20 minutes.
Bazel seems to have a lot of tradeoffs, from setup time of the sandbox for each task, to ergonomics that lead folks to maintain parallel 'normal' tooling.
Plus, 'with a warm cache' is doing heavy lifting, what's the real cache hit rate for a week of development? Investing in improving the cold build and frequent actions is still important with bazel or any incremental builder.
I'm not sure it's useful to talk about bazel broadly, it's actual performance and behavior comes down to the rules you use. You can configure bazel like turbo/nx and cache tsc/vitest/eslint on each package.json module, and get course cached units that are evicted on every change, or you can use gazelle and target per-file actions which are only invalidated when their dependencies change. But that trades off batching unless you use workers.
Can't comment on Bazel specifically, but having worked with both nx and turbo, the bottleneck was usually network and disk IOPS rarely compute.
Even fully cached outputs needs to fetched and read from a remote server[1]. A step n-1 outout fetched from remote cache server need to written to disk and then again read by step n[3] - all disk I/O and network bound operations.
10s may be achievable/realistic goal in the Java/C++ world where Bazel normally seen. In TS eco-system most people would be over the moon to get into ballpark of 1-2m for a decently large monorepo.
We should define Build more clearly here, if you mean running just transpile/compile steps or the full series of steps that includes tests (as the linear post here is talking about). It is hard to see even a small sub-set of a large suite of test that require a virtual DOM or a real browser can run in 10s or less.
[1] Typical for say managed CI setup .
[3] Common run-of-the-mill frontend + backend stacks in different languages etc.
If you don't need to rebuild anything, bazel can fetch only the final artifact (not the intermediates) from the remote cache.
Also, if you have persistent CI workers with a persistent bazel instance, you save on some network roundtrips, but that's obviously harder to set up and make bulletproof.
Agentic Coding has been a huge strain on CI,
I have been using Bazel to improve our build times and ultimately building customized runners to improve our CI
Anyways great blog post from linear team a lot to learn from it
Mostly around, at least the majority of 'stuff' I've worked on (both before and after the rise of coding agents) myself or others took enough time to make sure that anything done locally, if you run the tests locally, you're at least 90% of the way there as far as what CI/CD does.
I suppose the flipside being, most of those projects had less churn (i.e. one person was working on a service at a time, and we had good contracts between services.) Also, Our local boxes were way better than our CI boxes, so there was incentive to run locally versus waiting 2-10x the time for CI to run...
My take: it seems like systems should become smaller, more isolated, and contract-oriented.
I have been a long time proponent of monoliths, but it seems like agents would be happier with smaller, more isolated services. The more isolated, the better. Contracts between the service components only. Then it can iterate internally as long as it satisfies the contract. If it needs to, it can version the contract and keep iterating.
I wonder if the GitHub actions outages we keep seeing is due to themselves making self-hosted runners paid, hence bringing broke/cheap users back from hosted runners to their garbage infra.Meanwhile I'm running my Codeberg Actions on the free Oracle ARM machine 2 cores 12GB ram (previously 4 cores 24GB) and way more reliable. Keep winning bozos.
Edit: After second thought, I guess "alternative runner" providers still have to pay the self-hosted tax. So M$ actually saves and makes money by not scaling their infra and driving people to alternative providers they can tax freely. Actual geniuses.
(Disclosure: I'm a cofounder of RWX) For anybody wanting to solve similar problems and considering Bazel, take a look RWX. It's built around the same concepts of content-based caching and graph-based task execution, but it's far more runtime agnostic and easier to adopt. https://rwx.com
I just feel apps like linear are increasingly getting in the way of full send agentic development where sub agent orchestration is done through agent to agent messaging, work trees, on demand git restructuring and epoch specific coordination plains, often .md files. The smaller the human component of total product development gets, the more this may be the case.
Linear is a key part of my prompting technique. I write specifications in Linear, and the history I generate in it becomes a critical source of context for my agents. Prompting my agents has become “look at XYZ-124 in Linear and ask me any questions you have.”
> where sub agent orchestration is done through agent to agent messaging
How do you expect to see the history/record of what the agents did and why? Is it enough to see it in PRs? Do you expect tickets that have the design and history? How are you thinking of agents being able to historically resolve reasoning/why/decisions made in earlier passes?
Genuine open question here. My assumption is that a GH or Linear or Jira is still useful as a decision store. It may as well be a custom app over Postgres, but it seems like something is needed to store this and for observability. A GH/Linear/Jira is nice if only because of standard APIs and integration points (whatever you build would likely end up duplicating a subset of those).
My recent winnowwallet.com still has build times of over 10 minutes. I'm convinced the agent really wants build times around 5m to move at a quick pace. Also to not drive me insane. It took a ton of work to get it down from 45 minutes because my tests launch a full version of the app, and walk it through major usecases while recording video and screenshots. I then use AI to qa this. It also generates its webpage this way. All from CI/CD
I cannot believe we are in 2026 and CI/CD hasn't evolved enough to even consider hot updates. Burn your CI/CD pipelines to the ground and start over again without any of the slop. Don't let anyone who calls themselves DevOps Engineers design it again, only people who are System Administrators.
Maybe I'm alone in this, but as someone who is in tech, I don't know what CI is, and I don't think it's unreasonable to expect it to have an expansion within the article the first time you use it...
AI is in the cultural zeitgeist, but you gotta expand most other things at least once.
So if you don't know what CI is, what does "I am someone in tech", mean? Marques Brownlee probably doesn't know CI is, but he's also in tech. Because if you by mean "I am in tech" that "I am a software engineer", and you don't know what CI is, then boy; I'd be worried for you.
Also the case of 'learning that someone did updates directly on AWS console instead of terraform and losing 0.5 or more days cleaning up the resulting mess'
Within the engineering tech sphere, CI/CD have also been terms that have been standard for at least a decade now.
You would probably be rejected from most interviews at the first stage if you didn't vaguely know what they mean at this point.
Linears entire product is tailored towards software engineers/engineers in general or people who work alongside engineers, so its not surprise their posts have a bit of assumed knowledge.
Most days I feel like I must be the dumbest person on HN. I don't understand what 80%+ of submissions are about. But if it sounds interesting, I'll dig into it a bit and learn a few things along the way.
I recommend making a habit of googling or asking AI about terms you run into. As someone who is in tech, you will hear tech jargon your whole career that will not be explained if it's considered standard terminology for the audience being spoken to. It is better to learn to educate yourself than expect others to go out of their way.
Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
For the question where are the alternative OSes? Here is one that I've seen. There's probably more - https://www.reddit.com/r/ClaudeAI/comments/1wfpydl/i_asked_c...
For that other stuff you mentioned like the right click menu. Those huge corporate projects suffer more from layers of institutional dysfunction and will be very very slow to show any improvement. Their dysfunction can't be solved with just faster coding.
Our QA, formerly a fairly frequent blocker of all our releases, are doing more in-depth reviews and catching issues earlier in our release process. They have become unblocked to the point they are actively chasing down work that starts to slip.
We have cleaned up and tuned both our security alerts and operations logs and improved our tenant isolation in our service in a way that makes customer and formal audits SIGNIFICANTLY easier.
We're setting ourselves up for faster human development of the hard-things. Our development environment and infrastructure are faster, cleaner, more auditable processes, and cheaper overall to operate.
These fixes mostly don't show up in our product change logs, and definitely don't fall into "new features". It would largely be invisible to the outside world, but our costs are going down (though to be fair, not offsetting the spend on AI to date), internal productivity has improved, operational incidents are down, and customer satisfaction is up.
A lot of people may be more empowered to create things now with less up front effort but it doesn’t lead to having better ideas or more actual system architects.
What exactly are you using to back up this claim?
> QA reviews [...] fully automated
At some point, with all this velocity, human users become the bottleneck, unable to keep up with and learn all the changes and new features. Luckily, there's a simple solution: just replace the human users with agentic AI users.
> Henry Ford II: Walter, how are you going to get those robots to pay your union dues?
> Walter Reuther: Henry, how are you going to get them to buy your cars?
https://quoteinvestigator.com/2011/11/16/robots-buy-cars/
Because every developer is now a slop cannon by default, by default users will experience churn and whiplash, and things will break all over the place. As you point out, this is bad. It's also impossible to fix without deploying agents on the QA side. Like it or hate it, agentic testing is inevitable to protect users from the churn and noise caused by the slop cannon. I don't think that replaces test engineers at all - if anything it makes the job more fun. If you've ever had to keep playwright tests in sync with the target manually, and kept the CI environment up to speed with toolchain changes, you know what I mean.
Whether the "slop cannon by default" situation could have been avoided in the first place, is another question... But we're here now and there's no going back. Might as well deal with it as best as we can.
TLDR: it's not all bad :)
Yeah, was not surprised to read this. Actions is convenient if you already use GitHub, but it can also be pretty slow. Given reliability is also a major issue with GitHub these days I expect to see more orgs moving to different pipelines
Not saying that they're shooting for either of those segments, but someone should. Plenty of enterprise orgs that need (or are convinced they need) Jira's featureset.
For a less complex project (1 programming language, still shipping to all 3 major OSes), with my knowledge and agents I got the bazel conversion done in 2 weeks.
The setup cost for bazel just went down by a lot, and I don't think the industry as a whole is aware of that yet.
Tsgo, oxlint, caching dependencies etc. what linear outlined in their blog post would be more impactful for the average TS project I've worked on.
Plus, 'with a warm cache' is doing heavy lifting, what's the real cache hit rate for a week of development? Investing in improving the cold build and frequent actions is still important with bazel or any incremental builder.
I'm not sure it's useful to talk about bazel broadly, it's actual performance and behavior comes down to the rules you use. You can configure bazel like turbo/nx and cache tsc/vitest/eslint on each package.json module, and get course cached units that are evicted on every change, or you can use gazelle and target per-file actions which are only invalidated when their dependencies change. But that trades off batching unless you use workers.
Even fully cached outputs needs to fetched and read from a remote server[1]. A step n-1 outout fetched from remote cache server need to written to disk and then again read by step n[3] - all disk I/O and network bound operations.
10s may be achievable/realistic goal in the Java/C++ world where Bazel normally seen. In TS eco-system most people would be over the moon to get into ballpark of 1-2m for a decently large monorepo.
We should define Build more clearly here, if you mean running just transpile/compile steps or the full series of steps that includes tests (as the linear post here is talking about). It is hard to see even a small sub-set of a large suite of test that require a virtual DOM or a real browser can run in 10s or less.
[1] Typical for say managed CI setup .
[3] Common run-of-the-mill frontend + backend stacks in different languages etc.
Also, if you have persistent CI workers with a persistent bazel instance, you save on some network roundtrips, but that's obviously harder to set up and make bulletproof.
Anyways great blog post from linear team a lot to learn from it
Mostly around, at least the majority of 'stuff' I've worked on (both before and after the rise of coding agents) myself or others took enough time to make sure that anything done locally, if you run the tests locally, you're at least 90% of the way there as far as what CI/CD does.
I suppose the flipside being, most of those projects had less churn (i.e. one person was working on a service at a time, and we had good contracts between services.) Also, Our local boxes were way better than our CI boxes, so there was incentive to run locally versus waiting 2-10x the time for CI to run...
I have been a long time proponent of monoliths, but it seems like agents would be happier with smaller, more isolated services. The more isolated, the better. Contracts between the service components only. Then it can iterate internally as long as it satisfies the contract. If it needs to, it can version the contract and keep iterating.
Edit: After second thought, I guess "alternative runner" providers still have to pay the self-hosted tax. So M$ actually saves and makes money by not scaling their infra and driving people to alternative providers they can tax freely. Actual geniuses.
Genuine open question here. My assumption is that a GH or Linear or Jira is still useful as a decision store. It may as well be a custom app over Postgres, but it seems like something is needed to store this and for observability. A GH/Linear/Jira is nice if only because of standard APIs and integration points (whatever you build would likely end up duplicating a subset of those).
AI is in the cultural zeitgeist, but you gotta expand most other things at least once.
Within the engineering tech sphere, CI/CD have also been terms that have been standard for at least a decade now.
You would probably be rejected from most interviews at the first stage if you didn't vaguely know what they mean at this point.
Linears entire product is tailored towards software engineers/engineers in general or people who work alongside engineers, so its not surprise their posts have a bit of assumed knowledge.
Its best thought of as the testing systems that are run as part of pull request review / merge to main / build processes.
Most days I feel like I must be the dumbest person on HN. I don't understand what 80%+ of submissions are about. But if it sounds interesting, I'll dig into it a bit and learn a few things along the way.