The joys of maintenance programming (2011)
Comments
I gave a presentation many years ago to a software process group in Silicon Valley entitled, "Unsolved Problems of Software Maintenance". The problems are all still there.
Having done lots of both greenfield and maintenance work over four decades, I can unquestionably say that software maintenance is way, way more difficult.
I heard this first from Joel Spolsky, "it's easier to write a program than read it."
Which led me to write an article on the subject, "How To Read A Program," about the cognitive aspects of trying to learn unfamiliar code. https://www.itjungle.com/2009/01/28/fhg012809-story01/
The article uses AS400 applications as an example, but is general enough that it applies to applications in most languages.
This is a field begging for help from AI.
Someone needs to start doing comp-lit style studies of computer code instead of literature.
Most of the problems of API design could be remedied by people stepping through their own code and looking askance at the whole process.
I love maintenance programming. I love legacy programming too. I have a story where the two worlds collide.
At my last job I had to work with a PowerShell codebase that `cloc`ed in at over 100,000 lines, and was absolutely ancient to boot, having been added to in bits and pieces by sysadmin after sysadmin over the course of 12 years total, in 3 different human variables-and-comments languages, all of whom were Windows power users with very clear ideas of what needed to be done, but also all of whom were very clearly learning to program for the first time, at least in the wonky PoSH paradigm. AKA: Classic big ball of mud situation. Every new customer had to have their details run through this codebase, however, and I saw the potential to eventually boil this down to a simple 6- or 7-parameter cmdlet (PoSH for "shell script"). I got my undersized chisel and toy hammer and got to work.
In some ways starting work on the proj felt more like beginning to dig into an ancient COBOL database in a System/360 mainframe than anything resembling the state of the art in 2023, or even 2011. There was a single VM that everything had to run on, and the script couldn't even be called remotely via SSH because the only way in was via a web browser based remoting session. The connection was spotty and would often drop halfway through coding in there in my dinky little Notepad++ session, requiring minutes of reauthentication just to get back to work. Unit testing (which PowerShell had a surprisingly good third party library for, called Pester) was out of the question. And: There was no version control! No wonder it got to 100k lines - most of the code was being entirely unused! But what?
The first few Saturdays I spent didn't go very far. I was mostly trying to figure out where the logic even began and ended in this mess of spaghetti. But eventually I started to piece together an overall structure for the thing. I made a pact to myself not to delete any of the many, many files that had accrued over the years until I knew every file that the code would touch first. I started doing some lightweight encapsulation with the -WhatIf flag, and with firebreaks (sticking `return` after each tiny function) so I could work on one linear piece at a time, and gradually tiptoe my way forward to the finish line. Eventually I got there, and managed to strip the whole thing down to about 1500 lines.
The joy of this kind of programming is that, even if you don't start with regression tests or modern practices, you do know from having ran through the original process a million times exactly what the end product is supposed to look like. Since no one is paying attention to your refactoring work in the first place, you're allowed to take your sweet time. To me it was obvious this mess of code was going to become a blocker for the business at the pace we were going: We were a SaaS, and yet we had to manually provision customers behind the scenes like this, in a process which took days to weeks? And we didn't even have a proper CLI we could call for this? Jackpot!
I left the company about 2 months after wrapping this project up and demoing it to my coworkers in search of better pay, but I'm really glad I got the chance to take such a fearsome codebase and tidy it up like that. There is just something satisfying about that kind of work, it's like gardening. I also got really, really good with Notepad++'s find and replace across all files functionality. There is in fact a skill to it! (Mostly that you should always, always find and replace the longest strings first!) Although I think these days I'd probably reach for a similar function in vanilla vim first, or even in bare `sed` if I was feeling gutsy.
(I'd end with "If you, dear reader, are in a similar PowerShell quagmire you need someone to pull you out of, I'm available for consulting at $X00/weekend", but alas. My weekends for the next few months are already sworn to another equally wonky project for my current employer. At least I'm getting overtime this time. ;) )
I felt so much joy just reading this! Yeah, it is truly great to just simplify.
One more benefit is to learn how to tame complexity. The most complex piece of software doesn't come from nowhere; it is built over years, if not decades. People will never experience the same level of complexity in the "greenfield" programming. And that experience changes the entire mindset, from how to write the codes to how to structure the entire engineering org.
After the NX bit, and cookies, and shadow stacks, and now tagged pointers (again) I'd rather stop pushing the guardrails for shitty code down into the hardware. It costs performance and the bandaging never ends.
Let's say I wanted to do this. How do I actually get started?
Let's say I make a website that showcases my qualifications and lists out the services I offer.
How do I actually get a client?
The author of the article also has some thoughts on those topics: https://typicalprogrammer.com/how-to-start-freelancing-and-g...
I’ve only skimmed it and have started reading but the parts about highlighting/phrasing of one’s marketable skills seems good.
Nice post. In the past I've had colleagues (also senior people) who disliked touching "legacy" parts (e.g. Java based microservices compared to newer Express.js or Kotlin based microservices which don't provide half the value that the legacy parts do). I also think that maintenance programming can be rewarding. However, I do believe that it does not make sense to throw entry-level engineers into the maintenance mess alone: they still need some guidance in my experience. Then you'll be able to reap the benefits such as getting better in debugging and understanding more about the business problems that the software is solving.
Do people understand code by line-by-line reverse-engineering what the code is doing, or do they understand it by relating it to what they've written before?
If the latter is the case, then you get better at reading code by writing code. Writing lots of code puts those code-patterns into your long term memory, and then when you see those code-patterns again you'll recognize them.
For system design too - if you've designed lots of systems yourself, then when you see a new system, you'll be able to relate it to the systems that you've designed before.
So maybe building greenfield projects also makes you better at maintaining existing projects?
It'd be great if someone could point me to some existing literature on this topic. I've looked around and can't find any.
I haven't had to do it a lot, but in previous cases where I came into a codebase blind and had to fix bugs or figure out what's going on, I've always started by mapping the flow first and then digging into the details of what's going on. When looking at the flow, I go breadth-first through everything and then dig in from there.
I maintain two major maps, which are what is the code logically trying to do, but also what exactly it's doing. (i.e. Sanitize inputs, check error conditions, do work, return result as a logical model and then "ensure foo is < 0 and > -10" etc. as the specific checks).
> So maybe building greenfield projects also makes you better at maintaining existing projects?
I think it does, because it builds a higher-level sense of how something "could" or "should" be and familiarity with thinking at the system level.
I've had a lot of problems with people who (seemingly) only have experience maintaining projects. They seem to have a tendency to focus narrowly on "fixing" a bug close to its source, and often lack understanding of a lot of fundamental technologies (because they only attend to things that have broken), and get stuck by a kind streetlight fallacy where they fix things in more familiar areas. The end result is ugly kludges and whack-a-mole bugs where a superficial "fix" in one area causes a new bug to pop up in another.
>> So maybe building greenfield projects also makes you better at maintaining existing projects?
Only to an extent, i think.
> I've had a lot of problems with people who (seemingly) only have experience maintaining projects.
And similarly I've had problems with people who (seemingly) only have experience with starting a new project and then simultaneously over-engineering and piling on tech debt.
I think "i've never bootstrapped a project before" is easier to cure than "I don't have a good sense of what's expensive vs what's cheap tech debt."
Some tech debt is basically 0% interest. Was this small hack bad? Yeah. Will it need to be fixed, eventually? Yes, for sure. But does it compound every day, or does it just sit there being a little yucky? Very easy to determine in hindsight. Very hard to predict as you write the hack. The end result being the simultaneity in my example. People will over-engineer things that won't end up being problems and under-engineer things that will turn out to require compounding hacks and it would've been cheaper to just get it right the first time.
>> I've had a lot of problems with people who (seemingly) only have experience maintaining projects.
> And similarly I've had problems with people who (seemingly) only have experience with starting a new project and then simultaneously over-engineering and piling on tech debt.
The best thing is to build a greenfield project and then maintain it for several years, fix your mistakes, then do the same thing on a new project again (including the maintenance).
You really need the full spectrum of experience, and put yourself in the shoes of the future maintainer when you're building something new.
This. I've worked in both greenfield and brownfield areas. The chief failuremode of greenfield is that it works, but it is unscalable and shoddily done. The chief failuremode of brownfield is some kind of feature cardinality explosion that becomes impossible to maintain after some critical juncture. Reading between the lines, they're sort of the same failure mode happening at different times, but both driven by results-driven-programming rather than architecture.
>If the latter is the case, then you get better at reading code by writing code. Writing lots of code puts those code-patterns into your long term memory, and then when you see those code-patterns again you'll recognize them.
Let's hope what you write the first time is a good practice.
There are lots of greenfield-only programmers who build something (by glueing stuff together), then they proceed to ship their barely working software to get that promotion / bonus and they run away to the next project
They never have to maintain anything, some poor soul has to fix the broken software.
They also dont even know what mistakes they made, since they arent there to solve it.
I think there’s something to be said for taking a job on a late stage project early in your career, participating in RCA analysis of all of the problems, and then picturing yourself in the room when those decisions are made and asking if you would have made the same decisions or better ones. Then work next on a greenfield project to test your theories out and adjust/grow.
I never felt like I had 1 year of experience 5 times because I moved between several kinds of projects at different lifecycle phases and with different constraints, and drew lots of parallels between them. At the end of five years I had project histories in my brain that spanned more than ten years. And got jobs that should have gone to someone with 8 years’ experience. I do not think this was a coincidence.
The joy of Greenfield is short lived. At some point, it becomes too costly to fully refactor and you are stuck with what you chose and edging closer and closer to legacy dev.
Some projects though just keep on trucking along and doing the next part with the newest shiny thing. You end up with five ways to access data, three ways to send emails and fifteen different methods of report generation. Frankenstein architecture! This architecture also emerges when you have leaders that don't make decisions and peeps debating all day on slack and then doing their own thing anyway because they didn't really want to offend anyone and everyone's ideas are great.
I'm currently on a sort of sabbatical from greenfield research projects. I'm doing maintenance programming on a large widely used open source project. It's been really refreshing. I didn't realize how much I was feeling the pressures of 15+ years of greenfield research. By contrast the maintenance programming is clearly defined, easier to put down at the end of the day and unambiguously useful. I'm also rapidly improving in my debugging, code understanding, profiling and optimizing skills which will make me a better programmer when I do return to writing new code.
I’ve described this to other programmers before. Greenfield projects are fatiguing because you still have options—too many options. Every layer is still open for you to rethink and rework. Let the hand wringing commence!
As the months and years wear on though, things change. Each bug has an obvious solution, and you’ll take the path of least disruption. Implementing new features is a matter of figuring out which trade offs you’re most ok with. Maybe you need to refactor something first, but that’s an obvious path itself.
You’re painted in a corner. A beautiful corner where the only safe thing to do is to tread carefully. But you don’t mind, because you know where you are and what you need to do to get to where you want to go.
Nicely put. I'd also add that for me being a researcher became my identity and so my self-worth became tied to the perception and adoption of the research. With an established project (especially one I didn't write) there's already a stable acceptance of it's value and so I'm free to just maintain and improve it.
Discussed at the time (of the article):
The joys of maintenance programming - https://news.ycombinator.com/item?id=2267261 - Feb 2011 (29 comments)
The dupe detection on HN during submission doesn’t find much older submissions?
For much older submissions it doesn't try, because reposts on HN are fine after a year or so (https://news.ycombinator.com/newsfaq.html).
There are a bunch of reasons for that, but two interesting ones are (1) historical material is particularly welcome here, and (2) we want cohorts of new users to get an 'education' in the 'classics'. It's too easy for grizzled veterans to assume that everyone has already seen the perennials, but even they once read The Story of Mel, or whatever, for the first time!
>Skilled programmers recognize what makes good code good, and what makes bad code bad
You're still somewhat biased, so you can get put on projects where the style is different from yours. If you happen to think that style is bad, it won't be fun to work on.
You shouldn't want to change the style. Chances are you'll do it incompletely (job change) or the time spent won't be worth it. If someone else comes along and doesn't like your style, we're just going to repeat this process.
There's been way too many different approaches that work for developers to be unified on what styles are best.
Style is not what make code good or bad.
Maintenance programming for pay can be rewarding iffffffffffffffff management has a deep understanding of and appreciation for the task. There can be a lot of joy and satisfaction in adding tests, fixing edge cases, fulfilling user needs, "quality of life" improvements, optimizing. Honing.
My experience is that this is almost never the case.
Generally, management does not understand or appreciate why it's a slog, why you might want to refactor this or that, etc.
Obviously a lot of this falls upon the engineer. It's your job to communicate and demonstrate. Helps if you have other engineers backing you up and presenting a unified message. Also helps if management is technical enough to understand what you're saying instead of simply believing -- or not believing -- you.
But maaaan. Uphill battle! I'm burnt out on it. I've decided I need to seek greenfield stuff. Maybe forever. But definitely for now.
I was almost bounced by HR during a screening interview a couple of years ago when they found out I had been doing maintenance work for the previous six months. I had to explain that it was a tiny part of my overall work experience. The hiring hypocrisy is strong.
Also as you point out, getting funding for preventative maintenance or continuous improvement is extremely difficult in a corporate IT environment, especially if there isn't a product owner to advocate for the system and to secure funding. Although I do believe you need a product owner to drive the work to ensure it delivers business value, and not have it driven solely by developers.
Nobody cares about "eliminated a whole class of tickets" aside from the support staff, and really who cares about the support staff at most companies. What would the phone monkeys know about our software. /s
Agreed. Imagine you’re a CEO: Why on earth would you want your expensive engineers reworking something that already makes money? It seems much more sensible to leave the working code alone and focus on new revenue-generating opportunities.
> focus on new revenue-generating opportunities
in mature industries, there aren't new revenue generating opportunities, there is only getting better at what you ordinarily do; all marketshare growth comes at the expense of competitors.
> Imagine you’re a CEO: Why on earth would you want your expensive engineers reworking something that already makes money?
CEO will start looking at this topic if app performance/stability metrics will start making impact on revenue stream, aka app is slow and stop working every day and customers go away because of that.
Which happens if the CEO is conscientious enough to insist on collecting and looking at those metrics.
Heh. Going to be a lot less maintenance work if they are.
True, but app performance/stability metrics are (often extremely laggy) trailing indicators of code quality. So by the time the app starts falling over, things are extremely rotten, and it's not necessarily correlateable to any specific maintenance decision(s).
By the time the app starts failing, usually the people who wrote it got burned out and left :/
Security and risk. You can run several different code/pom/package.json scanner tools that will articulate a risk that will send your IT security department into a panic attack, which you can sell as time you need to spend fixing things.
That's not what the post is about. They describe the process as mostly fixing issues with existing software, so the motivation is that something is broken or not working as required anymore.
Refactoring it is just one of possible parts of the process, much opposed to 'yeah this legacy code is broken dogshit and needs to be rewritten even though the current software is already written, proven and works for 99% of the stuff we need' approach.
> Refactoring
I personally don't believe refactoring should ever be part of "maintenance".
Maintenance needs to avoid YAGNI and avoid the engineers natural desire for over-engineering and unnecessary perfect polishing and abstruse abstraction.
Refactoring is something that should be driven by new features (or worst case towards rewrite), and few have the skill to do it well.
I'm not arguing the sibling strawman that maintenance be "the fastest, sloppiest, filthiest duct-tape maintenance imaginable."
Excelling at maintenance is highly skillful work and often poorly rewarded.
> I personally don't believe refactoring should ever be part of "maintenance".
Of course it shouldn’t, because software maintenance does not exist. Programs don’t degrade with use. All “maintenance” on software is, like new development, the implementation of new context->behavior combinations, and calling it something different is a result of applying an inappropriate physical-product metaphor.
Of course it shouldn’t, because software maintenance
does not exist. Programs don’t degrade with use
The word "maintenance" doesn't imply some kind of physical product metaphor. Sorry. That's not what the word means.
Well, I certainly agree that engineers tend towards YAGNI and this must be avoided.
Refactoring is something that should be driven by
new features (or worst case towards rewrite),
Assuming we're talking about a program that does something useful and isn't just a screensaver, it's probably ingesting data from the outside world. That data can change in a variety of ways. API partners and users can introduce changes on their end and will eventually find new and annoying edge cases. Or maybe the volume of data increases by 100x and changes must be made on your end.
Or maybe your software sort of rots as the security vulnerabilities pile up. Eventually you've got to make major version or point upgrades to the framework and/or various dependencies.
We want to handle these demands judiciously and as a rule, yeah, refactoring isn't generally going to be the answer but sometimes it is.
[Refactoring] is not what the post is about.
Clients and management generally understand that old code needs to be maintained. They don't love that reality, but that's not generally a tough sell unless they're completely clueless. And they're usually not completely clueless.
The tension in my experience comes when an engineer wants to perform something other than the fastest, sloppiest, filthiest duct-tape maintenance imaginable.
Management accepts the need for fixes, but often does not have the technical/domain knowledge to understand the short- and long-term merits of Proposed Fix A versus Proposed Fix B. And naturally, incentives heavily push management toward the quick fixes by default. Time is money, after all. And managers are expected to meet quarterly and annual OKRs. We don't have a good way to quantify the long-term value of a "proper" fix versus a "duct tape" fix. And so on.
So for me maintenance work is difficult to separate from notions of longer-term software maintenance such as refactors, dependency upgrades, etc.
Why would you turn off manufacturing equipment to do routine maintenance? It’s making you money when it’s turned on, and not when it’s turned off, after all.
Did ball bearings in application's garbage collector wear out?
CEOs tend to be short-sighted when it comes to engineering management, that's true.
and yet the CEO would not find it ridiculous to spend money maintaining plant and equipment in the factory.
It's just a lack of understanding. They imagine they understand, because software doesn't get worn out, and doesnt degrade obviously!
It is not at all obvious that trying to develop new revenue-generating opportunities will actually bring in more revenue long term compared to optimising and improving existing revenue generating sources. And it's not like the old code stops working once you start reworking it. Give your engineers N months to rework the old code, and if it's not better you don't have to deploy it.
Every larger company I've worked at had enough engineering overhead that they could afford to have small teams working on both reworking old stuff and creating new stuff, while the main teams kept the working stuff generating revenue.
It's worse that that. Most companies give career incentives to new projects. Google is probably the most well known offender, but they aren't the only one.
Very few companies understand that the most valuable asset of a programmer is his mental model of your system. You need people who can do the hard work because they understand the system at an intuitive level.
Google has turned it into a sport though. That deserves special mention.
They’ve even got their own mascot. It’s a gravestone.
Are you being serious right now? Pics. I need pics.
> the most valuable asset of a programmer is his mental model of your system.
Which is actually something a lot of management would prefer to not have. They often want people to deeply document a system, on the mistaken belief that this knowledge could be transferred out easily.
Unfortunately, it's not. Even a well documented system has inherent complexity that only working on the system would reveal. Otherwise, merely reading the source code would allow for a complete understanding and deskchecking of the whole system!
We can also break it down even more, managers are only allowed to pick one of the mutually exclusive options:
- Well documented, well maintained code that everybody could work on
- Quickly finished projects that nobody ever touches again, but only the programmer(s) have a mental model of it
The first option is more expensive at first, but way less expensive if you happen to have to rewrite it from scratch at some point (and if you are not maintaining and document it, that moment will come and it might not come at a point in time that fits the schedule).
The second option is less expensive first, but it will bite you in the arse if the programmer(s) are gone or forgot how it is done.
Everywhere else managers try to minimize risks, here they often just don't aeem to get the trade-offs involved.
The missing bit you forgot is that even if the first option was chosen, it very quickly devolves into the second option, whether the managers like it or not.
Does anyone have any tips on this beyond the additional "sales" type stuff where you have to put in a ton of extra effort to measure and sell your work?
I've found this to be true... to the point where some places even prefer manual maintenance to spending (potentially) a little more time on automation, even if that time would be paid back over the next couple of years.
I'm not adept at planning, selling my work, doing the manual maintenance, and doing the extra automation, as eventually that work becomes homework unless you get a boss that's willing to put in some effort here, which I haven't found yet.
I think one of the keys is consensus-building. If one engineer tells management that it would be a great idea to do XYZ, management doesn't necessarily know whether to believe them.
But if the whole team is on board and has reviewed your proposal it's going to be more credible. And that's just good engineering anyway.