Are senior developers even worth the money?
Experience may have less to do with a programmer’s efficacy than other attributes, a study led by the Saarland Informatics Campus has shown.
Financial institutions paying bigger salaries to experienced programmers might be skipping out on other potential metrics of success, it suggested.
The study*, which was done as part of the European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), says: “Commonly used experience measures do not correlate well to observed efficacy. Instead, we underline to use self-estimation and learning eagerness as more accurate measures for programming experience.”
The study defined programming efficacy as speed and correctness. As the chart below shows, years of programming experience were shown to have a minimal correlation with programming efficacy compared to other factors, including self-estimation of coding ability compared to peers, time spent reviewing other people's code and the number of languages a developer can code in. Strangely, actual time spent on pure programming each week was minimally correlated with efficiency.
Correlation between various experience measures and programmer efficacy:
The researchers wired research subjects up to EEGs and discovered that coding efficacy appeared related to 'cognitive load.' It's a concept that may not weigh heavily on the minds of recruiters or HR departments, but the study suggests it has a critical impact on developers' proficiency. A developer who can achieve the same results while activating less brain power is more efficacious.
"Programmers with higher efficacy can not only comprehend source code faster, but also with less mental effort," the researchers noted, adding that "because of their lower cognitive-load levels, programmers with high efficacy likely can sustain longer periods of work."
Although expert developers should be able to complete tasks with lower levels of cognitive load, the study suggested this isn't always the case. As banks try to cut costs in their technology divisions, they could trim senior developers with long years of experience and make do with junior teams that spend a lot of time mentoring each other and reviewing each other's code instead.
Levels.fyi suggests that a Goldman Sachs VP in software engineering takes home almost double the pay packet of a less experienced analyst in the same department, with $243k instead of $129k. The numbers are similar at JPMorgan ($186k vs. $99k), Citi ($151k vs. $95k), and Deutsche Bank ($206k vs. $87k).
The study also suggests that banks' method of hiring, using stressful whiteboard and coding tests, might be flawed. "A lot of hiring processes use technical interviews in front of a whiteboard, which artificially introduce stress and high cognitive load. An alternative solution to evaluate potential talent with measures that allow for accurate and quick responses, without inducing unnecessary stress and cognitive load, would be private interviews, comprehension tasks, or other alternative interview methods,” the study recommends.
Not everyone is convinced by the study, though. Eric Vergnaud, a veteran technologist of JPMorgan, HSBC, and a former Credit Suisse technology chief for the Asia Pacific region, called the study “completely flawed”, saying that “what makes a programmer efficient is their ability to deal with the unknown, the unstructured and the incorrect, not the speed at which they replicate known patterns.” It takes 5-10 years for a developer to become truly proficient, said Vergnaud.
Writing on LinkedIn, however, Capital Fund Management’s head of Data Sourcing Eric Lebigot welcomed the study, attacking the notion of experience as a measure of efficacy more broadly, saying that he had “never seen anybody change their relative skill level by going through a PhD program.”
The researchers themselves admit that the experience, expertise, and efficacy in programmers are poorly understood, especially in the way that the interact.
Have a confidential story, tip, or comment you’d like to share? Contact: sbutcher@efinancialcareers.com in the first instance. Whatsapp/Signal/Telegram also available (Telegram: @SarahButcher)
Bear with us if you leave a comment at the bottom of this article: all our comments are moderated by human beings. Sometimes these humans might be asleep, or away from their desks, so it may take a while for your comment to appear. Eventually it will – unless it’s offensive or libelous (in which case it won’t.)
