Best of…: Best of 2015: The A(nti)-Team

Building a good, healthy team environment is hard. In this story, they just don't bother. --Remy

In the 1980's, there was a TV show called The A-Team. There was the scrounger, who could scam anyone out of anything. He would make promises that were sort of true to get what he wanted (sound like marketing?) There was the tough guy who could intimidate anyone into doing anything. He knew how to get things done, but underneath it all, was a nice guy. There was the leader, who could always come up with a plan to save the day. And there was the one guy who was a little crazy (the good kind of crazy), but who you could count on in a pinch. There was also the occasional outside helper who would run interference and recon. This was a group of folks who worked as a well-oiled machine to get the job done. Failure was not an option! They were a team!

The A-Team never filed a project methodology document. No wonder they were wanted criminals.

Alex had taken a job on a new greenfield development effort to replace an aging and unsupportable birds-nest-o-wtfa. Naturally, the position was advertised as "we intend to do things right!" The project is fully funded. We will have the proper equipment and team personnel to get this job done. We have the full support of six layers of management plus all of the users. Alex was optimistic.

The first thing they did was spend several months wrapped in those numerous layers of management, end users, support folks, senior people who used to support the project (to explain the problems that plagued the old system), and the three architects of the new system. The new architecture was heavily documented, presented to and signed off on by all of the above. It was even reviewed with a critical eye by an independent third party regulatory auditing agency to ensure that the overseeing authorities were confident that the correct approach was being taken.

An 8 page document detailing development coding guidelines (e.g.: code formatting settings, naming conventions, unit tests, code coverage and other such team-wide items) was created, reviewed and decreed to be followed by all who worked on the project.

The project was off to a good start.

Job one was to hire the development part of the team. For this, they looked (very far) offshore to find the cheapest possible talent. After all, anyone can be trained, right? A team of 11 developers who collectively had 13 years of experience, and a team leader with 5 years of experience were hired and put in place.

The next major decision was which database should be used. There were three in widespread use at the company. Since all of the databases were hosted on centralized servers, one was immediately ruled out because the hardware that hosted the data servers was insufficiently powerful to handle the expected load in a reasonable time frame. Of the other two, one was widely used by everyone on the team. They knew its syntax, quirks and limits. The the third was mis-configured to have a reputation as being flaky. However, that one also was the corporate standard. In spite of the objections of the team, they used the third one.

Project management decided that QA folks could be brought in later.

Finally, it was time to begin doing detailed design. The offshore lead decided that a lot of time could be saved by doing design on-the-fly as required. Of course, the architects objected, but the project manager agreed to it.

And so the architects started working on building the controller engine and other such mainstays of the project. The junior team, which was to query numerous remote systems for input data, merge, filter and pre-process it, decided that they knew better than what was specified in the architecture document, and started designing their own way of doing things. Without telling the architects or management.

Come time for the first sprint check-in and all sorts of red flags flew up during code reviews. The junior lead decreed that the architecture document was only a suggestion that could be ignored in favor of the developers desires. Naturally, this spawned lots of are-you-f'g-kidding-me's and emails up the chain. The project manager and above seemed disinterested, saying that the junior developers shouldn't be doing that, but we trust them to do the right thing.

This went on, with the architects pointing out implementation flaws and shortcomings that would not support the requirements. All suggestions were ignored, because the offshore lead said "Google fosters an environment of innovation and creativity; we should too!" He was reminded that Google is (in large part) a think-tank, and that this was a highly regulated project within a highly regulated industry. The architecture, which had been signed off by more than 40 managers, was not optional or a suggestion, but mandatory. This was not kindergarten, where creativity is fostered; you had to stick to the approved plan! Now, we're not talking about how to write a subroutine, or encapsulate an object; we're talking about using threading incorrectly and in the wrong places, doing database accesses and interprocess communication in such ways that would not be scalable, or provide enough throughput to finish daily runs by regulatory deadlines. Spawning multiple processes instead of just using threads. Using files to act as semaphores, because that's how they did it in school. The list goes on.

None of that mattered. The junior developers resented that they were not consulted on the architecture, and so were bent on ignoring it - with the blessing of their lead. The project manager continued to acknowledge the problems, but didn't do anything about them. The problems were reported up the chain, and nothing was done. Everyone on the team should have an equal say in things.

In the real world, if a student thinks the teacher is wrong, he doesn't get to change his grade. The surgical resident cuts where the surgeon says and not the other way around. The general doesn't discuss strategy with the privates. If you join a union, and as the new guy demand to have equal say on policy with the union bosses, you'll be bunking with Jimmy Hoffa. Experience speaks with exclamation points. Inexperience speaks with question marks.

Except on this "team".

The junior developers continued to do what they thought was best, ignoring the architects at every turn. Much of their code was written and rewritten several times over because the designs by the juniors didn't take things into account. Things more experienced folks know to plan for. By the time 8 months had passed, so much damage had been done that some of the more complex requirements simply couldn't be hooked in, and more than a month of back-pedaling had to be done on a greenfield development project.

About this time, management acquiesced and asked some of the business users to write business-level tests (e.g.: via a spreadsheet that would be fed into JBehave to JUnit test things). The developers would provide the underlying code and some sample entries in the spreadsheets. The architects said that QA folks should be hired because business folks rarely know how to deal with edge cases, precision issues, etc. But the money was not to be spent. After six months of effort, the business users proudly decreed that all the tests for the entire application (e.g.: the entire requirements document) had been set up. A five minute glance showed that they didn't handle edge cases, null cases, precision cases, or most of the other things that usually require tests. In fact, they had put all of the records that could possibly be processed (at least in their minds) into one giant pass-fail test. Of course, when something changed and it inevitably failed, there was no way to know what failed.

Finally, it got so bad that the architects built a physical wall in the code between the setup code (written by the offshore folks) and main engine (written by the architects) sections of the application. Immediately before the main engine began to grind the data, every single variable in the system would be flushed to a state table in the database, so that when something would inevitably be challenged, they could show the inputs that were provided and send the fix-it work to the offshore team. At least this way, they could insulate the main engine from the debris.

The department saved a lot of money by using cheap labor, no QA folks and the politically expedient database. Of course, all of the code of the setup portion done by the offshore team was a disaster, and for the most part, very difficult to learn, support, debug and enhance.

The product hadn't even been deployed yet, and the users were already complaining that it took too long to diagnose and fix problems (one of the main reasons the whole rewrite project was authorized), that perhaps the rewrite wasn't satisfying the main purpose of the rewrite, and that perhaps something might be wrong"

[Advertisement] Scout is the best way to monitor your critical server infrastructure. With over 90 open source plugins, robust alerting, beautiful dashboards and a 5 minute install - Scout saves youvaluable engineering time. Try the server monitoring you'll today.Your first 30 days are free on us. Learn more at Scout.