Having worked in the same place for nearly a decade tends to give you some perspective on the life cycle of a small business. I’ve found that every 2-3 years a fresh crop of engineers are hired and a tribe is formed. After the hardships of a mostly failed project that somehow pinched out a lukewarm victory, the tribe goes on a journey through the company ranks to push for a centralized or core engineering team.
A while ago I wrote a post about what it takes to bake a 762 layer cake; a thinly veiled metaphor of modern real-time software development. In the end, the sheer size of what we must maintain for today’s standards has created something of comical Rube Goldberg approach to keeping it all together.
I’ve recently ventured outside of real-time simulations and video games to peek into the world of the internet. I’ve discovered that there is no better example of the threats we all face than to look at what happens in an environment of rapid growth and profit. Welcome to the World Wide Web.
A friend and I had a long discussion about our future as software developers. We are getting older each year and, though our skills continue to branch and strengthen, it’s really a series of diminishing returns. Salaries eventually cap, and certain branches turn into dusty skills that are reserved for those few and far between contracts that pay generously just for remembering how that dated system still works. Video games are something of a young man’s dream; long hours, little pay, and a decreasing chance of personal success in an increasingly crowded and turbulent industry. My friend has left the real-time world behind and found new gold in old hills; web applications. He has convinced me to at least stick my nose in the door but I must admit I don’t like what I’m smelling.
Not having touched client-based web technologies since HTML 1.x in college, I spent a few days reading about web technologies. In my ventures I followed a trail of dead and dieing technologies, and I watched libraries abandoned and forked over and over. It was like a class in digital World History to watch these frameworks and languages thrive then die… but did they? Not really.. Many hosting sites continue to support aging technologies while some websites and web applications continue to push forward with a blend of old and new. There is only this unsettling sensation, but without any definitive facts, that most of what exists out there is in perpetual deprecation; and yet it’s still there, all of it.
I started looking into AngularJS. I’m told it’s the new hotness right now. As I learned more about Angular I fell deeper into the rabbit hole. It lead me back to my old stomping ground of HTML. This brief homely sensation was replaced with dread as I dug deeper into CSS hell. Magical tags of :before and :after and .show and .hide were everywhere. A face-punch of intrinsic appendages were scattered in there as if one was simply expected to know. I mean why wouldn’t you always include webkit-box-shadow and moz-box-shadow with box-shadow in your style template right? Duh! This is further complicated by the insistence to include other styles such as Bootstrap to help reduce the pains of mobile support. To add insult to injury most of this was processed through “Minification” which decimates the readability to save a few bytes of download and improve parsing performance.
The documentation and vast majority of training material seems highly geared toward Linux or OSX users. Most of the tools don’t even run on Windows. I spent what felt like days, installing various software packages to avoid dual booting into Linux or building virtual machines on my PC just to compile these libraries. Eventually I just gave up and used the pre-compiled Angular and Bootstrap files. I still don’t know if I’ve executed enough command lines to actually boot a local web server. I still don’t know what half of these packages do; many of which have home pages that only say, “A tool used to make your life easier. It does other stuff too.” Gee thanks.
The Internet is broken. I’m convinced of it. It’s built on a collection of systems that require an institutionalized knowledge of how things work now and how they have worked for the last 15 years or so. It’s a backwards compatibility nightmare that I don’t ever see going away. I’m certain that technologies like Angular are just another stop-gap in the road to figuring out what the web is supposed to be. I’m not sure we’ll ever get there in my lifetime, but it’s a pretty filthy place to be in right now.
Days later, I’m still working on my Angular “Hello World”, but only after I figure out why Bootstrap glyph icons aren’t rendering… as soon I figure out why I need to:
Ohh.. Bower not… Nevermind.
I’ll just come out and say it. I’m something of a fan of redneck engineering. Most of life’s hardest problems often have a genius and somewhat unorthodox solution. When you are like most people, you look on a shelf at your local store. If the shelf is empty, the problem remains until you can buy that something to plug that hole or cool your beer or just have a place to put your feet up. When you are a redneck engineer you see it as a opportunity to dig through your scrap pile and marry seemingly useless bits and pieces into a solution. It’s never pretty, it looks incredibly dangerous, and yet… it works.
Often times we see this same dilemma in software development. The proliferation of open APIs and frameworks has given rise to the similar mentality that a common person may express about redneck engineering. If there is a library to solve problem X you use it because… well… it’s there, it’s quick, and you don’t have to think about it. You are either the guy who wants a hot tub and buys one with LED chroma-therapy lights and a built-in iPhone dock, or you are the guy who builds what he needs; hot jetted water.
I’m not saying there is anything wrong with being the guy who just drops the cash for a hot tub. Speed is how companies stay in business and sometimes it’s just cheaper to buy from someone who only knows how to build that one thing quickly. Still, I think most engineers working in today’s environment would probably agree that we’ve all become consumers much more than problem solvers.
Sure we ask ourselves every day, “How can I work with this framework to solve my problem?”. But how often do you still ask yourself, “How can I solve this problem?” It’s becoming a question we ask ourselves less and less. I’d like to think that in a world of such rigid constraints and an industry shackled by buzzwords and new “solutions” and standards being created each day that there is still room for the redneck software engineer. Otherwise, we may wake up one day and realize just how lazy we all are, and how little we actually know about a whole lot because it’s all been abstracted from us.
Customer: “Oh hello there. I’d like to buy a cake.”
Salesman: “We have a wide selection behind our counter here…”
Customer: “…no. You see I need a cake that looks like that one. But I need it for about 1000 people.”
Salesman: “We can bake 100 custom cakes. That will be…” *cha ching* “$10,000″
Customer: “Whoa.. Sorry, I’m going to have to shop around…”
Salesman: “Wait wait. It’s half price to add a layer to one cake. So what about one cake to feed everyone for…” *cha ching* “$1,000″.
Customer: “I’ll take it but only if that cake is treated with the same care and quality.”
Salesman: “Baking cakes is what we do here. A few more layers is not a problem. We just delivered a cake with almost as many layers (5) just last week.”
Customer: “I’ll need it in 2 weeks.”
Salesman: “I’ll have it at your door with a bow on it.”
Baker: “…you’re fired.”
How do you balance a 762 layer cake? The answer is lots of wooden stakes and wires and shoving PVC spines down the center. It’s an amazing achievement to behold untouched. The fondant is a smooth and aesthetically pleasing curtain over the abomination just below the surface, and maybe 10% of it is actually edible cake.
Unless you are watching a reality show baking competition it’s not too often that someone will try to buy or sell such a ridiculous cake. There are laws of physics that most of us intuitively learn in our everyday lives. We know that it’s not easy to balance a car on a toothpick or erect a 762 layer cake without consequence. And yet, we see this in software every single day.
You work all the time with a client that doesn’t understand the “physics” of software development, a sales team that is trying to keep the cash flowing in, and a development team who can only keep adding layers to the same code because that it what was sold to the client.
You’ve done 3D graphics? Everything from this point on will have it. Multi-player? Everything from this point on will have it. Buddy systems, chat support, VOIP, server clusters, massive meetup scheduling database, and VM back ends? It’s all there, some parts left dormant for the what-if promises made behind closed doors.
Like some sort of code hoarders, we are afraid to start over. There’s not time after all, and clients are always looking for the best bargain. We know that we are just one handshake away from feature X being promised at no cost to the next customer because, “baking cakes is what we do.”
The reality of code re-use is something of a lie in many ways. It’s only true if you understand and accept it’s caveats and faults. It sells a dream to non-programmers that code is more like a bowl of dry cereal – easily separable – than the muddy mess that happens when you let it sit in milk for too long. Most programmers who aren’t too close to their “precious” code can see that code is almost always backstroking through a deep pool. It’s a product of all of those promises, and the need to fix bugs in age old features that were assumed to be rock solid. But the programmer knows, the programmer always knows. Eventually something will come loose, but we aren’t paid to fix old features. We are paid, to keep adding layers.
Rarely can we simply pull a piece of code and use it untouched, if only because it is tied to a dozen or so user interface hooks or event listeners and other loosely coupled but critical bits. Some features need to plug into other features, and even component-based architectures haven’t completely resolved the dependency issues.
I wish I had an answer to the 762 layer cake but I’m afraid this is just the world we live in. Products are too big to fail now and we are forced to add new layers onto an already rocky foundation.
When the icing becomes more important than the cake we lose sight of the physics involved in keeping it together.
Peter Molyneux… For many young hopefuls who are eager to earn their gray hairs before the age of 25, the name Peter Molyneux doesn’t ring much of a bell. Most game designers going through school today probably associate with Notch more than Molyneux. If you grew up in the Playstation generation, after the golden age of PC and the print publications of “Gaming Gods” in magazines, then you’ve probably not heard of this guy. Or maybe you have.
Molyneux has had a dark cloud over his head for quite possibly his entire gaming career; the kind of reputation I think would drive most people out of the industry. I mean, can you blame the guy? He started out as a baked bean salesman in some factory. The guy was quite literally a bean counter. It’s practically in his DNA to up-sell you something you may clearly regret hours later. (In classic Fable fashion, this post needed at least one good fart joke).
In all seriousness though, I’ve probably followed Molyneux through his entire career. I was personally more invested in his vision than the actual execution however. Even by his late days at Bullfrog, Molyneux had a special way of speaking to the press. The man knew how to make headlines as colorful as an 80’s themed nightclub. His quotes were always especially inspiring, though I read them with the kind of inspirational skepticism you get when your too-old-to-care grandad tells you how much “tail” he got because of his car. You know only a fraction of it is true, if any, but it leaves you thinking… “what if”
I feel like there is this loose thread that people must connect between the man and the kind of games he makes. It’s as if the fact that he makes god games requires him to be as perfect as a god himself. To be honest, gods aren’t exactly perfect either. Mistakes are made, promised are broken; they are practically meant to be. What is more important than the end product to me is the wishful dream that sparked it.
You see, we need a guy like Molyneux to promise that you can step on a flower and it will die or drop an acorn and it will grow into a massive tree. We need designers to have a childlike untetheredness about them, while understanding that not every idea will make the cut. But here’s the thing, not every idea is good for every game. And though we never saw dynamic flowering meadows in Fable, we have seen games focus almost exclusively on that kind of mechanic. We’ve seen many ideas that seemed almost birthed from this man’s mouth, even if it didn’t come from his studio. We need more of that.
The first word most of us learn is, “no”. From the moment of our birth we begin the life lessons of what to say, how to act, and which dreams are worth telling someone. We are taught what it means to be cool, and how to be attractive, and what are the best things to say for every occasion. Being a likable person often means sticking to the script. We’ve all had those moments in our lives when we wanted to speak and maybe didn’t. Eventually we all reach a point in our lives where the dreaming stops, and we’ve heard “no” so much that we find ourselves saying it quietly. We mutter in our minds, “people will think I’m crazy for saying this,” but Molyneux seems to be the only person who hasn’t changed. He may add caveats and speak in “maybes” but the dreams are still there for the world to see.
In an industry filled with people ready to pounce on your failures there is still a guy who isn’t jaded enough to be the same boisterous dreamer that made games like Populous, Black and White, Dungeon Keeper (the good one), and even the Fable series. Where technical behemoths like John Carmack were my motivator to seek out Engineering as a future, it was boisterous dreamers like Molyneux that inspired me to think of games as more than just guns and corridors.
Admittedly Molyneux’s public persona has developed into a caricature of it’s own, even spawning a fake Twitter account that tweets outlandish game ideas. But would we really want it any different? Do we want to live in a world where people just have “designs”, but rare dreams? Do we want to only play what is possible and never hear about the impossible? That sounds like a boring place to be, and yet here we are; slowing bashing each other down until everyone sits comfortably somewhere in the middle.
Molyneux has recently suffered some hard blows, yet again, to his credibility but I hope that he doesn’t stop. If he walks off into the sunset it will quite literally be the end of dreamers. All that is left out there seems to be quiet voices timidly hoping not to become him just for speaking up, or others who would rather bury their idea before they let someone else make it. If he leaves we will be left with people selling us the product, not the dream. Games may stop being about what is possible and simply be about what is here now. I’d encourage all designers to speak as loudly and shout about their crazy ideas from the tallest soapbox and become the next Molyneux. If you look at the roster of successful games under his belt you might agree with me in saying, there is nothing wrong with that.
Video games are big business. Ubisoft, EA, Activision; no one is exempt from using multiple studios, each employing hundreds of people, to get a single product to ship. Even indie studios have started to grow as that space settles into its’ own. Indies have elevated their game to the AA status and those bite-sized experiences from lone developers we once called indie don’t seem to have a name anymore; perhaps hobbyists? Everything is moving up and onward. And with all of this growth, where does that leave the process of finding those hundreds of people to ship the next game?
For the first time in my life I had an experience that I’m still not certain that I can put into words, but I’ll try. It was such an experience that I owe it to myself to try to explain. I had what felt like an uncanny interview, something that I feel could only be possible due to the success that video games have enjoyed.
I was contacted by a recruiter about a year ago, maybe more. I humored her because I did admire the studio she represented. We chatted and she seemed like a knowledgeable and friendly person. I turned down the offer at the time but the conversation was struck up again recently. This time I decided to at least take the first step and possibly answer my proverbial question about the relationship of this company and myself, “what if?” What if I passed the test. What if I got to meet the team. What if I got an offer. What if I accepted.
These were all questions that were always there, so I took the plunge and accepted the test. What transpired after that very human, very friendly interaction was… interesting.
I took the programming test. I finished it in what felt like a reasonable amount of time. Some questions took longer than others, but there was nothing terribly difficult. It was mostly questions that were masking a genuine engineering problem through humorous context. This is the point where I started to feel a little paranoid. Why would such a successful studio throw out a bunch of softball questions? And to add to that, why would they give me so much time to complete it?
When I finished the test, I reviewed it thoroughly. I ran inputs, good and bad, through every answer to validate their outputs. My answers were stable, consistent, and produced the demanded outputs. I sprinkled in comments to state my assumptions throughout my answers. As with all code, there is a large permutation of answers to any software problem, this is something that the test itself mentioned. Sometimes the answer relies on your objectives. Are we coding for readability, re-usability, performance, memory conservation, or is the real test just a veiled attempt to look at your choice of coding standards or see if you like tabs or space?! *cue dramatic chipmunk!*
I had many concerns at this point, but I was honestly having too much fun with the test to worry. My biggest concern was that the test not only stated there were multiple solutions, but it also stated that it would be processed by an automated system… Wait what? No human hands would touch this random Word document I was about to upload to a webpage that had little more than a “good luck, you’ll need it” message? I thought it was pretty funny at first… I mean, the answers were right, so I’d at least get a callback to discuss why I chose option 4 of 999, right?!. Sadly no. In less than 12 hours (many of those being sleeping hours) I was rewarded with a kindly worded generic email that confirmed my greatest fears.
Only after I got that email did I abashedly realize that they weren’t looking for a right answer, they were looking for THE right answer. The questions were a lot like a pixel hunting puzzle in a classic Adventure game; not nearly enough information, and mostly wild guessing based on what little clues were on the screen. I knew what the answer was, I just needed to figure out how to tell the game that I knew what the answer was. Alas, it was too late, and I had been judged and filed away by the system. My final interaction was a response that politely explained that I was but a drop of water in an ocean of applicants. That was the end it.
I wasn’t sure how to take all of this. I have never NOT gotten a callback and I can’t ever recall talking to a company I wanted to be at and not gotten an offer. It was a strange experience for me. It was rejection, sure; but it was the kind of rejection that I find harder to accept. An algorithm told me I wasn’t good enough, like something out of Minority Report or Gattaca. How am I supposed to feel about that?
Taking that test was an eye opening experience. It had honestly been a while, years in fact, since I decided to sit on the other side of the interview table. This may have been an isolated incident and in many ways I do hope so. In an age of Twitter, “Social” Media, You Tube, and Goggle Hangout, it’s easy to forget the value of just shaking someone’s hand. There is irony in trying to find intelligent and sociable engineers in a creative space by grinding them through a machine first. In the end though, my first and most important question was answered. I had little else to say about it after the following few tweets. As crushing as it may have felt, I’d like to think that it was in my best interest that it didn’t go any further. That’s what I’ll keep telling myself until I believe it.
I suppose the bright side to all of it is.. Now I know.
— Ben Quintero (@Ben_Quintero) January 7, 2015
Not to be such a Debbie Downer so soon after Christmas, but this is something that fell onto my lap. As with many things online, one hyperlink lead to another and I found myself reading about the poetry of Michelangelo. I can’t say that I studied nearly as much as I should have about the great Michelangelo but his poetry seemed very relevant to that of a game developer… I thought I might present some excerpts and ponder on his plight. Continue reading The Tortured Artist
Top 5’s seem to be a popular thing right now, as they often are at the end of a long and crazy year. This really got me thinking about the events that profoundly changed my perspective of an industry I’ve cherished for so long… Though some of these are arguably a multiple year affair, 2014 is the year that I felt affected by them the most. Continue reading Top 5 Hmm Moments of 2014
I’m not sure what to call it. Some might call it agism I suppose, but I’m still struggling to find a word that accurately describes what happens after you’ve gotten pretty good at doing your job, mostly because you’ve been around long enough to see how the Matrix works… Maybe if I talk it out I’ll get some suggestions or divine inspiration, but one thing is certain; career choices are never easy. They only seem to get harder as life gets more complicated.
Software is fundamentally broken; or at least the creation of it. Think for a moment about how video games get made. A bunch of eager over-achieving creatives sit in a room and throw pasta at the wall until something sticks. They go their respective ways to flesh out the ideas and come back together to fight over who’s baby doesn’t get thrown into the volcano. Finally the idea is further enhanced, concepts are drawn, and the game is pitched to the development team who will almost assuredly and universally hate it. The design is clearly overambitious and, while the concepts are alluring, everyone takes the large gaping holes in the design and interprets it as behaving like their favorite game of last year. It should be noted that everyone is already wrong…
In time, the team begrudgingly comes to an agreement and the real work begins. If the team can’t leverage the technology from their past game things get really interesting. It’s back to the boardroom where whiteboards are filled with diagrams and flowcharts. The question will always come up; do we have time to fix the systems we are keeping? The answer will inevitably be no, but the team hasn’t realized it yet. Some people are already jotting numbers and formulas while Billy in the back row keeps rolling his eyes because the project is 6 months from even considering if that feature will get prototyped. Feelings are hurt, dreams are crushed, and an overwhelming sense of dread haunts the team.
Everyone is starting to see the vision of the game design but even after a couple short months of grinding at the foundation for the game it’s obvious that the design was overreaching. Managers are lost because the schedule in no way reflects what is going on in the pit of Engineers and Artists. It’s The Walking Dead down there and his/her How-To book said nothing about this. Rapid Development, Scrums, Standing Meetings, Waterfall, none of the material alphabetically sorted on the bookshelf stand up to the reality that is software development; they act more like a SpongeBob band-aid stretched over a deep laceration. Designers are turning gray as their precious vision is being ripped apart and the development team is slowly leaching all creative decision-making away. Producers are leaning over the programmers, asking why the conversation system isn’t implemented yet. Engineers have been asking to hire someone to handle that task and more but it has been sitting on the Manager’s desk for 3 months now; before this project even started. The Executive team looks at the numbers and can’t seem to justify the cost of bring in more people. They remember being in the trenches; back in the day when it was just 2 people cranking out games. This team will just need to step up their efforts. And every two weeks, the Business Developers keep pestering the team for a demo to start shopping this game around or taking it on the road for shows. It’s an 18 development window but everyone wants it yesterday.
The project is coming to a close and everyone is looking slightly pale, a little bushy, and carry a hint of, “I feel like destroying something beautiful” in their eyes. Management suspects that station 6 in the render farm was taken out to pasture with bats in hand by the art team but they don’t want to “poke the bear” right now. The game is actually kind of fun, looks amazing, and there is real promise but a lot was left on the cutting room floor. Now, all the team can do is find some institutionalized beauty in what they have, nothing more nothing less; like a pet rock or a prized collection of various lengths of wire. As amazing as the game is, there is still much in the way of polish to be done and bugs to be squashed. As a kind gesture, and partially out of self-preservation, Management brings in a couple more Engineers to keep the tasks on this mythical schedule. Just move these tasks over to this new resource and done! Instant success; back on target. Yeah, those guys you asked for before this project started? Here they are, 4 months til ship date. Awesome… Just enough time to train them with the time the Engineering Lead doesn’t have because he has somehow been delegated the Project Manager’s job, while still expected to code.
The day finally arrives… The game has made it through the approval process and is on it’s way to print. Oh by the way, Management wants to talk to the Engineering Lead… It seems those extra Engineers put a strain on the budget and they are going to have to do some “restructuring” in the company. Don’t worry, the Engineering Lead is probably okay; the casualties will certainly be the new hires and maybe 3-4 others he/she has befriended to make up the loss. Management has asked the Engineering Lead to break the news to the team; they have another meeting to catch with Business Development about, “some things coming down the pipe”. A glimmer of false hope does little to ease the Engineering Lead as he receives a short stack of pink slips and is asked to fetch the Art Director for what can only be assumed as a similar conversation.
The SERIOUS Difference
“But..but.. I just want to make a mini-game about saving lives!”
If you’ve ever worked in the Military contracting business, creating Serious Games, you’ll know that the process is actually not too dissimilar; though the financial climate is controlled more by politics than consumer demand. The main differences are in the people you work for and the hurdles you have to jump over. The (wannabe) Designer is usually the point-of-contact in the Military branch; the guy who holds the money over you. They are kind of like the Publisher when you’re dealing with that Executive Producer that wants to make his game, not yours. Though he barely knows where to find the power button on his 15 year old business class laptop your point-of-contact will insist on being involved every step of the way; serving only to slow everything down. On top of that, you’ll be asked to replicate some preexisting training game that you’ve never seen before; killing all creative rights you thought you had. You’ll be asked to make it do what that other game does, but prettier and able to run on “current” PC architectures (his laptop). You’ll have to deal with security clearance or Government Furnished Information (GFI) which is a fancy way of saying, “Please fill out these forms, requesting to view a screenshot of some 20 year old flight simulator, and someone might get back to you in 6-8 weeks.” Repeat this process just about any time you want to make a decision that involves some form of creative choice; be sure to C.C. the point-of-contact. Also do your homework because they are not allowed to do software updates without admin approval so you might have to work with aging brittle versions of libraries and browsers locked in time. This is why it takes 7 Managers, 2 Engineers, and 1 Artist to screw in a light bulb, a bulb that had to be purchased by completing a purchase request form, then an expense sheet filled and approved, and a signed order from the Management teams boss saying it’s okay…
There is something fundamentally flawed about Big Software, and yet we can’t go back; the world expects bigger and better. Consumers and technology users want platforms to appear and make their experiences simpler and more unified. This is a rare sight for one person’s garage project. I’m not sure if the world of software is destined to be this broken or if there will ever be a light at the end of the tunnel. I’d imagine that, with business ruling over the science of software development, we are pretty much screwed.
If you’ve ever seen a duck swim you know exactly what it’s like to be in software development. On the surface, everything looks smooth and confidently organized. That duck appears to glide across the water, barely making a ripple. Just underneath the surface, his legs are paddling comically fast. Beneath the surface tension, they are stuck in a frantic motion that seem almost otherworldly to what the we see above…
“Me to, duck… Me to…”