Evaluated Node.js, moving 100% to JavaScript – am backing off for a while

It has been fun. Following a month ramping-up on JavaScript, AngularJS, Node.js and Git conclusions are:

  • AngularJS looks great
  • Hold off on Node
  • Keep JavaScript close, but not a best buddy
  • JavaScript transcompilers are promising
  • Git is really easy to install/ use; embrace over SVN for disconnected commits/ simplicity

Surprisingly easy to get up and running with Node

It is surprisingly easy to get up and running with node, see the below screenshot for what took a little over an hour after deciding to install Ubuntu and use Eclipse as an IDE for node development:

NodeJsInUbuntu

Git is trivial to install on Windows/ Linux. It took minutes to create a Git repository in Dropbox.

What is Node.js

Node has a lot of what you are already used to. For almost everything we can do in .Net there is a corresponding Node package. Socket IO, http communication, async, MySQL provider and so on. These are called modules in Node; installed trivially using the Node Package Manager (npm) via a terminal prompt. Many popular JavaScript libraries are also available as Node Modules: Underscore, Mocha and CoffeeScript being particularly popular.

Node literally has one thread and an event loop which cycles through pending events. This means any part of your codebase can block other requests. A major difference is the style of coding in Node. ASP.Net etc are implicitly multi-threaded, pre-empting threads to ensure each server request gets its share of CPU time. Node code must be crafted in a manner so-as nothing blocks a thread.

While learning Node, much tutorial code required piping streams and nesting JavaScript callbacks. Apparently most Node code is like this. Such code soon becomes difficult to follow and comprehend. With familiarity this will improve, but well crafted OO code will always be easier to understand.

Why Node?

JavaScript is everywhere; many developers know JavaScript so why not use it server-side too?

We write validation logic in C# on the server and JavaScript in the browser. Using Node we can reuse the code.

Performance is a huge seller. Apparently Node.js is bad-ass rock star tech that can blow ASP.Net etc out of the water performance-wise. Let me debunk this: performance is an area where I really kick-ass; have tuned many systems (small and large) generally seeing ~100->400 times improvement under load with surprisingly minimal tweaks. Most were systems that had already been tuned.

Performance is a function of your developers and/ or having someone on staff who understands performance holistically. Do not select a technology because its theoretical maximum load is 20% higher. At a fortune 10 I tuned two maxed out Datacenter installs (~thirty machines each) to all the machines using almost zero CPU. Undoubtedly several people spent weeks or months analyzing which machines to buy for ‘peak performance’. Architecture, sensible implementation and tuning are where real performance gains are found.

Performance of Node can be killed by any one bad section of code. Tools to tune Node are very immature. With .Net we use WinDbg/ sos.dll to analyze production systems – it is very difficult to analyze Node in production.

JavaScript is the Future?

As many tech friends said: on my third read of JavaScript the Good Parts it really made sense. Quality coding can be achieved in JavaScript but it is far from a perfect language.

Google’s DartMicrosoft’s TypeScript and CoffeeScript all bring real OO concepts including classes and even static typing to JavaScript. Currently they transcompile to JavaScript. Within five years expect a language in this category to have gained traction + adopted into all browsers. Current versions of all browsers self-update.. Once most of the world is running self-updating browsers it becomes possible for new standards to roll out quickly. Powers that be in the Internet world will settle on a standard; that is why Microsoft threw TypeScript into the ring.

JavaScript was cobbled together quickly in 1994 as a scripting language for Netscape. It is very weak and will eventually be ousted. Transcompiling is an intermediary step.

Final Conclusions/ Predictions

Node.js is hot today, but not a good fit for the kind of applications I personally work on: large systems with a traditional RDMS back-end and a lot of inter-system messaging to slow legacy systems.

Node.js is helping building a great base of JavaScript’s frameworks for the enterprise. Be-most are from small untrusted sources. It is only a matter of time until a serious security breach occurs via someone slipping malicious code into an open source JavaScript library. Once a high profile incident occurs the JavaScript community will figure out how to mitigate such attacks.

A Node.js rival with multithreading will appear, or Node itself will be extended. Ruby gained multithreading; after years of its user base stating single threaded web servers are fine.

JavaScript will morph into a real language within five years.

Posted in Development (General), Technology | 3 Comments

Technology Fragmentation is a Project Killer

Project failure is very real

Standish Group research shows ‘Of 3,555 projects from 2003 to 2012 that had labor costs of at least $10 million, only 6.4% were successful‘. 41% were total failures and the rest vastly over-budget, did not meet expectations etc.

StandishGroupPieGraph

This post documents twenty-five years in the Software Development industry. Project failure has become commonplace today; I hope to convince you that Technology Fragmentation is a key cause.

1989 -> 1999: One Project Failure – mostly PowerBuilder/ Oracle

Starting as a summer engineer with Siemens in 1989 to my final PowerBuilder project in 1999 only one failed. Seriously. (Failure was with a Big 5 IT consultancy, staffed with mostly non-technical resources did not want to be on that particular project).

All projects from ’94->’99 were PowerBuilder/ Oracle – many people became experts in them, there were no additional frameworks. Virtually every project used similar architectural techniques.

During this period few individuals were essential to the project success. Certainly some people were standouts ability-wise and undoubted shaved months off the schedule, but project would have continued without them as many others had similar knowledge (even if not quite as sharp/dedicated/productive). At least twice I saw several contractors who between them almost certainly rescued the project, but for the most part they did this by training existing staff, not super-human 80+ hour week coding.

1999->2001: Fought off Failures – Java + major Frameworks:

Technically I had only one outright failure with Java; the final one when I opted not to burn-out and resigned from an insanely well paying contract. The burned through ~$10m in six months.

On the first two Java projects I averaged ~80 hours per week, exceeded 100 hours per week over one seven week period and even had some 24 hour days! I was the main technical resource, unfortunately hired late in their SDLCs. Developers lack of knowledge with the entire technology stack was a core issue. This required me to learn quickly and work incredible hours to stabilize/ educate others.

On these successful projects several developers followed close behind – it was a team effort to succeed against the unknown technologies, but only between ~20->30% of the team in each case. A significant problem was the number of technologies to learn.

With PowerBuilder/ Oracle even the weakest team members were somewhat competent in one of the two technologies, and they generally soon improved (lots of other people to learn from). With Java and its increasing number of frameworks/ app-servers/ etc it was not uncommon for a project to only have one expert per framework/ tool. This meant several people became critical to project success. If they had bluffed their way through the interview their area was a ticking time-bomb. With a plethora of frameworks it is very hard to interview screen technically; unless an expert is used for interviews bluffers can be very hard to weed out.

Some Stability with .Net: 2002-> 2007

My first .Net project failed outright, but was my only outright failure with .Net until ~2012. No one understood the technology on our first project; about a year in I was making great breakthroughs, solving most of the long standing issues. Unfortunately due to missing long-term deadlines our strong manager was ousted was replaced with a ‘yes-man’; we disagreed and he soon ousted me. That project failed within a year and I received several supportive emails from client staff. Approaching $10m dollars of tax-payer money wasted.

Once up to speed with .Net virtually every project called me a ‘superstar’, ‘insanely productive’ etc and I did not see a single failure.  Unfortunately there was plenty of evening and weekend work to put out fires/ meet deadlines.

Why were these projects all successful? We knew the entire technology stack. In particular I knew .Net and Oracle/ SQL Server very well; this enabled extinguishing fires quickly and permitted time to educate their developers. Many people were almost OTT thankful for taking the time to assist them.

Stress with .Net: 2007 -> Present

Still no failed .Net projects but most were stressful; typically from overwhelming amounts of evening/ weekend work.

Since ~2007 the .Net market really exploded. This caused two major issues:

  • Quantity of frameworks sky-rocketed
  • Quality of developers plummeted

By 2007 most projects I arrived at had a fair number of frameworks/ tooling: Microsoft Patterns and Practices, ASP.Net Membership Provider, Log4Net and NUnit were particularly popular. As time progressed ORMs came into the mix; I have worked with at least five ORMs so far. Templating/ code generation frameworks and IDE helpers like Re-Sharper were also popular.

Of those I listed above only log4net saves time in the short term. Most frameworks/tools only reduce time/money/complexity the second or third time used. This is a well documented fact. A rule of mine has to been to never use more than two new technologies on a single project. Learning curves and risks are just too great.

~2007 -> 2012 was my ‘Project-Rescue Phase’. The majority required enormous effort to understand the technology stack and generally battles with managers/ architects to stabilize. Typically removing unnecessary technologies and performing major refactorings to simplify code/ make it testable via Continuous Integration.

On my final ‘project-rescue’ contract: within three weeks we met a deadline despite the team having zero to show for the previous three months. Their Solution Architect left them a high level design document littered with fashionable buzzwords; nothing useful had been produced. Two previous architects made no progress; including the original solution architect. I was their forth architect in about three months. One other developer and I began from scratch coding everything to meet phase 1 in three weeks; the other five people did little but heckle. The consulting company I assisted was still being difficult so I left them to it; they lost that client. Being fair they put something into production, but it took three times than longer and was a terrible product. They burned through three more architects during that time.

Performance Tuning: 2010

Around 2010 I advertised for and landed about ten performance tuning/ defect fixing contracts. Massive stress but great intellectual challenges fixing issues customers could not squash. I had a 100% success rate with these, taking a maximum of four days :)

An opportunity to observe many systems over a short period. Unnecessary Complexity was the only constant. Frameworks and trendy development techniques being primary offenders. One customer had a mix of Reflection and .Net Remoting that hindered debugging most of their code base for years. I removed that problem in ~twenty minutes which stunned their coders – they were amazing with wide open eyes and gaping mouths cartoon style :) [Topic-change: this is where experience counts and that minuscule piece of work was in the 100x Developer zone. Such times are rare, do not let anyone tell you that they are consistently a 10x developer.]

100% Failed .Net Projects: 2012 -> Present

2012 forward I decided to stop the ‘workaholic rescue’ thing and try to talk sense into managers/ architects/ stakeholders. This was ineffective. Two projects failed outright and the third is a Death-March (Stakeholders believe all is fine as they ‘tick-off progress boxes’ but reality is a long way from their current perception).

Two of these projects suffered from ‘resume-driven architectures’; the other classic wishful-thinking timescale-wise, but they hired about two hundred Indian contractors to compensate which always works (not). I was tempted to give each member of the leadership team three copies of the Mythical Man-Month; three copies so they could read it three times faster.

Quantity Up/ Quality Down for additional Frameworks and Tooling:

From 1989 -> ~1998 the number of technologies was modest.

About 1999 the Internet-Effect really began. Ideas took center-stage in the early days. Certainly in the Java world many were rushing to use latest techniques they had just read about online: EJBs, J2EE, Distributed Processing, Design Patterns, UML etc… Most teams were crippled by senior staff spending their time in these trendy areas rather than focusing on business needs. This coincides with the beginning of my ‘project rescues’ and being told way too often that I was “hyper-productive compared to the rest of the team” (that has never been an aim).

By ~2007 open source frameworks and tools were center-stage in most projects. Since then we have seen exponential growth and large companies (Sun, Microsoft, even Google) proliferated their dev spaces with low quality framework/tool after low quality framework/tool presumably hoping some would stick. Apple is one of the few to exercise real restraint. The Patterns and Practices group within Microsoft is particularly shameful example most of us are familiar with.

Over Twenty Languages/Frameworks/ Tools is now common?

It is tough to single out one project, but below I quickly listed forty-two basic technologies/ core concepts a sub-project at one company used:

“VS 2012, .Net 4+, html5, css3, JavaScript, SPA/ MVC concepts, Backbone, Marionette, Underscore, QUnit, Sinon, Blanket, RequireJS, JSLint, JQuery, Bootstrap, SQL Server, SSIS, ASP.Net Web API, OData, WCF, Pure REST, JSON, EF5, EDMX, Moq, Unity DI, Linq, Agile/Scrum, Rallydev, TFS Source Control, TFS Build, MSBuild scripts, TFS Work Item Management, TFS Server 2010/ 2012, ADSI, SSO, IIS, FxCop, StyleCop, Design Patterns (classic and web)”

Beyond the buzzword virtually every area of the application was custom implemented/ modified framework rather than taking standard approaches. It was certainly job-security and helped lock out anyone new to team.

That is just one of three projects since 2012 where I “complained three times, was not listened to so walked away as calmly as possible rather than fight like I used to”.

Resume-driven Architectures / Ego-driven Development

Why do most contemporary projects so many frameworks and tools? I see three key drivers that happen during the Solution Architecture phase:

  • Strong External Influence
  • Resume Driven Architectures
  • Ego Driven Development

Strong External Influence is a key driver. SOA appearing on magazine covers, Microsoft MVPs all singing the same tune etc. Let’s look at how these work.. What appears on magazine covers is driven by the major advertisers. SOA sold more servers, hence it was pushed on us. Many friends are MVPs so I must take care in explaining them: many try hard to stay independent but most are influenced but their MVP sponsor to publish material around certain topic. Over the years I have seen many lose their MVP status and generally it was after outbursts against Microsoft, or they stopped producing material that Microsoft Marketing wishes to see. Apologies to MVPs, but you all know this is the truth.

Resume Driven Architectures is the Solution Architect desiring certain buzzwords on his/her resume to boost their own career, and/or being insecure about finding their next role without the latest buzzwords. On one project I had to leave early the Solution Architect mandated an ESB for a company with under 2,000 employees! Insanity. Of course they failed outright, but not before going three times over schedule and having almost 200% turnover in their contract positions during a six month period! It is not fair to single out any one individual; we have all seen a platoon/ company sized number of such people over our careers.

Ego Driven Development: Bad managers tend to compare ‘number of direct reports’ when trying to impress one-another. Bad architects do the same thing, just with latest buzzwords.

What needs to happen?

With fewer technologies one or two key players can learn them all and stabilize a project. This is not feasible with over forty technologies.

Already in the JavaScript community we are seeing a backlash against needing large numbers of frameworks, but this is causing further fragmentation. A core concept of AngularJS (and others) is to not rely on a plethora of other frameworks. Of course early stages of learning ‘stand-alone’ frameworks like AngularJS are tough. Frameworks generally do not save time until the second or third project we use them. We could learn AngularJS but what if our next project does not use it? Time wasted, likely no efficiency gained.

No-brainier: Reduce additional frameworks

Doh, virtually all of us realize this! The problem is how? Personally I am learning AngularJS and node.js with the intent of waiting until a suitable position appears. This approach vastly limits the projects we can work on.

Personal experience shows that once a project with a vast number of frameworks is given the green light it takes a Herculean Effort to change even tweak its Solution Architecture. As independent contractors we can avoid clearly-crazy projects including those loaded with buzzwordsUnfortunately that limits the projects we can work on. It may keep us sane though; I believe:

“Beyond two new technologies project success is inversely proportional to their combined complexity”

Avoiding clearly-crazy projects means we avoid more failed projects so could be wise choice in the longer term.

Ensure the Solution Architect implements

I used to believe this was a perfect solution, if only Architects had to build what they preached then they would constrain Architectures to the bounds of reality. Unfortunately this appears not to be universally the case; perhaps it reigns them in a small amount? I have witnessed more than one case of someone attempt to implement their own bizarre architecture.

In the worst case having to implement themselves, must reign in an Architect’s craziest ideas. Personally I shy away from pure architecture, especially on multiple teams. Before now when under pressure I have resorted to semi-bluffing and palming quickly cobbled up ideas off onto others safe in the knowledge I did not have to implement. I soon became cognizant of what I was doing and brought it to a swift halt. Do you think others will be so honest? Let’s try to ensure Architects are on implementation teams.

Ignore vendor influence

Guy Kawasaki has a lot to answer for. IT vendors have long tried to sell us what we do not need, but Guy Kawasaki introduced many techniques we see today.

Attended a free conference or user group with quality speakers lately? Receive free trade magazines? Java ones are funded by larger players in that space, Microsoft ones often by (indirectly) by themselves. There is great value in these resources, but please keep your eyes open for manipulation.

SOA is a primary one I refer too. Magazine after magazine had SOA emblazoned their front covers, many conference talks were around SOA.. it became the buzzword du jour for years. SOA used conservatively is fantastic, but from about 2002->2010 I saw project after project with SOA sprinkled around as if salt from a large salt-shaker. Re-factoring to remove/ short-circuit SOA was a key technique of mine – ‘strangely’ removing much of it led to much maintainable and performant code. Why was SOA so heavily hyped by our industry? Distributing code leads to more servers; which increases hardware sales and more importantly server license sales. Server license sales are where the big-players make their real money. Costs of even a smaller companies SQL Server or Oracle licensing soon ring up to millions-of-dollars. High costs accompany CRM, ERP, TFS, Sharepoint and most other common sever based software.

Younger Architects are particularity susceptible to vendor influence. Younger people are more easily influenced, tempted by implicit promises etc and soon saddle their projects with many trendy buzzwords. How could a project possibly fail if every buzzword is hot on Reddit and our vendor representatives cannot stop talking about them?

Embedding consultants/ evangelists into large companies is very common. Received free conference tickets from a vendors? Free training and elite certifications? Sorry to lift the curtain, but clearly these are tricks which exist to coerce you into using a particular technologies.. and buy more severs! The consultants and evangelist are of course generally not evil, but they are trained to believe in what they are selling.

Become a Solution Architect

Being the Architect certainly works. Every project I had significant control over was a tremendous success. Unfortunately most projects select their Architects based on popularity with management and other non-technical attributes.

Frequently senior leadership believes Solution Architects should manage multiple projects and not be hands-on with code. This is a mistake. Personally I have turned this role down several times as it leads to poor Architectures – as stated above I have caught myself bluffing before in this role. Solution Architects should not span multiple projects. Staying hand-off for long leads to believing marketing of technologies. Marketing is often far from reality.

Most companies in-house Architects tend not to be the strongest technically. Senior leadership looks for softer skills – can they convince/ bully others, have a large physical presence etc. Notice how often we see tall white male Solution Architects? ~80%+ of the time yes? When this is not the case the Architect is almost always technically sound – because they attained the position on technical merit. All too often leadership looks for someone ‘with weight to throw around’ – at the Fortune 10 discussed above virtually all ‘thought leaders’ in our department had a large physical presence, and shouting at subordinates was second nature. It was amusing when I was asked to review the work of the two worst because so many people were complaining about them.

Conclusions

Hopefully it is clear that we must reduce the number of technologies in our projects. For the foreseeable future it is unlikely we can return to the stability/ predictability of pre-Internet/ tech-boom days.

This post is far longer than my notes/ original outline predicted. In future I will partition posts into more digestible subtopics with more focus on how we can improve. There is good information here, so under time-pressure I decided to publish as-is. Being between contracts AngularJS and node.js are calling my name. These two appear the most likely to emerge as victors from current JavaScript framework fragmentation.

Posted in Development (General), Management, Productivity, Technology | Tagged , , , , | 1 Comment

10x Developers and 10x Projects

In addition to Scott’s and many, many other 10x Developer posts here is an opinion from someone labeled ‘hyper-productive’ on most projects during the past twenty years.

Key Points

  • 10x is a zone we enter at times, it cannot be sustained
  • 10x has historically (in Mythical Man Month etc) meant the difference between worst and best developers
  • Being 10x better than average is a rare occurrence with short duration
  • 10x zone is only achieved after months or even years perfecting required skills/ preparing conditions for the highly productive period to commence
  • Many important tasks take the same/ very similar amount of time regardless of the individual – e.g. attending the daily stand up, liaising with another department, manual testing etc

Common Characteristics of being ‘in the 10x zone’

  • Task that does not lend itself to parallelism – i.e. efficiency is gained from reducing communication, discussing unimportant implementation details etc
  • Full stack developer avoids need to interface with knowledge silos – this is a very common with hyper-productive developers but often leads to resentment from the team unless great care/preparation is taken to handle political backlash
  • Using a library/ technique highly suited to task at hand – a great example is deserializing/ serializing xml by hand vs. employing a tool like xsd.exe or JAXB. Many thousands of lines vs. a trivial library call can lead to man-months of saved effort
  • Already familiar with business/ technical problem – second time around is always faster. By the third time most problems are solved many times faster
  • Bust through politics – managed to obtain access to all necessary sub-systems, and are shielded from political backlash of stepping on toes so can sustain the access

10x Projects are a far Bigger Deal

Over the years I have seen around fifty projects. Excluding the really crazy ones 5->10x is a rough measure for the difference in productivity we see at the project level. Probably more if one considers operational issues; too many systems are moved to production before being fully stable. Many projects fail, so technically there is an infinite difference in productivity. 5-10x is a rough approximation between ones that make it to production.

Common Characteristics of very inefficient Projects

  • Too many people – and/or too many of the wrong people
  • Cumbersome, un-enjoyable process
  • Developers treated as commodities – no praise for quality work
  • Better developers left the project early
  • Rude, non-technical leadership

Anyone with a few software development projects under their belt knows the main issues that lead to poor projects. Unfortunately after all my years in software development I am now of the opinion that many are inefficient intentionally by design. Yes, the leadership team actually desires this! A sizable proportion of senior management prefer to have very large teams. The root cause appears to be Empire Building; a larger team brings the manager/director more power. Many resist promoting others that could potentially challenge them, and several times I have witnessed life being made difficult for quality individuals with the sole intention of forcing them off the project. As time progresses leadership weakens and it can decimate large companies, especially during ‘tough economic times’. This is why many very large companies look outside the firm when time comes for a new CEO.

Conclusions

Obviously there is a massive productivity difference between the best and worst developers. That does not mean someone highly productive on Project A will immediately be highly productive on Project B.

A badly run project can cripple even the best individual’s ability to do great work. This is of far greater importance than bickering about if one developer is actually 10x or not. One great developer is just one great developer with a limited skill-set. The team wins the war, great developers are often decisive in key battles but they cannot win wars alone.

Developers have specialties. Personally I still struggle until up to speed with a new technology – it takes time and hard work to be ‘10x’ again with new technologies. Beware anyone who tells you otherwise, especially the ones who claim to be ‘10x in everything’, everyday on every project’.. That is not reality. Every project failure I have witnessed had at least one guy who would publicly state he was expert in a very long list of technologies.

Posted in Development (General), Management, Productivity, Technology | 2 Comments

Productivity distribution is not Gaussian, closer to Pareto

Productivity Distribution within an IT Team

For decades we have been taught that productivity is a Gaussian Distribution (bell-curve) with most team members contributing about average performance:

Gaussian Distribution

A recent article in the journal of Personnel Psychology caught my eye. It argues performance generally follows a Pareto distribution:

Pareto distribution

This much more closely correlates with what we often see in IT. Most teams have a minority of amazingly productive members while the majority unfortunately commits little. This is crux of the issue I am trying to analyze and address. Often most of the ineffective people do have the ability but lack the motivation. Many classic texts repeatedly state that in our field human motivation is the single, strongest contributor to productivity.

Educating and motivating moves many members from the left of this curve towards the right.  I have done this many times; transforming what management saw as weak people into confident and able producers. This curve is always going to exist, but we can much to reduce its height; by bringing more individuals to the right.

I have lingered on the left of the curve for too long several times; it is an unpleasant place to be and personally it is intolerable for an extended period. Everyone joins towards the left and progresses to the right as they learn the stack and are better able to collaborate with the rest of team. Not everyone has the technical skills, people skills, experience or raw dedication to move to the right; this is where leaders should step-up, helping team members improve and not hindering ones they disfavor (color of skin, educational background, physical appearance etc – we see it all).

Many will disagree

Non-technical managers and Scrum Masters often resent the hyper-productive and coined a derogatory term for them: hero-developers. Googling this term finds much derision accusing them of being bottlenecks and a worst practice in Agile.

Story time.. (yes, I have lots of stories): On a recent Fortune 10 engagement I learned our team of ~thirty previously had one member who could dive anywhere into our stack; virtually everyone said he held the project together. Mere mention of his name to their ex-Manager made him angry; “I am, sick of hearing his name” etc. Many told me that the manager (now senior Director) drove the hyper-productive developer out of the company. On my arrival the project was very much in disarray with shockingly serious production issues; issues the entire team could not solve, and had not solved for over a year.

But they are wrong

Once ramped-up at the Fortune 10 I methodically worked through and solved all the systems serious production issues. It took many months of evening and weekend work before I was able to effectively and efficiently attack larger problems. This was an absolute extreme in my own career and after eighteen months on that project I was absolutely exhausted. Evidently a ‘hero’ was needed, but trust me that no one wants to be one for long – it is too draining. We should avoid the need for heroes by enabling others to be more productive.

Technical types will guess a key to that staggering success was becoming familiar with their full stack, acquiring access rights to sub-systems and establishing enough trust to ‘step on toes’ without serious political backlash.

Why was that particular project in such trouble?

That particular project is at the extreme of what I have witnessed, but serves as a useful example.

Essentially non-technical manager after non-technical manager forced great engineer after great engineer out of the door. Despite ‘saving the day’ again and again even my own manager fought tooth and nail against most initiatives I proposed, despite backing from virtually the entire team. This division was an extreme and has now been dissolved at that Fortune 10; their own politics almost certainly led to their demise.

A salient point is that everywhere I observed failure this category of culture existed – many potentially highly productive members kept their heads down and pretty much just stayed around for the paycheck hoping their next manager will be better. Those are almost direct quotes from several (very honest) people!

“Not a Team Player” – common accusation from weak leadership

Many reading may be thinking that I am not a Team Player.. There is a great image doing the rounds on social networks:

There-is-an-I-in-TEAM

Which can be countered with the following memes:

every-group-project-the-hangover

Unfortunately it does not get much better after graduation!

stack-of-ladders
This would work better if it could convey the manager as intimidating the ladders and actively preventing them from performing their usual function.

Who is Right? What to do?

Almost everyone laughs at the above images; because there is uncomfortable truth in each extreme.

Common sense suggests establishing measurable metrics you believe are critical to project success. Decide who the producers are based on those metrics, not who takes you to lunch most often, is the most agreeable, appears confident etc.

One simple metric I use is to scroll through source code commits and look for the names appear most often; one or two names generally make up the bulk of commits. Think twice about sharing your metrics with the team; political players will game them. Of course watching for gaming could be a metric you keep to yourself :) Over the years I have seen several play ‘a numbers game’ for measurable tasks like fixing defects; they select only trivial defects and even then many of their ‘fixes’ are re-opened. This can be tricky to detect; one particular ‘gift-of-the-gab’, ‘charming to most women’ etc individual would take the QA team (both female) for lunch/ dinner and ensure that they never re-opened his defects but raised new ones.

Long term team/ project success

Too often on software development projects we see a very uneven distribution of work. When this is addressed we see very successful projects. I have done this many times, but unfortunately many times told “when you left the project/ team fell apart”. In this series of blog posts I hope to make headway into creating teams that sustain themselves even after the ‘hero-worker’ has departed. A gut feeling is that those teams had no-one else with courage/persistence to challenge poor decisions. Most in our workforce do posses the necessary technical skills (or at least the ability to learn); problems lie elsewhere. Motivation is certainly a common one.

Observations during Success, Total Failure and Death Marches

With little deviation this is what I have observed in various categories of software development projects:

Successful Projects:

One or two very productive people drive the team. Viewing the commit logs show a vast percentage of commits are from one or two people. Generally they work many evenings and weekends. Hero-driven development is often used a derogatory term. In reality on a very successful project virtually all ‘heroes’ try to spread their knowledge to take pressure off themselves.

Why does the ‘hero’ thing work? For years in interviews I summarized myself as someone who ‘pulls the strings together’. Being able to traverse the full-stack and nail down problems fast is key. Without at least one full-stack engineer teams fall victim to finger pointing between knowledge silos and technical debt rises fast. Siloed development leads to many issues no single sub-team/ person will take responsibility for. Such problems tend to linger.

Highly successful projects occur when there are several highly effective members who can bring others up to speed. At the Fortune Ten discussed above my second project had another very talented .Net architect; our productivity was amazing and we brought at least one other architect up to speed (it was a cloud based framework so everyone had 20+ years of experience, but all of us lacked true full-stack knowledge before the collaboration). Good collaboration and decent motivation meant we did not work evenings or weekends, yet the project ran smoothly.

Unsuccessful Projects:

Since about 2012 I left projects very early if they required intense ‘hero-driven development’; all three projects I left failed/ are failing. Prior to 2012 I stayed at many disasters, performed the ‘hero-developer thing’ and rescued several projects. This led to varying degrees of burnout.

Death March Projects:

This is the category of project that generally hits production but is way over budget and schedule; people are exhausted and operational costs sky high.

Very large companies tend to operate this way; they virtually always get systems into production because they have the financial resources/ big-name pull to keep hiring through the code-and-fix phase; even when several times over the original budget. When close to failure often they hire an army of contract staff; generally a small subset of those are ‘genius-level’ who stabilize the system (and enjoy charging many hours of overtime!). This latter solution is something I saw often early in my career working as a junior developer for very large companies.

Over the years I mostly kept a low profile on such very large projects (politics were overwhelming). A key observation is that potential heroes were trampled on and generally left the project/company or retreated to the safety of a development silo. This was most evident on Government Projects.

But my projects have never failed!

Do not hire anyone who states this. In email correspondence with leaders of projects that failed many state that it was a success. Bizarre, some seem to believe their own lies.

Everyone with experience has seen several failures because most IT projects fail.

How common is failure? A recent study by Computerworld reveals that 94% of IT projects in the past decade with budgets of greater than $10 million, in government and out, launched with major problems or simply failed.

My experience with large companies and government organizations is that ‘tall blades of grass are lopped off’ before they ‘take over the garden’. Recently on 300+ person project cost exceeded a million dollars per screen delivered! I expect that project will make it to production, but at enormous cost with major issues. They are a gigantic, cash-rich company but the managers/ Scrum Masters were not tolerating stand-out individuals (other than themselves of course). Apparently their vision is a Software Factory – masses of cheap workers churning our code at a consistently high quality. Wishful thinking; this is nowhere close to reality in software development.

Solutions?

Bringing others up to speed is a key point here.

Hopefully we can accept that distribution of effectiveness is closer to a Pareto Distribution than Gaussian. Once accepted we can target reducing the height of the distribution and head towards success with reduced chance of project failure/ serious overruns.

Posted in Development (General), Management, Productivity, Technology | Tagged , , | Leave a comment

Should I buy a 3D Printer?

Thinking of buying a 3D Printer? Let’s give that thought a quick reality-check… I was ready for a somewhat hobbyist experience but it took way longer than expected to have the printer assembled and printing acceptably.

Mine is a $549 Printrbot Kit. Interestingly perusing forums of the current generation $2,200 MakerBot Replicator 2 showed their owners experience similar issues to what I’ve gone through.

Assembly of the kit took five to six hours. Until it printed reliably using PLA was probably a further forty hours but I did fabricate a new platform and adjustable bed. If time permits I’ll follow this post up with common issues + solutions so maybe you’ll be up and running in under twenty hours. Either way, expect to become a 3D Printer technician and be sure to have an abundance of patience handy.

The image below sums up the experience nicely. See the tools? Notice the spool holder made from DIY parts? Surprised to see DIY screws, power drill etc nearby? Well… forums and blog posts are littered with people who purchased a 3D printer and abandoned it before making good prints. Surf some forums and you’ll notice virtually everyone showing off their prints has their printer in some kind of hobbyist workshop; typically they’ll have loads of tools around and are veterans of past fabrication/ advanced DIY projects.

Complete Printer (1) (Medium)

But apparently a child can build one? Yes, some manufacturers are suggesting you buy one for your eight year old and he’ll have it assembled and printing in no time… Not a chance! I did see one blog post where a young boy had assembled the kit but “so far they were having trouble printing”. Younger than twelve I’d say to totally forget it and buy him/her a Lego Mindstorms or similar. High school age is probably more appropriate; even then a tinkerer Father on hand is almost a necessity.

Should I spend $550 or $2,500?

Now my $549 Printrbot is dialed-in and has had a few modifications its prints are excellent. Given the explosion in 3D Printer popularity, anything built in 2013 is going to look like a Dinosaur in a few years. Unless you need to print very large objects now I recommend buying a lower priced printer today and upgrading in a couple of years when mass production techniques should bring costs way down and take quality/ ease of use way up. I purchased Make: Ultimate Guide to 3D Printing for $6 as they reviewed about fifteen current generation printers.

Should I buy a Kit or Fully Assembled?

Assembly is typically only ~$100 extra, but the experience [frustration!] of building your first printer is invaluable. I was too young to build my first home computer in 1981 so was bought an assembled one. I did not make that mistake this time; the kit came with poor/incorrect instructions, but it was the correct route to take for both experience and bragging rights in twenty years time.

What to expect from a Kit?

Until the last couple of years most 3D Printers were built from open source online designs. Hobbyists sourced components themselves and spent unbelievable amounts of time tweaking. My PrintrBot is a well priced kit building on these many years of open sourced achievements. I doubt there is much profit margin for PrintrBot. Also, you will have noticed a lot of plywood on my printer; that’s laser cut plywood. It has issues but is an inexpensive way of producing the parts – once a laser printer is purchased the cost of chassis etc is next to nothing for the smaller scale manufacturers compared to the material/hours required to 3D Print parts as was common in the early days.

A kit gets to you to where these serious hobbyists were far quicker for a reasonable price.

Take a look at these pictures for an impression of what to expect:

Printrbot Kit (1) (Medium)

Printrbot Kit (2) (Medium)

Their website says it takes two hours to build. Perhaps once I’d built a few that would be true. In reality it takes about five hours. Documentation is poor and was incorrect in several places as incremental design changes have occurred since the videos/ online help were created.

How long until Printing is possible after Assembly?

Likely some owners obtain a decent print on their first attempt; most do not – actually possibly most never do before putting it in a closet or on eBay! Expect several hours to several before anything reasonable is printed.

ABS filament emits fumes which made my eyes sting, but is much easier for a beginner to use than PLA filament. Since my workshop is small and unventilated I have moved exclusively to PLA, ensure your printer is in a well ventilated area and start with ABS.

Hopefully time permits me to write a separate post on getting started, but main tips are:

  • Level the bed
  • Ensure z-home is set correctly (distance from extruded nozzle to the bed)
  • Ensure the printer extrudes at the correct rate (pretty easy for ABS)
  • Figure out the slicing and printing software
  • Use calipers to roughly calibrate movement along the x, y and z-axis (fine tune when printing ok)

A common beginner issue is the clog up the hobbed bolt with filament (tension springs set incorrectly will do this).  I’ve had mine out for cleaning at least twenty times, but it’s been fine for a good while since I figured out the correct spring tension for PLA (springs compressed to ~13.5mm). The following image shows a clogged hobbed bolt being cleaned with a needle (I lost one needle so now store it using a strong magnet to fix it to a drywall screw):

Hobbed bolt cleaning (19) (Medium)

What kind of quality can I expect?

Quality is really good in my opinion. I’ve printed several printer upgrades and the precision is incredible; hex bolts drop right in where they should etc.

It will take a while and likely much frustration to get the printer dialed in. After about sixty hours I seem to have the basics down and when an issue occurs now know what to tweak. Take a look at the next photos to see my progression with PLA:

Example Printrbot Prints (1) (Medium)

Example Printrbot Prints (2) (Medium)

Example Printrbot Prints (3) (Medium)

The final print shown is a case for a Raspberry Pi. It fits perfectly, and this was printed before I added most printer upgrades!

Tools required?

Hmm… this is a tricky one. As a long time DIY type I have access to a vast array of tools. The Drill Press and cross vice shown below has been particularly useful but these are not beginner tools.

At a minimum you need:

  • Precision Calipers (only about $20 on Amazon)
  • Quality screwdriver set (one with lots of quality bits is fine)
  • Jewelers precision screwdrivers
  • Small wire cutters and pliers
  • Tweezers (to tease strands of stray extruded filament away from the nozzle and lift prints)
  • Very sharp craft knife

These are almost essential:

  • Quality precision pliers, angled pliers and wire cutters (Xuron or similar)
  • Quality oil/ grease
  • Circlip pliers
  • Telescopic Magnetic Tool to hold awkward nuts in place during assembly

Semi essential tools (1) (Medium)

Ensuring bolts thread perpendicularly:

Semi essential tools (2) (Medium)

Final Words

Have a spare fifty+ hours, $550 and buckets of patience? You should buy one now!

Remember this is not Software Engineering; interacting with the real world is a whole different ball game. Fellow classmates and I discovered this during our postgrad Robotics Degree . Software is predictable and repeatable. The real world often not so much. In many ways 3D printing is similar to robotics – some software is involved but there is a lot of trial and error + tinkering.

Posted in DIY, Other, Technology | Tagged , , , | 3 Comments

Compacting Virtual Machines (VirtualBox and VMWare)

Google has never linked me directly to this information, just theories. One day time permitted me to run careful tests so I am sure these techniques are correct/ efficient:

Simple tricks to reduce size VM’s Disk Needs:
These will wipe GBs from your vmdk/ vdi.

  • Disable Windows hibernation (hiberfil.sys is the size of installed memory, you don’t need it)
  • Disable the memory paging file (paging file in a VM makes little sense to me)



Compact a VMDK (VMWare including VMWare Player):

  • Launch the VM
  • Inside the VM defragment its disk (defraggler works great, Windows degfrag is ok)
  • Inside the VM run “sdelete.exe -z” from DOS (as admin). This zeros out the free space and is an essential step
  • Shut down the VM
  • From VMPlayer: Edit Machine Settings -> Hard Disk -> Utilities -> Defragment (optional step, sometimes helps – official documentation is poor)
  • From VMPlayer: Edit Machine Settings -> Hard Disk -> Utilities -> Compact

At the final step you should see a huge reduction VMDK size.

Below is a screenshot showing the features in VMPlayer. Remember this is next to useless unless you run Mark Russinovich’s “sdelete.exe -z” to mark free space with zeros. Compacting VMs has been this way for years, it’s April 2013 now and surely soon ‘detect and zero free space’ functionality will be built into their compact options.

Compact_VMWAre_Player_VMDK

The image above shows a VM that reached 20Gb once, before being compacted back down to 10.7GB. These are typical results. Once compressed my two work VMs zipped down to ~4GB each; fine for archiving working databases, dev environments etc. One customer’s backup procedures left me concerned so weekly the VMs were AES encrypted and copied to a USB key chain flash drive.

Compact a VDI (VirtualBox):

Until very recently I have used VirtualBox since about 2008. Here are the steps to compact it.

  • Launch the VM; inside the VM defragment its disk (defraggler works great, Windows degfrag is ok)
  • Inside the VM run “sdelete.exe -z“. This zeros out the free space and is an essential step
  • Shut down the VM
  • From DOS (as admin):
    • cd <location of your VDI>
    • “C:\Program Files\Oracle\VirtualBox\VBoxManage” modifyhd –compact <your disk’s name>.vdi

Hope this helps folks. Any issues/errors please post in the comments and I’ll update the post.

Posted in Technology | Leave a comment

Filter log4net to a single thread using Notepad++

Over the years I’ve Googled and Googled for a solution to this. Heck I even tried Bing! Finally time + an urgent need permitted figuring out a simple solution. To filter down to one thread using Notepad++:

  • Copy the angle brackets and thread number to clipboard. E.g. “[12]”
  • Menu -> TextFX Viz -> Hide Lines without (Clipboard) text
  • Press Ctrl-A (Select all text)
  • Menu -> TextFX Viz -> Delete Invisible Selection
  • Press Ctrl-A (Select all text)
  • Menu -> TextFX Edit -> Delete Blank Lines
  • That’s it! You are now viewing one logging from only one Thread

This works very quickly even with 70,000+ line 10MByte log files. IMO it avoids the need for xml log4net logging and Chainsaw (or similar).Simpler is always better.

Demo: Two threads counting to 100:

Copy the angle brackets and thread number to clipboard. E.g. “[12]”

Menu -> TextFX Viz -> Hide Lines without (Clipboard) text

Press Ctrl-A (Select all text)

Menu -> TextFX Viz -> Delete Invisible Selection

Press Ctrl-A (Select all text)

Menu -> TextFX Edit -> Delete Blank Lines

That’s it! You are now viewing one logging from only one Thread

Posted in Development (General) | Tagged , , , , , | 6 Comments