We are currently looking at hiring a new senior developer. We've interviewed alot of applicants and have narrowed it down to 2 candidates (both of which have now attended 2 interviews). Typically our process follows somethng along the following lines.

The first interview is a sit down and chat, we ask a couple of technical questions, at the end of the interview we give them a set of multiple choice questions to answer, the idea being trying to gauge where their strengths and weaknesses are rather than simply look at the total number of correct answers.

The 2nd interview we set them 3 tasks, the first is in VS.NET where basically we get them to refactor a simple winforms application to better utilise object orientated techniques, most applicants take between 5 and 20 mins to complete this. The 2nd task invloves them talking to us as though we were a client, when they feel they have enough information from us they are asked to draw out a basic database schema that might model the clients needs. For the third task we ask them to go online for 15 mins and find out as much about a particular technolgy as they can, we then get them to tell us what they have learnt.

I've been asked by my superior to set a challenging codeing question to see if we can seperate the 2 candidates we are currently considering. The brief I've been given is that it should take them something in the order of 1hr to 1.5 hrs to complete and it should involve ASP.NET.

I've been considering a number of options, one of them is to get them to write a web page where a user can play naughts and crosses. The page would need to enforce a couple of simple rules, such as once a square has either a 0 or X in it the user could not change it and the page should also be able to recognise when the game was won.

Is this a fair task? What other questions or techniques do people prefer to use?

I've found this site to be an excellent source of information in the recruiting/interviewing process.

In answer to your question, I would ask what it is you're trying to discern about the candidate; which skills are you attemping to assess?

Is it important the candidate is familiar with ASP.NET and is that what you're trying to assess? Are you trying to determine architectural abilities, design abilities, performance, code maturity, object-orientation, reusability, cleanliness, knowledge of patterns, etc, etc. Or are you asking the candidates to do their best at solving a problem and then make a subjective comparison between the two? If the latter is the case, then just about anything reasonably complex could work, within the realm of the skills the candidate is expected to have. If the former is the case, that there is something in particular you're trying to assess, then structure the problem accordingly. Meaning, if you're interested in performance, give him an existing set of code and ask him to identify the bottlenecks and optimize it accordingly; if it's architecture ask him to rough out a framework for a given scenario and explain his technique.

Jeff...

Thanks for the link, will take a look.

I totally agree with your remarks. I guess we've been lucky in the past as particular candidates stood out a mile from the rest in regards to a match for the positions we were looking for making the selection process alot easier. This time round however the 2 candidates we are considering have slightly different skill sets with one candidate being more senior than the other.

The more senior candidate probably overshoots what we initially set out to get, however there is plenty of room for us to expand their role if they are capable of taking it on. The less senior candidate probably comes in a little under what we'd hoped for, however we can see that personality wise they are a good fit and are prepared to go through a learning curve with them. The issue really comes down to finding out..

1. That our appraisal of their skills is accurate.
2. That any weaknesses we believe they might have aren't gaping holes in their knowledge.

I don't want to set a question that simply tests that they can interact with a database or know SQL. We make extensive use of LLBLGEN so traditional ADO.NET skills aren't all that relevant. Which is why I'd thought about naughts and crosses as we get a look at how they go about implementing some simple business rules and how they hook that up to a front end. Even if they don't finish the task we can still have a look at how they approach a problem.

Once again thanks for the reply and the link.

(Warning: unhelpful bomb throwing post below...)

I'll take this opportunity to express my long-standing opposition to testing job candidates. Having been on both ends of it, I believe that it has no more value than lie detector tests.

Go with your gut. You'd mentioned that, in the past, people had stood out and you weren't concerned. Only hire those that you look over and feel don't need testing. If you have a question and need testing, you shouldn't be hiring that person. If you have no choice but to hire, you've compromised anyway. By that point, the testing only confirms that you compromised and helps you cover your butt.

But there's a downside to your company in testing. I stopped even applying for jobs where testing is performed. Is it beneath me? Perhaps. More importantly, it means that the company is willing to accept mediocrity, as long as it's certified mediocrity - the company is compromised. I don't want to work for mediocre companies, I want to work with great people who know they are great. Ask all the challenging questions you want; I'll answer them. But if you can't form sentences and thoughts well enough to be able to challenge me with questions, I don't want to work for you.

There are not that many great programmers. Testing is a way of covering your butt because you aren't getting the resumes of the truely great. But you likely aren't getting those resumes because you're doing testing.

I once headed up a job where it seemed we were just burning client hours with contractor wages. Every hour of every day that didn't have a developer at a PC was money lost. Sometimes you have to hire because not having somebody doing something, no matter how worthless, is worse than the damage they might do. I understand that sometimes you have to make a sacrifice.

The alternative is to wait for your reputation to get around, that you're a place where great developers want to work. You want for those people to arrive, and you don't hire until then, until you know it without testing.

So, either you're comprimised and hoping for the best, or compromised and testing to cover yourself, or you're free to wait for the best and know how to spot them.

Tell me this. Would you test Otis? Would you just know? Are you skilled enough to be confident of your questions and just make the decision about hiring Otis?

So, if you're just filling cubicles, by all means, test away. Just don't be surprised if you still end up with crappy developers.

</rant>

swallace wrote:

(Warning: unhelpful bomb throwing post below...)

I almost wrote something like this myself last night, but was too tired. But you did a better job!

I like to ask applicants for code samples. About 70% of the time they are able to provide some, and that frequently tells me what I want to know.

But there's no substitute for asking the right kinds of questions. And I don't mean trick interview questions. Just plain ones.

I think Scott has a point. Here in the netherlands, it's by law guaranteed that there's a 2-month 'trial' period. So you hire a person and you can fire that person within 2 months without problems whatsoever. The person hired can also quit directly without problems. After that 2 month trial period, firing a person is much more difficult and leaving directly can be a problem too (contract clauses, a period you have to work after the date you stopped, like a month or so).

I think that's an excellent system, because you can only see if a person is really fitting the job when the person actually does the job he/she needs to do, for some period.

Testing people, like psychological tests or IQ tests are meaningless. Even filling in long lists of solutions to trick questions to see if you can think 'outside' the box. That doesn't proof anything.

It's always a gamble though. A person can pretend as if the person can do everything very well, but in reality would fail miserably.

What's often overlooked is that most jobs don't require Einstein himself to get the job done, however the employer wants Einsteins brain^2 with 4 hands and 150 years of experience in a gazillion languages. As Scott said, there aren't many great developers, yet there are a lot of good developers which can perfectly do the job at hand.

What the employer should look for (IMHO) are: - does the person have any education, so the theory is there? - motivation. Not motivation to work long hours for no money, but the motivation to finish the job because the job is so cool and the person simply wants to FINISH things which are started. - Ability to work alone and stay motivated. 'Team work' is important, but most of the work is done alone, and the person has to be able to self-motivation, so the project runs itself (more or less )

People who are great project starters but lousy project finishers are not that useful in most jobs: when the deadline approaches, the finishers will make it, the starters will give up. That's not to say you only should hire finishers, but if a person is solely a starter, forget it.

Might sound harsh, but let me explain: some people think I'm a great developer. I'm not. Great developers are people like Dijkstra. I'm good at what I do, I can say that without shame now , but I think what makes me be on the level I'm at now compared to some other people is that I hate it when I can't finish what I started, it's a motivator for me.

So that gives results, because things get done. If I would have given up mid-project, nothing would have been finished and no result would have been there. Mike Abrash, ex-id software and now D3D hotshot @ MS , once said: the best feature you should give to your software is 'ship it!'. I.o.w.: finish what you start.

swallace wrote:

(Warning: unhelpful bomb throwing post below...)

I'll take this opportunity to express my long-standing opposition to testing job candidates. Having been on both ends of it, I believe that it has no more value than lie detector tests.

Go with your gut. You'd mentioned that, in the past, people had stood out and you weren't concerned. Only hire those that you look over and feel don't need testing. If you have a question and need testing, you shouldn't be hiring that person. If you have no choice but to hire, you've compromised anyway. By that point, the testing only confirms that you compromised and helps you cover your butt.

But there's a downside to your company in testing. I stopped even applying for jobs where testing is performed. Is it beneath me? Perhaps. More importantly, it means that the company is willing to accept mediocrity, as long as it's certified mediocrity - the company is compromised. I don't want to work for mediocre companies, I want to work with great people who know they are great. Ask all the challenging questions you want; I'll answer them. But if you can't form sentences and thoughts well enough to be able to challenge me with questions, I don't want to work for you.

There are not that many great programmers. Testing is a way of covering your butt because you aren't getting the resumes of the truely great. But you likely aren't getting those resumes because you're doing testing.

I once headed up a job where it seemed we were just burning client hours with contractor wages. Every hour of every day that didn't have a developer at a PC was money lost. Sometimes you have to hire because not having somebody doing something, no matter how worthless, is worse than the damage they might do. I understand that sometimes you have to make a sacrifice.

The alternative is to wait for your reputation to get around, that you're a place where great developers want to work. You want for those people to arrive, and you don't hire until then, until you know it without testing.

So, either you're comprimised and hoping for the best, or compromised and testing to cover yourself, or you're free to wait for the best and know how to spot them.

Tell me this. Would you test Otis? Would you just know? Are you skilled enough to be confident of your questions and just make the decision about hiring Otis?

So, if you're just filling cubicles, by all means, test away. Just don't be surprised if you still end up with crappy developers.

</rant>

I'd have to agree in general. What you want are good developers and I think those people that are good developers know a good developer when they meet one; go with your gut, as you say.

However, a good developer doesn't necessarily have the skills you need. I think of it like cake and frosting. Everyone wants the frosting, but a cake isn't a cake without the cake. Use behavior questions to understand whether the person you're dealing with is quality, but at some level you need to know whether the person in front of you can do the job - has the frosting you like.

Ideally, people would hire on quality alone. Let's assume for the moment that I'm a good developer. I have very minimal skills in ASP.NET. I know that, because I'm a good developer, ASP.NET skills are just another set of skills to acquire. But most projects can't wait for that. Given two developers of equal talent, they'd be foolish to hire the one without the necessary skills for the job. However, taking two developers, one quality and one crap, but the crappy one has the skills, your best long term decision is to pick the one with quality.

Picking a solid developer is the baseline, but there's more that needs to be discerned. I think asking the developer to solve problems they would be expected to solve as part of their job is good practice, so long as you already know you're dealing with a developer who understands fundamentals, is smart, is a team player, who "gets it", etc.

Jeff...

Jeff...

Picking a solid developer is the baseline, but there's more that needs to be discerned.

I agree with that. However, many of the qualities that make a developer great are intangible, and therefor not testable. They come from the person's sense of professionalism, work ethic, desire to learn, and general character.

Difficult to gauge. I stand against testing on principal, and know many developers who will not interview at companies that test. Those companies that test are thought, by good developers, to be filled with those who don't consider themselves good enough to stand firm against it, usually those without experience, quality references, or the least amount of social graces (hmmm, perhaps that's me after all...)

My point is that testing can lead to an unnecessary downward spiral of quality within the organization. Unnecessary because the things you really want to know can't be tested.

I'll point out, for clarity, that I live in a relatively smaller city, something I refer to as a big small town, and reputation here is everything. I know which developers are good, and what companies are blowing smoke, by reputation. I'm certain that's a factor.

I'm with Otis on my own personal concerns about finishing projects. I've said many times that good ideas are a dime a dozen, but acting on a single idea is worth millions of dimes. A friend once said that the world rewards action, and nothing else. And yet, ...

Do others face that problem?

swallace wrote:

Difficult to gauge. I stand against testing on principal, and know many developers who will not interview at companies that test. Those companies that test are thought, by good developers, to be filled with those who don't consider themselves good enough to stand firm against it, usually those without experience, quality references, or the least amount of social graces (hmmm, perhaps that's me after all...)

My point is that testing can lead to an unnecessary downward spiral of quality within the organization. Unnecessary because the things you really want to know can't be tested.

Tough, because there are many quality shops out there that use testing as a filter. Microsoft, for example. Say what you will about them, it's hard to argue that MS isn't filled with an insane amount of great developers. Sure, there are some poor ones in there, but when your developer count is in the 5 digits you've got to have some.

My point is that you shouldn't rely solely on testing, but that testing is what's going to get you that final bit of information to distinguish between two good developers. But I agree: relying solely on testing probably reveals more about the quality of the employer than the quality of the candidate.

Jeff...

swallace wrote:

I'm with Otis on my own personal concerns about finishing projects. I've said many times that good ideas are a dime a dozen, but acting on a single idea is worth millions of dimes. A friend once said that the world rewards action, and nothing else. And yet, ...

Do others face that problem?

Funny, I've been thinking about this myself lately... http://www.gilbertnet.net/jeff/default,date,2006-01-10.aspx

Jeff...

<Edit>Hey Frans, your forum doesn't like commas.

I have to agree with swallace,

I've been a software developer for 15 years now - 10 of those have been with VB and the last year has been spent working with VS2005 (VB.NET). I've spent much of this time continually challenging my skills so that my work stands out from the croud.

I've now reached the point where I feel that tests are a waste of my time and I typically take code samples and screen shots to interviews. Where there is a test involved I feel compelled to walk away.

It is of course equally frustrating when you take up a position only to find that incompetence, mediocrity and corner cutting is rife.

Jax.

jeffreygg wrote:

<Edit>Hey Frans, your forum doesn't like commas.

comma's are illegal characters in a url

Otis wrote:

jeffreygg wrote:

<Edit>Hey Frans, your forum doesn't like commas.

comma's are illegal characters in a url

Hmmm... http://www.w3.org/Addressing/rfc1738.txt

Jeff...

I guess I couldn't remember it because after the word 'MEMO' my brain went blank

A very good discussion and I very much agree with the majority of the points made. I do like the idea of getting candidates to bring their code in and have them guide us through it.

We also have a 3 month trial period over here. While that does give you a get out clause, the damage the wrong person can cause in a short period of time is somewhat scary. I've hired the wrong person before and it costs money to put things right. I guess thats the risk you take when hiring and all the interview process is intended to do is minimise this risk.

I also believe that psychological or IQ tests are totally meaningless. I also take little notice of a candidates certifications. Its also totally true that a good portion of a developers skills are none technical and impossible to test. So I can acknowledge the pont,

'why test when you can't actually measure a good portion of the parameters that go into making a good developer?'

If interviewers simply hand out exams and then hire on the basis of candidates scores, then yep their just covering their backsides for when the crap hits the fan. 'Hey how could this have happened, we hired developers who scored above 75? Can't be my fault, I hired the best there was.'

But I can't imagine people just hiring on that basis. For me asking people a technical question or two, either verbal (what's an interface?) or written is just a sanity check to make sure the person is who they say they are. If your C.V says you've had x number of years experience developing .NET applications and utilising OO design, you should be able to answer a question like that.

So onto the 'challenging' codeing task.

I really can see people's objections to this. Since a good developer is more than just someone that knows how to code, what exactly do you get out of giving someone a codeing task where they have limited time, can't ask a collegue a question or two and aren't really in an enviroment where they feel comfortable yet? Normally I'd say, not a great deal. However we have 2 very different candidates in mind with 2 very different salary expectations, neither candidates expecations are unreasonable for who we believe them to be. Both seem to be likeable people with a good working ethic and both demonstrate a history of commitment to seeing projects through. The question for us is, do we go with a candidate that is less senior than we'd hoped for but thats within our original budget or a candidate that is more senior than we'd anticipated that is someway outside the original budget? Either way we make a sacrifice and take a risk, which is fine, but it would be nice to gauge how much of a sacrifce/risk either way. Which is were my superior is coming from when he asked me to come up with a challenging codeing task. We are as convinced as we are ever going to be from interviews that personality/work ethic wise that both candidates are strong.

Once again thanks for everyone's input.

Let's top off the discussion with this:

http://www.msnbc.msn.com/id/10296177/site/newsweek/

Google: Ten Golden Rules Getting the most out of knowledge workers will be the key to business success for the next quarter century. Here's how we do it at Google.