Hi all,

Does any one have any suggestions for unit testing generated code.

My first idea is to just test the entity objects, I will be adding all my business logic to the derived entity classes. I will only access the entity objects in the UI. All calls to data layer will be within entity classes.

I think this is the "domain model" that I am using. _(off topic, but are there any good references to manager model, I see a lot of threads that mention manager model, but can't really find an example of what they are or how they should be implemented. Are manager classes static? do they hold instances of entities? any good examples?) _

What are others doing for unit testing?

Any comments are greatly appreciated.

Thanks,

Gabe Nodland

My approach really is that it isnt my job to unit test llblgen. I would consider that like unit testing the .NET Framework.

Controllers / Managers on the other hand, if they are hand rolled should be unit tested. However, if you use the template parser to generate managers / controllers, they should generally all be created the same way. So, IMO, if you have unit tested one very robust manager/controller then you have unit tested them all because they will all be created the same way each time.

If you want to code generate unit tests for your controllers using the template parser, you can do that too. Here is a link:

http://www.llblgen.com/tinyforum/Messages.aspx?ThreadID=1949

My general approach is to only unit test when I am solving a business problem.

For example, I just finished a mini framework that manages state. Here is my problem statement: "I need to know when a wizard has initialized, when its state changes, what the current state is, what the current, next, and previous steps are, and when the wizard is complete. I need this information when the client signals the wizard to navigate forward or backward as well as when the client signals that they are finished with the wizard"

I need to track state in a self contained way, so I wrote the following unit tests that test my business problem. Now that my unit tests are proven, I can move my unit test code into my web user control responsible for providing the UI for the wizard.

    [TestFixture()]
    public class StateTests
    {
        public StateTests()
        {
        }

        private WebWizard wizardHost;
        private Step step1;
        private Step step2;
        private Step step3;
        private Step step4;

        WizardState stateManager = null;

        
        [TestFixtureSetUp()]
        public void Setup()
        {
            wizardHost = ConfigurationHelpers.Instance().WizardById("1001");
            step1 = ConfigurationHelpers.WizardStepByStepId("1", wizardHost);
            step2 = ConfigurationHelpers.WizardStepByStepId("2", wizardHost);
            step3 = ConfigurationHelpers.WizardStepByStepId("3", wizardHost);
            step4 = ConfigurationHelpers.WizardStepByStepId("4", wizardHost);

        }

        [SetUp]
        public void TestSetUp()
        {
            stateManager = new WizardState();
            stateManager.ChangeState += new StateChangeHandler(this.StateChanged);          
        }

        [Test()]
        [Explicit()]
        public void CheckData()
        {
            Assert.IsNotNull(wizardHost, "Wizard host not initialized");
            Assert.IsNotNull(step1, "Step1 not initialized");
            Assert.IsNotNull(step2, "Step2 not initialized");
            Assert.IsNotNull(step3, "Step3 not initialized");
            Assert.IsNotNull(step4, "Step4 not initialized");

            Assert.IsTrue(step4.StepName == "Step 4", "Step4.StepName != 'Step 4'");
            Assert.IsTrue(step4.StepId == "4", "Step4.StepId != '4'");
            Assert.IsTrue(step4.PathToUserControl == "Step3.ascx",
                "Step4.PathToUserControl != 'Step3.ascx'");

        }

        [Test()]
        public void NavigateStep1()
        {       
            Console.WriteLine("<---------Start Test::NavigateStep1------------------>");
            stateManager.Navigate(wizardHost, step1);
            Assert.IsTrue(stateManager.CurrentStateName() == "FirstStep","Invalid State");
            Console.WriteLine("<-------------------------End Test:------------------>");
            Console.WriteLine();
        }

        [Test()]
        public void NavigateStep2()
        {
            Console.WriteLine("<----------Start Test::NavigateStep2------------------>");
            stateManager.Navigate(wizardHost, step2);
            Assert.IsTrue(stateManager.CurrentStateName() == "InteriorStep","Invalid State");
            Console.WriteLine("<--------------------------End Test:------------------>");
            Console.WriteLine();
        }

        [Test()]
        public void NavigateStep3()
        {
            Console.WriteLine("<----------Start Test::NavigateStep3------------------>");
            stateManager.Navigate(wizardHost, step3);
            Assert.IsTrue(stateManager.CurrentStateName() == "InteriorStep","Invalid State");
            Console.WriteLine("--------------------------<End Test:------------------>");
            Console.WriteLine();
        }

        [Test()]
        public void NavigateStep4()
        {
            Console.WriteLine("<----------Start Test::NavigateStep4------------------>");
            stateManager.Navigate(wizardHost, step4);
            Assert.IsTrue(stateManager.CurrentStateName() == "LastStep","Invalid State");
            Console.WriteLine("<--------------------------End Test:------------------>");
            Console.WriteLine();
        }

        [Test()]
        public void Finish()
        {
            Console.WriteLine("<-----------------Start Test::Finish------------------>");
            stateManager.Navigate(wizardHost,step4);
            Assert.IsTrue(stateManager.CurrentStateName() == "LastStep","Invalid State");
            stateManager.Finish(wizardHost, step4);
            Assert.IsTrue(stateManager.CurrentStateName() == "WizardComplete","Invalid State");
            Console.WriteLine("<--------------------------End Test:------------------>");
            Console.WriteLine();
        }

        [Test()]
        [Explicit()]
        public void VerifyAllStates()
        {
            Console.WriteLine("<--------Start Test::VerifyAllStates------------------>");
            stateManager.Navigate(wizardHost, step1);
            Assert.IsTrue(stateManager.CurrentStateName() == "FirstStep","Invalid State");

            stateManager.Navigate(wizardHost, step2);
            Assert.IsTrue(stateManager.CurrentStateName() == "InteriorStep","Invalid State");

            stateManager.Navigate(wizardHost, step3);
            Assert.IsTrue(stateManager.CurrentStateName() == "InteriorStep","Invalid State");

            stateManager.Navigate(wizardHost, step4);
            Assert.IsTrue(stateManager.CurrentStateName() == "LastStep","Invalid State");

            stateManager.Finish(wizardHost, step4);
            Assert.IsTrue(stateManager.CurrentStateName() == "WizardComplete","Invalid State");
            Console.WriteLine("<--------------------------End Test:------------------>");
            Console.WriteLine();
        }

        private void StateChanged(StateChangedEventArgs args)
        {
            Console.WriteLine("State Changed for Wizard:" + args.WizardHost.WizardName);
            Console.WriteLine("Current State:" + args.CurrentStateName);
            Console.WriteLine("Current Step:" + (args.CurrentStep == null ? "null" : args.CurrentStep.StepName));
            Console.WriteLine("Next Step:" +  (args.NextStep == null ? "null" : args.NextStep.StepName));
            Console.WriteLine("Previous Step:" + (args.PreviousStep == null ? "null" : args.PreviousStep.StepName));
            Console.WriteLine();
        }


    }
}

Devildog74 wrote:

My approach really is that it isnt my job to unit test llblgen. I would consider that like unit testing the .NET Framework.

exactly

So, what I would do (and what I do myself) is what DevilDog also says: test what you have to implement: write testcases which should succeed if you've done your job correctly. With tests you always have to assume things, like '.NET's methods are bugfree' or 'LLBLGen Pro's methods are bugfree'. If you run into failing tests because of bugs in llblgen pro, you have a repro case to send to me (your test ) and it will be fixed a.s.a.p., however as a lot of people are hammering the code daily, most bugs are weeded out already, so in this case you can safely assume that tests you write to test your code should be enough.

Devildog: thanks for the lengthy answers, as always

Otis wrote:

Devildog74 wrote:

My approach really is that it isnt my job to unit test llblgen. I would consider that like unit testing the .NET Framework.

exactly

I don't trust ANYTHING... Not LLBLGen, not the .NET Framework... Not IIS... Nothing.

When you deploy an application and the customer is standing there or 1000s of people wake up in the morning and logon your customer's web site... I don't think your client will want to hear something like "Sorry about that, but the problem is not in our code".

While testing LLBLGen or the .NET Framework is certainly not our responsibility, testing the final application IS. So make sure your test suite not only includes Unit Tests, but Integration Tests, System Tests and User Tests aswell.

Think of Unit Tests like an "onion". Microsoft have written Unit Tests which (hopefully) cover the lower layers. Frans has written tests which cover the middle layer above Microsoft. You have written unit tests which conver the outer layers. But if you are writing unit tests correctly. You tests won't actually call LLBLGen at all. Unit Tests by their very nature should only test 1 specific piece of functionality in isolation and in a known and repeatable context. I use NMock whenever possible to substitute external or 3rd party software and services so that my tests are more focused.

The downside to true Unit Testing is that you end up only testing your code and your code alone, not the fact that your code works with LLBLGen or that LLBLGen works with the database or that the database works on the particular target operating system and service pack combination and so on... This is why you need Integration Tests which test the fact that your code works with the external or 3rd party software or services that you call...

When all these tests pass, you still can't be sure your application is working. You need System Tests which test that all the parts of the systems are working as a whole. These are your end to end tests.

Finally User Tests are either manual tests or automated via a tool such as NUnitASP.

In reality, it is very difficult and time consuming to actually achieve this level of testing, but I for one always strive towards this achievement. I've had too many style experiences in the past where everything was tested but still managed to fail on the day...

Still, if your routine has to fetch X based on filter Y and it does that, your routine works. I don't think you should spend a lot of time writing tests which seem necessary, you should test your algorithm implementation, i.e.: do pre- and post conditions defined match? If so, your routine is correct. If not, fix it.

That's the whole purpose of unit testing: you define an interface, with specifications, pre and post conditions, this method does this based on this input etc. So you write unit tests to test that interface (interface being any API exposed) if these specifications match the actual implementation. If they do, the code works.

The advantage of this is that when you refactor code behind the scenes, your unit tests guard the API functionality exposed: if tests still succeed after the refactoring, the code is still correct. (correct as in: within the specification). If after refactoring the tests fail, you unintensionly altered the API's functionality, and thus introduced a bug, i.e.: the implementation doesn't match the implementation.

I really don't see the point of "But what if I pass in null into this method, it crashes!", if null is not within the spec of the routine. I mean: if you format your harddisk or remove c:\windows, your computer will not work as planned either.

Otis wrote:

That's the whole purpose of unit testing: you define an interface, with specifications, pre and post conditions, this method does this based on this input etc. So you write unit tests to test that interface (interface being any API exposed) if these specifications match the actual implementation. If they do, the code works.

Totally agreed

Otis wrote:

I really don't see the point of "But what if I pass in null into this method, it crashes!", if null is not within the spec of the routine. I mean: if you format your harddisk or remove c:\windows, your computer will not work as planned either.

Except for public interfaces like Web Services... In this case you need to test these kind of input ranges to make sure the correct errors are returned. We wont even start talking about security tests This is a real area where tests benefit from being generated...

However, you still need to provide at the very minimum a set of end to end tests which are outside the scope of unit tests.

Also tests should be written with the same care and design principles that are used for production code. When the number of tests starts to rise, they can become a burden in themselves as they too now need to be maintained.

Thanks for all of the feedback!!

I will only unit test my business layer to make sure it follows the correct business logic.

I will not test generated code, business layer will use generated code, but if i test all of the functionality that i will be using, then the generated code the UI will use should be tested as well.

I guess this isn't true unit testing because I'm also testing the sub components, by doing this. I think that this is an acceptable compromise, and it should be enough. It is a relatively simple project with under 30 entities.

Gabe

GabeNodland wrote:

_(off topic, but are there any good references to manager model, I see a lot of threads that mention manager model, but can't really find an example of what they are or how they should be implemented. Are manager classes static? do they hold instances of entities? any good examples?) _

What are others doing for unit testing?

Any comments are greatly appreciated.

Thanks,

Gabe Nodland

Gabe:

I struggled with the "Manager" concept as well. Previously, I was working with the Domain Model Pattern (Fowler) and it was a challenge to re-think and re-design my business layer with the Manager/Controller pattern. If you have the option, I will suggest that you read "Pattern Languages of Program Design Volume 3". It includes the "Manager Pattern" by Sommerlad which is the one suggested in the mentioned threads.

Regarding your quetions, Sommerlad suggests that indeed, your manager class should be static (singleton) and that it acts as a Factory [GoF]. Once it has all the "management" functions like retrieving, searching, saving, etc. you should be able to write your client code that will get a subject reference from the manager and then use it directly. When you need to save it for example, you send it back to the Manager class which in turn will access the persistence layer and voila....

Hope it helps

Alfredo

I will suggest that you read "Pattern Languages of Program Design Volume 3". It includes the "Manager Pattern" by Sommerlad which is the one suggested in the mentioned threads.

You got an Amazon link for that? I couldn't find it.

JimFoye wrote:

I will suggest that you read "Pattern Languages of Program Design Volume 3". It includes the "Manager Pattern" by Sommerlad which is the one suggested in the mentioned threads.

You got an Amazon link for that? I couldn't find it.

http://www.amazon.com/exec/obidos/tg/detail/-/0201310112/102-0749210-0715341?v=glance

Be aware that the mentioned pattern is explained in only 10 pages out of 574 pages! The book is interesting and has a lot of useful patterns. Anyway, if you want to have in your library a good book on patterns, besides the original GoF, buy it (Don't buy the other two volumes....not worth it)

Alfredo

Alfredo,

Thanks for the info, I will definately look into it.

Gabe

My approach really is that it isnt my job to unit test llblgen. I would consider that like unit testing the .NET Framework.

One of the rules of extreme programming is to NEVER unit test third party code. Its assumed this has been tested properly. Of course assume = "assumptions can make an ass of u and me"

:-)