Today I was showing off the product to a customer who is also technical, and he wanted to see the generated code. He complained about boxing/unboxing when he saw this property:

public virtual System.Int32 FileID
{
    get { return (System.Int32)GetValue((int)SWS_DataFileFieldIndex.FileID, true); }
    set { SetValue((int)SWS_DataFileFieldIndex.FileID, value); }
}

I didn't have a good answer to this, because, well, because it does appear to be to be boxing/unboxing the int. Generally speaking, I don't worry too much about this anymore (does seem that when you're first learning .NET boxing is presented as an unspeakable horror to be avoided at all costs). Any comment?

[P.S.]

I just realized that the emitted code is inconsistent, it has a space between "get" and "{" but a tab between "set" and "{" - which is why it looks funny above, though everything lines up properly in Visual Studio (mine, at least). That should just be a space. More unspeakable horror

As far as I know, value types are implicitly boxed when they need to be converted to reference types. In your exemple, boxing should occur when the GetValue and SetValue methods are called because both take objects as parameter.

Here's what I think :

I agree that you should avoid causing boxing and unboxing to occur if you can... but although boxing can play a role, it's not the key element of performance. I do think that the overall quality of an application and the way its database is designed are much more important.

About your customer then... I don't know what experience he actually has building software but it is clear that if you were looking for absolute speed of execution, you would not use a managed language like .Net.

You could answer that you made the choice of building an application with a rich user interface and maintainable code instead of an extremely fast one. That you are trading nanoseconds for developer productivity and code reliability.

The boxing indeed takes place, but as Stefcl said, it's not a big deal. The thing is that the container of the value is a 'field' object. This is a general class, so the value could be anything, thus Currentvalue is 'object' typed.

I could have used EntityField<T>, and made CurrentValue typed as 'T'. The disadvantage is that in a lot of cases you would want to use the field in a general way where you don't know 'T'. This is in general solved by an interface, so instead of referring to the field as EntityField<T> where you don't know the T, you could use IEntityField.

However, that gives a problem: CurrentValue then again has to become object typed, not T.

Generics are great, the fun part begins when you want to write code which has to work with any type but you don't know the type up front. There, .NET generics fail, as they require you to specify the type at compile time.

It's the same issue as with EntityCollection<T>. Sometimes you want to consume the collection but you don't know the type T. You then can use IEntityCollection2 to consume and use the collection without knowing or specifying T. However with fields this isn't always possible as illustrated above.

Boxing and unboxing is only really noticable if you store thousands of ints in an arraylist and you look over them a lot and THAT is the core of your application. However profiling showed that to fetch a set of entities etc., the boxing is really not even on the radar.

Things which really take performance are event handlers. 1 single event raise is slower than storing 10,000 ints in an ArrayList, boxing them all.

Thanks for the detailed discussion, Frans, that's what I needed.

Hey,

I'm the 'customer' that Jim was talking about. Amoung other things, I've been building a high-performance ETL tool that is much better than DTS so I do routinely deal with millions of records.

For a typical application that pulls a couple of records, makes a modification or two, and saves them, you would never notice the performance hit. The other overhead will dwarf the box/unbox cost. But if you need to process a lot of rows, it will be an issue.

What I was specifically hoping for is an approach used by a package called Neo, which I looked at but never fully adopted. They had the same generated code entities implementation but they made each field into an instance variable on the derived class (they were using 1.1 so no partial classes). The values are copied in and out of arrays during persistence. I do not know the design trade off but it felt pretty clean and powerful.

Otis -- your response is wrong on performance costs. I didn't know that for sure so I built a test. The C# compiler appear to be good about minimizing the number of boxing operations but there is still x25 hit for boxing over direct int array access. An event handler that does nothing is about the same cost as accessing a value in an array list – I don’t know where your 10K cost comes from. The performance of GenPro is about 50% slower than the object[] array test – which is perfectly reasonable. The bulk of the performance cost is tied to the boxing operations. The real value of GenPro comes from minimizing database access (or in our case avoid Microsoft’s binary persistence, which has sucky performance).

The implication of this analysis is that LINQ entities should x10 faster than GenPro if you are dominated by access cost because Microsoft is going to go through the trouble of avoiding the boxing overhead. Based on a cursory review, they plan to use an attribute driven design and language hooks to handle the persistance, similar to how XML serialization works.

The surprise for me was that the list<int> test was x7 slower than an int array. The last test case is based on a Neo-like implementation. It avoids the boxing penalty. It is also x7 slow than direct int[] arrays – the C# optimizer is not as smart as I gave it credit for. This implies that GenPro could pick up about x5 by a redesign that minimize the use of object[] arrays.

The stuff below is the output of my attached test program. I subtracted out the loop overhead. Enough fun for today… back to work for me.

Josh

Performance Test -- Boxing vs Event Handler.

TEST SECONDS RATIO loop overhead 0.45313 --- int[] 0.21875 1.0 object[] 5.56250 25.4 ArrayList 6.64063 30.4 list<int> 1.51563 6.9 list<object> 5.90625 27.0 Event Handler 6.62500 30.3 Test Entity (GenPro) 8.10938 37.1 Test Entity (Int Index) 8.09375 37.0 Test Entity (unboxed) 8.21875 37.6 Test Entity (unboxed - generics) 8.17188 37.4 Test Entity (unboxed, no virtual) 8.40625 38.4 Test Entity (Neo-derived) 1.51563 6.9

Press any key to exit.

With the optimizer, it looks like it inlines the method calls. The x3 cost on list<int> is still a mystery.

Josh

Performance Test -- Boxing vs Event Handler.

TEST SECONDS RATIO loop overhead 0.78125 1.0 int[] 0.51563 1.0 object[] 5.35938 10.4 ArrayList 6.42188 12.5 list<int> 1.54688 3.0 list<object> 5.67188 11.0 Event Handler 4.62500 9.0 Test Entity (GenPro) 6.14063 11.9 Test Entity (Int Index) 5.87500 11.4 Test Entity (unboxed) 5.87500 11.4 Test Entity (unboxed - generics) 5.96875 11.6 Test Entity (unboxed, no virtual) 5.90625 11.5 Test Entity (Neo-derived) 0.57813 1.1 Press any key to exit.

Hi Josh,

Thanks for taking time to post these benchmarks. They are interesting. But are you sure that an Array can be compared to a List?

array[x] = y is almost like setting an int variable to a value except that the array bounds are checked (and depending how your loop is written, CLR might even disable bounds checking).

List.add() may do more than just setting a value, perhaps it has to check if the List should grow or reallocate. Unfortunately, I can't read your source code because the attachment hasn't been approved yet but did you define an initial capacity for the List?

Best regards.

joshe wrote:

Hey,

I'm the 'customer' that Jim was talking about. Amoung other things, I've been building a high-performance ETL tool that is much better than DTS so I do routinely deal with millions of records.

Then the db roundtripping is totally crushing the time spend on boxing/unboxing a value type. You can say I'm wrong, but we've profiled the living daylights out of our code and if the boxing issue would be significant we would have changed it. However as I also explained, it's very cumbersome to change it in our code, i.e. with a generic class: it would lead to problems in other areas where boxing will still occur simply because the field is boxed through the interface used.

For a typical application that pulls a couple of records, makes a modification or two, and saves them, you would never notice the performance hit. The other overhead will dwarf the box/unbox cost. But if you need to process a lot of rows, it will be an issue.

Oh, so loading them from the db is faster than your processing routines and boxing will then be an issue? I doubt it.

What I was specifically hoping for is an approach used by a package called Neo, which I looked at but never fully adopted. They had the same generated code entities implementation but they made each field into an instance variable on the derived class (they were using 1.1 so no partial classes). The values are copied in and out of arrays during persistence. I do not know the design trade off but it felt pretty clean and powerful.

I know neo. The decision why LLBLGen Pro has fields is because they can be used to formulate dynamic lists and typed lists, something not a lot of other o/r mappers have, and also can be used in the typed query system, so no string-based crap.

Otis -- your response is wrong on performance costs. I didn't know that for sure so I built a test. The C# compiler appear to be good about minimizing the number of boxing operations but there is still x25 hit for boxing over direct int array access. An event handler that does nothing is about the same cost as accessing a value in an array list – I don’t know where your 10K cost comes from. The performance of GenPro is about 50% slower than the object[] array test – which is perfectly reasonable. The bulk of the performance cost is tied to the boxing operations. The real value of GenPro comes from minimizing database access (or in our case avoid Microsoft’s binary persistence, which has sucky performance).

Try writing a winforms app with a couple of event handlers which apparently do nothing more than changing some string in a form element. A list isn't an array btw, it has to resize every time.

My remark wasn't about being specific with 10,000 ints, but to avoid the discussion you're starting now: loops with a lot of value types which try to proof that boxing is very significant. Well, for THAT loop it might be, but in the total process it's insignificant. I tried to avoid a synthetic benchmark discussion like this with a routine with a loop which runs 100,000+ times to tell how slow boxing really is and how important it really is. That loop won't show that. Remember: your loop runs 20 million times. In 20 million iterations, it takes 5 seconds. However to fetch 20 million rows it will take perhaps half an hour of not more due to time spend inside the db, data transport to the client, object creation etc. (wild guess). Will these 5 seconds be significant on 30 minutes? IMHO no. That's why I said that it's insignificant.

LLBLGen Pro uses fields objects which use object typed value storage. I have explained why this is and that it's a bit cumbersome to change it as usage will often show it will still use boxing through the non-generic interface. This is making the point rather moot. I'll tell you something else: the value read from the DB is already boxed when it enters the entity. To fetch as fast as possible, it uses datareader.GetValues(), to read all values in a row at once. This will box value types, but will also avoid conversion logic in the datareader which is often slower plus it will avoid a long switch or if/else list for each type which is also slower. The boxed type will be stored as-is in the entity field.

The implication of this analysis is that LINQ entities should x10 faster than GenPro if you are dominated by access cost because Microsoft is going to go through the trouble of avoiding the boxing overhead. Based on a cursory review, they plan to use an attribute driven design and language hooks to handle the persistance, similar to how XML serialization works.

Then you should use Linq to sql. You'll quickly learn that their in-memory IL generation is fast in some areas but the framework falls flat on its face in many way more time consuming areas, like fetching entity graphs.

They don't use attributes per-se and I don't know what you mean with language hooks... linq queries are expression trees passed to the provider of the IQueryable<T> implementing object which will result in a query.

The surprise for me was that the list<int> test was x7 slower than an int array.

That's no surprise. A List<int> starts with I think 4 positions. So if you're storing 1000 elements, it has to resize a lot, which means memcpy's.

The last test case is based on a Neo-like implementation. It avoids the boxing penalty. It is also x7 slow than direct int[] arrays – the C# optimizer is not as smart as I gave it credit for. This implies that GenPro could pick up about x5 by a redesign that minimize the use of object[] arrays.

It doesn't use object arrays... It uses entityfield objects which have a property which is of type object, and not of the type of the value it stores. I tried to explain that in my previous post. When LLBLGen pro was written (in vs.net 2002), generics weren't there and it was the only option. Now generics are there, it could have a re-design, however it will give problems as well, as the generic type HAS TO BE KNOWN to be able to use a field if it's generic. So you then implement an interface which isn't generic. However it then has to have the property of type object which holds the value.

The stuff below is the output of my attached test program. I subtracted out the loop overhead. Enough fun for today… back to work for me.

Josh

---

Performance Test -- Boxing vs Event Handler.

TEST SECONDS RATIO loop overhead 0.45313 --- int[] 0.21875 1.0 object[] 5.56250 25.4 ArrayList 6.64063 30.4 list<int> 1.51563 6.9 list<object> 5.90625 27.0 Event Handler 6.62500 30.3 Test Entity (GenPro) 8.10938 37.1 Test Entity (Int Index) 8.09375 37.0 Test Entity (unboxed) 8.21875 37.6 Test Entity (unboxed - generics) 8.17188 37.4 Test Entity (unboxed, no virtual) 8.40625 38.4 Test Entity (Neo-derived) 1.51563 6.9

Press any key to exit.

If you want to avoid boxing, you shouldn't store the value in an object typed member. As soon as you do that, it's boxed. So your test entities all box values, which is the reason all their test results are roughly the same.

In LLBLGen Pro, you can use the field objects in its Fields array to produce filters, lists etc. Making the field objects generic types will perhaps gain overall a few % performance, but it's not noticable compared to the time spend inside the DB and inside the datareader. Trust me, you won't notice it. . 2/3rd of the time a select takes is now spend inside the datareader+db and the rest is inside the object fetch routine + object creation routines.

Btw, Linq to sql can be faster because it doesn't have inheritance over multiple tables. That alone kills a lot of performance inside an o/r mapper because a lot of extra code has to be added. Sure, you can unroll loops, avoid if/else statements with IL code generated at runtime, however how far will you go? With every IL statement generated at runtime, the framework itself becomes rather fragile, as debugging it will be a nightmare.

To use strongly typed members, like neo and POCO based mappers use, you have to either use reflection at runtime (very slow) or generate an adapter in IL which adapts to the object and is fast in set/get values for members. We avoided that by using field objects. It's a design which isn't seen often, and today, with easier IL generation tools at hand, it would have been a good thing not use the fields objects, however we decided to do it this way, as it was easier to write, easier to debug and cleaner for the developer using our code: no magic generated at runtime.

Because we do know where the bottlenecks are, Joshe, they're in object creation when fetching a query, even though we use factories. Strange, isn't it? Simply creating objects (entity has fields object which contains field objects) is slower than anticipated. 'Bottlenecks' is a big word, let's describe them as 'hotspots' when fetching data. As the entityfield object is the building block of our system, abandoning that would be out of the question, so we optimized it a lot and we got very far. The thing is, at some point you simply realize that optimizing further is actually not really helping much as some feature added is actually slowing things down perhaps, or ease of use-code is slowing things down. What to do? Should the feature go? Should the ease-of-use code be stripped out?

It's a slippery slope. I agree, performance is very important, and for a long time (2-3 years back) we thought other things were more important than brutal performance, because nothing beats a datareader. However since a couple of years we try to make it very fast, memory conservative etc. etc. however also without compromises: we won't cut out a feature if it hurts performance a bit, we also won't cut out ease-to-use code because it limits performance. It's a tradeoff.

If the boxing could be avoided, I'm more than willing to make the changes, but not to every price: if I have to use EntityField<int>() and our customers have a hard time using the fields now, it's not really helpful. In v3, where .net 1.x will be dropped, perhaps we could do some tests with this to see how bad it turns out to be. Though I dont expect any significant gains in a complete process.

Otis,

Thanks for the response.

Of course, toy benchmarks (like my program) and isolated performance optimization never translate into real performance gains. Sure, if you are writing numeric or scientific code, you can know the critical loops in advance -- otherwise, use a profiler before talking about performance.

I was asking to see what level of thought had gone into the design choices. What I look for in a tool vendor is that they have gone through all the tradeoffs and isolate me from having to deal with them. I want to delegate to them without it coming back to bite me.

I'm convinced that the trade-off choices are intentional and derived from real world experience, and that's all that really matters to me.

The heart of the matter is that if you want to leverage relational databases, you've already given up on the box/unbox debate. The smart thing to do as a developer is to use the tools that make you productive, profile your application after it has been developed, and then optimize the bottlenecks.

When I look at our web application code, it's always the SQL queries that dominate. Or the I/O cost of transmitting rendered pages with massively redundant formatting directives because an underlying UI component was poorly written.

On the ETL side, where I am routinely manipulating millions of records, GenPro is not the appropriate tool for accessing the primary data store. But no O/R mapper would be appropriate -- that's why people build/use ETL tools.

I'll be posting some more 'useful' questions once I've purchased licenses for my team and had a chance to work with it a while.

Josh

Thanks for your thoughts, Josh. It keeps us sharp when things in our framework are discussed and when we're asked why it's this way and not another way, so maybe some train of thought we completely missed is initiated and we all benefit from it (and that has happened in the past so we're open for these ).

ETL tools are indeed better off with low-level direct-access tools which leverage bulk-copy API's as otherwise the performance loss is simply too big.