- Home
- LLBLGen Pro
- LLBLGen Pro Runtime Framework
Performance issue
Joined: 04-Jan-2007
Dear all,
I'm facing a performance issue that maybe other have encoutered. I have two tables Serie and Style. I have more than 4000 records in Serie and 30 records in Style. Serie as a foreing key to style.
I'm trying to retreive 25 styles with 4 serie for each style. When I execute the linq query it takes around 15 seconds to execute. This is crazy. I'm returning 25 style with 4 series for each. 100 records... 15 seconds... This definitevely two long...
this is a code snippet I have used for this.
//I start by creating a prefetch Path private static IPathEdge[] styleInfoPathEdge = new IPathEdge[]{new PathEdge<SerieEntity> StyleEntity.PrefetchPathStyleSeries, serie=>serie.SeriePictureId!=null && serie.SerieApproved,4)};
//then I run the query var entities = metaData.Style.WithPath(styleInfoPathEdge).Skip(0).Take(25).ToList();
//I tried something else var styles = metaData.Style.Skip(0).Take(25).ToList(); foreach (var style in styles) { var series = metaData.Serie.Where(serie => serie.SerieStyleId == style.StyleId).Take(4).ToList(); }
//This is done in less than 1 seconds....
Any idea why this could be?
This the execution log for the first code snippet:
_<SNIP> Sorry, the length of this post was causing issues so I had to truncate it. If you need to send us large amounts of data please attach it to the post instead. Matt </SNIP> _
Joined: 08-Oct-2008
Do you need to use the .Take and .Skip if you are just retrieving the first 25 ? What is the performance like if you just use
var entities = metaData.Style.WithPath(styleInfoPathEdge).ToList(); ?
Could you also post the generated SQL in your slow case and in the above case ? Can you confirm (using SQL profiler) if the slow performance is from the SQL or from the code (it's almost certainly the SQL but would be could to be sure )
Matt
If you use paging, which you do, make sure the ParameterPrefetchPathThreshold is set to # of pk fields * # of parent nodes. By default it's set to 50.
Also, you posted a lot of tracing (which is useless if it's above a couple of lines), and you forgot a lot of info: http://www.llblgen.com/tinyforum/Messages.aspx?ThreadID=12769
Joined: 04-Jan-2007
Thanks for your reply.
-
make SURE you're using the latest build of LLBLGen Pro and the runtime libraries. Yes I'm am. the latest version downloaded from the customer area.
-
enable tracing of the DQE through the config file and obtain the generated query. I did that I gess. But it was cropped from the initial post as it was too long... I'm going to attach it here
-
enable LinqExpressionHandler tracing (see documentation) at level 3 to obtain the expression tree in text. I have added that to the tracing. I was not tracing the linq expression
-
if you run into a crash, always provide the FULL stacktrace. Not a crash
- always provide the full query in linq code the full code is:
private static IPathEdge[] styleInfoPathEdge = new IPathEdge[]{ new PathEdge<SerieEntity>(StyleEntity.PrefetchPathStyleSeries, serie=>serie.SeriePictureId!=null && serie.SerieApproved, 4), };
public PagedResultDto<StyleInfoDto> FindStyleInfos(int pageIndex, int pageSize, StyleSortEnumDto sort, bool ascending, string queryText) {
if (pageSize <= 0)
throw new ApplicationException(ErrorMessages.PAGESIZE_MUST_BE_BT_0);
if (pageIndex < 0)
throw new ApplicationException(ErrorMessages.PAGEINDEX_MUST_BE_BT_0);
using (var adapter = new DataAccessAdapter())
{
var metaData = new LinqMetaData(adapter);
var query = (IQueryable<StyleEntity>)metaData.Style;
var filter = DefautlFilterOnStyle();
if (!string.IsNullOrEmpty(queryText))
filter = filter.And(x => x.StyleNameCi.Contains(queryText));
query = query.Where(filter);
var totalCount = query.Count();
var sortExpression = ToSortExpression(sort);
if (ascending)
query = query.OrderBy(sortExpression);
else
query = query.OrderByDescending(sortExpression);
var entities = query.WithPath(styleInfoPathEdge).Skip(pageSize * pageIndex).Take(pageSize).ToList();
return new PagedResultDto<StyleInfoDto>
{
PageIndex = pageIndex,
PageSize = pageSize,
TotalCount = totalCount,
Items = entities.ToStyleInfoDtos()
};
}
}
private Expression<Func<StyleEntity, bool>> DefautlFilterOnStyle() { if (CurrentUser.CanViewUnApproved()) return (Style => true); else return (Style => Style.StyleApproved || Style.StyleCreatedBy == CurrentUser.UserId); }
private Expression<Func<StyleEntity, object>> ToSortExpression(StyleSortEnumDto sort)
{
Expression<Func<StyleEntity, object>> sortExpression;
switch (sort)
{
case StyleSortEnumDto.Name:
default:
sortExpression = (x => x.StyleNameCi);
break;
}
return sortExpression;
}
- always provide information about inheritance structures of ALL entities involved in the query No inheritance.They are Flat entities.
- always provide DDL SQL for the tables involved. The tables are quit simple one foreign key from a table to another. But I'm gonna attach it here at well.
[Update] I had to split the Trace in two files because of the 1024 KB file limit on the forum I tried to play with the threshold but it had no real influence. I do not understand why it retreives the 4000 series for the 25 styles.
DDL
CREATE TABLE [dbo].[bd_Style]( [StyleId] [int] IDENTITY(1,1) NOT NULL, [StyleName] varchar NOT NULL, [StyleApproved] [bit] NOT NULL, [StyleScoreTotal] [int] NULL, [StyleScoreNbrOfVote] [int] NULL, [StyleViewed] [int] NULL, [StyleLanguage] varchar NOT NULL, [StyleCreatedOn] [datetime] NOT NULL, [StyleCreatedBy] [uniqueidentifier] NOT NULL, [StyleModifiedOn] [datetime] NOT NULL, [StyleModifiedBy] [uniqueidentifier] NOT NULL, [StyleNameCi] varchar NOT NULL, [StylePictureId] [uniqueidentifier] NULL, CONSTRAINT [PK_bd_Style_1] PRIMARY KEY CLUSTERED ( [StyleId] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY]
CREATE TABLE [dbo].[bd_Serie]( [SerieId] [int] IDENTITY(1,1) NOT NULL, [SeriePublisherId] [int] NOT NULL, [SerieBDCollectionId] [int] NULL, [SerieStyleId] [int] NULL, [SerieName] nvarchar NOT NULL, [SerieNameCI] nvarchar NOT NULL, [SerieNameFormated] nvarchar NOT NULL, [SerieNameCIFormated] nvarchar NOT NULL, [SerieResume] nvarchar NULL, [SerieStatus] [int] NOT NULL, [SerieApproved] [bit] NOT NULL, [SerieScoreTotal] [int] NULL, [SerieScoreNbrOfVote] [int] NULL, [SerieViewed] [int] NULL, [SerieLanguage] varchar NOT NULL, [SerieCreatedOn] [datetime] NOT NULL, [SerieCreatedBy] [uniqueidentifier] NOT NULL, [SerieModifiedOn] [datetime] NOT NULL, [SerieModifiedBy] [uniqueidentifier] NOT NULL, [SerieHistoryScoreTotal] [int] NULL, [SerieHistoryScoreNbrVotes] [int] NULL, [SerieDrawingScoreTotal] [int] NULL, [SerieDrawingScoreNbrVotes] [int] NULL, [SeriePictureId] [uniqueidentifier] NULL, [SerieDrawingScore] [int] NULL, [SerieHistoryScore] [int] NULL, [SerieScore] [int] NULL, CONSTRAINT [PK_bd_Serie_1] PRIMARY KEY CLUSTERED ( [SerieId] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY]
ALTER TABLE [dbo].[bd_Serie] WITH NOCHECK ADD CONSTRAINT [FK_bd_Serie_bd_Style] FOREIGN KEY([SerieStyleId]) REFERENCES [dbo].[bd_Style] ([StyleId]) GO
ALTER TABLE [dbo].[bd_Serie] CHECK CONSTRAINT [FK_bd_Serie_bd_Style] GO
Hi there,
I see a lot of strange things on your code. For instance this generated sql:
SELECT TOP(1)
COUNT(*) AS [LPAV_]
FROM [ordoMANIA].[dbo].[bd_Style] [LPLA_1]
WHERE ( ( ( ( 1 = 1))))
which apparently corresponds to this linq query:
var query = (IQueryable<StyleEntity>)metaData.Style;
query = query.Where(style => style.StyleApproved || style.StyleCreatedBy == CurrentUser.UserId);
var totalCount = query.Count();
So I don't know if you post the real code or where that sql comes from.
From that query starts the problem. As you reuse it. The next linq query...
var entities = query.WithPath(styleInfoPathEdge)
.Skip(pageSize * pageIndex)
.Take(pageSize)
.ToList();
... generates this for Style:
SELECT TOP(25)
[LPA_L1].[StyleApproved],
[LPA_L1].[StyleCreatedBy],
[LPA_L1].[StyleCreatedOn],
[LPA_L1].[StyleId],
[LPA_L1].[StyleLanguage],
[LPA_L1].[StyleModifiedBy],
[LPA_L1].[StyleModifiedOn],
[LPA_L1].[StyleName],
[LPA_L1].[StyleNameCi],
[LPA_L1].[StylePictureId],
[LPA_L1].[StyleScoreNbrOfVote],
[LPA_L1].[StyleScoreTotal],
[LPA_L1].[StyleViewed]
FROM [ordoMANIA].[dbo].[bd_Style] [LPA_L1]
WHERE ( ( ( ( ( ( ( ( 1 = 1))))))))
ORDER BY [LPA_L1].[StyleNameCi] ASC
and this for Series:
SELECT
[ordoMANIA].[dbo].[bd_Serie].[SerieApproved],
[ordoMANIA].[dbo].[bd_Serie].[SerieBDCollectionId] AS [SerieBdcollectionId],
[ordoMANIA].[dbo].[bd_Serie].[SerieCreatedBy],
[ordoMANIA].[dbo].[bd_Serie].[SerieCreatedOn],
[ordoMANIA].[dbo].[bd_Serie].[SerieDrawingScore],
[ordoMANIA].[dbo].[bd_Serie].[SerieDrawingScoreNbrVotes],
[ordoMANIA].[dbo].[bd_Serie].[SerieDrawingScoreTotal],
[ordoMANIA].[dbo].[bd_Serie].[SerieHistoryScore],
[ordoMANIA].[dbo].[bd_Serie].[SerieHistoryScoreNbrVotes],
[ordoMANIA].[dbo].[bd_Serie].[SerieHistoryScoreTotal],
[ordoMANIA].[dbo].[bd_Serie].[SerieId],
[ordoMANIA].[dbo].[bd_Serie].[SerieLanguage],
[ordoMANIA].[dbo].[bd_Serie].[SerieModifiedBy],
[ordoMANIA].[dbo].[bd_Serie].[SerieModifiedOn],
[ordoMANIA].[dbo].[bd_Serie].[SerieName],
[ordoMANIA].[dbo].[bd_Serie].[SerieNameCI] AS [SerieNameCi],
[ordoMANIA].[dbo].[bd_Serie].[SerieNameCIFormated] AS [SerieNameCiformated],
[ordoMANIA].[dbo].[bd_Serie].[SerieNameFormated],
[ordoMANIA].[dbo].[bd_Serie].[SeriePictureId],
[ordoMANIA].[dbo].[bd_Serie].[SeriePublisherId],
[ordoMANIA].[dbo].[bd_Serie].[SerieResume],
[ordoMANIA].[dbo].[bd_Serie].[SerieScore],
[ordoMANIA].[dbo].[bd_Serie].[SerieScoreNbrOfVote],
[ordoMANIA].[dbo].[bd_Serie].[SerieScoreTotal],
[ordoMANIA].[dbo].[bd_Serie].[SerieStatus],
[ordoMANIA].[dbo].[bd_Serie].[SerieStyleId],
[ordoMANIA].[dbo].[bd_Serie].[SerieViewed]
FROM [ordoMANIA].[dbo].[bd_Serie]
WHERE
(
[ordoMANIA].[dbo].[bd_Serie].[SerieStyleId] IN
( SELECT [LPA_L1].[StyleId]
FROM [ordoMANIA].[dbo].[bd_Style] [LPA_L1]
WHERE 1=1
)
AND (
( [ordoMANIA].[dbo].[bd_Serie].[SeriePictureId] IS NOT NULL)
AND ( [ordoMANIA].[dbo].[bd_Serie].[SerieApproved] = True)
)
)
So, from all that I presume you have a problem in your first filter. Please check that (1=1). I recommend you to test a basic query. Then add a basic filter, then add the pathedge, so on, so you can identify where the problem is.
If you want we reproduce it, please provide a more straight-forward code.
Also, you attached 1.4 MB of trace info. I don't know if you had a look at it, but reading it all takes days, and it's not 'to the point': if a problem exists with 1 or 2 queries, submit the query executed. We don't need 1.4 MB of trace info. I removed it, as it only eats up space for no reason.
Please next time, post a SHORT trace, if required and only the bare minimum, i.e. the query which fails. Thanks
Joined: 04-Jan-2007
Hey Gentlemans,
Thanks for the quick reply.
[ToDaelmo]
Yes this is the log of the code.
SELECT TOP(1) COUNT(*) AS [LPAV_] FROM [ordoMANIA].[dbo].[bd_Style] [LPLA_1] WHERE ( ( ( ( 1 = 1))))
Is the result of the line var totalCount = query.Count();
Well Maybe the query is the problem, but it is quite simple. A simple where clause on columns of the style with an order by... Nothing fancing here.
I'm reusing it because I have implemented a paged result. On all my queries I returned: -The total amount of the items (this is the count) -The current page and the page size (the Skip and take) -The items of the page
For every Style returned on the query I want to retreive four series wich are linked through a 1:M relation... Also nothing fancy. I limit the number of record in the prefetch path and I have a limiting lambda expression on the prefetch path as well... Again I do not beleive I'm doing anything fancy :'(
What I do not understand is why the 4000 series are processed and addedd to a collection... You can see that in te log... You will find the two queries on the style (the count and the paged query) then 25 records are added to a collection. Then you have a query on the series. only one that return the 4000 series and add them to a collection?... weird.... I would have expected more 25 calls with a Select Top 4 on the series... But not... And it is this adding to the collection of the 4000 series that take a lot of time...
[ToOtis] My apologise for the long log. But I beleive it was bringin something... It was showing the 4000 series to be added.. Ok i could have drop 3999 and replace them by a comment but then It would not have been the real log. After all the long log was cropped on my first post and I quote "Sorry, the length of this post was causing issues so I had to truncate it. If you need to send us large amounts of data please attach it to the post instead. Matt."
But I'll keep that in mind for the next time.
Thanks for your help.
Regards,