- Home
- LLBLGen Pro
- Architecture
Auditing and Temporal Object Best Practices
Joined: 17-Aug-2003
jeffdeville wrote:
Thanks Frans, I'll replace them. What field type would you recommend (Image vs ntext as I don't have sql 2k5) Is there any reason that you know of why the triggers I was looking at were not using text/image fields?
I don't know, probably because they did something I wouldn't do.
I assumed there must be a brutal penalty to pay to give up the ability to audit large fields. If there is a high cost, could it be remediated by having two fields to save data in, a binary/ntext and an nvarchar? Thanks
Well, I think if you want to have auditing, you should think about: - do you want to log WHO changed data WHEN? - do you want to log WHICH data was changed WHEN? - do you expect a lot of data in your db/lots of mutations?
If you want the first, you don't need data comparison. If you choose the second, you DO need datacomparison. This means with text/image fields that you need to compare perhaps megabytes of data, which can be very slow, and it also can be tedious, because large block compares are not always doable in SQL. This is likely the reason why the triggers you saw weren't using text/image fields. (which made them IMHO useless. Why have auditing but not on all fields?)
If you choose the second option and you don't expect a lot of data/changes, you could simply copy the row completely into an auditing table. This is of course not efficient if you have a lot of data and/or a lot of modifications.
Seriously consider if you want to keep track of old values of the data. Do you really need every previous version of a field? If you DO: consider modeling the text/image field out into a separate table. You can then add a trigger to that table for just the updates/inserts, which means that if a user modifies that field, your trigger will fire.
If you DON'T, don't track original values, but just the WHO did which action WHEN.
Also a side question, is there a benefit to storing the after values beyond making it easier to identify what a particular person changed? It seems like only the before values are strictly necessary so long as you have access to the current state of the object in question.
If a row gets deleted, you won't have the after value for that particular action. Though I still doubt if it's really necessary to store all values which have ever changed at any given date. Unless it's required by law of course.
Joined: 09-Feb-2005
Gah... ok leave it to Frans to point out a multitude of other reasonable design goals. Punk.
To be honest, I don't know what I'll need. This project is one of those evolving deals, and I'm constantly asked to provide data I didn't expect to need. So since performance isn't really that big of an issue yet, I think I'll just go w/ all data for now. (Who, What and When) Since the adapter allows me to centralize all of this functionality, it should be fairly easy to change it if things slow to a crawl.
As for moving the large fields out to another table, and setting up a trigger there, that sounds like an excellent idea, but not an easily portable one, which is what I was hoping for. Perhaps this is an instance where a portable solution simply can't be a great solution yet, or at least not w/out investing more time than I have at the moment.
The deletion of the row removing the last value makes sense unless I store a copy of the data before the deletion in the same manner I do for inserts/updates. So long as I do that, it seems like I could still get away w/ just the before field.
Thank you for the input Frans. Very enlightening.
Joined: 09-Feb-2005
Here is an update to what I am using now. If you feel the need to comment, please feel free. :-) For the moment, I'm only posting it in the hopes that it helps someone else get started. Apologize for the length. There's a bit in here that is being leveraged by other aspects of the system.
Data Model tbl_DAA_AuditEntry:
CREATE TABLE [tbl_DAA_AuditEntry] (
[AuditId] [int] IDENTITY (1, 1) NOT NULL ,
[ChangeType] [char] (1) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
[DateModified] [datetime] NOT NULL CONSTRAINT [DF_tblAudit_DateModified] DEFAULT (getdate()),
[ModifiedBy] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
[EntityType] [varchar] (100) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
[PrimaryKeyName0] [varchar] (25) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
[PrimaryKeyValue0] [varchar] (36) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
[PrimaryKeyName1] [varchar] (25) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[PrimaryKeyValue1] [varchar] (36) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[PrimaryKeyName2] [varchar] (25) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[PrimaryKeyValue2] [varchar] (36) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
CONSTRAINT [PK_tblAudit] PRIMARY KEY CLUSTERED
(
[AuditId]
) ON [PRIMARY]
) ON [PRIMARY]
GO
tbl_DAA_AuditEntryChanges
CREATE TABLE [tbl_DAA_AuditEntryChanges] (
[AuditEntryChangeId] [int] IDENTITY (1, 1) NOT NULL ,
[AuditId] [int] NOT NULL ,
[FieldName] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
[OldValue] [sql_variant] NULL ,
CONSTRAINT [PK_tbl_DAA_AuditEntryChanges] PRIMARY KEY CLUSTERED
(
[AuditEntryChangeId]
) ON [PRIMARY] ,
CONSTRAINT [FK_tbl_DAA_AuditEntryFields_tbl_DAA_AuditEntry] FOREIGN KEY
(
[AuditId]
) REFERENCES [tbl_DAA_AuditEntry] (
[AuditId]
) NOT FOR REPLICATION
) ON [PRIMARY]
GO
Code IAuditManager
public interface IAuditManager
{
EntityCollection<AuditEntryEntity> FetchCollection(AuditCriteria criteria);
IList<AuditCriteria.PrimaryKeyStruct[]> FetchRecentMultiKey(string entityType, string userName, int numToRetrieve);
List<string> FetchRecentSingleKey(string entityType, string userName, int numToRetrieve);
}
AuditManager
public class AuditManager : AbstractManager, IAuditManager
{
/// <summary>
/// Goal is to return a list of the AuditEntries. I won't check the session, since this is likely to change a lot.
/// </summary>
/// <param name="criteria"></param>
/// <returns></returns>
public EntityCollection<AuditEntryEntity> FetchCollection(AuditCriteria criteria)
{
EntityCollection<AuditEntryEntity> _auditEntries = new EntityCollection<AuditEntryEntity>(new AuditEntryEntityFactory());
PrefetchPath2 prefetch = CreatePrefetch(criteria);
RelationPredicateBucket bucket = CreatePredicate(criteria);
SortExpression sort = new SortExpression(AuditEntryFields.DateModified | SortOperator.Descending);
using (DataAccessAdapter adapter = new DataAccessAdapter())
{
adapter.FetchEntityCollection(_auditEntries, bucket, 0, sort, prefetch);
}
return _auditEntries;
}
public IList<AuditCriteria.PrimaryKeyStruct[]> FetchRecentMultiKey(string entityType, string userName, int numToRetrieve)
{
RelationPredicateBucket bucket;
SortExpression sort;
SetUpFetchRecent(entityType, userName, out bucket, out sort);
ResultsetFields fields = new ResultsetFields(6);
fields[0] = AuditEntryFields.PrimaryKeyName0;
fields[1] = AuditEntryFields.PrimaryKeyValue0;
fields[2] = AuditEntryFields.PrimaryKeyName1;
fields[3] = AuditEntryFields.PrimaryKeyValue1;
fields[4] = AuditEntryFields.PrimaryKeyName2;
fields[5] = AuditEntryFields.PrimaryKeyValue2;
IDataReader reader = null;
using (DataAccessAdapter adapter = new DataAccessAdapter())
{
reader = adapter.FetchDataReader(fields, bucket, CommandBehavior.CloseConnection, numToRetrieve, sort, false);
IList<AuditCriteria.PrimaryKeyStruct[]> primaryKeys = new List<AuditCriteria.PrimaryKeyStruct[]>();
while (reader.Read())
{
AuditCriteria.PrimaryKeyStruct[] keyStructArray = new AuditCriteria.PrimaryKeyStruct[3];
keyStructArray[0] = new AuditCriteria.PrimaryKeyStruct(reader[0].ToString(), reader[1].ToString());
keyStructArray[1] = new AuditCriteria.PrimaryKeyStruct(reader[2].ToString(), reader[3].ToString());
keyStructArray[2] = new AuditCriteria.PrimaryKeyStruct(reader[4].ToString(), reader[5].ToString());
primaryKeys.Add(keyStructArray);
}
reader.Close();
return primaryKeys;
}
}
public List<string> FetchRecentSingleKey(string entityType, string userName, int numToRetrieve)
{
RelationPredicateBucket bucket;
SortExpression sort;
SetUpFetchRecent(entityType, userName, out bucket, out sort);
ResultsetFields fields = new ResultsetFields(1);
fields[0] = AuditEntryFields.PrimaryKeyValue0;
IDataReader reader = null;
using (DataAccessAdapter adapter = new DataAccessAdapter())
{
reader = adapter.FetchDataReader(fields, bucket, CommandBehavior.CloseConnection, numToRetrieve, sort, false);
List<string> keys = new List<string>();
while (reader.Read())
keys.Add(reader[0].ToString());
reader.Close();
return keys;
}
}
#region Private Methods
private PrefetchPath2 CreatePrefetch(AuditCriteria criteria)
{
if (criteria.SearchType == SearchTypeEnum.RecentlyModified)
return null;
else
{
PrefetchPath2 prefetch = new PrefetchPath2((int)EntityType.AuditEntryEntity);
prefetch.Add(AuditEntryEntity.PrefetchPathAuditEntryChangesCollection);
return prefetch;
}
}
private RelationPredicateBucket CreatePredicate(AuditCriteria criteria)
{
RelationPredicateBucket bucket = new RelationPredicateBucket();
if (criteria.BeginDate != DateTime.MinValue)
bucket.PredicateExpression.Add(AuditEntryFields.DateModified >= criteria.BeginDate);
if (criteria.EndDate != DateTime.MinValue)
bucket.PredicateExpression.AddWithAnd(AuditEntryFields.DateModified <= criteria.EndDate);
if (!criteria.ArrayIsNullOrEmpty(criteria.EntityType))
bucket.PredicateExpression.AddWithAnd(AuditEntryFields.EntityType == criteria.EntityType);
if (!criteria.ArrayIsNullOrEmpty(criteria.ModifiedBy))
bucket.PredicateExpression.AddWithAnd(AuditEntryFields.ModifiedBy == criteria.ModifiedBy);
if (criteria.PrimaryKeys != null && criteria.PrimaryKeys.Length > 0)
{
IPredicateExpression peAggregate = new PredicateExpression();
foreach (AuditCriteria.PrimaryKeyStruct pk in criteria.PrimaryKeys)
{
IPredicateExpression pe = ((AuditEntryFields.PrimaryKeyName0 == pk.Name) & (AuditEntryFields.PrimaryKeyValue0 == pk.Value))
| ((AuditEntryFields.PrimaryKeyName1 == pk.Name) & (AuditEntryFields.PrimaryKeyValue1 == pk.Value))
| ((AuditEntryFields.PrimaryKeyName2 == pk.Name) & (AuditEntryFields.PrimaryKeyValue2 == pk.Value));
peAggregate.AddWithOr(pe);
}
bucket.PredicateExpression.AddWithAnd(peAggregate);
}
return bucket;
}
private void SetUpFetchRecent(string entityType, string userName, out RelationPredicateBucket bucket, out SortExpression sort)
{
AuditCriteria criteria = new AuditCriteria();
criteria.EntityType = new string[] { entityType };
criteria.ModifiedBy = new string[] { userName };
criteria.SearchType = SearchTypeEnum.RecentlyModified;
bucket = CreatePredicate(criteria);
sort = new SortExpression(AuditEntryFields.DateModified | SortOperator.Descending);
}
#endregion
internal override RelationPredicateBucket CreatePredicate(ICriteria criteria)
{
throw new Exception("The method or operation is not implemented.");
}
}
AuditAdapter
public class AuditAdapter : DataAccessAdapter
{
ISecurityInfoProvider _securityProvider;
ICompressionProvider _compressionProvider;
List<string> _entityTypesToIgnore;
private bool _disableLogging = false;
#region Constructors
public AuditAdapter(bool disableLogging) : this(null, null, null)
{
_disableLogging = disableLogging;
}
public AuditAdapter() : this(null, null, null) { }
public AuditAdapter(ISecurityInfoProvider securityProvider) : this(securityProvider, null, null){}
public AuditAdapter(ICompressionProvider compressionProvider) : this(null, compressionProvider, null) { }
public AuditAdapter(ISecurityInfoProvider securityProvider, ICompressionProvider compressionProvider) : this(securityProvider, compressionProvider, null) { }
public AuditAdapter(ISecurityInfoProvider securityProvider, ICompressionProvider compressionProvider, List<string> entityTypesToIgnore)
{
_securityProvider = securityProvider == null ? new WebSecurityProvider() : securityProvider;
_compressionProvider = compressionProvider == null ? new NoChangeCompressionProvider() : compressionProvider;
_entityTypesToIgnore = entityTypesToIgnore == null ? new List<string>() : entityTypesToIgnore;
}
#endregion
protected override void OnSaveEntityComplete(IActionQuery saveQuery, IEntity2 entityToSave)
{
base.OnSaveEntityComplete(saveQuery, entityToSave);
if (_disableLogging)
return;
if (IgnoreThisEntity(entityToSave))
return;
AuditEntryEntity aEntity = CreateAuditEntry(entityToSave, entityToSave.IsNew ? "I" : "U");
if (!entityToSave.IsNew)
foreach (IEntityField2 field in entityToSave.Fields)
if (field.IsChanged)
CreateAuditEntryChangeField(aEntity, field);
using (DataAccessAdapter adapter = new DataAccessAdapter())
{
if (!adapter.SaveEntity(aEntity, false, true))
throw new ORMException("AuditTrail could not be saved.");
}
}
protected override void OnDeleteEntityComplete(IActionQuery deleteQuery, IEntity2 entityToDelete)
{
base.OnDeleteEntityComplete(deleteQuery, entityToDelete);
if (_disableLogging)
return;
if (IgnoreThisEntity(entityToDelete))
return;
AuditEntryEntity aEntity = CreateAuditEntry(entityToDelete, "D");
foreach (IEntityField2 field in entityToDelete.Fields)
CreateAuditEntryChangeField(aEntity, field);
using (DataAccessAdapter adapter = new DataAccessAdapter())
{
adapter.SaveEntity(aEntity, false, true);
}
}
private AuditEntryEntity CreateAuditEntry(IEntity2 entity, string action)
{
AuditEntryEntity aEntity = new AuditEntryEntity();
aEntity.ChangeType = action;
aEntity.ModifiedBy = _securityProvider.UserName;
aEntity.EntityType = entity.GetType().Name;
SetPrimaryKeys(aEntity, entity);
return aEntity;
}
private bool IgnoreThisEntity(IEntity2 entity)
{
return _entityTypesToIgnore.Contains(entity.GetType().Name);
}
private void SetPrimaryKeys(AuditEntryEntity aEntity, IEntity2 entity)
{
for (int primaryKeyIndex = 0; primaryKeyIndex < entity.PrimaryKeyFields.Count; primaryKeyIndex++)
{
aEntity.Fields["PrimaryKeyName" + primaryKeyIndex].CurrentValue =
entity.PrimaryKeyFields[primaryKeyIndex].Name;
aEntity.Fields["PrimaryKeyValue" + primaryKeyIndex].CurrentValue =
entity.PrimaryKeyFields[primaryKeyIndex].CurrentValue.ToString();
}
}
private void CreateAuditEntryChangeField(AuditEntryEntity aEntity, IEntityField2 field)
{
AuditEntryChangesEntity changedField = new AuditEntryChangesEntity();
aEntity.AuditEntryChangesCollection.Add(changedField);
changedField.FieldName = field.Name;
changedField.OldValue = _compressionProvider.CompressField(field.DbValue);
}
public override int DeleteEntitiesDirectly(string entityName, IRelationPredicateBucket filterBucket)
{
if (_disableLogging)
return base.DeleteEntitiesDirectly(entityName, filterBucket);
else
throw new NotSupportedException("You may not delete entities directly when auditing changes");
}
}
ICompressionProvider / NoChangeCompressionProvider
public interface ICompressionProvider
{
object CompressField(object input);
}
public class NoChangeCompressionProvider : ICompressionProvider
{
#region ICompressionProvider Members
public object CompressField(object input)
{
return input;
}
#endregion
}
ISecurityProvider / WebSecurityProvider
public interface ISecurityInfoProvider
{
string UserName { get;}
}
public class WebSecurityProvider : ISecurityInfoProvider
{
#region ISecurityInfoProvider Members
public string UserName
{
get
{
if (Thread.CurrentPrincipal as RolePrincipal == null)
{
RolePrincipal rp = new RolePrincipal(System.Security.Principal.WindowsIdentity.GetCurrent());
Thread.CurrentPrincipal = rp;
}
return Thread.CurrentPrincipal.Identity.Name;
}
}
#endregion
}
AuditCriteria
public partial class AuditCriteria : AbstractCriteria
{
public AuditCriteria()
{
SearchType = SearchTypeEnum.ConstructHistory;
}
private SearchTypeEnum _searchType;
public SearchTypeEnum SearchType
{
get { return _searchType; }
set { _searchType = value; }
}
public struct PrimaryKeyStruct
{
public string Name;
public string Value;
public PrimaryKeyStruct(string name, string value)
{
Name = name;
Value = value;
}
}
string[] _entityType;
public string[] EntityType
{
get { return _entityType; }
set { _entityType = value; }
}
PrimaryKeyStruct[] _primaryKeys;
public PrimaryKeyStruct[] PrimaryKeys
{
get { return _primaryKeys; }
set { _primaryKeys = value; }
}
string[] _modifiedBy;
public string[] ModifiedBy
{
get { return _modifiedBy; }
set { _modifiedBy = value; }
}
DateTime _begin, _end;
public DateTime EndDate
{
get { return _end; }
set { _end = value; }
}
public DateTime BeginDate
{
get { return _begin; }
set { _begin = value; }
}
}
AbstractCriteria:
public interface ICriteria
{
bool Personalize { get;set;}
}
[Serializable]
public abstract class AbstractCriteria : ICriteria
{
private bool _personalize;
private SearchTypeEnum _searchType;
public virtual SearchTypeEnum SearchType
{
get { return _searchType; }
set { _searchType = value; }
}
public bool Personalize
{
get { return _personalize; }
set { _personalize = value; }
}
public override string ToString()
{
return Utilities.Serialize(this);
}
public override bool Equals(object obj)
{
if (this.GetType() != obj.GetType())
return false;
return this.GetHashCode() == obj.GetHashCode();
}
public bool ArrayIsNullOrEmpty(int[] input)
{
return ((input == null) || (input.Length == 0));
}
public bool ArrayIsNullOrEmpty(string[] input)
{
return ((input == null) || (input.Length == 0));
}
public bool ArrayIsNullOrEmpty(object[] input)
{
return ((input == null) || (input.Length == 0));
}
}
Joined: 30-Jun-2005
Just thought I'd mention how we decided to do auditing. Instead of having an AuditAdapter, we made an auditor object that would listen for events coming out of the DataAccessAdapter (raised on OnSaveEntityComplete, OnSaveEntityCollectionComplete). When the transaction is complete, another event is raised to the auditor object, that signals it to go ahead and persist its auditing information. The auditor uses its own DataAccessAdapter to do this, which is not connected with an auditor object because you don't need to audit the auditor
Joined: 13-Dec-2006
Hello,
This thread has been very helpful as I'm facing very similar issues. However our DataLayer project was orignally generated using the template group 'Self Servicing' rather than 'Adapter'.
So how would I achieve this auditing in a 'self servicing' project? I presume I need to over-ride the OnSaveEntityComplete equivilant? But I cannot figure this out in self servicing yet?
Any ideas/examples?
Currently I can apply this to each individual entity - but I would like to extend (or over-ride) the save method.
Protected Sub updateWithAudit(ByVal entity As EntityBase)
If entity.IsDirty() Then
Try
Dim i As Integer
While i <> entity.Fields.Count
Dim field As IEntityField = entity.Fields(i)
If field.IsChanged Then
Dim audit As New DaaAuditEntity()
audit.ChangeType = "Update"
audit.ModifiedBy = "USER NAME HERE"
audit.EntityType = entity.GetType.ToString
audit.PrimaryKeyField = entity.Fields.PrimaryKeyFields.Item(0).Name.ToString
audit.PrimaryKey = entity.Fields.PrimaryKeyFields.Item(0).DbValue
audit.IsNew = True
audit.OldValue = entity.Fields(i).DbValue
audit.NewValue = entity.Fields(i).CurrentValue
audit.DateModified = Now()
audit.FieldName = entity.Fields(i).Name.ToString
audit.Save()
End If
i += 1
End While
entity.Save()
Catch ex As Exception
ShowError(ex.Message, ex.Source, Page)
End Try
End If
End Sub
Thanks, Connor
Joined: 17-Aug-2003
Well, I think you want to do that in the same transaction, so you need to make the audit entities participate in the same transaction as the real entity you're auditing.
To do this, please read this thread: http://www.llblgen.com/TinyForum/Messages.aspx?ThreadID=8417
and the last post of mine in that thread explains it further.
You can automate this by creating your own version of the entityusingentitybase.template and by creating your own templatebindings file which overrides the existing binding (see 'Using the generated code: Adding your own code to the generated classes' for details as well as the SDK if you want deep info about this)
THough it might be that you don't want auditing on a lot of entities, just on a couple, so it might be enough to just add this code to a partial class of the derived entity classes (two-class scenario).
We are aware of the fact that this is an often used feature which isn't straight forward to implement, as it has the difficulty to save new entities during an active save process, and the code which decides the auditing has to take place isn't able to do so without complex hand written code. So we'll try to address this in v2.1 to make it easier to add auditing code. Either by explaining existing constructs better with a proper example, or by adding new features so adding auditing is easier.
Joined: 04-Mar-2007
jeffdeville wrote:
Here is an update to what I am using now. If you feel the need to comment, please feel free. :-) For the moment, I'm only posting it in the hopes that it helps someone else get started. Apologize for the length. There's a bit in here that is being leveraged by other aspects of the system.
Jeff
I'm not sure if your approach is 100% appropriate for us, but I'm wondering if you completed this auditing solution and if you would be willing to share the completed code?
The thing that I like about the "recording" aspect (as opposed to the "reporting" aspect) of your approach is that it's generic. Instead of separate <TableName>Audit tables, you just use AuditEntry and AuditEntryChanges tables to record all changes. And since you are auditing at the field level, not the object level, you're not generating a lot of unecessary overhead when minimal changes are made. (Did the use of SqlVariant cause any problems?)
My concerns primarily pertain to the "reporting" aspect of your approach. In particular: How well does your solution accomodate object graphs? For example, say I had a Customer/Phone hierarchy (a Customer has 0..* Phone records), and the user changed a phone # associated with a customer. How easy is it to report things like: "John changed Sally's home phone # from abc to xyz on xx/yy/zz." It's seems like the fact that there are no actual DB relationships between the audit tables and the tables being audited is an impediment to reporting. Perhaps in practice, however, you've surmounted these obstacles. (In our case, however, efficient and performant recording is much more important than reporting, which would occur infrequently.)
I am also wondering whether you considered adding a transactional context that would allow one to know that a series of AuditEntry records occured as part of the same user activity. In the example above, if the user changed their address (in Customer) and a phone # (in Phone) in the same form and the clicked Save, we would know that both chnages happened at the same time. I imagine doing this by generating some sort of transaction ID (e.g. returned by StartAuditTransaction) when the form is opened. This transaction ID is saved to all the AuditEntry records associated with this user until EndAuditTransaction is called.
In light of the entire discussion, are you satisfied with your approach and would you do anything differently if you could start over (how, for example, does your approach differ from Phil's and why?)
Finally, I'm wondering what kind of audit support will be in 2.1 and when it will be available. Do you have any more information in this regard Otis?
Thanks so much.
Rob
Joined: 17-Aug-2003
We'll provide a pluggable system for auditing as we've also done for validation, so we can revert control to you over what's audited when and how. The system isn't designed in all its details yet, but it will be possible to audit a lot of fine details, like changes to fields/ entities, save actions, load actions etc. As auditing often creates new entities these are also saveable within the same transaction.
We'll also add a dependency injection system out of the box so it will be very easy for you to simply write a class and setup injection at runtime so you don't have to worry about whether auditing (or validation for that matter ) is performed, as the object which takes care of it is plugged in always.