You should always limit the amount of time spend in the slowest part of the process (well, that's obvious of course, but often this is overlooked ).
Serialization -> network transport over a wire -> deserialization is often the bottleneck, but it depends on the network speed. For example, are you on a gigabit ethernet connection, the network is faster than most harddisks can deliver data. This means that the bottleneck can be in the database instead.
Nevertheless, serialization and deserialization is not the fastest way of handling data. It's often slower than expected and especially if the datablocks are becoming huge. that said, for paging through data, the user is often not interested in 1000's of rows so if you have a paging system which shows 20-50 entities at a time, this will never be slow.