Entities which are cached, are kept in memory by the entities referencing them.
Consider Customer - Order. If you fetch both, the Customer references its orders through its Orders navigator, the collection it contains, and all orders inside that collection reference the customer.
So if you then cache the Customer, you also cache the orders. Now this is obvious. What's not obvious is when you e.g. fetch an order, myOrder, and you associate it with this cachedCustomer:
myOrder.Customer = cachedCustomer;
this now also adds 'myOrder' to cachedCustomer.Orders, and thus is kept in memory too.
In general if you want to cache entities, make sure they don't have navigators, or at least not the navigators on the PK side (so in my example, the Customer.Orders navigator should be deleted in the designer). This way, doing myOrder.Customer=cachedCustomer; won't add myOrder to the Orders collection as it's not there
So in my example, if you cache myOrder, you also cache the associated customer, and its entities. It's a graph, so things point to one another.
You have to make sure the data you cache doesn't have references to entities it shouldn't be having. The large object you're fetching, is that used somewhere where you create associations with it?