So your xml string is 1 billion (with a B) characters long? Unicode chars are 2bytes per char, so that'll give you 2GB I think.
An xml string of 1 Billion characters takes ages to parse, why do you use such a big document? In any case, you're running into limitations of .NET (at least the version you're using) I think, as the issue occurs inside the string builder.
Writing the xml, it will append the fragments to a string builder internally which will then be used to construct the string. Your set of entities is simply very very very large, so serializing them together to 1 document is running into this limitation.
I'd serialize less entities to xml, as it must be a massive amount of data you're serializing...