The string specified between <$ $> is specified as a referenced assembly on the compiler directive. If the assembly is in scope (i.e. in the probing path) it will be found and it will be loaded. If you specify a full path, it will always be found of course.
It's not possible when compiling in code, to specify a folder where to look for referenced assemblies as there is when you compile on the command line with csc, so a full path OR a location in a probed path (so .NET's fusion finds it) is required.