You should only have 1 provider registered under an invariant name in the machine.config. It will always give an exception if there are two or more for Firebird client. So remove the one that's not used (e.g. the older one, as you installed 2).
For firebird embedded, if I recall correctly, you have to store the embedded dll in the application's executable folder. Additionally, it has to know about the Firebird client dll, as our runtime works with DbProviderFactories.
There are two ways a factory of an ADO.NET provider (such as the Firebird client) is registered in the DbProviderFactories: through machine.config or through the application's app/web.config file. If you don't want to run an installer on the machine where you're deploying, you can copy the firebird client dll into the app's executable folder and register the factory in the app's web/app.config file. For an example of how to do that, see the LLBLGenPro.exe.config file which has a section (commented out) for that. I've copied it below
part of your app's app/web.config file
<system.data>
<DbProviderFactories>
<add name="Firebird Client Data Provider" invariant="FirebirdSql.Data.FirebirdClient" description=".Net Framework Data Provider for Firebird" type="FirebirdSql.Data.FirebirdClient.FirebirdClientFactory, FirebirdSql.Data.FirebirdClient, Version=5.3.0.0, Culture=neutral, PublicKeyToken=3750abcc3150b00c"/>
</DbProviderFactories>
</system.data>