Should I include Entity Framework in my class library or directly into application?

EntityFramework just has to be "installed" once, at Project.Library.Models. If you encapsulate the context object in a Data Access Layer and use common code-first objects as your DTOs, you won't need a direct reference anyplace else.

Your programmes, websites, etc. will automatically include the required DLLs if you provide a reference to that assembly. Be aware that Visual Studio won't recognise the transitory dependence if you are using EntityFramework.SqlServer, thus you will need to do the following actions:

class Include
{
    private Include() { }
    private SqlProviderServices IncludeSql;
}

Somewhere in the models project to ensure that the DLL is also copied.