"The * field is required" when the EF object has the field already

My datamodel is the following:

[Table("Jobs")]
public abstract class Job
{
    public int Id { get; set; }
    public DateTime? StartTime { get; set; }
}

[Table("PbnJobs")]
public abstract class PbnJob : Job
{
    [Required]
    public virtual PbnSite Site{ get; set; }
    [ForeignKey("Site")]
    public int SiteId { get; set; }
}

[Table("DomainJobs")]
public class DomainJob : PbnJob
{
}

public class PbnSite
{
    public int Id { get; set; }
}

Site is just a foreign key object.

My code looks the following:

var context = new LmDbContext();
var job = context.Jobs.First();
job.StartTime = DateTime.Now;
_context.SaveChanges();

The thing is, the above code throws the exception of type

System.Data.Entity.Validation.DbEntityValidationException

, saying:

"The Site field is required"

However when I add:

var site = ((PbnJob)job).Site;

Making the code:

var context = new LmDbContext();
var job = context.Jobs.First();
var site = ((PbnJob)job).Site;
job.StartTime = DateTime.Now;
_context.SaveChanges();

Then the context is saved just fine.

What's the issue here?

EDIT: By the way I'm using proxies and lazy loading.