EF6, lazy loading not working as expected

Using EF 6, Lazy Loading Enabled is set to True in the model. Here's an example of my problem:

var agent = context.AgentDetail.Where(a => a.Agent.GroupCode == "1234");

Running that will return 5 results. If after that I run (for the purpose of testing only)

var code = agent.FirstOrDefault().Agent.GroupCode;

I get a null reference exception because Agent is null.

Here are my entities:

    public partial class AgentDetail : Entity<int>
    {
        public Nullable<System.DateTime> Date { get; set; }
        public string Name { get; set; }
        public decimal Balance { get; set; }
        ...

        public virtual Agent Agent { get; set; }
    }

    public partial class Agent : Entity<int>
    {
        [System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Usage", "CA2214:DoNotCallOverridableMethodsInConstructors")]
        public Agent()
        {
            this.AgentAspNetUsers = new HashSet<AgentAspNetUsers>();
            this.AgentDetail = new HashSet<AgentDetail>();
        }

        public string GroupCode { get; set; }
        ...

        [System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Usage", "CA2227:CollectionPropertiesShouldBeReadOnly")]
        public virtual ICollection<AgentAspNetUsers> AgentAspNetUsers { get; set; }
        [System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Usage", "CA2227:CollectionPropertiesShouldBeReadOnly")]
        public virtual ICollection<AgentDetail> AgentDetail { get; set; }
    }

How could it give me 5 results in the first query, then? I can't figure out what's wrong here, any help would be appreciated.