Moq & EF6 - Mocking EF6 Remove method not removing an object

I am trying to mock my EF6 DbContext and it all works for Add, Update, Find methods. But it's not working for Remove method for unknown reason.

In theory, after removing, the Students collection should have only 1 return left. But it keeps returning the Count - 2.

I put 3 Moq.Verify checks to ensure that all methods are called and they are executed indeed. But it's not actually removing the item from the Students Collection.

If I commented Assert.Equal line which check the counts, the whole test passed.

Delete XUnit Method

[Fact]
public void Delete()
{            
    Mock<DbContexts.MVCWebAppDbContext> dbContext = new Mock<DbContexts.MVCWebAppDbContext>();
    IStudentsService studentService = new StudentsService(dbContext.Object);

    var students = new List<Student>()
    {
        new Student() { StudentID = 1, RefNo = "12456343", FirstName = "John", LastName = "Smith", DateOfBirth = DateTime.Now.AddYears(-10), DateCreated = DateTime.Now },
        new Student() { StudentID = 2, RefNo = "87984564", FirstName = "Pete", LastName = "Luck", DateOfBirth = DateTime.Now.AddYears(-20), DateCreated = DateTime.Now.AddDays(1) }
    };

    var mockSet = new Mock<DbSet<Student>>();
    mockSet.As<IQueryable<Student>>().Setup(m => m.Provider).Returns(students.AsQueryable().Provider);
    mockSet.As<IQueryable<Student>>().Setup(m => m.Expression).Returns(students.AsQueryable().Expression);
    mockSet.As<IQueryable<Student>>().Setup(m => m.ElementType).Returns(students.AsQueryable().ElementType);
    mockSet.As<IQueryable<Student>>().Setup(m => m.GetEnumerator()).Returns(students.AsQueryable().GetEnumerator());

    mockSet.Setup(m => m.Remove(It.IsAny<Student>())).Callback<Student>((entity) => students.Remove(entity));

    dbContext.Setup(c => c.Students).Returns(mockSet.Object);

    int idToDelete = 1;

    dbContext.Setup(s => s.Students.Find(idToDelete)).Returns(students.Single(s => s.StudentID == idToDelete));

    // call delete method now
    studentService.Delete(idToDelete);

    // 1 object deleted, it should return 1
    Assert.Equal(1, students.Count());  // <----- Error here

    dbContext.Verify(s => s.Students.Find(idToDelete), Times.Once);
    dbContext.Verify(s => s.Students.Remove(It.IsAny<Student>()), Times.Once);
    dbContext.Verify(s => s.SaveChanges(), Times.Once);
}

StudentService.cs Delete method

MVCWebAppDbContext _context;

public StudentsService(MVCWebAppDbContext context)
{
    _context = context;
}

public int Delete(int id)
{
    var objToDelete = _context.Students.Find(id);

    if (objToDelete != null)
    {
        _context.Students.Remove(objToDelete);
        return _context.SaveChanges();
    }

    return -1;
}

Could you guys please help me with this Remove method mocking?