Entity Framework 6 One-Way relationship using Code First

I have the following scenario:

1. Color Class

   public int ID
   {
       get;
       set;
   }
   
   public string Name
   {
       get;
       set;
   }
   
   public string Hex
   {
       get;
       set;
   }
   

2. Widget Class

   public int ID
   {
       get;
       set;
   }
   
   public int HeaderBackgroundColorID
   {
       get;
       set;
   }
   
   public Color HeaderBackgroundColor
   {
       get;
       set;
   }
   

Using Code-First, I am trying to create a one-way relationship between Widget to Color classes with the HeaderBackgroundColor / HeaderBackgroundColorID Fields.

normally i would do this in the config class:

this.HasOptional(r => r.HeaderBackgroundColor )
    .WithMany(m => m.Widgets)
    .HasForeignKey(fk => fk.HeaderBackgroundColorID);

but i am not intrested in adding a Widgets collection to the Color class. tried this:

this.HasOptional(r => r.HeaderBackgroundColor )
    .WithMany()
    .HasForeignKey(fk => fk.HeaderBackgroundColorID);

but that throws a validation error.

What's the way to do this right?