Records that are going to be cascaded must have a foreign key back to the record being deleted in order for the cascading delete to function. You consequently remove a Contact record from your example. The cascade functions because there is a foreign key from ContactPhoneNumber to Contact. The cascade ends since there is no foreign key from PhoneNumber to ContactPhoneNumber (the foreign key travels the opposite way).
This is due to the fact that you described the relationship as one of many to many. If a ContactPhoneNumber is deleted and then its related PhoneNumbers are deleted, there may now be other ContactPhoneNumbers that don't have a valid PhoneNumber, if you consider trying to execute a cascade deletion on your model as you would like (Because there can be many ContactPhoneNumbers to one PhoneNumber). The process would then proceed when these were eliminated. This cyclical cascading is unpopular with databases.
Why you require the many-to-many relationship is not totally clear to me because, if you really need it, you won't be able to do a cascade on remove. If you can maintain your connection:
The cascade might then be configured to function how you want it to by using the formula 1 Contact - * ContactPhoneNumber 1- * PhoneNumber.