Music dips and sounds poor whilst on call

[SaltyD191]

So sometimes I like to listen to music whilst I'm on a call if I'm on a long call mainly.  However, when I do, the background noise coming from the other end makes the music dip in volume and it gets really jittery and it just sounds really bad.  It only happens when there's sound coming from the other end.  Is there a way for me to stop this from happening?  Or could it just be the earphones I'm using?