Stop Spotify from taking control of headphone button ?

[FluffyXai]

I've only been able it find posts about trying g to get Spotify to work with external headphone buttons. My problem is the opposite; it works too well. 

I use both Spotify and Aimp to listen to different sets of music. A lot of the things I listen to aren't on Spotify, and oppositely, Spotify makes it easy to search for songs I like. But my problem is whenever I'm using another music app, when I press the button on my headphones, it immediately switches to Spotify (then proceeds to pause/skip as I asked) 

I've checked the other apps and they are set up to be able to make use of the controls, and I even tried using other apps like Smart Key to control it. But Spotify could tunes to assume all control. Any help?