Spotify Connect switches to phone automatically

[GCWright]

OK this isn't a major problem but it is kind of annoying... I've found that often when I'm listening to the desktop version of Spotify on my laptop, if I open Spotify on my Android phone (e.g. so I can download tracks to an offline playlist on the phone), the box that says 'You are listening on (name of laptop), continue/listen on this phone', will pop up but immediately disappear and the music will suddenly start playing on the phone instead of my laptop. Is there a way to stop this happening?

Thanks in advance.