Request song by voice does not use songs from my library

[portcanuck]

Plan

Premium

Country

Canada

Device

Pixel 2

Operating System

Android 10

My Question or Issue

When using "OK Google" to ask for a specific song it rarely plays the song I've liked in my library. Instead it usually opts for a different version of the song or from a different album. This is frustrating as this is usually used while driving so it ends up costing me data to stream the song instead of using the one I have downloaded. Couple this with automatically playing random songs after the requested song, my limited amount of data starts to take a hit.

I can't switch to downloaded only mode. Voice requests do not work in this mode. This has been my biggest annoyance so far since switching from Google play music.