There's definitely a complete skill in itself in sightreading - translating from a note on a page to a finger position on the instrument without any conscious effort involved. It needs a lot of practise, but that's all. It's no different really from translating a number on a tab to a finger position, just that it takes a bit more work to figure out where the position is.
Just to add to that... this approach is what many people start with, but it encourages you to treat music notation as tablature, rather than sound. A dot does not represent a finger position but a sound (which can be achieved by a finger position on some instruments, multiple positions on other instruments e.g. guitar, violin etc, or requires a finger position plus other factors on an instrument like trumpet flute etc). To read music as sound there is no substiture for singing, where you have to hear what the note is and then produce that sound.
Edinburgh Recording Studio Windmill Sound