Using the default Video Conferencing app from the app store, we tried the Pixel 6 Pro and by default, the audio is played back by the loudspeaker. We couldn't make the app playback anything through the ear speaker.
The only thing i can think of that could make your phone do that is another app that uses the ear speaker by default is open in the background and is locking your device to use only the ear speaker. Try closing all open apps except the VideoConferencing one and see if this resolves the issue. If the issue continues, you'll need to look for and force-stop any special apps you may have installed that to your knowledge play back audio and play it through the earphone specifically
That being said, we have a version of the app in development that has dedicated buttons for switching between devices/modes.