There is a report called "Agent Login History" that you can run.
If you use the Office Hours and also set your extensions to use Auto Switching then they will change status to DND which effectively takes them out of the queue at the end of the shift, and logs them in at the beginning of the next shift. This will also reflect on the report next time you generate it.