What should I do with orphan files?

[dxburge]

As part of my testing, I deleted my whole media library using the "Manage Media" screen and Bulk Actions Delete.
However, I've noticed that there are still some media files in both /media_files/original/user subdirectories as well as the /media_files/uploads directory.

I'm assuming I can simply delete these files manually, or should I wait for some garbage collector process to clean up these folders?
It also makes me wonder if there are orphan database records that should be cleared as well.
If anybody knows what should be done, please let me know.