Tom,
     Yes you should remove the old nests and clean out the boxes as much as 
possible to eliminate ectoparasites.  Often second nestings of barn swallows 
and phoebes which use the same nest again are totally infested with lice.
       Bluebirds sometimes lay white eggs so if you had a bluebird on the 
nest that's whose eggs they are.

Paul Woodward
Fairfax City, VA