A book about an old, haunted house by not horror.

I love hearing stories about old homes and the current owners digging into the past and learning all the secrets of the families before.
The story could be set anywhere but in my mind I'm picturing an old Victorian mansion or an old abandoned castle. The type of story where you find hidden letters or photographs under floorboards that prompt the main character to do some digging.
The story can have some paranormal aspects but ideally not a horror story, something more historical fiction. A thriller would be okay too.