Poltergeists initially have the condition Invisible, and cause damage by throwing furniture. If a party member has the condition See Invisibility or Volo's Ersatz Eye, they become Revealed for one turn.
Poltergeists are only drawn into battle if they are revealed, otherwise they can be ignored entirely (although they continue throwing furniture).
They can fly down from the upper levels of the estate, so it may be best to wait for them to come closer rather then traversing into areas where Cursed Skulls are still active.