Why does cold air rush in when i open the door on a chilly day?

Inside our house it's warmer than outside, so the pressure in here is greater due to the kinetic energy of the particle motion then the pressure out there. Wouldn't the warm air rush OUT to equalize it?