If we assume that the smell of smoke in the house means that smoke exiting the top of the chimney is entering the home then the house, as an envelope, is below outdoor pressure and smoke is being driven in by the higher outdoor pressure. The most obvious reason for that is that the combustion air is being extracted from the building, but warm air passively exits from the upper stories, and air is blown out by kitchen fans, bathroom fan, clothes dryers, and of course the biggest of all exhaust fans, the attic fan. For cubic foot that goes out a cubic foot must come in somewhere.
Supplying outside air to the appliance has the advantage of raising the building pressure and delivering the air where it’s exiting. The fire won’t mind a little smoke smell. That said, the house does need some air intake rate, and that means solving the smoke problem to be completely rid of the smell.
Not using a barometric damper maximizes the chimney’s draft, but it also makes it difficult or impossible to regulate the amount of air passing through the firebox, and the oil fire will become sooty as outdoor temperatures rise unless it’s set for up for so much excess air that it’s viciously inefficient in cold weather. Using a barometric damper will dilute the stack smoke with the end result being less air up the flue to achieve a stable combustion. The secondary question of “why is smoke sinking” may be easier to resolve if combustion is more consistent. There will always be cold still air that smoke sinks in, but if that’s an every day thing the site isn’t suitable for burning solid fuel.
It’s normal for people to try to burn wood more slowly than it will burn efficiently and end up with a smoky fire. That smoke is vaporized solid fuel going up the flue, it should be burned up and the flue gas should be hot enough to rise.
It’s also normal to buy enough flue to meet code and then discover that the way wind wraps the air around the structure entrains the flue gas, and e gthe concealer smoke pipe has to be taller to get the smoke up out of the recirculating boundary layer. The air on the downwind side of a house is a whirlpool on its side, that recirculation can take air from the stack and bend it around and deliver it rising up from the ground along the wall. A taller stack can put the smoke up above the layer of the air flowing over the roof that recirculates on the downwind side.
All this assumes the stack isn’t leaking smoke directly into the home. If it’s all outdoors and exposed that’s easy to visually confirm. If it’s concealed in whole or in part that’s not a sure thing. It can be tested.
There’s also a tendency to burn wood that isn’t dry enough and make creosote and it does smell like smoke, and it drools out from pipe joints. If the smoke smell doesn’t need a current fire, it’s something to look for.