Usually had it within an 1/8 th of an inch from fully closeond. Flames are something I also wondered about. Seem like even when fully/almost fully dampered and basically smolering, I still get one or two spots burning off the wood. Dont know if it should be.
In a modern EPA stove you should not be able to fully damper down the fire until it just smolders. What you should see is that the wood itself has very few flames but you will see flames in the top of the firebox. If the stove is really hot it may even look like the secondary burn tubes are like little gas burners. The way you describe it, I would usually say your wood is not dry enough. However, the other option is that you are reducing the air too early before the stove is hot enough to sustain the secondary burn. Try the thermometer on top and get the stove to at least 500 F before turning down the air that much.