Haven't visited the forum for awhile, and just saw this posting now that there are 13 response pages. Might have an idea for you.
We see a lot of "heatform" fireplaces in our neck of the woods. Offered by a variety of manufacturers, the heatform was intended to create a flow of heated air into the room while the fireplace was burning. The masonry firebox was built oversize, to accomodate the heatform (a metal inner firebox), with an airspace between the masonry and the heatform. The theory was, heat from the fire would transmit through the metal heatform, warming the air in the airspace between it and the masonry. The heated air would rise and flow into the room through openings above the fireplace, pulling room air into similar openings to the room below the fireplace to be heated in turn. The photo in Roospike's post on page 4 of this thread shows a typical heatform installation, with two intake grills near the floor and two output grills above.
We've seen a few heatform installations where the intake openings in the room below the fireplace were omitted, and a single intake opening to outside air through the back of the fireplace structure was substituted. This outside air opening wasn't to supply combustion air directly to the fire, but into the airspace between the heatform and the masonry structure to be heated by the fire and delivered into the room. I think the theory was that heating air from outside the house and delivering it into the room would counteract room depressurization from the chimney updraft, while introducing pre-heated fresh air into the room instead of "stuffy" recirculated room air.
Whatever the theory, the outside-air heatform didn't work very well, and our Sweep customers complained of anemic performance when burning in colder weather, and cold airflow into the house when the fire wasn't burning.
If this fireplace is an outside-air heatform, your experience is going to be even worse: the outer convection shell on your insert will minimize heat transfer to the airspace around the heatform, likely warming it just enough to draw barely-warmed outside air into the room and counteracting the heated air output of your Summit.
Try blocking the air intake grill on the outside of your fireplace structure and see if that doesn't improve your situation. If this seems to inhibit the flow of combustion air into the Summit's firebox, remove the knockout plate and burn room air.