I have messed with perlite for other projects, you couldn't pay me enough to deal with trying to keep a liner centered in the flue while trying to pour that stuff down from the top. I don't think you have the room to make it worth doing that either.