how do you determine the cost of removing carbon

Estimate. Start with a low estimate for the social cost of carbon and see how the market reacts. At the same time, we can provide grants for carbon sequestration projects, but no subsidies for categories of solutions.

Does creating a new solar/wind power plant count?

No, solar/wind would also pay a carbon tax based on their manufacturing processes, though that would be a lot less than fossil fuel generation.

I'm not a fan of subsidies since those encourage "creative accounting," and instead prefer simple, quantitative penalties.

This is what the carbon offset economy

No, the carbon offset economy was supposed to be a way to allow creative accounting to limit responsibility.

If an org wants to install renewables to offset some of their energy use, then they need to actually use the energy to offset their energy use, not just tally it up. I don't care about generation numbers, I care about tons of CO2 and other emissions.

if the chance of getting caught is low people will still do it

Right, so increase the chance that cheaters will get caught. Set default emissions numbers to a high (but reasonable) number based on worst case estimates, and require orgs to prove they're emitting less. Do it for all imports and domestic industries alike so it's fair.

Then randomly audit after approval. If companies get caught, fine and revert to the high estimate until they prove they've fixed their accounting (perhaps after some number of years of correct reports). This should be highly automatable, and I'm guessing most domestic orgs already have high quality numbers.

That's a really simple solution since there's no complex adjustments based on local offsets, just number of tons emitted. The only tricky business is sequestration, and orgs would need to prove it's actually sequestered.