It wasn't as simple as just "he got fired". There was a bunch of dodgy shenanigans around how the GG handled the whole affair. There is a really good ABC podcast about it.

That's missing the rest of the story. It's true, Governor-General John Kerr participated in many of these groups, both CIA linked and not, but it's important to note that those front groups aren't just some talking club, they're the primary means of asset-building that the US intelligence community uses in leveraging local actors to act in America's interest. The same clubs, charity balls, conference meetings etc. were happening all over Europe, Latin America, and East Asia, with many of them becoming pro-American Heads of State, government bureaucrats, military officers, business leaders, news magnates, religious leaders, and in some cases actual far-right terrorists (especially in Italy, Greece, Turkey, Chile, Argentina, Brazil, Mexico, Indonesia, Japan, Republic of China).

If that's not enough, in 1977, the United States Deputy Secretary of State Warren Christopher made a special trip to Sydney to meet with Whitlam and told him, on behalf of US President Jimmy Carter, of his willingness to work with whatever government Australians elected, and that the US would never again interfere with Australia's democratic processes.

Lastly, Gough Whitlam himself denied the CIA was involved for most of his life because he was unaware that the then-Head of ASIO Peter Barbour had disobeyed orders to sever ties with the CIA and worked closely with the CIA in the overthrow of Allende. In an interview 8 months before Whitlam's death, the interviewer informed him that Barbour's betrayal and Whitlam lamented. Friendlyjordies had a video on the moment of truth (https://www.youtube.com/watch?v=4Dpx4o_uaGg).

It's one of those deep politics things like the Gulf of Tonkin Incident or the Assassination of Julius Caesar (not that the Dismissal is as monumental) in that journalists and analysts of the moment have little to work from except social connections and similar ideologies of the participants, but there's no definitive evidence that would be enough to satisfy a criminal trial or even a newspaper article. Even historians today aren't definitive one way or the other on the Dismissal, but its very likely that historians a few hundred years from now wont have those biases or pretensions and will just cut to the chase and ask 'who benefits from this?'