It's not that simple. Each time that an agency was dismantled, it was always replaced by something else. If we were to look at the history:

Aboriginal and Torres Strait Islander Commission - established by Labor, dismantled by Liberals
Ministerial Taskforce on Indigenous Affairs - established by Liberals, dismantled by Labor
Office of Indigenous Policy Coordination - established by Liberals, dismantled by Labor
National Indigenous Council - established by Liberals, dismantled by Labor
Department of Families, Housing, Community Services and Indigenous Affairs - established by Labor, dismantled by Liberals
Indigenous Advisory Council - established by Liberals and still exists
National Indigenous Australians Agency - established by Liberals and still exists

Looking back through the history, it could be argued that Abbott was responsible for the heaviest dismantling, but it wasn't really connected to election cycles.

The current structure under the NIAA seems to be the most detailed, transparent, and accountable body that we have had so far. The Corporate Plan and Reconciliation Action Plan are worth a read. It definitely makes you wonder why we need a Voice when the plans, structure, and hierarchy is already in place.
https://www.niaa.gov.au/sites/default/files/publications/niaa-rap-2022-25.pdf
https://www.niaa.gov.au/sites/default/files/publications/niaa-corporate-plan-2022-23_0.pdf