Yeah, as mentioned by Aielman15, Fighters tend to not have a good Charisma. So it's actually not just that Martials only have skills, but they are usually worse at those skills compared to Casters (barring Rogue, who is only outclassed by Bard). (Note: I use Martial here to mean "class without the Spellcasting pr Pact Magic features in their base class" and Caster to mean "class that counts fully for Spellcasting multiclassing and Warlock.)

Martials tend to excel in Strength, Dexterity, and/or Constitution and usually be middling at best in other stats. Since there is only one skill associated with Strength: Athletics, while, other stats have at least 3 each (except Constitution) a Strength based character will be worse at skills than other characters. Of course the relative strength of the different skills will vary depending on your table, but I think we can at least all agree that the Charisma based ones tend towards the top. They also don't get any more Proficiencies or Expertises than any other class (Bard and Rogue are again the exceptions).

In essence, pointing to the skill system doesn't really help, as Martials aren't usually any better than Casters at them, or are even worse in some cases.