I think hemp would be the best material for clothes, but in most places it's still an illegal plant.

Cotton and wool can at least be returned to the earth naturally. Cotton can be grown places where water shortages aren't an issue.

Personally the greenest option for me is trying to buy clothing made from nature textiles at a second hand store. I also wear what I own until it is basically rags, if a garmet gets a hole or a stain it becomes work clothing for when I'm doing dirty work. Obviously everyone on the planet cannot do that, but as it stands we already waste tons of clothing with fast fashion and many garmets are only worn a handful of times before being thrown away or even never worn or sold at all before becoming trash.

Someone already suggested hemp, but there is also other fibres like linen.

At the end of the day clothing would not be an issue at all, if clothes were made to last and worn accordingly. Unless you work in blue collar jobs, the wear on clothes is minimal and there is no reason why a set of shirts shouldn't last you a decade.