I'm not sure if it is entirely accurate to compare them in this way, as "Matrix" refers to simply the protocol, whereas "Signal" could refer to the applications, server, and protocol. That being said, is there any fundamental difference in how the Matrix ecosystem of federated servers, and independently developed applications compares to that of Signal that would make it less secure, overall, to use?
The most obvious security vulnerability that I can think of is that the person you are communicating with (or, conceivably, oneself, as well) is using an insecure/compromised application that may be leaking information. I would assume that the underlying encryption of the data is rather trustworthy, and the added censorship resistance of federating the servers is a big plus. However, I do wonder if there are any issues with extra metadata generation, or usage tracking that could be seen as an opsec vulnerability for an individual. Signal, somewhat famously, when subpoenaed to hand over data, can only hand over the date that the account was created, and the last time it was used. What would happen if the authorities go after a Matrix user? What information about that user would they be able to gather?
Because of its federated nature, Matrix shares tremendously large amounts of metadata between instances, even though the messages (and I believe the usernames you send to, but someone correct me on this) themselves are encrypted.
On the flip side, Signal requires you to sign up using your phone number. Some Matrix instances only require you to choose a username and be done with it.