Just the act of refusing makes the act of seizing your phone legal or not. If you legally give them your phone by your own will, they are able to use all evidence they find in the courts. If you deny to give them your phone, and they seize it anyways and access it you have a valid path to throw the evidence they discover out as an illegal search and seizure of your property. I'm not a lawyer but that is the general thought process on denying them access to your property.

Edit: Just want to say this mostly pretains to United States law and similar legal structures. This advice is not applicable everywhere and you should research your countries rights and legal protections.