We know from the movies that Stan Lee driving a truck couldn't budge it, nor could Stark's Iron Man glove when he was wearing it. To me that signals that inanimate objects being wielded by others can't move it. However it was also on the floor of the S.H.I.E.L.D. Helicarrier at one point so that could be indicative that intent is important too. Hence I'd posit that Thor could leave it on a truck without the owner knowing, and it could be driven around, or leave it on a elevator and the guy calling the elevator from the ground floor isn't going to end out staring at wreckage.

Vision isn't being wielded by others. One can also argue that the absence of darkness in Vision made him worthy, however, he didn't get powers, so I feel like that rules that theory out. I think the hammer seeing Vision as both not alive along with not acting upon it as a vessel of something/someone else attempting to move it is the more logical take. Basically Vision is the aforementioned elevator in this scenario. Ground Floor, Hammers.

Or the writers could be implying something in the hammer handover scene, and poorly writing elsewhere. Take your pick really.