Can't use deferReply and permissions at the sametime

[PenT1x]

When I use deferReply and have an permission set. Every time there is someone who doesn't have the permission it gives me this error:

Error [InteractionAlreadyReplied]: The reply to this interaction has already been sent or deferred.