If you decline the challenge, you will be penalized.
You have just 1 minute to accept/decline a challenge, else you will face the penalty automatically.