There are many reasons why your pairing may not work out, whether it’s a person you don’t feel comfortable with or a temple that isn’t the right style for you.
Here is a message you can send to your buddy if you don’t want to continue with him or her:
Thank you for taking the time to show me around the temple, it is much appreciated. However, I think that this is not quite the right fit for me.
You can also let me know if there is anything I should know about your experience and if you’d like me to find you a new buddy by replying to your buddy assignment email.