Surprisingly, this is a really easy answer and don't panic, this is not something that will hurt your professional image out there.
I would just simply send the agent a quick note and say thank you for the opportunity to discuss the story but you have decided to hold off on the submission right now. There is certainly no need to go into detail about things. Just let the person know.
I know for myself, that I keep a log of submissions. If someone decides to sign with another agent or not do anything, I close out the account in my data base. It doesn't bug me. Things happen.
And honestly, the only way it would hurt your career is if you:
- Emailed the person and was abusive or rude.
- Got on the internet and started slamming the agent
- Tweeted, blogged and anything else about the agent.
Get the idea?
Thanks for the question.