Does Replying to YouTube Comments Hurt Video Performance?

I have just been thinking this. Say my new channel was to get 20 comments and in the morning after sleeping I reply to them similar to checking morning emails. Wont this harm the channel if the viewers then come back to reply and obviously only watch 20 seconds or so of the video actually replying? What are your thoughts on this? Should we be replying to commenters during the early growth of a video? I know this is thinking deep into stats and how actions can affect growth. Really curious if this actually has an affect.