You can dislike Non-binary people and not be a Transphobe
You can dislike the idea of non-binary; how you act depends on if it’s transphobia or not.
There is no issue to me with someone who dislikes non-binary people. Someone who dislikes non-binary people that is their right, to hold their opinion. This dislike isn’t something I find inherently transphobic. This dislike I view as something that is more based on the fact it’s still — at least in western colonial lands — a new phenomenon that people have to get used to. I state western colonial areas because of the number of areas originally used to have identities for people outside of the gender binary. So when does a dislike of nonbinary people become transphobic?
Regardless of your dislike — how you treat people is key.
If you keep your dislike to yourself, treat people as people with feelings and opinions, regardless of their gender identity and pronouns, then that is not transphobic. Nor is your dislike — it is likely one born from misunderstanding or unsure as to if there are any effects on you. Hint — someone else being non-binary does not affect you, beyond asking you to remember their pronouns.
If however, you feel the need to police another individual, and to try and police how they present. Suppose you feel a need to ignore pronouns — something that is deeply personal, and a basic level of respect for anyone. If you feel the need to take away a person's rights because of their gender identity and pronouns. That? That is transphobic behaviour right there. I would ask what makes you want to police something that isn’t their choice and doesn’t affect you beyond needing to remember pronouns and be a decent human being.
Your actions make all the difference between your dislike being just a dislike or transphobia.
Remain respectful, treat others with respect, utilise the pronouns they ask you to, and educate yourself on what it is to be non-binary. You won’t ever be called out for your dislike, as it won’t be transphobic. Attempt to remove rights, be disrespectful and make yourself feel comfortable by putting other people in boxes that they have told you they don’t fit in? That — that is transphobic behaviour, and it is on you to fix that. We can educate and explain, but you have to be willing to listen.