It's definitely part of the reason, but not the main reason. It is not a sufficient or necessary condition, but more people adds more weight for concern when there already is a concern. All I really care about is if there is a plan of action, not simply hope on their part. There might be an argument to stop them in terms of visiting the country (as in, their intent is "hit and run" propaganda), but I can't see why I should call something a threat if it isn't associated with imminent violence.
Are you seriously missing how Hitler used violence to come to power? He toned it down a little bit after being imprisoned, but it's not like he came to power and then unleashed the violence. It wasn't anti-German belief that was ever the threat, it was the constant use of violence. Hitler basically got people to ignore the law (or rather, he got the right people to ignore the law), he didn't use the law of Germany to get his way.