There is a distinction to be made between a floating abstraction and the arbitrary.
You can arrive at a floating abstraction in your mind, without accepting any arbitrary statements, by accepting statements without judgment, or holding concepts before you have tied them to reality.
E.g. Someone first introduces you to "justice" before you have the conceptual framework or experience for you to fit it into your hierarchy of concepts. As a word, a part of the language, you keep a tenuous hold on it in the framework of semantics, without really understanding (before validly forming) a concept of justice. So you speak with other people using the word "justice"... perhaps accept what other say about "justice" and what it means and how it relates to other concepts, but until you go through the exercise of thinking, and until your concept finally has some attachment to a part of your (valid) knowledge, it remains "floating". Holding a concept as floating temporarily is not necessarily a vice... sometimes it is a necessary stage, prior to your integration of it. Rationally you "should" (according to prioritization of time, effort, and your value hierarchy etc.) decide how important the concept is to your life, and if it is important, to undergo the process of thinking required to anchor it to knowledge.
Observe the statement about "justice" might not have been arbitrary, indeed it could have been true. Suppose, having never really thought of politics or even ethics, you heard it directly from, say Leonard Peikoff, and your closest family and trusted friends, all of whom told you they thought very long and hard about it, and even provided you with an explanation tied to reality, which, unfortunately you could not fully understand... yet. You can see some basis but cannot form all the connections. You also have independently judged the quality of thought of these people based on other claims they have made. Here there is at least some evidence for the statement, i.e. that it is not arbitrary, and sweeping it from your mind would be a mistake. [[If you insist on personally re-investigating the sum of human knowledge in every minute detail ALL THE TIME, and expecting omniscience for validating knowledge, you would never take any medication, step on any plane, or do anything which involved ANY INFINITESIMAL LEVEL of dependence or trust on others knowledge of reality. Rational trust in something someone says is not blind faith in a statement which is arbitrary, but an assessment of everything you know about, reality, the person, and what they have said]] Here, the concept "justice" could be a floating abstraction for a time, but with the kinds of non-arbitrary statements of Peikoff, you could start thinking about it, chewing and building the ladder of abstractions connecting justice to reality until the concept is no longer floating.
In the final equation the hierarchy of knowledge is yours, thinking is something you do by yourself, and the knowledge you build must be built by your own mind.
In a sense, a floating abstraction is not (yet?) a validly formed concept (contextually for you), but there is enough evidence not to dismiss it altogether...i.e. that although you have not yet gone through the process of conceptualization and integration, there is some indication or evidence that it is a valid concept capable of integration. Of course you might conclude after enough thought and weighing of evidence that a floating abstraction is actually an invalid and arbitrary concept.
The arbitrary is not so much a floating abstraction as an invalid concept, a concept for which no evidence exists, i.e. which was reached entirely arbitrarily. This bespeaks Rand's genius in her naming of it.