Country X with all it’s recognized borders becomes Objective Republic X and the government of that country ceases to own “unclaimed land”.
This question sorta belongs in the “how to make a transition” subforum, which we probably don’t have here. Before we go invoking Y, let’s consider how an Improper Government X can become POG X (I don’t mean “what actions can bring about this change in government”, I mean “given such a change, how does it happen”). The government will have a bunch of stuff that it improperly controls. Examples would include (developed) real estate and money taken by force from individuals, by court-ordered confiscation or general taxation methods. Some of this property is traceable, and can (and should) be returned to the actual owner, but some of it cannot be returned to the rightful owner. The remainder has become unowned property – it’s not limited to land – and the government finds itself the custodian of such property; its obligation is to dispose of that property if and when a valid claim is made. Going back to the good old homesteading days, the US government was the custodian for land that individuals could claim as their own, following certain objectively stated rules. This is the correct relationship between government and unowned land. Unreturnable property would then leave government control according to objectively stated and justified law: this does not mean sudden abandonment.
Country Y to the north is paying attention and decides to either claim the unowned land for itself or to encourage its citizens to move in and claim land, which if I’m not mistaken with an Objective immigration policy they would be permitted to do.
The latter, yes (it’s not clear to me that a foreign government automatically has the right to act as an individual with respect to country X). So there could be a large-scale Newfie immigration. However, they would have to compete with the native Michiganders for these claims, and I don’t think it’s reasonable to expect all of the newly available land in Michigan to be claimed by foreigners – that presupposes that Newfies are really clever and fast, and Michiganders aren’t. But it’s your hypothetical, to let’s assume this happens, eh?
Then these people could decide that they want to be a part of country Y again and announce that in keeping with O’ist principals they have the right of property and can do so, thus effectively granting ownership to of their property to Country Y.
Not exactly (as in, “at all”). When a person from country Y acquires property Z in country X, they don’t get to change what country Z is in. Z is in country X. Well, they can decide or wish that they were still part of country Y, but wishing doesn’t make it so. And none of this gives any support to the idea that a government should own all property not owned by an individual. Furthermore, there is no general and conventional recognition of such a position, and even though there is very little unowned land in the US, as far as I know it is not the case that the government owns all unclaimed / unowned land, so you can’t even say “even though it’s wrong, that’s just the way it is” – I’d need some substantial evidence to persuade me that it actually is that way. US law actually prohibits the government from claiming ownership of land off of the planet.