I think it is because countries invest effort in developing their economy, infrastructure, laws, etc. to serve their people and represent their citizen's shared values.
As a concrete practical example, let's say a country invests heavily in infrastructure paid for by decades of taxes and sweat. Shouldn't they have a say in whether a person who didn't invest in the infrastructure could reap the benefits of it? In addition, if there was no such thing as state borders, people might not ever choose to make investments in the common good.