I personally wouldn't care if it were like that, God knows I have bigger things to worry about in my life, and I think most people could handle it without any major problems arising....
But, you can't ever underestimate the kind of creeps that are out there in the world.....Would you really want to send your 8 year old daughter into a public restroom at a GreyHound station or something, where just anyone could be creeping around trying to look through the cracks in the stall door frame?
Granted, there's still a chance that you could be accosted by some "level three sex offender" type guy in any public place, but having some sort of "international rest-room revolution" where there were no gender restrictions on who could come and go, I think that would be inviting the creepers to creep out of the woodwork!
If it actually happened, like if the US passed a federal law universally banning "gender segregated restrooms", I give it two weeks until there's stories all over the national news about peeping toms, child-molesters, how "uncomfortable it makes me feel"....on and on and on
I had to take a shit one time really suddenly while walking down the street, I was with a friend of mine and we were on a street mostly populated by higher end apartments, nice restaurants and little yuppie, pseudo/Bhuddist trinket stores....The kind of stores that sell incense, glass beads, little brass statues of hindu Gods, shit like that....I knew there was nowhere on that street I could take a shit....So I ask my friend where the closest place that would let me use their bathroom was....
well, one street over there's a smoke shop/deli/convenience store place, no chance of using their bathroom, AND there's a Adult Bookstore right next door, which has public restrooms....haha...
My friend suggests I go use the bathroom in the bookstore but kind of gives me a smirk and tells me, "Be careful in there..."
Long story short, and I didn't know it at the time(he did), a lot of Adult Bookstores in the US are frquented by gay men and prostitutes who often use the bathrooms to "take care of business"....I had like three people follow me in there and I kind of knew what was going on....It was an interesting shit....had at least two dudes peek in on me! Which hey, I asked for it I guess going in there....when you gotta go