Thats going to depend a lot on the circumstances and local law. Here in NC, CXS and NS were deeded the property for the tracks plus 15 feet on either side in 1902, I believe it was. They own the property the rails sit on all over the state consisting of the land under the rails and 15 feet on either side. There was an issue several yeras ago where the railroad closed the railroad crossing on a busy street to replace the crossing, then something wasn't available they needed to fix it and they kept the road closed for nearly a month waiting on whatever it was to be delivered. The Mayor sent the railroad a letter saying that city ordinance made it a city offense to close a street crossing without a city permit and if it wasn't fixed within some deadline he gave them, the city was going to start fining CSX $250 per day until the crossing was open. The railroad attorney replied in writing something to the form of, "Dear Mayor, CSX transportation owns the railroad crossing and the land 15 feet on either side, and in fact, Royal Avenue which runs parallel to our tracks is on CSX property. If you do not cease and desist, CSX will bring in our earth-moving equipment and we will remove Royal Avenue from our property." It was all crickets from the mayor after that.
Usually railroad property is clearly marked with signage. I had to contact CSX several years ago and have a railroad police officer escort me to take pictures at the CSX railyard for the railroad clubs werbsite. After 9/11 you'll certainly get unwanted attention from the law if you're in a place you shouldn't be and taking photos. I'd do as I did and contact the railroads police department if you want to take photos, even from legal right of way. Otherwise they'll likely show up and "check you out" to make sure you aren't a terrorist or saboteur or something.
While you can legally take photos from public right of way anywhere except in certain areas like a military base or if you're in the way of a LE investigation, you'll surely get attention from the law if you don't advise someone in authority what you're doing before hand when it comes to photographing railroads, ports, important buildings, bridges, etc.
I think though that "innocent trespass" is hardly ever enforced. I see kids walking on the railroad tracks here in town every day walking home as the schoo is right beside the railroad tracks, which is illegal. When I was a cop I'd advise the kids it was trespassing and asked them to leave railroad property until I was "strongly advised" by my supervisor to leave them alone, that's the railroad police's problem not mine. Unless the rairoad police notifies the locals about vandalism or theft on RR property the locals don't really care from what I've experienced.