In basketball how come the offensive player can scoot back up against the defensive player and push him?

like i was watching a playoff game yesterday and i saw an offensive player nudging the defender with his back towards him, and he basically push the defender back a little more and a little more until he was right next to the basket and then dunked it. why is that not considered an offensive foul if hes pushing the defender out of the defenders fixed spot on the floor? i see this happen all the time and it just seems like it would be considered a foul