Dog walking and unfriendly neighbor

Hi All,

I have a big dog that I walk around two or three blocks everyday.
I am a very considerate dog owner: I *always* clean up after my dog and I make sure that my dog walks on the curb side at all time. I have even taught my dog to pee on the street.

Recently one guy has been harassing me every time I passed in front of his house with my dog. He's always yelling at me asking me to walk my dog only in front of my building. Lately he threatened me that he would take physical actions in addition to his verbal aggression.

I know that some other dog owners don't clean after their dog and understand the frustration of that guy when he sees the damages on the sidewalk. Technically the sidewalk doesn't belong to him. Besides I am not doing anything wrong. What can I do to make him understand that he's talking to the wrong guy and that I will not change my itinerary?

Any input would be very much appreciated.

Thanks,
Tennisfan