I feel ya, I like terraforming, and I'd love to do more projects towards making Bastion more beautiful, but people insist on interrupting me. I can get away usually because I have blink, but why am I obligated to fight you when I'm trying to build? Sure that makes me a coward, it also means I'm not stupid. I like fighting too, but some people are too tied up in the pvp all the time mentaility, I have other things to do, that's what makes the server more interesting.
If I ever have any free time, since I've been working mostly on kingdom projects, I'd be happy to work with you. I figure that I can fight off one person, and distract them if more people come.
I guess the solution is just to shrug it off, even if it is frustrating. Working in groups might help a bit.