Can neighbor sue me for land survey cost?

Around 6 months ago I installed a picket fence between my property and my neighbors. At the time we both agreed where to put the fence. A month after he has a survey done and it shows that I am on his property in two spots about 6 inches but I'm also on my property in some areas 6 inches. I offered to move the fence onto my property but he told me he was going to take me to court and if I touched the fence it would be trespassing. I have received a letter stating he is going to sue me for the full cost of his survey 850$, his legal fees, and for us to move the fence. I don't mind moving the fence but can he make me pay for a survey and legal fees? I've tried talking to him about it but he insists we pay for the survey and says we can not move the fence until he takes us to court for the survey cost.