I know Brian can but do the others or is it just at certain points.

I believe they make this joke on the show. They never clarify whether the others can understand him or not.

Chris can as well. The others do sometimes. Its never a for sure thing though.

Yes. I think the idea basically is that they can in fact understand him but they just choose not to pay any attention to him. Chris anyway has been shown to fully understand him.