Its only refined and simple carbohydrates that can cause trouble.  
A person could eat all the brown rice or whole wheat pasta he wanted and would not get fat.  Complex unrefined carbohydrates are just not dense enough in macro-nutrients to cause obesity.
Foods with high AE (assimilation efficiency) are what causes weight gain:  Meats, oils, whole milk, eggs, nuts, refined grains, simple carbohydrates.
Most unrefined plant foods have an AE of 15-30%.  Meats are around 80% and up.