Its only refined and simple carbohydrates that can cause trouble.
A person could eat all the brown rice or whole wheat pasta he wanted and would not get fat. Complex unrefined carbohydrates are just not dense enough in macro-nutrients to cause obesity.
Foods with high AE (assimilation efficiency) are what causes weight gain: Meats, oils, whole milk, eggs, nuts, refined grains, simple carbohydrates.
Most unrefined plant foods have an AE of 15-30%. Meats are around 80% and up.