What sets the glide angle is the aerodynamic efficiency which is total drag to total lift ratio. If your airspeed (v) is constant, the rate of decent is v*sinx, where x is the glide angle.
The rate of energy loss to drag (P=d*v) is compensated by potential energy loss (P=mg*v*sinx):
d*v = mg*v*sinx.
Since the rate of decent is constant it requires that the lift balance the weight:
L = mg
Therefore, we plug this to the 1st equation and cancel out the "v":
d=L*sinx
or
sinx = d/L
(I cheated a little with the angles for simplicity, but this holds well for small angles)
Weight: we see that weight is not directly involved. It is indirectly involved by increasing the induced drag contribution to the total drag.
Now, for gliders you do not necessarily need the best glide angle. For souring on vertical winds, what you need is a slow rate of decent - slower then the velocity of the rising air, not the best angle. So your angle might be poor, but it is achieved at a very low velocity. This is how para-gliders work. They just want to sour, not cover a distance. An aerodynamically efficient fighter can have a shallow glide angle, but at very high velocity, which also mean very high rate of decent.
And one final note: Frontal area has little to do with the drag. Its not even a rough indicator, unless you are just talking about scaling up an identical smaller model.