A look back at the past 15+ years of FSO

[RedBeard]

I've been doing some data analysis on the past 15+ years of FSO and thought I'd post some of what I've found here.  All of this is based on FSO log data from the AHEvents site.  Here's a look at participation since Jun of 2009.



As you can see, there's some anomalies in the data.  I believe the large spikes are scenario events that were somehow logged as FSO, even though they weren't.  I dropped some logs from the collected data that appeared to be other events like King of the Hill as well.  Still, it's probably 95% good data.

The trend doesn't look good.  I would like to start or renew a topic elsewhere to discuss how to address this.

It shows a decline in population and total flight time.  Mostly, this is understandable as FSO used to be a single life event, but I didn't see a jump in total flight hours after multilife was enabled, which surprised me.  We do see an increase of sortie rates with multilife, but again that hasn't seemed to have much impact on total flight hours.  I may have to go back to check my data.

Anyway, if you are interested, I'll be posting more info here.

[fudgums]

Before the thread turns sour, we should be thankful for the CMs and the job they do/did. It’s tough.


And also 450 pilots were the days.

*insert The Wire Quote about the good ole days

[hazmatt]

Anybody know what caused that big drop at the beginning of the chart? The rest seems pretty gradual but that one was pretty drastic in relation to the rest of the chart.

[RedBeard]

Sorry, I didn't mean to turn things sour, nor did I mean to imply any fault with the CMs.  My original intent was to look at how to make missions more effective, but there are always other things you can learn along the way from the data you get.

In particular, I wanted to look at bomber effectiveness.  There's some data missing from the logs to get a real look at effectiveness, but the data does show the following:

Plane	Total Sorties	Sorties With Dmg	Total Landed Sorties	Sorties Landed With Dmg	Chance to Make Target	Chance to Destroy Target and Get Home
A-20G	2499	1884	530	450	75.39%	18.01%
Ar_234	508	404	390	343	79.53%	67.52%
B-17G	5373	4714	3431	3222	87.73%	59.97%
B-24J	3803	3016	2616	2347	79.31%	61.71%
B-25C	7465	6063	3485	3026	81.22%	40.54%
B-25H	457	310	92	72	67.83%	15.75%
B-26B	2796	2274	1902	1696	81.33%	60.66%
B-29	2932	2813	1270	1257	95.94%	42.87%
B5N2	1153	350	217	72	30.36%	6.24%
Boston_III	3157	2435	1353	1145	77.13%	36.27%
C-47A	74	2	21	0	2.70%	0.00%
D3A1	2760	1841	451	300	66.70%	10.87%
G4M1_Model_11	2240	1454	839	613	64.91%	27.37%
He_111H	3243	2431	1956	1686	74.96%	51.99%
Ju_87D-3	3702	2626	564	403	70.93%	10.89%
Ju_88A-4	11119	9181	5820	5221	82.57%	46.96%
Ki-67	2208	1663	924	742	75.32%	33.61%
Lancaster_III	937	873	519	509	93.17%	54.32%
Mosquito_Mk_XVI	276	201	144	119	72.83%	43.12%
SBD-5	3465	2454	883	625	70.82%	18.04%
TBM-3	1619	755	353	172	46.63%	10.62%
Tu-2S	2094	1720	740	640	82.14%	30.56%

There's a number of things we don't have in the published log data that would help with analyzing bomber  effectiveness.  These include formations used or not, load out carried, what ordnance was dropped and when, and who did damage, but didn't destroy a target.  I understand that some of this is actually available, but not published, and I may take a crack at exposing more of it at some point.

[hazmatt]

I think this number is way too high! lolz
                                                                               Chance to make target      Chance to destroy target and get home
G4M1_Model_11   2240   1454   839   613            64.91%                                27.37%

I remember one fso back in the day were I drew the short straw and had to fly a formation of these deathtraps and we all got slaughtered before ever getting sight of the target!

[AKKuya]

And also 450 pilots were the days.

I remember when we had 600 pilots and we were planning for 4 objectives to defend and attack.

[RedBeard]

Anybody know what caused that big drop at the beginning of the chart? The rest seems pretty gradual but that one was pretty drastic in relation to the rest of the chart.

I suspect it was an anomalous set of data.  I have found numerous cases where the data just doesn't jive.  Pilots die without taking off.  Bomber formation pilots die after only two kill claims.  Pilots take off more than once.  Some of this may be due to when logging for the event is turned on or it might be data getting dropped as it is transferred from the servers to the AHEvent site or it might be logging is a low priority item on the server that was sometimes overloaded and couldn't keep up.  Real-time programming is an art and the data we get is mostly all we have and we just need to accept that. 

There might be cases where we lost a server or internet mid Frame and noone returned after it came back.  I do remember times when large chunks of pilots would disco.

There might be cases where the event design just didn't work out well and it was a slaughter that was over quickly.

There are some cases where multiple logs were produced for the same frame, one log for Axis, one for Allied, and I haven't checked yet to see if my data set collection has merged those two correctly or not.

I will say that there are possibly still some non-FSO event data mixed in and that may be the spikes up or down.  I have added some sanity checks along the way as I look at data to help filter out stuff that doesn't belong.  Things like, FSO is Friday Squad Ops, so if data is dated on a Thursday or Sunday, it's probably not FSO data.

The big V around late 2010 is really just a down spike with not a lot of event data on either side of it. 

What I have at the moment is data on 177 Events (months), 536 Frames, 244 Squads, 4273 Pilots, 152000+ sorties using 123 different planes, vehicles, PT Boats, and an RV-8.  Most of the data is going to be good, but some isn't.

[RedBeard]

My intentions in looking back at all this data is to understand what works and what doesn't.  To do that, I have been trying to recreate what actually happened based on log data.  In essence, I have been looking at automated after action report generation.  I have sort of been able to do that, but with mixed success.  There's a lot of info that can be picked up from the log data, but it's not reliable.  For instance, I can determine who are the defenders and who are the attackers with about 90% accuracy based on log data.  I can also determine active fields, field ownership, and most target fields, but I need better accuracy for targets to get the data I'm looking for.

I am starting to integrate some Objectives data into the mix.  I have about 2-3 years of objectives data and I have orders for a bit more than that.  The objectives data gives a little more info about active base and target proximity, required targets vs. what was actually destroyed and the orders give expected flight plans.  I was hoping to look at animated maps of flight paths, but that is a long way away at this point. 

If anyone has Objectives going back to 2009 and wouldn't mind sharing it, I'd like to talk to you about it.  Any more than the last 2 years or so, would help.

[RedBeard]

Here are some of the things I'm looking at producing and sharing from the data.

I will post separately on where I'm at with after action reports.

I have been looking at building up a "leaders board" of top squads and/or top pilots with some metrics just for fun, but haven't decided yet what meaningful metrics should be used.  This could potentially show a historical perspective of things the squad did to get where it is.  Some of what pushed me in this direction is seeing how many friendly fire incidents there were.  Departures and during engagements seem to be notorious for FF.  Another factor is I wanted to identify what squads were good at and use that in planning missions.

Another thing I'm looking at is writing up some software to produce pilot log books.  It seemed to be a natural side project that would be fairly easy to do from the after action report data.  It would be a mix of similarity to  WWII pilot log books and current pilot log books used today.  Part of this will be dependent on my either integrating Objectives data for everything or using software to intuit target information into the sortie description.  It would be nice to get the bombs dropped/target damaged info in this, but as it stands now, I can't tell the difference between didn't carry bombs, didn't drop, dropped and missed, jettisoned for escape, and dropped and damaged but didn't destroy.

[Spikes]

I remember when we had 600 pilots and we were planning for 4 objectives to defend and attack.

I don't recall FSO ever having 600 pilots.

[RedBeard]

Here's a look at the kind of after action report I'm generating.  This one is hand generated based on the July Frame 1 log while trying to follow an algorithm in my head.  It doesn't include any flight path info from orders and does not include Objectives info such as coordinates, it is just what was shown in the logs.

22:05
    Allies
        1 defensive F4U-1C departed from Task Group #1 to cover Task Group #1
        14 defensive F4U-1C and 1 F6F-5 departed from #3 to cover Task Group #1
        8 offensive TBM-3, 8 offensive F6F-5, and 2 offensive F4U-1A departed from Task Group #2 for Airfield #16
        1 offensive F4U-1C departed from Task Group #1 for Airfield #16
    Axis
        6 offensive Ki-67, 7  Ki-61 departed from Airfield #16 for Task Group #1
        12 def N1K2 from Airfield #16 to cover Airfield #16
22:10
    Allies
        Pilot exchanges 1 F4U-1C for 1 F4U-1A out of Task Group #2
        1 F6F-5 is lost to defenders at Airfield #16
    Axis
        12 N1K2-J defenders at Airfield #16 engage 4 F6F-5s and shoot 1 down
        2 offensive Ki-61s depart from Airfield #16 for Task Group #1
22:20
    Allies
        4 F4U-1C + 1 F6F-5 defenders at Task Group #1 engage 2 Ki-61s and shoot 1 down
        4 F6F-5 jumped by 12 N1K2-J defenders at Airfield #16 and shoot 1 down, but lose 4
        10 F4U-1C defenders at Task Group #1 engage 3 Ki-67 formations and shoot 8 down and lose 1
        2 TBM-3 and 1 F4U-1A lost to defenders at Airfield #16
    Axis
        3 Ki-67 formations shoot down 1 F4U-1C, but lose 3 formations
        12 N1K2-J defenders at Airfield #16 engage 4 F6F-5s and 3 TBM-3s and shoot down 4 F6F-5s and 2 TBM-3s while losing 1 N1K2-J
        1 Ki-67 formation destroys a ship at Task Group #1 but the formation is lost
        3 Ki-61 damage Task Group #1 and sink 1 ship
22:30
    Allies
        3 F6F-5 departed Task Group #3 for Task Group #1
    Axis
        3 N1K2-J land at #13
        9 defensive N1K2-J, 5 Ki-61 depart Airfield #16 for Airfield #16
         10 offensive Ki-61 and 3 Ki-67 depart Airfield #16 for Task Group #1
...

I feel like this is better than the by squad,pilot time based facts, but is still a bit difficult to grasp all the moving pieces.  It is also the reason why I would eventually like to be able to replay with coordinates on a map.  Right now, we just know origination location, destroyed target location, and sometimes RTB location, and not a lot between unless an engagement occurs and then just who was involved in the engagement and not where.

I have some of this coded up, but I'm currently trying to deal with targets changing between hour 1 and hour 2.

If anyone has feedback on the style of the after action report or information presented, let me know.  I'd like to make it useful to someone other than just me. 

Once I get Objectives data integrated, I can add some additional distance and direction info, e.g. 6 offensive Ki-67, 7  Ki-61 departed from Airfield #16 for Task Group #1 100 miles to the NE

[AKKuya]

I don't recall FSO ever having 600 pilots.

Around 2009/2010, we hit 600 pilots for a few frames.  It was commented on.  i think I confirmed it looking at the logs.  15 years can make the memory a little fuzzy plus being on the other side of 50 now. 

[fudgums]

I don’t anyone did yet, but it will inevitably turn on the CMs at some point lol

[Devil 505]

Interesting data gathered.

There are a few elements that don't quite make sense though.

1. How are you calculating what a sortie is? I would expect to see the sortie number match the number of pilots except in FSO's where extra lives are given, but that seems to not be the case.

2. Your B-29 sortie count seems to be way too high. I can only think of 4-5 FSO's in total where the plane was used. No way their total use is 2900 sorties, even with drones and rearming counting as a new sortie.

[RedBeard]

1. How are you calculating what a sortie is? I would expect to see the sortie number match the number of pilots except in FSO's where extra lives are given, but that seems to not be the case.

2. Your B-29 sortie count seems to be way too high. I can only think of 4-5 FSO's in total where the plane was used. No way their total use is 2900 sorties, even with drones and rearming counting as a new sortie.

A sortie is defined as one pilot takeoff (launch) followed by one flight termination (shotdown, landed, ditched, bailed, disco).  You can get multiple sorties even in one life events when pilots are allowed to bail/replane at the start or disco/replane with CM permission.

There were 3 B-29 events: 2012-12, 2013-12, and 2014-10 with 3 frames each.  I just did a quick scan of the html files and you are correct.  There were only 318 sortie departures listed.  I just validated that the DB has 318 sorties, so now I need to go back and find what I did wrong with a SQL JOIN.