« previous next »
Pages: [1]   Go Down

Author Topic: Parser help  (Read 260 times)

Offline Wotan

  • Platinum Member
  • ******
  • Posts: 7201
Parser help
« on: December 18, 2004, 05:40:25 AM »
What we want to do is set up a log parser for our webpage so that designated squad members can upload their logfile (from FB/AEP/PF).

What we want is to upload the raw text file, to our website,  have it filtered out and record our data such as deaths, kills, hit %, number of sorties etc... (all this is contained in the logfile already) on to our 'killboard..

I haven't attempted to program anything like this before and am wondering if some one could offer some suggestions and point me in the right direction.

I am completely clueless on how to even get started... (besides a goggle search :p)

TIA
Logged

Offline fd ski

  • Silver Member
  • ****
  • Posts: 1530
      • http://www.northotwing.com/wing/
Parser help
« Reply #1 on: December 18, 2004, 08:50:36 AM »
I'd recommend Perl for job like this, basically it's reading text and categorizing it as you see fit.

Break down is as follows:
You have source files, logs of whatever sort that you'll be reading.
You have to have means of identifying them and categorising, unless you're only looking for "all sum" stats.
So maybe pilot nick or something in teh filename would be a good idea.

Then you have program read the logfiles from selected place ( could be same FTP that they are uploaded to ).
I don't know what logfiles of Il2 look like, but i'm sure they are in some sort of text line based format with something that delimits them.
Your parser has to read and understand each line ( at least those it wants to understand ) and then keep track of things for you for displaying later.

Perl is really helpfull here because you can do something called hashtables and nest them within each other. I wrote a parser for AH 4 years ago but don't have it up anywhere, just source code. Perl being inherent ***** to read, i won't send it to you :)

I'll give you an example though.

Let's say you have following lines in the log:
( format : pilot, event )
fd-ski, takeoff
wotan, takeoff
fd-ski, kill
wotan, death
fd-ski, landing

your parser basically can do this:
( in pseudo code )
read line
split it into two parts
in hashtable called "pilots" write: $pilots{$data1}{$data2}++
do same for each next line.

Result of this would be hashtable as follows ( tab denotes a root of nested hashtable )

Pilots
         fd-ski
                   takeoff 1
                   kill 1
                   landing 1
         wotan
                   takeoff 1
                   death 1


Now, logfiles are notoriously more complicated and you have to find a logical way to handle stuff in it, but with hashtables making stats of any sort is very each and elegant from the programing point of view.

here is a chunk of AH log and i'll post the main stats engine in the reply below. Hope this helps.

12-09-01 13:15:49,RAM,Start Flight,22,Fw 190A-5
12-09-01 13:15:49,RAM,TO Load,100 %,1,0,0
12-09-01 13:15:50,Marco8X,Start Flight,18,Bf 109G-6
12-09-01 13:15:50,Marco8X,TO Load,100 %,1,1,0
12-09-01 13:15:51,Naudet,Start Flight,17,Fw 190A-8
12-09-01 13:15:51,Naudet,TO Load,100 %,1,1,0
12-09-01 13:15:52,Wail,Start Flight,17,Fw 190A-8
12-09-01 13:15:52,Wail,TO Load,100 %,1,1,0
12-09-01 13:15:53,mora,Start Flight,18,Bf 109G-6
12-09-01 13:15:53,mora,TO Load,100 %,1,1,0
12-09-01 13:15:53,Hazed,Start Flight,23,Fw 190A-5
12-09-01 13:15:53,Hazed,TO Load,100 %,1,0,0
12-09-01 13:15:55,Westy,Start Flight,20,Fw 190A-8
12-09-01 13:15:55,Westy,TO Load,100 %,1,1,0
12-09-01 13:15:56,Morsa09,Start Flight,23,Fw 190A-5
12-09-01 13:15:56,Morsa09,TO Load,100 %,1,0,0
12-09-01 13:15:57,KKEN,Start Flight,18,Bf 109G-6
12-09-01 13:15:57,KKEN,TO Load,100 %,1,1,0
12-09-01 13:15:58,MrLars,Start Flight,20,Fw 190A-8
12-09-01 13:15:58,MrLars,TO Load,100 %,1,1,0
12-09-01 13:15:58,Mathman,Start Flight,16,Fw 190A-8
12-09-01 13:15:58,Mathman,TO Load,100 %,1,1,0
12-09-01 13:15:59,Tex,Start Flight,22,Fw 190A-5
12-09-01 13:15:59,Tex,TO Load,100 %,1,0,0
12-09-01 13:15:59,Furious,Start Flight,22,Fw 190A-5
12-09-01 13:15:59,Furious,TO Load,100 %,1,0,0
12-09-01 13:16:00,Steve74,Start Flight,20,Fw 190A-8
12-09-01 13:16:00,Steve74,TO Load,100 %,1,1,0
12-09-01 13:16:00,Broesy,Start Flight,19,Fw 190A-5
12-09-01 13:16:00,Broesy,TO Load,100 %,1,0,0
12-09-01 13:16:00,airbull,Start Flight,23,Fw 190A-5
12-09-01 13:16:00,airbull,TO Load,100 %,1,0,0
12-09-01 13:16:01,Nibbio,Start Flight,26,Bf 109G-6
12-09-01 13:16:01,Nibbio,TO Load,100 %,1,0,0
12-09-01 13:16:01,Pei,Start Flight,23,Fw 190A-5
12-09-01 13:16:01,Pei,TO Load,100 %,1,0,0
12-09-01 13:16:01,pinner,Start Flight,16,Fw 190A-8
12-09-01 13:16:01,pinner,TO Load,100 %,1,1,0
12-09-01 13:16:02,DrDea,Start Flight,16,Fw 190A-8
12-09-01 13:16:02,DrDea,TO Load,100 %,1,1,0
12-09-01 13:16:02,Greif,Start Flight,23,Fw 190A-5
12-09-01 13:16:02,Greif,TO Load,100 %,1,0,0
12-09-01 13:16:02,xavimm,Start Flight,23,Fw 190A-5
12-09-01 13:16:02,xavimm,TO Load,100 %,1,0,0
12-09-01 13:16:02,maik,Start Flight,17,Fw 190A-8
12-09-01 13:16:02,maik,TO Load,100 %,1,1,0
12-09-01 13:16:03,StSanta,Start Flight,23,Fw 190A-5
12-09-01 13:16:03,StSanta,TO Load,25 %,1,0,0
12-09-01 13:16:03,ManeTMP,Start Flight,26,Bf 109G-6
12-09-01 13:16:03,ManeTMP,TO Load,100 %,1,0,0
12-09-01 13:16:04,Wlfgng,Start Flight,16,Fw 190A-8
12-09-01 13:16:04,Wlfgng,TO Load,100 %,1,1,0
12-09-01 13:16:04,agent89,Start Flight,16,Fw 190A-5
12-09-01 13:16:04,agent89,TO Load,100 %,0,0,0
12-09-01 13:16:04,Logan,Start Flight,16,Fw 190A-8
12-09-01 13:16:04,Logan,TO Load,100 %,1,1,0
12-09-01 13:16:05,LJKBear1,Start Flight,22,Fw 190A-5
12-09-01 13:16:05,LJKBear1,TO Load,100 %,1,0,0
12-09-01 13:16:06,MadMaxx,Change Field,16,26
12-09-01 13:16:07,LUPO,Start Flight,19,Fw 190A-5
12-09-01 13:16:07,LUPO,TO Load,100 %,1,0,0
12-09-01 13:16:07,airmess,Start Flight,23,Fw 190A-5
12-09-01 13:16:07,airmess,TO Load,100 %,1,0,0
12-09-01 13:16:07,GLASSESs,Start Flight,23,Fw 190A-5
12-09-01 13:16:07,GLASSESs,TO Load,100 %,1,0,0
12-09-01 13:16:07,JoGee,Start Flight,17,Fw 190A-8
12-09-01 13:16:07,JoGee,TO Load,100 %,1,1,0
12-09-01 13:16:07,Buzzbait,Start Flight,22,Fw 190A-5
12-09-01 13:16:07,Buzzbait,TO Load,100 %,1,0,0
12-09-01 13:16:07,kratzer,Start Flight,22,Fw 190A-5
12-09-01 13:16:07,kratzer,TO Load,100 %,1,0,0
12-09-01 13:16:08,Lizard3,Start Flight,16,Fw 190A-8
12-09-01 13:16:08,Lizard3,TO Load,100 %,1,1,0
12-09-01 13:16:08,PegBrown,Start Flight,18,Bf 109G-6
Logged

Offline fd ski

  • Silver Member
  • ****
  • Posts: 1530
      • http://www.northotwing.com/wing/
Parser help
« Reply #2 on: December 18, 2004, 08:52:25 AM »
#!/usr/bin/perl

use ahparser;


print <Content-type: text/html\n\n



   





END_HEAD

$logfile = @ARGV[0];
@events = ("Takeoff","Landed","Killed","Crashed","Ditched","Captured","Bailed","Disco","LossRatio");


#first portion opens the logfile with settings for the scenario and outputs them
#opening sorted list of all vehicels .. so that everything is nice and sorted :)
@sortedplanes  = getPlaneList();

#open configuration file and get the array of it
@cfg = getArr( "$logfile.cfg");

#open the ban file for the logenties banned from the scneario
@badloglines = getArr("$logfile.ban");


#open the out file for the pilots banned from the scneario
@badpilots = getArr("$logfile.out");



#outputting the sceanrio information
print '
';
$test = getvalue (title,\@cfg,1);
print $test;
print '',"\n",'
';
$test = getvalue ("date",\@cfg,1);
print $test;
print '',"\n",'
$test = getvalue ("link",\@cfg,1);
print $test;
print '">Event page',"\n",'

$test = getvalue ("image",\@cfg,1);
print $test;
print '">',"\n",'
_',"\n",'
Event:',"\n",'
';
$test = getvalue ("disc",\@cfg,1);
print $test;
print '',"\n",'

Result:',"\n",'
';
$test = getvalue ("result",\@cfg,1);
print $test;
print '',"\n",'

Explanation of results:',"\n",'
';
$test =  getvalue ("explain",\@cfg,1);
print $test;
print '',"\n",'

CM notes:',"\n",'
';
$test = getvalue ("cmnotes",\@cfg,1);
print $test;
print '




';


#THIS SECTION PUTS THE PROPER COUNTRIES INTO @COUNTRIES ARRAY. ONLY TWO COUNTRIES SHOULD EXIST THERE.

$country0 = getvalue ("Country 0",\@cfg,1);
$country1 = getvalue ("Country 1",\@cfg,1);
$country2 = getvalue ("Country 2",\@cfg,1);
if ( $country0 ne ""){
@countries = ("Country 0");
}
if ( $country1 ne ""){
@countries = ( @countries, "Country 1");
}
if ( $country2 ne ""){
@countries = ( @countries, "Country 2");
}




#in this portion we pharse the log file. We find the country beginnings and catalogue the events into hashes
#if header found we create:
# - list of pilots in their current planes ( hash )
# - list of planes used by this country ( single array )
# - kills by each plane type ( if kill or assist event found - look at what pilot killed flew ( in hash ) and add to the kill counter for that plane ( array of arrays ? )
# - take off events/landing/disco/bail/killed events are found - find owners by the nick in hash and add to proper column. ( array of arrays )
#now we open the logfile
open (BHANDLE, "<$logfile") or die "can't open logfile";
#now we read each line of the logfile
my %planes;
my %pilotplane;
my %totals;
my %enemyplane;
my %pilotstats;

foreach $line (){
chomp $line;
if(isOutlawed($line, \@badloglines)){
if($line =~ m/^Country/){
$country = $line; #now country stores the current country

foreach $d ( @countries ){
if ($d ne $country){
$opposition = $d;
}
}

}#end if line has country in it
#now we start working on record level.
@rec = split (",",$line); #this breaks down the log by , - and now we need to break the time and date down too
@rec1 = split (" ",@rec[0]);
$time = @rec1[1]; #now time contains the time of the event
$handle = @rec[1];


#check for take off event
if(@rec[2] eq "Start Flight" && isOutlawed($handle, \@badpilots) eq "Valid" ){
if (checkPlane($country, @rec[4]) ne "Invalid"){
#pilot takes off - therefore we add a sortie to his count and new life

$pilotstats{$handle}{Sortie}++;

#put pilot in "in Flight" status
if ($pilotstats{$handle}{status} eq "In Flight"){
#we add a disco to his previous plane sortie
$prevPlane = $pilotplane {$country}{$handle};
$planes{$country}{$prevPlane}{Disco}++;
$pilotstats{$handle}{Errors}++;
}
$pilotstats{$handle}{status} = "In Flight";



#now we mark off the pilot taking off as being in this plane
$curplane = @rec[4];
$pilotplane {$country}{$handle} = $curplane;
$planes {$country}{$curplane}{Takeoff}++;
$enemyplane{$opposition}{$curplane}++;
$enemyplane{$opposition}{$curplane}--;
}else{
$pilotstats{$handle}{errors}++;
$pilotstats{$handle}{status} = "Down";
}


}#end of if for start flight event



#check for landing event
if(@rec[2] eq "End Flight"){
#since we can have different ways of ending fligth - we have to account for all of them
$event = @rec[3];
$curplane = $pilotplane{$country}{$handle};
$planes{$country}{$curplane}{$event}++;

#and how we deal with the individual pilot and thier status
if ($pilotstats{$handle}{status} ne "In Flight"){
$pilotstats{$handle}{Errors}++;
}
$pilotstats{$handle}{status} = "Down";
$pilotplane{$country}{$handle} = "";



}#end of if for start flight event

#kills baby kills ;)
if(@rec[2] eq "Kill Awarded"){
$planekilled = @rec[4];
#now we check if pilot killing was in fligth ( if not - he was a gunner )
if($pilotstats{$handle}{status} eq "In Flight"){
$curplane = $pilotplane{$country}{$handle}; #we need to know what plane killing pilot was in
}else{
$curplane = "Gunner";
$pilotstats{$handle}{status} = "In Fligth";
$planes{$country}{$curplane}{Takeoff}++;
$enemyplane{$opposition}{$curplane}++;
$enemyplane{$opposition}{$curplane}--;
$pilotplane{$country}{$handle} = $curplane;
}


if($curplane eq ""){
$curplane = "Unknown";
}
#and now the entry ;)
$planes{$country}{$curplane}{$planekilled}++;
$enemyplane{$country}{$planekilled}++;
$pilotstats{$handle}{Kills}++;



}
}
}#end of foreach line BHANDLE
close (BHANDLE); #end of reading of the logfile

Logged

Offline fd ski

  • Silver Member
  • ****
  • Posts: 1530
      • http://www.northotwing.com/wing/
Parser help
« Reply #3 on: December 18, 2004, 08:56:34 AM »

#ok so we start clearing output ;)
#we loop though all the countries
foreach $c ( @countries ){
#now we check if there are actually any pilots that took off in this country
if($pilotplane{$c} >0){ #here we make sure that some pilots actually took off in this partocular country
#printing the header of the table
print '
Stats for ',getvalue($c,\@cfg,1),'',"\n",'

';

#start of the table
print '',"\n",'',"\n",'',"\n",'',"\n",'',"\n",'',"\n",'',"\n",'',"\n",'',"\n",'',"\n",'',"\n",'',"\n",'';

#now we make a table row for each fricking plane in the set for this country :)
$count = 0;
foreach $splane (@sortedplanes){
chomp $splane;
foreach $plane(keys %{$planes{$c}}){
if($plane eq $splane){

$count++;
if($count%2==1){
$back = "#FFFFFF";
}else{
$back="#C0C0C0";
}

#we print the plane name
print '',"\n",'',"\n";

#now we calulate the loss ratio for the plane - we have to make sure that it's not unknown  - since that one might have 0 for take off value
if(($planes{$c}{$plane}{"Takeoff"}-$planes{$c}{$plane}{"Disco"}) != 0 ){
$planes{$c}{$plane}{"LossRatio"}=sprintf("%01d%",(($planes{$c}{$plane}{"Killed"}+$planes{$c}{$plane}{"Ditched"}+$planes{$c}{$plane}{"Captured"}+$planes{$c}{$plane}{"Bailed"})/($planes{$c}{$plane}{"Takeoff"}-$planes{$c}{$plane}{"Disco"}))*100);
#print $planes{$c}{$plane}{"loss"};
}else{
$planes{$c}{$plane}{"LossRatio"} = "0%";
}

#now we get each single event :)
foreach $event ( @events ){
print '',"\n";
$totals{$c}{$event} += $val;

}#end of foreach $events
print '';
}
}#end of foreach $plane
}
#now we print out totals
print '',"\n",'',"\n";
foreach $event ( @events ){
print '',"\n";
}
print '
';
print getvalue($c,\@cfg,1);
#and here comes the first row of this table - THE ONE LINER FROM HELL :)
print '		Take offLandedKilledCrashedDitchedCapturedBailedDiscoLoss ratio
',$plane,'';
$val = $planes{$c}{$plane}{$event};
if($val eq ""){
$val= "_";
}
print $val;
print '
Total';
$val = $totals{$c}{$event};
if($val eq ""){$val = 0;}
if($event eq "LossRatio"){
$val = sprintf("%01d%",($val/count));}
print $val;

print '

',"\n",'
',"\n",'
',"\n";
#################now we make a table with kills :)
#printing the header of the table
print 'Kills for ',getvalue($c,\@cfg,1),'',"\n",'

';
#start of the table
print '',"\n",'',"\n",'',"\n",'',"\n";
}
}#end of foreach eplane
}
print '',"\n",'',"\n";
#and now we start the output in good old fashioned way ;)
#now we make a table row for each fricking plane in the set for this country :)
$count = 0;
foreach $splane (@sortedplanes){
chomp $splane;
foreach $plane(keys %{$planes{$c}}){
if($plane eq $splane){
$count++;
if($count%2==1){
$back = "#FFFFFF";
}else{
$back="#C0C0C0";
}
#we print the plane name
print '',"\n",'',"\n";

foreach $zplane (@sortedplanes){
chomp $zplane;
#now we get each single enemy plane that can be killed:)
foreach $eplane (keys  %{$enemyplane{$c}}){
if ($eplane eq $zplane){
print '',"\n";
$totals{$c}{$plane} += $val;
$totals{$c}{"enemy"}{$eplane} += $val;
}
}#end of foreach $events
}#end of foreach $zplane

print '',"\n";
print '';
}
}#end of foreach $plane
}
#and now totals line
print '',"\n",'',"\n";

foreach $splane (@sortedplanes){
chomp $splane;
foreach $eplane (keys %{$enemyplane{$c}}){
if($splane eq $eplane){
print '',"\n";
$totals{$c}{"Allkills"} += $val;
}
}
}

#and now we just output total of kills for the country - and everyone is happy :)
print '',"\n";
print '
';
print getvalue($c,\@cfg,1);
foreach $splane (@sortedplanes){
chomp $splane;
foreach $eplane (keys %{$enemyplane{$c}}){
if($eplane eq $splane){
#here we start a column for each enemy plane found in this hash
print '		';
print $eplane;
print '		Total
',$plane,' ';
$val = $planes{$c}{$plane}{$eplane};
if($val eq ""){
$val ="_";
}
print $val;
print '		 ';
$val = $totals{$c}{$plane};
if($val eq ""){
$val = 0;
}
print $val;
print '
Total ';
$val = $totals{$c}{"enemy"}{$eplane};
if($val eq ""){
$val = 0;
}
print $val;
print '		 ';
print $totals{$c}{"Allkills"};
print '

',"\n",'
',"\n",'
',"\n";
}#end of the if statement for checking if someone took off

}#end of foreach country

#print '


Trouble shooting data

';


####################and now for wonderful trouble shooting stuff ;)
#$cm = @ARGV[1];
#chomp $cm;
#if($cm eq "PASSWORDHERE"){


#foreach $pilot(keys %pilotstats){
#        foreach $event(keys %{$pilotstats{$pilot}}){
# if($event eq "Kills"){
# $val = $pilotstats{$pilot}{$event};
# if ($val > 0){
#print $pilot, "`", $val, "
\n";
# }
# }
# }
#}
#FUNCTIONS ###########################
sub isOutlawed{ #this function checks in the line sent to it is outlawed from the sceanrio
my $checkLine = $_[0];
my $in = $_[1];
foreach $line ( @$in ){
chomp $line;
chomp $checkLine;
if($line eq $checkLine){
return "Invalid";
}

}
return "Valid";
}


sub getvalue{ #this function takes name and an array then finds the cvalue for this name in the array
my $search = $_[0];
my $in = $_[1];
my $which = $_[2];
$ret = "";
if($which eq ""){
$which = 1;
}
foreach $z (@$in){
@obj1 = split("`",$z);
if(@obj1[0] eq $search){
$ret = @obj1[$which];
}
}
chomp $ret;
return $ret;
}

sub checkPlane{
chomp $_[0];
chomp $_[1];
$allowedPlanes = join("",$_[0], "planes");
$allowedPlanes = getvalue($allowedPlanes, \@cfg, 1);
chomp $allowedPlanes;
if ($allowedPlanes =~ m/$_[1]/ || $allowedPlanes eq "All"){
return $_[1];
}

#if ( $allowedPlanes =~ m/$_[1]/ ){
# return $_[1];
#}

return "Invalid";
}


Disclaimer:
I'm a ****TY programer, code above is barely recognizable to me and i'm pretty sure that soon it will show up on the net as an example of how NOT to code :)

Use at your own discresion.
« Last Edit: December 18, 2004, 08:59:54 AM by fd ski »
Logged

Offline fd ski

  • Silver Member
  • ****
  • Posts: 1530
      • http://www.northotwing.com/wing/
Parser help
« Reply #4 on: December 18, 2004, 08:58:16 AM »
oh, and scuzzy, feel free to whack this if i'm spilling any secrets here ( logfiles were changed by my program after being drawn from AH and are 3 years old, so i don't think this is against the law or anything :D )
Logged

Offline Wotan

  • Platinum Member
  • ******
  • Posts: 7201
Parser help
« Reply #5 on: December 18, 2004, 10:15:31 AM »
Thanks Fdski...

I copied and pasted so Skuzzy can butcher away :p
Logged

Offline montag

  • Parolee
  • Copper Member
  • **
  • Posts: 164
Parser help
« Reply #6 on: December 18, 2004, 01:29:53 PM »
10 GOTO 10
Logged
Pages: [1]   Go Up
« previous next »