Aces High Bulletin Board

General Forums => Open Beta Test => Topic started by: artik on January 05, 2016, 12:47:59 PM

Title: MakeAHMap and AH3
Post by: artik on January 05, 2016, 12:47:59 PM
Hello,

Also I've taken a break for a while from AH, I still would like to make sure that MakeAHMap utility for building terrains from geographical data continues to work with AH3.

Recall: http://www.hitechcreations.com/wiki/index.php/Terrain_Generation_From_Geographical_Data

It creates following files:

1. terrain elevations (.elv file)
2. high resolution shorelines and rivers (waterd.bmp/waterc.bmp)
3. ground cover (tiles) according to the geographical data set, i.e. farm, forest, dessert areas, etc. (gndtype.bmp)
4. Nice looking clipboard map.

Now I'd like to know what changes should I do to make the stuff work:

1. Are waterd.bmp/waterc.bmp relevant for representation for shore lines?
2. Ground cover gndtype - AFAIR it had changed am I right.
3. And of course AFAIR the resolution of elevations had changed how the alignment between all these types changed

And the last but not least - water elevations, AH2 has slope restriction that I fixed by modifying elevations (because real world slopes are higher quite frequently) have slope restrictions changed in new version and how.

What I would really appreciate is documentation of terrain file formats.

Thank You.

(Also I don't have to fly, I'm much more flexible with coding  :) )
Title: Re: MakeAHMap and AH3
Post by: hitech on January 05, 2016, 01:05:29 PM
The concept of water map is gone. All elevations below 0 are considered water.

Elevation is now signed 16 bit at 660 ft vertex spacing.

gndtype.bmp is similar but now the channel control is different . See the document for specifics on file types.

http://bbs.hitechcreations.com/smf/index.php/topic,376440.0.html

HiTech
Title: Re: MakeAHMap and AH3
Post by: Easyscor on January 05, 2016, 01:24:24 PM
Hi Artik, I see Hitech has responded while I was writing this but here it is anyway.


There are import options unless you obtain the file formats for the htx (I believe it is) format.

1. terrain elevations (.elv file)
This is now a .raw file. It can be either 16 bit signed (zero is zero elev.) or unsigned. I doubt you'll want the unsigned version but it's documented in this forum. This is a 4096x4096 raster as they all are. Search on .raw files in this forum.

2. high resolution shorelines and rivers (waterd.bmp/waterc.bmp)
You're going to need to do some of that coad stuff here. No water above zero elevation except the new river paths.
You can build custom objects containing lakes if you like but that's not in your program description.
Water is automatic if the elevation drops below zero. I'd suggest contacting htc to see if you can obtain the format for the shoreline htx files but the raw elevation files may be all you need. If you do retrieve the htx file format, please document it in here.


3. ground cover (tiles) according to the geographical data set, i.e. farm, forest, dessert areas, etc. (gndtype.bmp)
This is another 4096x4096 file but it's an RGB bitmap. Search on 'Splat' files in this forum.

4. Nice looking clipboard map.

Now I'd like to know what changes should I do to make the stuff work:

3. And of course AFAIR the resolution of elevations had changed how the alignment between all these types changed
Everything is shifted. iirc, 1/2 mile north and 1/2 mile east. waterd won't quite line up but you should play with the TE to see the changes.

Title: Re: MakeAHMap and AH3
Post by: artik on January 05, 2016, 03:47:43 PM
Thanks few quick questions:

@hitech - river objects (I assume like roads as well) how can I create ones? I need file format specifications. The geographical data contains rivers in already in piecewise linear format so all I need is to convert relevant part to internal TE river object. Am I right? If so can you please provide a format there?

Regarding the water below 0. It is problematic. It basically means that all lakes should be at 0 level.

I can think of following ways to resolve it:

1. I can either modify elevations and move lakes to 0.
1.a if I add rivers they would... Flow backward at some parts (assuming I'll know how to do it)
1.b Or I can "create canyons" where rivers/lakes exists this would cover in such a case both rivers and lakes - I can specify some slope threshold regarding canyon depths.
2. Remove lakes entirely, keep rivers as objects
3. Make lakes special kind of ground type with special tiles so they'll look like a water but behave like... grass. keep rivers as objects
4. Remove both rivers and lakes (but it can really badly hurt the impression of the map).

To be honest I think of either (1b) or 4. Maybe 1a.

What do you think?
Title: Re: MakeAHMap and AH3
Post by: hitech on January 05, 2016, 04:13:50 PM
Thanks few quick questions:

@hitech - river objects (I assume like roads as well) how can I create ones? I need file format specifications. The geographical data contains rivers in already in piecewise linear format so all I need is to convert relevant part to internal TE river object. Am I right? If so can you please provide a format there?

Regarding the water below 0. It is problematic. It basically means that all lakes should be at 0 level.

I can think of following ways to resolve it:

1. I can either modify elevations and move lakes to 0.
1.a if I add rivers they would... Flow backward at some parts (assuming I'll know how to do it)
1.b Or I can "create canyons" where rivers/lakes exists this would cover in such a case both rivers and lakes - I can specify some slope threshold regarding canyon depths.
2. Remove lakes entirely, keep rivers as objects
3. Make lakes special kind of ground type with special tiles so they'll look like a water but behave like... grass. keep rivers as objects
4. Remove both rivers and lakes (but it can really badly hurt the impression of the map).

To be honest I think of either (1b) or 4. Maybe 1a.

What do you think?

You can make decent lakes with rivers, file format is not finalized, but it consist of spline points along with widths.

HiTch
Title: Re: MakeAHMap and AH3
Post by: artik on January 05, 2016, 04:18:00 PM
You can make decent lakes with rivers, file format is not finalized, but it consist of spline points along with widths.

HiTch


Great, even if it is only preview format it would be great to know one to at least see how stuff works. Later the relevant module can be updated.

Looking forward.
Title: Re: MakeAHMap and AH3
Post by: hitech on January 05, 2016, 04:36:15 PM
The current format

The y value is height off ground for roads, and width for rivers.


   fprintf(Fp,"%d\n",TrackList->TrackCnt);
   for(TrackIndex= 0 ;TrackIndex < TrackList->TrackCnt;++TrackIndex)
   {
      Track = TrackList->Tracks[TrackIndex];
      if(Track == NULL)
      {
         fclose(Fp);
         return -1;
      }

      fprintf(Fp,"%d,Type\n",Track->Type);
      fprintf(Fp,"%d,Flags\n",Track->Flags);
      fprintf(Fp,"%d,ResourceType\n",Track->ResourceType);

      fprintf(Fp,"%d,DestFieldIndex\n",Track->DestFieldIndex);
      fprintf(Fp,"%d,SourceFieldOrObjectIndex\n",Track->SourceFieldOrObjectIndex);
      fprintf(Fp,"%f,SrcRadius\n",Track->SrcRadius);
      fprintf(Fp,"%f,DestRadius\n",Track->DestRadius);

      fprintf(Fp,"%s\n",Track->TrackName);
      fprintf(Fp,"%d,PntCnt\n",Track->SplinePath.NodeCnt);

      PntCnt = Track->SplinePath.NodeCnt;
      for(PntIndex= 0 ;PntIndex < PntCnt;++PntIndex)
      {
         fprintf(Fp,"%lf,%lf,%lf\n",Track->SplinePath.Pnts[PntIndex].x,
                                    Track->SplinePath.Pnts[PntIndex].y,
                                    Track->SplinePath.Pnts[PntIndex].z);
      }
   }


Title: Re: MakeAHMap and AH3
Post by: 715 on January 05, 2016, 07:51:13 PM
Hitech: in the .tra file are the text descriptors after the comma actually necessary, or can I leave them out (to save space, a 512x512 terrain I'm working on has about 25K points in several hundred river segments).

Edit: actually never mind.  Because the number of elements is way less than the number of points total, it wouldn't make much difference at all.
Title: Re: MakeAHMap and AH3
Post by: hitech on January 06, 2016, 11:44:21 AM
715:

They are not necessary, I simply put them there to make the file easier to read.


Also what is the spacing you are using between points? That's sounds like a very large number since we already smooth with splines?

If so I may wish to write an optimizer to remove points X distance from the spline curve.

BTW the hold up on the rivers & roads is how do do an efficient collision and culling system. Nothing has jumped into my head yet.

HiTech

Title: Re: MakeAHMap and AH3
Post by: artik on January 06, 2016, 12:23:38 PM
BTW the hold up on the rivers & roads is how do do an efficient collision and culling system. Nothing has jumped into my head yet.

HiTech

Just a thought...

Basically if you ignore a rare case of "road higher the terrain" you can just rasterize the roads/rivers and just check upon collision with the ground what kind of surface it is. Having a simple grid/sub-grid spatial index would make it easily O(1) operation.
Title: Re: MakeAHMap and AH3
Post by: 715 on January 06, 2016, 03:24:56 PM
Also what is the spacing you are using between points? That's sounds like a very large number since we already smooth with splines?

If so I may wish to write an optimizer to remove points X distance from the spline curve.

HiTech

I have no control over the spacing.  The numbers I quoted are for the WBDII portion of the fine resolution GSHHG GIS dataset.  That is actually a pretty coarse dataset (much coarser than the SRTM WBD set that I wanted to use).  However, it's the only one I can use as most rivers are defined by a line which allows importing into your river path file. SRTM WBD is way more detailed, but it defines rivers the same way as lakes, as closed polygons, thereby defining the path of both left and right bank shorelines.  So far I cannot conceive of an automatic way to convert a closed polygon into a spline path.  I've not even had much success doing it manually either in the TE and I can't seem to create anything but the simplest blob like lake.  Maybe I'm missing something.

If that's too many points then it's probably up to me to write an optimizer routine for my program to remove points so I'm not importing too many points. (Edit: or just use the lower resolution versions of that database instead of the fine one.)  It won't be easy to do that intelligently (but the lower res databases have probably already solved that).  (I can't see anyone using the TE to generate too many path points manually that you'd have to have a routine to reduce them in your code.)  First I'll try importing at full resolution and see if it chokes the TE.  Each path is limited to 256 points right?  (I'll have to break some river segments into sub segments as some have more than 400 points.)
Title: Re: MakeAHMap and AH3
Post by: artik on January 06, 2016, 03:30:26 PM
...The numbers I quoted are for the WBDII portion of the fine resolution GSHHG GIS dataset...

Are we working on same stuff?

makeahmap uses GSHHG data to create rivers on the map for AH2 terrains. But in AH2 it uses waterd to mark them. So in AH3 it probably should be path - something I want to do.
Title: Re: MakeAHMap and AH3
Post by: 715 on January 06, 2016, 03:43:06 PM
Are we working on same stuff?

makeahmap uses GSHHG data to create rivers on the map for AH2 terrains. But in AH2 it uses waterd to mark them. So in AH3 it probably should be path - something I want to do.

Yeah.  (But I'm just coding to keep my brain from atrophying.)  I actually wanted to use the SRTM WBD dataset as it is way more detailed than GSHHG.  GSHHG is kind of coarse; most rivers (virtually all in Europe) are defined only by path with only very crude information about width (4 values of river "importance").  Some sections of rivers, like those behind dams, are defined as lakes, i.e. closed polygons, and I'm at a loss on how to convert them to AH3 spline path + width.  I'd be interested in any ideas you have on that.
Title: Re: MakeAHMap and AH3
Post by: artik on January 07, 2016, 09:07:34 AM
Some sections of rivers, like those behind dams, are defined as lakes, i.e. closed polygons, and I'm at a loss on how to convert them to AH3 spline path + width.  I'd be interested in any ideas you have on that.

GSHHG has two sets: sea/land/lake/land sets of polygons and rivers as "lines" it actually makes it simpler to integrate it into TE. Of course the biggest problem is Lake as you need water above the sea level.

If HTC would be so kind to allow water at non-0 altitude it would make stuff much simpler :-)

What I can think of is something like that:

(http://s3.amazonaws.com/aww-imagedata/9f7545e2-a19b-475f-909d-f1585629f720.png)

It looks like the track has source radius and destination radius or in general I can create a circle with it. So I can start filling the lake with circles using Monte-Carlo method allowing some overlapping from big to small.

Or start from borders:

(http://s3.amazonaws.com/aww-imagedata/db301765-3b76-423a-b967-373af2ed9423.png)

And increase "river" width each time going inside.

Also I think it would be good idea if some lakes are huge or not to high above sea level just put them on altitude 0 regarding their geographical altitude.




This way or another currently TE brings my Laptop to BSOD once I try to add a river in it...   :mad:  So I'll start from basic terrain and than go to lakes and rivers.
Title: Re: MakeAHMap and AH3
Post by: 715 on January 07, 2016, 10:23:22 AM
I am almost certain that source and destination radius parameters are not what you think.  I've played with that in the terrain editor and it had no effect on the river appearance.  I suspect they have something to do with supply.  Also, using multiple overlapping river paths to construct a water feature creates artifacts as currently implemented because the river banks overlap.  However, I think the river rendering is maybe only preliminary.  Edit: I like your second idea, although it will probably also have problems with overlapping visible banks.  I'll try that manually and if it works then there is the hard topological problem of coding it automatically.  ;)

So, Hitech, how big a bribe would a closed polygon lake object cost us?  :)
Title: Re: MakeAHMap and AH3
Post by: hitech on January 07, 2016, 11:02:51 AM
I am almost certain that source and destination radius parameters are not what you think.  I've played with that in the terrain editor and it had no effect on the river appearance.  I suspect they have something to do with supply.  Also, using multiple overlapping river paths to construct a water feature creates artifacts as currently implemented because the river banks overlap.  However, I think the river rendering is maybe only preliminary.

So, Hitech, how big a bribe would a closed polygon lake object cost us?  :)

Actually I am pondering lake creation via splines. I.E. Create the lake shape similar to roads, Then I create the polygon mesh from the splines at load time.

HiTech

Title: Re: MakeAHMap and AH3
Post by: 715 on January 07, 2016, 11:07:14 AM
Actually I am pondering lake creation via splines. I.E. Create the lake shape similar to roads, Then I create the polygon mesh from the splines at load time.

HiTech

Woohoo!!!  That would be a BIG help!
Title: Re: MakeAHMap and AH3
Post by: artik on January 07, 2016, 12:26:39 PM
Actually I am pondering lake creation via splines. I.E. Create the lake shape similar to roads, Then I create the polygon mesh from the splines at load time.

HiTech

That is exactly what is needed :x

Few (not so) corner cases from real world maps:

1. Lakes can contain ilands - it can be nice to have an option for that
2. Confluence of multiple rivers and rivers and lakes

Title: Re: MakeAHMap and AH3
Post by: 715 on January 07, 2016, 01:48:48 PM
Artik: although it is moot if HiTech does the polygon spline lake, your clever idea of concentric rivers to make a lake works (except for the artifact of the overlapping river banks).  The attached image is an imaginary test terrain, but a real (random) Russian lake.  I only needed two concentric rivers (plus a small third one that is highlighted) to recover the shape of the lake.

The spline filled polygon is a MUCH better way to go if you can do it HiTech. 
Title: Re: MakeAHMap and AH3
Post by: artik on January 07, 2016, 02:13:06 PM
Artik: although it is moot if HiTech does the polygon spline lake, your clever idea of concentric rivers to make a lake works (except for the artifact of the overlapping river banks).  The attached image is an imaginary test terrain, but a real (random) Russian lake.  I only needed two concentric rivers (plus a small third one that is highlighted) to recover the shape of the lake.

The spline filled polygon is a MUCH better way to go if you can do it HiTech. 


Nice   :aok

It only remains to automate this process algorithmically  :confused:

My thought that big lakes would be brought down to 0 altitude and smaller would be created as shapes.

For example biggest and most relevant lakes for WW2 have relatively low altitude above sea level:

Ladoga 5m
Onega 33m
Vänern 44m
Saimaa 76m

So they can be safely brought down to 0 without hurting "geographical accuracy" too much.

I think that what I'll start with.. Especially when any river I add in TE crashes my Laptop to BSOD  :mad:
Title: Re: MakeAHMap and AH3
Post by: Easyscor on January 07, 2016, 04:23:37 PM
I think that what I'll start with.. Especially when any river I add in TE crashes my Laptop to BSOD  :mad:

I wonder if it's because the file doesn't exist yet.
Try this, create a NotePad file in your terrain folder and name it [myterrain].tra
Copy this text into the file and save.
See if you laptop will let the TE run with it. I'm pretty sure that it will.

~~~ 1st line starts with 1 below ~~~
1
2,Type
4,Flags
-1,ResourceType
-1,DestFieldIndex
-1,SourceFieldOrObjectIndex
5280.000000,SrcRadius
5280.000000,DestRadius
Track 0
4,PntCnt
132.386656,1.000000,-16848.007071
1023.029543,37.000000,-16844.570720
4276.734436,53.800000,-13824.979635
7575.369753,65.800000,-3347.292916

~~~~~~
A final CR(LF?) after the 6 brings the carrot to start the next blank line in NotePad but not in Word

It's a short river near the center of any terrain.

Good Luck
Title: Re: MakeAHMap and AH3
Post by: 715 on January 07, 2016, 06:54:19 PM
Nice   :aok

It only remains to automate this process algorithmically  :confused:

My thought that big lakes would be brought down to 0 altitude and smaller would be created as shapes.

For example biggest and most relevant lakes for WW2 have relatively low altitude above sea level:

Ladoga 5m
Onega 33m
Vänern 44m
Saimaa 76m

So they can be safely brought down to 0 without hurting "geographical accuracy" too much.

I think that what I'll start with.. Especially when any river I add in TE crashes my Laptop to BSOD  :mad:

Yeah, the algorithm looks like it could be difficult.  I think I'm just going to wait to see if HiTech adds a spline closed polygon lake object.

Pushing below 0 sounds viable for a lot of lakes.  (Remember you have to go a little distance below 0, not just to 0, or you will see the ground through the shallow water.)  Of course there is Lake Lausanne at nearly 400 m.  That might end up looking a bit weird if sunk to 0m.  ;)
Title: Re: MakeAHMap and AH3
Post by: artik on January 08, 2016, 12:59:24 AM
I wonder if it's because the file doesn't exist yet.
Try this, create a NotePad file in your terrain folder and name it [myterrain].tra
Copy this text into the file and save.
See if you laptop will let the TE run with it. I'm pretty sure that it will.

Great it worked, probably the issue with track editing itself - also note it is Graphics error as my Intel HD card becomes unresponsive and after 1 or 2 "recoveries" PC crashes into BSOD.

Now I can go with it.

Pushing below 0 sounds viable for a lot of lakes.  (Remember you have to go a little distance below 0, not just to 0, or you will see the ground through the shallow water.)  Of course there is Lake Lausanne at nearly 400 m.  That might end up looking a bit weird if sunk to 0m.  ;)

The biggest issue with pushing lakes to 0 is actually that rivers connecting lake and sea would flow at some points from low altitude to high one  :confused:
Title: Re: MakeAHMap and AH3
Post by: Easyscor on January 08, 2016, 01:37:02 AM
I'm glad it worked.
Title: Re: MakeAHMap and AH3
Post by: 715 on January 11, 2016, 03:38:54 AM
715:

They are not necessary, I simply put them there to make the file easier to read.


Also what is the spacing you are using between points? That's sounds like a very large number since we already smooth with splines?

If so I may wish to write an optimizer to remove points X distance from the spline curve.

HiTech

HiTech:  I imported all the rivers (and lakes, as outlines) from the fine resolution GSHGG dataset for a 512 mile terrain in the Kursk area of Russia/Ukraine.  There were 211 tracks and a total of 21,212 points.  The TE did not choke.  A cursory check showed the rivers appear to render properly in the TE.  I'll try Europe next (which has more rivers), but so far so good.  (I didn't try building the terrain as it doesn't do rivers in the game yet.)

715
Title: Re: MakeAHMap and AH3
Post by: hitech on January 11, 2016, 08:52:55 AM
HiTech:  I imported all the rivers (and lakes, as outlines) from the fine resolution GSHGG dataset for a 512 mile terrain in the Kursk area of Russia/Ukraine.  There were 211 tracks and a total of 21,212 points.  The TE did not choke.  A cursory check showed the rivers appear to render properly in the TE.  I'll try Europe next (which has more rivers), but so far so good.  (I didn't try building the terrain as it doesn't do rivers in the game yet.)

715

I'm curious do they look reasonable? Also post a zip of the files at some point, it will make a good base test terrain for when I get back to it.

HiTech
Title: Re: MakeAHMap and AH3
Post by: 715 on January 11, 2016, 01:50:51 PM
I'm curious do they look reasonable? Also post a zip of the files at some point, it will make a good base test terrain for when I get back to it.

HiTech

They looked reasonable except where my program messed up: I split rivers into 256 point sections (Easyscor told me that was a limit), but I forgot to duplicate the endpoints so the rivers have occasional gaps.  I'll post a zip once I fix that error.
Title: Re: MakeAHMap and AH3
Post by: Easyscor on January 11, 2016, 04:03:58 PM
They looked reasonable except where my program messed up: I split rivers into 256 point sections (Easyscor told me that was a limit), but I forgot to duplicate the endpoints so the rivers have occasional gaps.  I'll post a zip once I fix that error.

Yep, but I always want more. I'm a greedy bastage.
Title: Re: MakeAHMap and AH3
Post by: 715 on January 12, 2016, 01:32:04 AM
I'm curious do they look reasonable? Also post a zip of the files at some point, it will make a good base test terrain for when I get back to it.

HiTech

I zipped the terrain directory but it's still 140MB (way above the 20MB limit).  What are the minimum files you would need?

The only problem I saw in the TE was some of the rivers don't show the square box spline control points (but you can select them).

In many parts of the rivers the number of control points appears to be more than necessary.  In the future I will try the lower resolution database.
Title: Re: MakeAHMap and AH3
Post by: hitech on January 12, 2016, 09:18:55 AM
*.htz (splat and height) oba and your rivers ( i forgot the name)

HiTech
Title: Re: MakeAHMap and AH3
Post by: 715 on January 12, 2016, 01:53:08 PM
*.htz (splat and height) oba and your rivers ( i forgot the name)

HiTech

Still no go: heightmap.htz is 42MB (even bigger than the source .raw file) and it doesn't compress much at all in zip.  This terrain is based on high resolution DEM data (NASA 3 microsec SRTM) and it has a lot of fine structure.  Therefore it doesn't compress well.  I've attached the river track file and for testing you could just add it onto a blank flat 512 terrain (that is all land, this region has no ocean).

file: KurskRivers.zip -> Kursk.tra, data from GSHHG available under GNU Lesser General Public License.  Note: some lakes are added as a 99ft wide river outline (pending addition of a lake object or for filling in via Artik's concentric technique).
Title: Re: MakeAHMap and AH3
Post by: ghostdancer on January 26, 2016, 09:56:44 AM
Hey Artik I was wondering if it would possible to have a version of the current MakeAHMap that outputs the elevation to 4096x4096 instead of 1024x1024? Currently I still do most of my work / editing in Photoshop so it is easy enough for me to take the waterd and size it down to 4096. But sizing up the gndelv.bmp file to 4096 loses quality obviously. So if there was a version of your program that created a 4096x4096 BMP I could use that to construct the height.raw in Photoshop.

I am hoping that it would be just a quick and easy change to make and yes I know it would be a stop gap for now.

Thanks.
Title: Re: MakeAHMap and AH3
Post by: artik on January 26, 2016, 10:24:20 AM
Actually raw altitude file of 4096x4096 is already working. However water stuff is still very limited and does not work 100%. Also no ground cover.

The project now is at github: https://github.com/artyom-beilis/makeahmap/tree/ah3  I'll push changes to the repository ASAP, if you can't build the code I'll build the exe.


I was busy recently I'll try to get back into it soon.
Title: Re: MakeAHMap and AH3
Post by: ghostdancer on January 26, 2016, 11:03:55 AM
Not sure what you exactly mean by build? Do you mean bring it into Visual Studio and do a build / compile? If so yeah a new exe would be appreciated.