Aces High Bulletin Board

General Forums => The O' Club => Topic started by: deSelys on September 19, 2005, 05:13:11 AM

Title: How I shot myself in the foot last night...
Post by: deSelys on September 19, 2005, 05:13:11 AM
...or should I say in the HDD?

At work I manage 3 servers. The domain controller is a Windows 2003 server (std ed) with 3x72.8 GB HDD in RAID 5 ( 1 volume) = ~145 GB.
 We are running low on disk space in a couple of partitions so I had them buy a couple extra HDD to go to 5x72.8 GB HDD in RAID 5 (1 volume) =~290 GB.

To increase the volume size, there is no ther way than deleting everything and recreating the partitions = not fun at all! We are using Veritas Backup Exec 9 and, while looking for the best easiest way to recover evrything, I've stumbled upon horror stories about people not able to restore users, user rights or other Active Directory niceties from a BE backup.

So I was looking for another procedure and I realized that I could do everything with a KNOPPIX boot linux CD: I could copy bit-to-bit the whole volume onto an IDE HDD, increase the volume size, make a new partition table, copy back from the IDE drive the small partitions in the larger ones and then use ntfsresize to increase the file system size to match the partitions.

I just did that on a standalone computer and it worked.

This friday evening, I made a fresh backup, ran chkdsk -f on every partition and started to copy the volume onto the IDE drive.

Saturday atfernoon the copy was done and I could mount every partition in linux. I copied the original MBR and partition table onto a usbdisk as another backup...
I installed the 2 new HDDs, created the new volume and quickly installed Windows 2003 server on it just to be able to make new partitions "a la microsoft"... one can't be to careful.

Then I started to copy the original partitions onto the bigger ones in the new volume.

Yesterday evening I come back at the office, shut down the server and remove the IDE drive. I try to boot...no joy. The server was hanging at "attempting to boot from C". No big deal, I'm thinking, I'll just boot with the windows 2003 server cd and use the recovery console to see what fixboot and fixmbr have to say. But the administrator password was rejected every time (and yes, I had loaded the correct keyboard code). WTF??? I began to sweat buckets and tried continuously to boot this expensive piece of rotten junk......................... . until I made a baaaaaaaaad decision and replaced the new MBR with the old one. Then the Windows recovery console couldn't even find the previous install, and gpart couldn't recreate the partition table (too much inconsistencies, which is strange as the windows install was fresh and unmodified).

I had to replug the IDE drive and start again, this time removing the 2 new HDDs so the copy will go back on a drive with exactly the same geometry as before. This was at 4 am and the copy will last 20 hours

When people began to arrive this morning, I had to announce that the network is mostly down (only the database and mail are running) but they were cool with that. Fortunately for me, this is my first fediddleup since 5.5 years. Nevertheless, I feel too damn dumb for what I did (and retrospectively I can't even understand why I did that).

I've called this morning for some external help because I can't afford to lose everybody's time on this.  for A consultant will be there tomorrow to see if we can restart from the IDE restore, or if we have to use the tapes.

Wish me luck....
Title: How I shot myself in the foot last night...
Post by: Ghosth on September 19, 2005, 07:33:38 AM
OUCH!

Best of luck getting it back up bro.
Title: How I shot myself in the foot last night...
Post by: Mighty1 on September 19, 2005, 08:08:57 AM
Man that sucks.

I'm a Novell guy myself.

I can add drives to my raid 5 system without doing anything special.

I use Veritas BE myself and have had no problems with the rights being restored.

Not sure why your backup wouldn't go on but I know with the Novell version I can redirect the restore to another drive and it will keep it's rights.
Title: How I shot myself in the foot last night...
Post by: straffo on September 19, 2005, 08:13:02 AM
remind me one of my teacher :
teacher : some unix commands are dange rous

teacher : like su root ***
teacher : and rm / -rf
teacher : oooooppppssss

straffo : gmmmmppphhhh :D
Title: How I shot myself in the foot last night...
Post by: Trell on September 19, 2005, 09:15:43 AM
Ouch.

Becasue it is a domain controller you would not be able to boot into safe mode and normaly the recovery console gives me issues with dc.

you could try to install the system from scratch on a small partition(we normaly week the windows partition.  and try to restore using the systate.
With dcs it would restore your asdi information as well when you do this,  then manualy add back the rest of the information.

Your week   Sounds like mine.

I came in last tuesday to one of our domain controllers with a crumpted adsi information.  I could not get it up no matter what i did.  I ended up having to reinstal the os,  and add back everything else in (decided to not keep it a dc,  Was planning to pcpromo it anyways.  But it is hell.

Have been looking into some system resotr software like accronis for a whil;e and this gives me a reason to push it more to my bosses.
Title: Something I learned a while back...
Post by: rshubert on September 19, 2005, 01:16:37 PM
when I was still doing IT and networks.

NEVER NEVER NEVER screw around with the server hard drives.  It's easier and ultimately faster to add a new server with the new features you want, copy the data over, then decommission the old server.  It costs more, certainly, but then just how much is mr. consultant gonna gouge you for?  And frankly, any server more than a year or so old is hopelessy obsolete, anyway.

MS (and Novell, and...) all promise easy addition of hard drives to the RAID, but they LIE.  Something can go wrong, and your precious data can be lost.
Title: How I shot myself in the foot last night...
Post by: Trell on September 19, 2005, 02:13:28 PM
I agree that I allways cringe about adding drives, If I am moving data.

Normaly I would just Create a new raid set for this.  I also allways like having a second dc around for issues like this.

But compleatly disagree about servers baing out of date at only a year,  2/3 of our servers are over 1 year old with no plans to upgrade them.  We are more likly  to add more servers for the extra Tasks.
Title: How I shot myself in the foot last night...
Post by: 2bighorn on September 19, 2005, 03:10:43 PM
Well, that's why it's very important to choose the right controller for your server.
Most of decent hardware raid controllers allow easy raid expansion and most come with utilities with which you can resize volumes/partitions, if not, there's plenty of quality third party utilities which support raid 5.

It's also very good idea NOT to run OS on the same raid as your DATA is, and having ALL on the same volume equals suicide sooner or later.

Rebuilding raid 5 is slow and tedious process. Wish ya luck, you'll need it.
Title: How I shot myself in the foot last night...
Post by: Makarov9 on September 19, 2005, 03:27:34 PM
You can always blame it on Bush. :)
Title: How I shot myself in the foot last night...
Post by: Vulcan on September 19, 2005, 03:57:41 PM
http://www.acronis.com/enterprise/products/ATIESWin/
Title: How I shot myself in the foot last night...
Post by: deSelys on September 20, 2005, 03:53:54 AM
Hehehe the bit-to-bit copy of the data from the IDE back onto the RAID worked. I was also able to cancel the consultant intervention in time...W00t!
Of course, I'm back at the start grid as the 2 new drives aren't included in the volume. I need to make some more tests and then I'll start again...
Oh I now know why the administrator password wasn't accepted in the recovery console....because it wasn't asking for the administrator password (as stated on the screen) but for the Active Directory recovery password!!! Thanks a lot, MS.

I know that we aren't running the most failsafe configuration but our budget is limited. But I'll try to change our config in the future to make us more resilient to a DC failure.
Title: How I shot myself in the foot last night...
Post by: Saintaw on September 20, 2005, 06:53:07 AM
Does that mean this is a good time for you to loose my speeding tickets??? :D
Title: How I shot myself in the foot last night...
Post by: deSelys on September 20, 2005, 07:40:17 AM
Quote
Originally posted by Saintaw
Does that mean this is a good time for you to loose my speeding tickets??? :D


You got speeding tickets on your bicycle???? :huh
Title: How I shot myself in the foot last night...
Post by: Saintaw on September 20, 2005, 08:30:31 AM
a-it's a tricycle!
b-I'm fast!

Now, go jump off a plane or somethin' :D
Title: How I shot myself in the foot last night...
Post by: Boroda on September 20, 2005, 11:17:34 AM
Quote
Originally posted by Vulcan
http://www.acronis.com/enterprise/products/ATIESWin/


Had a funny problem with this tool this weekend. Failed to copy a system onto an identical (!!!) drive. Diagnosis: to gassenwagen. It's too damn smart.

They want 14000 rubles for a Linux version of True Image... It's $500 now. So they get awarded with a flag in their hands and a drum on their neck.

Looking for a good disk cloning programm now, trusted Norton Ghost failed to copy modern IBM/Hitachi drive ("fail to read data"), True Image tampered MBR copying _identical_ disks, where to look now?...
Title: How I shot myself in the foot last night...
Post by: Vulcan on September 20, 2005, 07:29:23 PM
True Image worked for me where ghost fell down. Dunno what happened with your experience but it gets the :aok from me (Win 2003 Server copy).
Title: Re: How I shot myself in the foot last night...
Post by: GRUNHERZ on September 20, 2005, 08:10:05 PM
Quote
Originally posted by deSelys
...or should I say in the HDD?

At work I manage 3 servers. The domain controller is a Windows 2003 server (std ed) with 3x72.8 GB HDD in RAID 5 ( 1 volume) = ~145 GB.
 We are running low on disk space in a couple of partitions so I had them buy a couple extra HDD to go to 5x72.8 GB HDD in RAID 5 (1 volume) =~290 GB.

To increase the volume size, there is no ther way than deleting everything and recreating the partitions = not fun at all! We are using Veritas Backup Exec 9 and, while looking for the best easiest way to recover evrything, I've stumbled upon horror stories about people not able to restore users, user rights or other Active Directory niceties from a BE backup.

So I was looking for another procedure and I realized that I could do everything with a KNOPPIX boot linux CD: I could copy bit-to-bit the whole volume onto an IDE HDD, increase the volume size, make a new partition table, copy back from the IDE drive the small partitions in the larger ones and then use ntfsresize to increase the file system size to match the partitions.

I just did that on a standalone computer and it worked.

This friday evening, I made a fresh backup, ran chkdsk -f on every partition and started to copy the volume onto the IDE drive.

Saturday atfernoon the copy was done and I could mount every partition in linux. I copied the original MBR and partition table onto a usbdisk as another backup...
I installed the 2 new HDDs, created the new volume and quickly installed Windows 2003 server on it just to be able to make new partitions "a la microsoft"... one can't be to careful.

Then I started to copy the original partitions onto the bigger ones in the new volume.

Yesterday evening I come back at the office, shut down the server and remove the IDE drive. I try to boot...no joy. The server was hanging at "attempting to boot from C". No big deal, I'm thinking, I'll just boot with the windows 2003 server cd and use the recovery console to see what fixboot and fixmbr have to say. But the administrator password was rejected every time (and yes, I had loaded the correct keyboard code). WTF??? I began to sweat buckets and tried continuously to boot this expensive piece of rotten junk......................... . until I made a baaaaaaaaad decision and replaced the new MBR with the old one. Then the Windows recovery console couldn't even find the previous install, and gpart couldn't recreate the partition table (too much inconsistencies, which is strange as the windows install was fresh and unmodified).

I had to replug the IDE drive and start again, this time removing the 2 new HDDs so the copy will go back on a drive with exactly the same geometry as before. This was at 4 am and the copy will last 20 hours

When people began to arrive this morning, I had to announce that the network is mostly down (only the database and mail are running) but they were cool with that. Fortunately for me, this is my first fediddleup since 5.5 years. Nevertheless, I feel too damn dumb for what I did (and retrospectively I can't even understand why I did that).

I've called this morning for some external help because I can't afford to lose everybody's time on this.  for A consultant will be there tomorrow to see if we can restart from the IDE restore, or if we have to use the tapes.

Wish me luck....


Phew!!


For a moment i thought  you were this guy... (http://www.limestonemedia.com/funny-video/video/cops_and_guns1.mov)
Title: How I shot myself in the foot last night...
Post by: MrCoffee on September 21, 2005, 01:35:16 AM
Quote
Originally posted by Makarov9
You can always blame it on Bush. :)


:D