Author Topic: How I shot myself in the foot last night...  (Read 602 times)

Offline deSelys

  • Gold Member
  • *****
  • Posts: 2512
How I shot myself in the foot last night...
« on: September 19, 2005, 05:13:11 AM »
...or should I say in the HDD?

At work I manage 3 servers. The domain controller is a Windows 2003 server (std ed) with 3x72.8 GB HDD in RAID 5 ( 1 volume) = ~145 GB.
 We are running low on disk space in a couple of partitions so I had them buy a couple extra HDD to go to 5x72.8 GB HDD in RAID 5 (1 volume) =~290 GB.

To increase the volume size, there is no ther way than deleting everything and recreating the partitions = not fun at all! We are using Veritas Backup Exec 9 and, while looking for the best easiest way to recover evrything, I've stumbled upon horror stories about people not able to restore users, user rights or other Active Directory niceties from a BE backup.

So I was looking for another procedure and I realized that I could do everything with a KNOPPIX boot linux CD: I could copy bit-to-bit the whole volume onto an IDE HDD, increase the volume size, make a new partition table, copy back from the IDE drive the small partitions in the larger ones and then use ntfsresize to increase the file system size to match the partitions.

I just did that on a standalone computer and it worked.

This friday evening, I made a fresh backup, ran chkdsk -f on every partition and started to copy the volume onto the IDE drive.

Saturday atfernoon the copy was done and I could mount every partition in linux. I copied the original MBR and partition table onto a usbdisk as another backup...
I installed the 2 new HDDs, created the new volume and quickly installed Windows 2003 server on it just to be able to make new partitions "a la microsoft"... one can't be to careful.

Then I started to copy the original partitions onto the bigger ones in the new volume.

Yesterday evening I come back at the office, shut down the server and remove the IDE drive. I try to boot...no joy. The server was hanging at "attempting to boot from C". No big deal, I'm thinking, I'll just boot with the windows 2003 server cd and use the recovery console to see what fixboot and fixmbr have to say. But the administrator password was rejected every time (and yes, I had loaded the correct keyboard code). WTF??? I began to sweat buckets and tried continuously to boot this expensive piece of rotten junk......................... . until I made a baaaaaaaaad decision and replaced the new MBR with the old one. Then the Windows recovery console couldn't even find the previous install, and gpart couldn't recreate the partition table (too much inconsistencies, which is strange as the windows install was fresh and unmodified).

I had to replug the IDE drive and start again, this time removing the 2 new HDDs so the copy will go back on a drive with exactly the same geometry as before. This was at 4 am and the copy will last 20 hours

When people began to arrive this morning, I had to announce that the network is mostly down (only the database and mail are running) but they were cool with that. Fortunately for me, this is my first fediddleup since 5.5 years. Nevertheless, I feel too damn dumb for what I did (and retrospectively I can't even understand why I did that).

I've called this morning for some external help because I can't afford to lose everybody's time on this.  for A consultant will be there tomorrow to see if we can restart from the IDE restore, or if we have to use the tapes.

Wish me luck....
Current ID: Romanov

It's all fun and games until someone loses an eye... then it's just a game to find the eye

'I AM DID NOTHING WRONG' - Famous last forum words by legoman

Offline Ghosth

  • AH Training Corps (retired)
  • Plutonium Member
  • *******
  • Posts: 8497
      • http://332nd.org
How I shot myself in the foot last night...
« Reply #1 on: September 19, 2005, 07:33:38 AM »
OUCH!

Best of luck getting it back up bro.

Offline Mighty1

  • Parolee
  • Silver Member
  • ****
  • Posts: 1161
How I shot myself in the foot last night...
« Reply #2 on: September 19, 2005, 08:08:57 AM »
Man that sucks.

I'm a Novell guy myself.

I can add drives to my raid 5 system without doing anything special.

I use Veritas BE myself and have had no problems with the rights being restored.

Not sure why your backup wouldn't go on but I know with the Novell version I can redirect the restore to another drive and it will keep it's rights.
I have been reborn a new man!

Notice I never said a better man.

Offline straffo

  • Persona Non Grata
  • Plutonium Member
  • *******
  • Posts: 10029
How I shot myself in the foot last night...
« Reply #3 on: September 19, 2005, 08:13:02 AM »
remind me one of my teacher :
teacher : some unix commands are dange rous

teacher : like su root ***
teacher : and rm / -rf
teacher : oooooppppssss

straffo : gmmmmppphhhh :D

Offline Trell

  • Nickel Member
  • ***
  • Posts: 693
How I shot myself in the foot last night...
« Reply #4 on: September 19, 2005, 09:15:43 AM »
Ouch.

Becasue it is a domain controller you would not be able to boot into safe mode and normaly the recovery console gives me issues with dc.

you could try to install the system from scratch on a small partition(we normaly week the windows partition.  and try to restore using the systate.
With dcs it would restore your asdi information as well when you do this,  then manualy add back the rest of the information.

Your week   Sounds like mine.

I came in last tuesday to one of our domain controllers with a crumpted adsi information.  I could not get it up no matter what i did.  I ended up having to reinstal the os,  and add back everything else in (decided to not keep it a dc,  Was planning to pcpromo it anyways.  But it is hell.

Have been looking into some system resotr software like accronis for a whil;e and this gives me a reason to push it more to my bosses.

Offline rshubert

  • Silver Member
  • ****
  • Posts: 1462
Something I learned a while back...
« Reply #5 on: September 19, 2005, 01:16:37 PM »
when I was still doing IT and networks.

NEVER NEVER NEVER screw around with the server hard drives.  It's easier and ultimately faster to add a new server with the new features you want, copy the data over, then decommission the old server.  It costs more, certainly, but then just how much is mr. consultant gonna gouge you for?  And frankly, any server more than a year or so old is hopelessy obsolete, anyway.

MS (and Novell, and...) all promise easy addition of hard drives to the RAID, but they LIE.  Something can go wrong, and your precious data can be lost.

Offline Trell

  • Nickel Member
  • ***
  • Posts: 693
How I shot myself in the foot last night...
« Reply #6 on: September 19, 2005, 02:13:28 PM »
I agree that I allways cringe about adding drives, If I am moving data.

Normaly I would just Create a new raid set for this.  I also allways like having a second dc around for issues like this.

But compleatly disagree about servers baing out of date at only a year,  2/3 of our servers are over 1 year old with no plans to upgrade them.  We are more likly  to add more servers for the extra Tasks.

Offline 2bighorn

  • Gold Member
  • *****
  • Posts: 2829
How I shot myself in the foot last night...
« Reply #7 on: September 19, 2005, 03:10:43 PM »
Well, that's why it's very important to choose the right controller for your server.
Most of decent hardware raid controllers allow easy raid expansion and most come with utilities with which you can resize volumes/partitions, if not, there's plenty of quality third party utilities which support raid 5.

It's also very good idea NOT to run OS on the same raid as your DATA is, and having ALL on the same volume equals suicide sooner or later.

Rebuilding raid 5 is slow and tedious process. Wish ya luck, you'll need it.

Offline Makarov9

  • Silver Member
  • ****
  • Posts: 949
      • http://www.332nd.org/
How I shot myself in the foot last night...
« Reply #8 on: September 19, 2005, 03:27:34 PM »
You can always blame it on Bush. :)

Offline Vulcan

  • Plutonium Member
  • *******
  • Posts: 9853

Offline deSelys

  • Gold Member
  • *****
  • Posts: 2512
How I shot myself in the foot last night...
« Reply #10 on: September 20, 2005, 03:53:54 AM »
Hehehe the bit-to-bit copy of the data from the IDE back onto the RAID worked. I was also able to cancel the consultant intervention in time...W00t!
Of course, I'm back at the start grid as the 2 new drives aren't included in the volume. I need to make some more tests and then I'll start again...
Oh I now know why the administrator password wasn't accepted in the recovery console....because it wasn't asking for the administrator password (as stated on the screen) but for the Active Directory recovery password!!! Thanks a lot, MS.

I know that we aren't running the most failsafe configuration but our budget is limited. But I'll try to change our config in the future to make us more resilient to a DC failure.
Current ID: Romanov

It's all fun and games until someone loses an eye... then it's just a game to find the eye

'I AM DID NOTHING WRONG' - Famous last forum words by legoman

Offline Saintaw

  • Platinum Member
  • ******
  • Posts: 6692
      • My blog
How I shot myself in the foot last night...
« Reply #11 on: September 20, 2005, 06:53:07 AM »
Does that mean this is a good time for you to loose my speeding tickets??? :D
Saw
Dirty, nasty furriner.

Offline deSelys

  • Gold Member
  • *****
  • Posts: 2512
How I shot myself in the foot last night...
« Reply #12 on: September 20, 2005, 07:40:17 AM »
Quote
Originally posted by Saintaw
Does that mean this is a good time for you to loose my speeding tickets??? :D


You got speeding tickets on your bicycle???? :huh
Current ID: Romanov

It's all fun and games until someone loses an eye... then it's just a game to find the eye

'I AM DID NOTHING WRONG' - Famous last forum words by legoman

Offline Saintaw

  • Platinum Member
  • ******
  • Posts: 6692
      • My blog
How I shot myself in the foot last night...
« Reply #13 on: September 20, 2005, 08:30:31 AM »
a-it's a tricycle!
b-I'm fast!

Now, go jump off a plane or somethin' :D
Saw
Dirty, nasty furriner.

Offline Boroda

  • Persona Non Grata
  • Platinum Member
  • ******
  • Posts: 5755
How I shot myself in the foot last night...
« Reply #14 on: September 20, 2005, 11:17:34 AM »
Quote
Originally posted by Vulcan
http://www.acronis.com/enterprise/products/ATIESWin/


Had a funny problem with this tool this weekend. Failed to copy a system onto an identical (!!!) drive. Diagnosis: to gassenwagen. It's too damn smart.

They want 14000 rubles for a Linux version of True Image... It's $500 now. So they get awarded with a flag in their hands and a drum on their neck.

Looking for a good disk cloning programm now, trusted Norton Ghost failed to copy modern IBM/Hitachi drive ("fail to read data"), True Image tampered MBR copying _identical_ disks, where to look now?...