Hey Guys... Long time...
Well I've never been so busy in my life. One of the things keeping me busy is straightening out my IBM Lowell campus IT situation. I was promoted from Tivoli tech support to IT Administrator. I'm responsible for 65 users on laptops, 250+ Development Sun servers running our product, and everything else. Basically if it has a port I support it.
The previous IT admin left us nothing in his path. This means we don't know WTF he did with everything. One of the biggest tasks is gaining control of our backup and recovery stuff. We have a Win2003 backup server pointing to 20 or so systems to be backed up. This is connected to a large ADIC Scalar 100 via Veritas Net backup.
We have around 500 tapes in circulation with Iron Mountain. This is our off site tape disaster recover site.
I think at this point we need to get a consultant in to straighten this out. There's tons of policies in place in veritas and no real way to figure this crap in time for our next tape swap out.
What I need to know is.. is there a consultant service for this. We need to have someone to come in and take a look @ what we got and figure out all our BU/recover needs and teach me what I need to do. We don't have time to goto Veritas school. We need a quick and dirty way of getting this done.
Where does me and my company turn for this??
Thanks
All you uber system admins out there... please read
-
[JiF]Crash
- Forum User
- Posts: 76
- Joined: Fri Aug 25, 2006 12:45 pm
-
[JiF][AARP]Grimp
- Moderator
- Posts: 3803
- Joined: Thu Aug 24, 2006 9:23 pm
- Location: Massachusetts
Yet another useless reply by Grimp:
Dear Mr. Baker,
As a graduate of an institution of higher education, I have a few very basic expectations. Chief among these is that my direct superiors have an intellect that ranges above the common ground squirrel. After your consistent and annoying harassment of my coworkers and me during the commission of our duties, I can only surmise that you are one of the few true genetic wastes of our time.
Asking me, a network administrator, to explain every little nuance of everything I do each time you happen to stroll into my office is not only a waste of time, but also a waste of precious oxygen. I was hired because I know how to network computer systems, and you were apparently hired to provide amusement to myself and other employees, who watch you vainly attempt to understand the concept of "cut and paste" for the hundredth time.
You will never understand computers. Something as incredibly simple as binary still gives you too many options. You will also never understand why people hate you, but I am going to try and explain it to you, even though I am sure this will be just as effective as telling you what an IP is. Your shiny new iMac has more personality than you ever will.
You walk around the building all day, shiftlessly looking for fault in others. You have a sharp dressed useless look about you that may have worked for your interview, but now that you actually have responsibility, you pawn it off on overworked staff, hoping their talent will cover for your glaring ineptitude. In a world of managerial evolution, you are the blue-green algae that everyone else eats and laughs at. Managers like you are a sad proof of the Dilbert principle. Since this situation is unlikely to change without you getting a full frontal lobotomy reversal, I am forced to tender my resignation, however I have a few parting thoughts.
1. When someone calls you in reference to employment, it is illegal for you to give me a bad recommendation. The most you can say t o hurt me is "I prefer not to comment." I will have friends randomly call you over the next couple of years to keep you honest, because I know you would be unable to do it on your own.
2. I have all the passwords to every account on the system, and I know every password you have used for the last five years. If you decide to get cute, I am going to publish your "favorites list", which I conveniently saved when you made me "back up" your useless files. I do believe that terms like "Lolita" are not usually viewed favorably by the administration.
3. When you borrowed the digital camera to "take pictures of your Mother's birthday," you neglected to mention that you were going to take pictures of yourself in the mirror nude. Then you forgot to erase them like the techno-moron you really are. Suffice it to say I have never seen such odd acts with a sauce bottle, but I assure you that those have been copied and kept in safe places pending the authoring of a glowing letter of recommendation. (Try to use a spell check please; I hate having to correct your mistakes.)
Thank you for your time, and I expect the letter of recommendation on my desk by 8:00 am tomorrow. One word of this to anybody, and all of your little twisted repugnant obsessions will be open to the public. Never f*** with your systems administrator. Why? Because they know what you do with all that free time!
Wishing you a grand and glorious day,
Crash
Dear Mr. Baker,
As a graduate of an institution of higher education, I have a few very basic expectations. Chief among these is that my direct superiors have an intellect that ranges above the common ground squirrel. After your consistent and annoying harassment of my coworkers and me during the commission of our duties, I can only surmise that you are one of the few true genetic wastes of our time.
Asking me, a network administrator, to explain every little nuance of everything I do each time you happen to stroll into my office is not only a waste of time, but also a waste of precious oxygen. I was hired because I know how to network computer systems, and you were apparently hired to provide amusement to myself and other employees, who watch you vainly attempt to understand the concept of "cut and paste" for the hundredth time.
You will never understand computers. Something as incredibly simple as binary still gives you too many options. You will also never understand why people hate you, but I am going to try and explain it to you, even though I am sure this will be just as effective as telling you what an IP is. Your shiny new iMac has more personality than you ever will.
You walk around the building all day, shiftlessly looking for fault in others. You have a sharp dressed useless look about you that may have worked for your interview, but now that you actually have responsibility, you pawn it off on overworked staff, hoping their talent will cover for your glaring ineptitude. In a world of managerial evolution, you are the blue-green algae that everyone else eats and laughs at. Managers like you are a sad proof of the Dilbert principle. Since this situation is unlikely to change without you getting a full frontal lobotomy reversal, I am forced to tender my resignation, however I have a few parting thoughts.
1. When someone calls you in reference to employment, it is illegal for you to give me a bad recommendation. The most you can say t o hurt me is "I prefer not to comment." I will have friends randomly call you over the next couple of years to keep you honest, because I know you would be unable to do it on your own.
2. I have all the passwords to every account on the system, and I know every password you have used for the last five years. If you decide to get cute, I am going to publish your "favorites list", which I conveniently saved when you made me "back up" your useless files. I do believe that terms like "Lolita" are not usually viewed favorably by the administration.
3. When you borrowed the digital camera to "take pictures of your Mother's birthday," you neglected to mention that you were going to take pictures of yourself in the mirror nude. Then you forgot to erase them like the techno-moron you really are. Suffice it to say I have never seen such odd acts with a sauce bottle, but I assure you that those have been copied and kept in safe places pending the authoring of a glowing letter of recommendation. (Try to use a spell check please; I hate having to correct your mistakes.)
Thank you for your time, and I expect the letter of recommendation on my desk by 8:00 am tomorrow. One word of this to anybody, and all of your little twisted repugnant obsessions will be open to the public. Never f*** with your systems administrator. Why? Because they know what you do with all that free time!
Wishing you a grand and glorious day,
Crash
http://www.sapphire.com/
I interviewed with them when I was looking for a new job. I found something before they got me anything but they do have a lot of resources. Might be worth a look if you are serious.
We use Veritas backup here for 5 servers that I setup myself, it's really not hard if you can spend some time on it. My setup is not nearly as complex as yours sounds though.
I interviewed with them when I was looking for a new job. I found something before they got me anything but they do have a lot of resources. Might be worth a look if you are serious.
We use Veritas backup here for 5 servers that I setup myself, it's really not hard if you can spend some time on it. My setup is not nearly as complex as yours sounds though.
I can't help you with specifics but:
Do you have the ex-admins PC to rummage through? if so look for who he used as support contacts. Get a list of all the vendors that have been used and contact them asap and have them look up previous trouble ticket notes. Don't toss even the dumbest looking note left at his desk for at least a year.
Honelty the best situation would be to hire the guy back as a consultant for some dumb over priced hourly fee and get the information you need. Even approch the person yourself (help I got stuck with your job attitude) and offer to pay for the information and have the company reimburse you. The time spent on another con-insultant will likely be as much since they will have to figure stuff out.
Even though your going to get heat for the things that will happen it is entirly the managments fault for not have another person know all of the information you need. There should always be a backup administrator, and documentation for if there isn't one left, thats part of DR. Ask them who the back up admin was, or even who worked with him when the equipment was installed/set up.
And find froums that deal with your software and equipment.
That all I can help with.
Do you have the ex-admins PC to rummage through? if so look for who he used as support contacts. Get a list of all the vendors that have been used and contact them asap and have them look up previous trouble ticket notes. Don't toss even the dumbest looking note left at his desk for at least a year.
Honelty the best situation would be to hire the guy back as a consultant for some dumb over priced hourly fee and get the information you need. Even approch the person yourself (help I got stuck with your job attitude) and offer to pay for the information and have the company reimburse you. The time spent on another con-insultant will likely be as much since they will have to figure stuff out.
Even though your going to get heat for the things that will happen it is entirly the managments fault for not have another person know all of the information you need. There should always be a backup administrator, and documentation for if there isn't one left, thats part of DR. Ask them who the back up admin was, or even who worked with him when the equipment was installed/set up.
And find froums that deal with your software and equipment.
That all I can help with.
-
[JiF]Dandyman
- Forum User
- Posts: 147
- Joined: Fri Aug 25, 2006 3:28 pm
- Location: South Central (PA)
- Contact:
Please under no circumstances should you hire the person back as a consultant. It sets a bad precedent that your company will not only tolerate people who do not document, but reward them in the end. I've worked in a lot of IT environment with hundreds upon hundreds of different IT folks throughout my career. People who do not document will never document. Sometimes these people are just lazy or overworked, but most of the time they hide knowledge intentionally. They squirrel away proprietary knowledge to make themselves more valuable and powerful. Bring them in as a consultant and they will absolutely get worse. They'll drag their feet because they're hourly. They'll drag their feet because they know they've got you over a barrel. They'll drag their feet because they are incompetent and didn't document because they never really understood it either. I've seen this situation more than a handful of times and every single one ended with the person either being fired or walking off the job without any real progress having been made.
You need to move forward at this point and I recommend one of three possible courses of action:
1. Contact Veritas and find our who your sales rep is and call them. They should be able to provide you with in an in-house consultant OR know a third party consultant service they can get a hold of quickly.
2. You're IBM, so call IBM. IBM Global Services has one of the largest consultant stables in the world and I know for a fact that they have people conversant with NetBackup.
3. Now might be the time to switch platforms. I'm not a big fan of Tivoli Storage Manager, but it works and the cost should be pretty low to you.
Last but not least you need to take some steps to protect yourself in the meantime. The important thing to realize is that if you don't know what state your backups are in, then they are BROKEN. Do not count on them, do not expect them to serve you when disaster strikes and start doing some quick and dirty backups the hard way. Try the following:
- Identify relatively static (changes rarely) data and burn it to DVD or CD. Make two copies and take a set home with you (if allowed) otherwise send it to another business location.
- Purchase two external 500GB hard drives. Connect one to a server, share the drive and go around and create some jobs to copy important data to the share nightly. Take the disk containing the previous day's backup home with you and attach the other disk. You might have to reshare the drive. Swap disks in the morning and take the other disk home that day.
- For really big data sets you're going to have a hard time with either of the two tips. If you have a data set that is too big or simply cannot be backed up through an easy method make sure your boss knows about it. If its something like an Oracle or SQL database try and find another business group that might be able to host a copy of the database for you.
-Don
You need to move forward at this point and I recommend one of three possible courses of action:
1. Contact Veritas and find our who your sales rep is and call them. They should be able to provide you with in an in-house consultant OR know a third party consultant service they can get a hold of quickly.
2. You're IBM, so call IBM. IBM Global Services has one of the largest consultant stables in the world and I know for a fact that they have people conversant with NetBackup.
3. Now might be the time to switch platforms. I'm not a big fan of Tivoli Storage Manager, but it works and the cost should be pretty low to you.
Last but not least you need to take some steps to protect yourself in the meantime. The important thing to realize is that if you don't know what state your backups are in, then they are BROKEN. Do not count on them, do not expect them to serve you when disaster strikes and start doing some quick and dirty backups the hard way. Try the following:
- Identify relatively static (changes rarely) data and burn it to DVD or CD. Make two copies and take a set home with you (if allowed) otherwise send it to another business location.
- Purchase two external 500GB hard drives. Connect one to a server, share the drive and go around and create some jobs to copy important data to the share nightly. Take the disk containing the previous day's backup home with you and attach the other disk. You might have to reshare the drive. Swap disks in the morning and take the other disk home that day.
- For really big data sets you're going to have a hard time with either of the two tips. If you have a data set that is too big or simply cannot be backed up through an easy method make sure your boss knows about it. If its something like an Oracle or SQL database try and find another business group that might be able to host a copy of the database for you.
-Don
----
Dandy Deeds Done Dirt Cheap
Dandy Deeds Done Dirt Cheap
-
[JiF]Timmay!
- Forum User
- Posts: 378
- Joined: Thu Aug 24, 2006 12:41 pm
- Location: MA, USA
I agree with pretty much everything Dandy says but would add a CYOA clause to it.. Document the situtation from your point of view and make sure your management sees this document. Keep it short, be impartial, stick to the facts and note areas that you either have a plan for or need a plan for. Personally, I like PPT slides for things like this - make it a living document, adding new pages at the front as status updates.
If you do this and something bad happens, you've Covered Your Own parts. If you do this and something bad doesn't happen, you'll come out looking good for managing the problem.
If you do this and something bad happens, you've Covered Your Own parts. If you do this and something bad doesn't happen, you'll come out looking good for managing the problem.
Way better advice than mine, I spoke to quickly, sry. I'm wasn't thinking of that kind of charicter of people, I'm used to good people[JiF]Dandyman wrote:Please under no circumstances should you hire the person back as a consultant. It sets a bad precedent that your company will not only tolerate people who do not document, but reward them in the end. I've worked in a lot of IT environment with hundreds upon hundreds of different IT folks throughout my career. People who do not document will never document. Sometimes these people are just lazy or overworked, but most of the time they hide knowledge intentionally. They squirrel away proprietary knowledge to make themselves more valuable and powerful. Bring them in as a consultant and they will absolutely get worse. They'll drag their feet because they're hourly. They'll drag their feet because they know they've got you over a barrel. They'll drag their feet because they are incompetent and didn't document because they never really understood it either. I've seen this situation more than a handful of times and every single one ended with the person either being fired or walking off the job without any real progress having been made.
~~~~~~~~~~~
-Don
-
[JiF]Dandyman
- Forum User
- Posts: 147
- Joined: Fri Aug 25, 2006 3:28 pm
- Location: South Central (PA)
- Contact:
-
[JiF]DispDave
- Forum User
- Posts: 244
- Joined: Thu Aug 24, 2006 11:39 am
- Location: Worcester, MA
Vinster....you wouldn't be the Admin in question, now would you? :-)Vinster wrote:
Honelty the best situation would be to hire the guy back as a consultant for some dumb over priced hourly fee and get the information you need. Even approch the person yourself (help I got stuck with your job attitude) and offer to pay for the information and have the company reimburse you.
Hardest part of the job is documenting. I have some but not nearly as much as I should. I am in a situation that when I leave or go on vacation, nothing better go wrong. I have cross trained a few people for a some of the vital things, but I am the only person here that soley deals with the PC's. 
Not to steer this thread in another direction, but since we are on the topic does anyone know of any resources that would help someone document? I'm thinking of doc templates and things of that nature. I hate documenting simply because any documentation is a constantly evolving monster that can be outdated tomorrow.
Not to steer this thread in another direction, but since we are on the topic does anyone know of any resources that would help someone document? I'm thinking of doc templates and things of that nature. I hate documenting simply because any documentation is a constantly evolving monster that can be outdated tomorrow.
-
[JiF]Timmay!
- Forum User
- Posts: 378
- Joined: Thu Aug 24, 2006 12:41 pm
- Location: MA, USA
-
[JiF]Crash
- Forum User
- Posts: 76
- Joined: Fri Aug 25, 2006 12:45 pm
Yea.. we have an internal swiki that I'm creating now to document. Better than M$ word[JiF]Mike wrote:Hardest part of the job is documenting. I have some but not nearly as much as I should. I am in a situation that when I leave or go on vacation, nothing better go wrong. I have cross trained a few people for a some of the vital things, but I am the only person here that soley deals with the PC's.
Not to steer this thread in another direction, but since we are on the topic does anyone know of any resources that would help someone document? I'm thinking of doc templates and things of that nature. I hate documenting simply because any documentation is a constantly evolving monster that can be outdated tomorrow.
-
[JiF]Dandyman
- Forum User
- Posts: 147
- Joined: Fri Aug 25, 2006 3:28 pm
- Location: South Central (PA)
- Contact:
-
[JiF]Crash
- Forum User
- Posts: 76
- Joined: Fri Aug 25, 2006 12:45 pm
Thanks again all..
Luckly I'm still friends with the old admin and has offered to answer any question I need. It's mostly why and where everyting is. I'm pretty good @ the admin stuff. A bump in the road is I have to shutdown the entire campus next thurs night for a total power down of the building to cut over a new doctors office on floor 1.
After getting over that I'm going to get certified in veritas netbackup.
Luckly I'm still friends with the old admin and has offered to answer any question I need. It's mostly why and where everyting is. I'm pretty good @ the admin stuff. A bump in the road is I have to shutdown the entire campus next thurs night for a total power down of the building to cut over a new doctors office on floor 1.
After getting over that I'm going to get certified in veritas netbackup.