A Technical Manager's Perspective

Event Monitoring and Support Tools

I have responsibility for both the Unix Operations and the Windows Operation within our company, and I struggle to find tools that work well in both environments. Both groups work autonomously and do a great job of supporting their respective environments, however as our responsibilities grow, I would like to find consistent tools for maintaining, monitoring and supporting our environments.

Two tools that do work in both environments are BMC Patrol for performance gathering and HP Openview for event notification and monitoring. We are not using Openview to monitor our network, instead we use it for system and event monitoring. We have a decent baseline of monitored events, however we struggle with event escalation especially when new errors or a new event occurs. We have installed IBM Director on the IBM nodes and it passes Hardware events to Openview, and we have scripted certain events which also hand off to Openview. There still is a lot of refinement needed in our process as many events enter Openview and are not reacted to because the criticality level is identified incorrectly or the followup and escalation are not defined in Openview. We are looking to reduce the number of system outages that impact our user community by improving our event escalation and notification.

Another tool that is in use in on our Unix platform is Tripwire.Tripwire is an auditing tool that will identify changes in files and will enable you to better track changes in your environment. We have started looking the Tripwire on the Wintel side and hope to have it deployed within a month.

I am open to others thoughts and experiences surrounding event monitoring and event notification in a multi-OS environment and would be glad to discuss this in detail.

Technorati : Infrastructure, Microsoft, Support, System Administration, Unix

Daylight Savings Time Patches

Most IT managers are aware of the change in Daylight Savings Time and the impact this will have on our environments. I think we can all agree that it is not going to crash our systems or impact us in a manner similar to that predicted for Y2K, however our challenge as Technical Managers is to ensure that our computer systems remain up, available and in sync from a time perspective.

These days with enterprise wide domain controllers, email, calendaring, blackberries, and multiple gadgets you do not want to have a problem with time synchronization.

Here is a list of some of the critical patches you may need for your environment:

- Windows Server Patches

- Windows Exchange Patches

- Blackberry Patches

Good Luck .....

Technorati : Infrastructure, Microsoft, System Administration, network-administration

Microsoft Virtual Hard Disks (VHD)

Last month I wrote about Microsoft's Virtual Lab Express which is a great tool for previewing and testing software.

As I have investigated further, I think I prefer the Microsoft Virtual Hard Disks (VHD), which are very similar, however the advantage is that you can download the VHD's to your environment, share them with your end users and then test the functionality in a very secure environment.

I feel that this is the new paradigm for software testing. You can create and entire virtual network within a VHD, along with the Active Directory, DNS, Front-End Application environment and Back End database server environment. You can use this environment for development, user acceptance testing or even offer this environment as a demo or testing environment to your customers.

If you are an IT professional with Microsoft based responsibilities, you should check out both the Microsoft's Virtual Lab Express and Microsoft Virtual Hard Disks.

Healthcare and technology

Working in Healthcare, you quickly realize that there are few adoptions of bleeding edge or newer technologies, instead we are lucky to have and deploy three year old technologies and strategies. We recently deployed a virtualization strategy that was piggy-backed on a Blade purchase. The Blade purchase was approved in 2005 for 2006 and we were allocated limited funds for a Blade chassis that would be populated with 6 or 7 servers.

After researching our options in early 2006, we realized that many folks were using Virtualization on Blade technology and we started looking at Virtualization solutions. We compared both Microsoft Virtual Server and VMWare ESX server and found the VMWare product to be the better product. We quickly realized the VMWare's ESX server was very stable and deployable, and we proceeded to spend funds allocated for the Blade project on virtualization.

We quickly realized that it was more cost efficient to deploy the VMWare 2 CPU licenses on a robust standalone server with alot of memory, than on a Blade server with limited memory and disk capacity. You can deploy more Virtual servers on a beefy server with 16 or 32 gigs of ram than you can on a Blade server with 8 to 16 gigs of ram. We proved the merits of VMWare from a cost perspective and a deployment perspective and expanded our resources in that area.

This deployment could not have happened if we did not have the Blade funds, however one thing that I have learned is that sometimes you need to be creative to get your job done. We spent more on Virtualization by spending less on the Blade infrastructure and in the long run this benefited the company.

Technorati : Management, virtualization, vmware

Oracle's Linux Strategy illustrates Oracles Power in Marketplace

At Oracle World last month, Chief Executive Larry Ellison told thousands of attendees at the Oracle OpenWorld conference last week, "If you are a Red Hat support customer, you can very easily switch from Red Hat support to Oracle support." as reported by CNET. This story has generated a lot of controversy in the Linux camps of the world, and is one of the top story's on digg and Linux Watch.

This direction illustrates the power that Oracle has in the Marketplace. Oracle is doing this to ensure another stable platform for its products, however it comes at the cost of RedHat. How would you feel if you were a developer at RedHat or a contributer to the RedHat distribution. I think one strategy that Oracle could take would be to add resources to both the RedHat and Oracle-Customized RedHat distributions. Oracle could and probably will reduce their Licensing structure for Linux based products in an effort to drive adoption to Linux.

Someone who is in favor of this new Oracle strategy is Dana Gardner at ZDNET who wrote ...

"If Oracle can subsidize the offering of subscription services to an acceptable and open (not too forked) Linux distribution (and these need to happen) through its commercial products' revenues, it will. It was okay for open source service providers to undercut Unix and Windows via their low-cost development and distribution means, right? What's good for the goose …" Follow this link for the complete article

Will current customers really move to Linux ?

For larger Enterprises with Oracle databases, this path needs to be cost effective and supportable. Oracle is trying to bring some credibility to the support side of its products on RedHat, and this may be an good strategy, if they really can provide the support. From a cost perspective, it is almost an even comparison of Hardware and OS choices. You still need to purchase RedHat and a support contract for your products. Hardware cost of Database Platform Intel based servers and Sun Servers are very similar.

So, in my opinion, there is not a compelling reason for Large Enterprise Organizations to move your production database structures to Linux, however this could be a nice strategy for Small and Medium sized businesses.

Technorati : Infrastructure, System Administration

Recovery of Damaged DVD

We have an older Pioneer DVD JukeBox that stores over 200 DVD's. This device was installed over 5 years ago and all of the folks involved in the installation have left our group. It was fairly dormant for the first 2 or 3 years however about 2 years ago we started using it as an archive repository. We use Point Software to virtually bind a number of DVD's together which then appear as a Share for folks and applications to use.

Over the last 6 months we have had a number of hardware related issues with the Jukebox. Internal switches would fail causing problems, shares would fail to mount and most recently two DVD drives had to be replaced. With this most recent failure we also realized that the bad DVD drive, damaged one of the DVD's in the share. There were 40 DVD's in the share and it would not mount because one of the DVD's was damaged. You could actually see the damage on one side of the DVD so we pulled the damaged DVD and worked through remounting the share with only 39 DVD's.

We tried to read the damaged DVD on multiple DVD readers however we could not read the DVD. We realized that we did not have a backup copy of the DVD, breaking Kevin's Rule Number 1 for all System Administrators, Backups are a Requirement. After further investigating, the original thought was that we backed up the data before archiving, so we would not need to backup or copy the archived data on the DVD server. However, with data spread across 40 drives, it was hard to determine which data was missing and a further complication was the fact that most of the data was from 2003 and we had selective backups from 2003 that probably would not work.

We were in trouble and Directors and VP's were looking for answers. I Goggled "Data Recovery" and identified thousands of companies that claimed they could restore your data. I called 4 different vendors and chose one close to work. I had a pay a premium to escalate our recovery however the vendor turned the restoration around in 2 days and recovered all of the data on the DVD. This allowed us to move the data back to the archive share however stressed the need for a backup or copy of our Archived DVD data.

Once the crisis died down, my team investigated how to copy a DVD within the Jukebox, and has started the process to make a duplicate DVD copy for all Archived DVD's in the Jukebox.

Though-out this process, all I could think about was RULE Number 1:

Backups are a Requirement and are Crucial to your success

Technorati : System Administration

Administrator Task List

You may have read my post from June on System and Network Administrators Tasks along with my little update to that post in July. If you happen to miss that post, I identified the tasks that I feel every System and Network Administrator need to understand to become better administrators. I am republishing this list, as I would like to touch base on a couple of topics related to this list over my next few posts.

1. Backups are a requirement and are crucial to your success
2. Monitoring Tools
3. Maintain Service Contracts with all vendors
4. Maintain Version Control
5. Consistent Hardware Builds and Software Installations
6. Documentation
7. Communication
8. Teamwork
9. Training and Schooling
10. Stay Current with Technology
11 Know your users or your customers
12 Know your line of business

Technorati : System Administration

Server Consolidation Update

At the end of June I posted and entry outlining our Server Consolidation Project, and I thought that I would update you on our progress and touch on our struggles and accomplishments.

We introduced an IBM Blade chassis along with VMware ESX Virtualization software for both the Blade environment and a separate dedicated virtual server environment. We purchased a IBM Blade Chassis H with 4 servers, along with 2 IBM x366 servers which have 4 Dual Core CPU's and 16 gig of ram in each server. We have also reallocated 2 IBM x345 servers for use in the Virtual Space. From a Virtual Licensing perspective we have purchased 2 VMware ESX 4 CPU license keys along with 6 VMware ESX 2 CPU license keys.

We struggled with the initial setup and configuration of the Blade Chassis, however with a little help from IBM and a lot of troubleshooting and hard work, my team has stabilized the Blade environment and we have started a pilot project on the Blade Servers. Our strategy will be to test a couple of applications on Blade servers running Windows 2003 Server and test a couple of applications running on a VMware ESX - Windows 2003 Virtual Server.

On the Virtual side we started with the 2 IBM x345's and then introduced the 2 IBM x366's. What we have found is VMware ESX server performs better on bigger faster servers with a lot of memory, than on the smaller Blade servers, however knowing this gives us deployment options. Smaller, low cpu, low memory, low IO utilized server requirements fit nicely on the blades, whereas high CPU and high Memory requirements will be deployed on the IBM x366 servers.

Heating concerns with the Blade Unit forced us to reorganize our data center cabinet layout and move 2 cabinets directly in front of a large libert air conditioning unit. We move the blade chassis into one of these cabinets and we are considering moving the IBM x366's into the second cabinet.

Overall, I am very pleased with the VMware Virtualization results. VMware is very stable and works well, however I am not as pleased with the Blade Unit. I think the blade unit can be a great solution for smaller data centers that do not have a lot of space, however my initial impression is that it was costly, technically challenging and a little under-power for our needs.

Technorati : System Administration, virtulization

Microsoft Security Patches

If you are in a Technical Role, you probably already know that Microsoft released 9 Critical and 3 Important patches this week. If you have not aware of this then you can get more information at http://www.microsoft.com/technet/security/bulletin/ms06-aug.mspx

One of the patches, MS06-40 fixes a critical problem that could allow hackers to take control of your system and run programs and bots from your system. This flaw is very similar to a flaw that was exploited 2 years ago called BLASTER, which took down many business networks. The US Department of Homeland security warned businesses that they could be at risk if they do not apply this patch http://www.infoworld.com/article/06/08/09/HNmspatchdhs_1.html .

You should consider applying MS06-40 to all servers, workstations, desktops and laptops as soon as possible to avoid problems.

Technorati : Microsoft, System Administration, network-administration

Continuation of my Top 10 System and Network Administrator Tasks

Top 12 System and Network Administrator Tasks

Last week, on the In the Trenches podcast, Kevin and George reviewed of my list of System and Network Administrator Tasks. I really appreciated the review and all comments. George and Kevin spoke of a couple of Tasks or Requirements that could be added to the list and I have to agree with them. The two other items were ….

1) Know your users or your customers

2) Know your line of business.

I certainly agree with these and I am going to change my list to a Top 12 System and Network Administrator Tasks. Thanks for the input and thanks for the link.

Here is Kevin and George’s Link for In the Trenches:

http://www.kevindevin.com

Here is the link to their Podcast:

In the Trenches Podcast.mp3