Tuesday, January 27, 2009

wireless challenges of the Vocera badge

THIS just popped up last week and is worth a quick read for anyone planning to deploy Vocera. Vocera is a unique product, and it has unique requirements for optimal operation. I know I say that all the time, but this article gives you a chance to learn about the "nuances" of Vocera wireless operation through someone else's pain.

Many of the things that typically "fix" wireless problems in data networks will degrade or completely break communication to Vocera badges. Things like increasing power on the AP's or adding amplifiers will actually cause more problems. Even doing the "right thing" and adding more AP's can actually further degrade your network unless you re-tune the channelization and power output of all nearby existing AP's when you deploy them.

I wasn't involved in this project, and I have no information about this deployment beyond what Network World published. I have seen other people use these techniques to "tune in" Vocera, I've heard the results first hand, and they never work. The only way to get predictable, high quality voice in a Vocera environment is to do it right:
  • Read the Vocera Infrastructure Planning Guide.
  • Ensure ALL of your AP's have ALL of the recommended settings configured.
  • Have a wireless survey performed with spectrum analysis.
  • Make sure surveyor knows VOCERA'S requirements for acceptable results before they start.
  • Make sure they can detect devices that complete with badges like microwaves and cordless phones.
  • If there are dead spots, add properly configured AP's, tune in channels and power settings, and resurvey the area.
  • Put on a badge and "play test tone." Walk everywhere any badge wearer may go and confirm a good steady stream is coming out of the badge.
Questions? Got a comment from your troubleshooting experiences? Post them here!

Monday, January 19, 2009

4.1 upgrades

I know most Vocera systems are still running version 4.0 software, so I'd like to take a few minutes to talk about 4.1 and the upgrade process since this will be relevant to most sys admins in the very near future.

In this post I'll talk about:
  1. 4.1 Overview
  2. Preparing for the Upgrade
  3. Upgrade!
  4. Post Upgrade Tasks


4.1 Overview
4.1 has many improvements over 4.0, I won't go over all the NEW FEATURES but the biggies for me are:
  • The latest firmware for B2000's
  • Asset tracking
  • Scheduled reporting
  • Phone Access to Genie
If you want detailed info about 4.1, this would be a great time to contact your sales team.


Preparing for the Upgrade
The very first step is planning! We need to consider how the existing system was sized and how old it is. If we have a single server and this has become a critical application, is it time to build it into a cluster? If the hardware is old, is it time to upgrade it or replace it? If our VTS is co-resident on our VS do we need to split them as we expect to add redundancy or increase capacity? For either upgrading an existing box or buying a few platform, please review the SERVER SIZING MATRIX.

The next step if we are recycling the servers themselves is to check them for requirements. On each server check the installed drive letter for Vocera and make sure there is at least 25 Gigs available. Sometimes backups, log files, or previous upgrade media clogs up the disk and causes the installer to fail. As long as you are logged into the server, check the amount of memory the system is reporting and make sure it's the same. If the wrong amount showed up, or came slightly dislodged in transit we want to get it corrected.

Some of the new features will take advantage of having email integration configured. If your system doesn't have it configured you will probably need to coordinate it with the email team. Now is the time to line up those conversations and get the account created. Get a copy of the 4.1 ADMINISTRATION GUIDE so you can give them the section on the requirements.

A few security points; if you're not using https for the web console, consider if you want to enable Secure SSL on the 4.1 installation. Check the group in your system that has rights for Vocera administration. Is voice prints enabled? If not you might want to consider this too. These are both great conversations you can have with your engineer while planning for your upgrade.

This is also a great time to check our backups: make sure that we have them, make sure they are running every night, and be sure they are being either backed up off box or COPIED someplace where they are. If we have a STAGING SERVER we can restore one of those backups and take a test upgrade on the database before taking down the production boxes!

OK, with all of that done we need to contact our sales team and setup for the upgrade. If there is some lead time, and you are on ANY version of 4.0 other than SP8, then you should consider PATCHING your system to get firmware updates out to your badges sooner than later.



Upgrade!
Most of the responsibilities for the upgrade will fall on the engineer coming in to do it. The engineer should be ready when they arrive onsite with media, licenses, a few utilities to get through the process. As one of these engineers, I appreciate having a few things lined up:
  • a place to lay out a laptop and some papers
  • confirmation of Vocera Support entitlement
  • if possible, an upgrade window during hours covered by that contract
  • clear downtime window communications to end users well in advance
  • physical access to servers if needed
  • three badges with fresh batteries
OK, I hope that doesn't sound too needy.

I did have one upgrade that encountered a database issue during the process. Even with my experience and training, I wouldn't have been able to find the root problem on my own. This only happened once, but this is why I am so keen on aligning a safety net incase it's needed. I mention this as a reminder to eveyrone that they should treat their critical systems as such.


Post Upgrade Tasks
Your upgrade project plan should include a testing phase where all parts of the system are confirmed working. Things that should be included:
  • flashing badges
  • badge.properties has all servers
  • badge to badge by name
  • badge to badge by group
  • badge to internal extension
  • badge to external local number
  • badge to external long distance number
  • internal phone to badge by group
  • external phone to badge by name
  • broadcast one to many
  • broadcast response
  • VRS data load (check schedule)
  • VS backup (check schedule)
  • VTS has all VS servers
Once all system checks are out of the way, I'd say it's time for update training. Go check out VOCERA UNIVERSITY for 4.1 training. With that under your belt you will be ready to dig into the asset tracking, scheduled reporting, and Phone Access to Genie functionality!

I'm sure someone has something to add on this post, please share it!

Wednesday, January 14, 2009

the case for a staging server

All Vocera customers holding an enterprise license are allowed to build a lab server for testing purposes within certain guidelines, it's called a staging server. If you look at your original license file and you see these lines you should be entitled:

*** snip
VOCERA_LICENSE:
Staging Server License:
Report Server:
*** end snip

A few things:
  • If you just see the "VOCERA_LICENSE" line it means you are probably looking at a standard license (but if you've upgraded from standard to enterprise it might be the wrong copy of the license,.)
  • If you don't see the staging server key but the "Type Of System:" contains the word "Enterprise" you should contact support.
  • If you aren't sure or can't find the file, call Vocera support and ask for a copy of your current license.
One quick side note: if you have a VRS and that line is blank in your license, please make sure you have your license key for it backed up somewhere!

Assuming you see your staging server key in that file, you should do a little background reading in the VOCERA INSTALLATION GUIDE VERSION 4.1 You should find a whole section on what a staging server is, what it's for, and suggestions for deployment architectures for them.

I think you should build a staging server somewhere! I like the "Staging Server connected to Production VLAN" option because it allows for testing badges on the same AP's used in production, but that's just personal preference.

Whatever architecture you choose, this will give you the ability to:
  • Restore a production database to the staging server
  • Test new settings during the day without effecting end users
  • Take your time practicing with service pack updates before requesting a downtime window for patching production servers
  • Upgrade a VS during business hours and document the process for change control
  • Perform remediation steps on an offline system to see if they are successful
  • Test new versions of anti-virus, backup software, and other third party application additions

With that I will rest my case, I hope you can see benefits of having such a device at your disposal. If you have any questions or comments please feel free to post them below!

Saturday, January 10, 2009

adopting an existing system

I've held off on updating the blog for a while to keep LUKE'S POST at the top of the page. This is really great info so if you missed it please take a little time to go back to it.

As I've been running around doing 4.1 upgrades I've found that lots of existing systems have been inherited by people who weren't involved in the original deployment project. Since most people in this position don't have a clear path to ask the questions they have, or maybe even know what questions they should ask, I thought I would take a post and put out some advice on the matter. Here's our plan:

  1. contact Vocera support
  2. gather system info
  3. look at current usage
  4. check the database
  5. get training!


Contact Vocera Support

First step, hopefully before you even have a problem to report, you need to contact Vocera support and get setup with them. You are going to want to document some things so have a pen or keyboard handy.

First thing open up the VOCERA SUPPORT page, take note of the email address and save that to your contacts. Also pick a phone number to put in there and your speed dials too;
408-882-5100 or
800-331-6356

OK now that we know how to get in touch with them, go ahead and setup a support login, you should see a list of info you need to send them on the main support page. It could take a little time to get that account created, don't worry if it does.

Once your account is created go ahead and log into the Support Center. The first page is the "Home" tab and it tells you contact info, hours and holidays, the current top Knowledge Base articles, and any open cases you have. Very good info!

Just for a little practice, go to the Find Knowledge Base Article tab, and search for "ftp" and you should get a hit "How should large amounts of data be transmitted to Vocera? (FTP)" go ahead and click that link. I suggest you copy that info, print it, keep it handy somehow. If it changes you now know how to find it.

Let's give them a call during normal business hours. You should get a live, friendly, and helpful person on the phone in pretty short order. I would ask them for the following:
  • support contract type (standard or 24x7)
  • renewal date
  • names of people allowed to open cases
  • if there are any open tickets you can't see
  • a copy of your license key if you don't already have one
  • your account managers name and contact information

There is your crash course on Vocera Support, now that we have a safety net in place we are moving on!


Gather System Info

Time to look at what we have installed, here is the lingo:

VS or Vocera Server; any server running Vocera. There is often more than one of these running in a cluster.

VTS or Vocera Telephony Server; a server with a Dialogic card that acts as a PBX gateway for VS servers.

VRS or Vocera Reports Server; this server crunches advanced reports for you, these are worth their weight in gold, I hope you have one.

Staging Server; this is an isolated VS that can be used for testing and lab work.

BCWS or Badge Configuration Workstation; there should be a pc somewhere used to flash new badges, it will probably have nothing connected to it but a wireless access point.

For each of these systems we need to do make sure we have the following:
  • IP address
  • hostname
  • a local user account with remote access permissions
  • a way to connect to it; RDP, VNC, IP KVM, whatever you use - BUT TEST IT!
  • check OS patch levels, and how much available disk, RAM, and CPU you have
OK something we need to make special note of: when you log into the local keyboard and look at the monitor you should swee two windows open on each VS and VTS. When you login via RDP you will need to use either "/console" or "/admin" to get to it. I'm not going into all the details here, google can be your friend on this one.

For this next part I have to assume someone left you a link and credentials for the VS and VRS applications. If you don't have these links and login ID's you need to get them or contact Vocera Support and see if they can help you.

On the VS:
  • Login to the console and look into every groups permissions tab noting any that have Vocera Administration, usually one group to itself. Look at the group membership for any groups that have it and review all names to see if anyone has a backdoor into your system.
  • If the previous admin is gone, remove the account. If they have transfered and still use a badge carefully check the permissions to make sure they are appropriate. If they are the backup admin don't be too rash.
  • Make sure backups are running nightly and you have them going off box somewhere that you can get to them if you need them.
  • Check your version and patch level, plan to apply service packs to current (read the release notes) and determine if you need to coordinate a upgrade.
On the VTS
  • Take a look at your VTS and note the wiring going to your PBX. Either you have analog phone lines terminated or a T1 cable (looks just like a network cable) between the two.
  • If you have analog you should find out what the extension on EACH line is and test them individually.
  • If you have a T1/ISDN PRI find out where the crossover cable is between the VTS and the PBX and MAKE SURE IT IS LABLED WITH BIG BOLD LETTERS.

On the VRS:
  • If you can, log in as administrator and reset the password. If this account/password is shared amoung people you may not be able to. If your not up to 4.1 yet and you use VRS heavilly you should probably be talking to your account team about some benefits of upgrading.
BCWS:
  • I'd go for a function test, if you factory reset a badge does it flash from the BCWS and work on the active production VS? If not it's time to roll up your sleeves.
You can configure email alerts for things like VTS failure and cluster failovers (and even more if you have upgraded to 4.1 like scheduled report delivery) if you want it/need it, set it up! If you have another management software running, setup some alerts there for basic network connectivity, and low disk space.



Gather Usage Info

With all the basics out of the way, you need to get a pulse on the health of your system.

If you have a VRS, start with running these reports:
  • low rec by access point
  • low rec by badge
  • low rec by user
The threshold on these reports is 70% so you will see everything below that. I'd take this approach:
  1. low rec by access point, any below 50% go look at the antennas, check the configuration and code level, finally see if there are any other AP's close enough for badges to roam too. Often the last AP heading out parking or smoking areas, I wouldn't worry as much about the fringe AP's as one with low rec in a unit.
  2. low rec by badge, any badge that is being passed between users that reports less than 50% I would want to look at and test myself. If it's bad you need to replace or RMA it, don't let it back out to the user community.
  3. low rec by user, here's where it gets tough. Generate this report and look for where you can make the most difference. It will take time and training, and you need the support of the people running the departments but taking the time and getting the users trained up is well worth the effort.
This will get us an idea of what is giong on, but we should also check our ticket system, however the users were instructed to report problems and see what has been opened (and what was closed.) Sometimes we find that real issues were closed without resolution by people who didn't know how to escalate the issue. It's at least worth a look.

As part of our pulse checking, go get a badge and a headset and go walk the entire environment while doing a "play test tone." Wherever you are getting drops so are the users.

I'm going far out with my personal opinion here, but I think sometimes people spend too much time looking at network stats and heatmaps, the only thing that users care about is what they experience first hand, everyday. Your wireless tools are invaluable in finding and fixing problems, no doubt, but there is nothing like a badge for gauging the actual user experience.


Check The Database

So for the basics of getting to know our production database we are going to keep this a little light and fluffy.

First step is to log into the web console and go to the Maintenance menu, Data Check tab and run an integrity check for Names, Phone Numbers, and Groups. Pay attention to high priority items, but if you're not positive on how to resolve a conflict it's best to ask for help first! Look for duplicate names, for example a doctor who is a user and has the same name as an address book entry for his main practice number. This little tool can catch quite a few problems.

For a day or two, log into the console a few times a day and on the Status Monitor menu, on the Group Status tab, look for groups with zero active users. Any of these you find you need to report to the appropriate department head. If those people are not using Vocera find out if they should instead be an Address Book Entry with a pager, or just removed from the system.

Grab a badge and call several Address Book Entries, extensions, pagers, local area code, long distance, and 800 numbers. Document any types of numbers that do not work.


Get Training!

As the admin people are going to expect you to have all the answers, nearly all of them are already in the VOCERA DOCUMENTATION the ones to live and die by are the Administration Guide and the Infrastructure Guide but there will be important info in all of them!

OK, with our virtual bookshelf filled with the guides we need it's time for training! VOCERA UNIVERSITY is what you are looking for. You'll want to do the intro to Vocera class as well as the system admin class, if you are also responsible for training end users (or re-training) I would suggest the End User Training class. There is a nice little write up on each on the pdf's on this page.

If you want to know about other opportunities for meeting other admins you should ping your account manager, they may know of regional events or about an upcoming user conference and be able to get you informtion.


Final Thoughts

I know this is a long post and there is a lot of information in here, but it's far from complete. Some sections really just give you enough info to create a query for google or to ask for help intelligently, lets use the comments section to ask questions and fill in the gaps where we need to!