Monday, December 29, 2008

Vocera databases – who’s where & why?

Finally time to tackle another big subject, the Vocera database. While I am very comfortable with the subject, there a couple of people who could do it even better, Luke Ludeman is one of those people.

Luke is a Implementation Engineer at Vocera, it's his job to design databases and build systems for Vocera's customers. He has also trained many of the partner engineers that are out in the field right now, myself included. Given his knowledge and experience, how could I refuse when he graciously offered to write the article below? From here on, in his own words.


Vocera Databases – Who’s Where & Why?

Yea – this post is long. Not War and Peace long, but a Vocera database is a topic huge topic. Vocera systems have a few key components that need to function well together in order to provide a great end-user experience: a pervasive wireless network, great training, and one component that seems to receive the least amount of focus when setting the initial system up – the database. The database in a Vocera system is it’s heart and soul. How well the system functions relies significantly on how well the database is structured and maintained.

I’d like to take a moment to discuss some Vocera database practices regarding the structure. For a Vocera database, there are a few key concepts that contribute to a healthy, scalable, and efficient system:

  • Departments, Groups, Group Nesting, and Group Membership
  • Call Flows and Call Forwarding

Setting up a Vocera database properly does not take any more time to build, and positions everyone involved for success with the system. A properly structured Vocera database will be easy to administrate and ready to support sustained growth. Admittedly, the examples below are all healthcare related, but very applicable across all industries. Here are some of the keys to Vocera database success:


Groups & Group Membership

Let’s start with Departments and work our way “down”. Departments, simply stated, are they are large groups of people working towards a common goal (typically) in a geographically similar area (a unit/department). Departments can usually be sub-divided into smaller groups by role/position as not everyone in a department usually does the same job – a role-based group. There are typically nurses, techs (nursing assistants), unit secretaries, physicians, etc... Each with a somewhat specific duty to perform. Here are some assumptions and schema tricks to setup each unit/department:

  • Departments only have role-based groups as members (almost never users)...
  • ICU (department) only has members ICU Nurse, ICU Tech, ICU Unit Secretary, etc...
  • Role-based groups (ICU Nurse, ICU Tech, etc.), typically has users as members, and maybe other groups as well, but do not contain special (managerial) role-based groups (ICU Nurse Manager, etc.) as members. This type of group can be static or dynamic with regard to the users that are members.
  • Special role-based (managerial) groups (ICU Nurse Manager) are nested as members of the departmental group, but not a member of the “associated” role-based group (ICU Nurse – see the bullet directly above) For example, the ICU Nurse Manager group is not nested within ICU Nurse. Why? Typically, the members of special (managerial) role-based groups have ascended to this role and do not want to receive general calls to the “associated” role-based group, and if they did, would probably not be much help due to their meeting schedules. As such, the ICU Nurse Manager would typically not want to receive calls destined for the ICU Nurse group...


Another key point - almost every department (in healthcare) has a "Charge" or “Supervisor” role in some form and/or title. By default, allow an entire group within the department to add/remove themselves from their particular "Charge" group (dynamic role-based group). For example, all ICU Nurses can add/remove themselves from the ICU Charge Nurse group. By making this permission somewhat broad and allow the entire "nurse" group to add/remove themselves even though there may only be a subset of nurses that will fill this role - it dramatically simplifies administration as the staff that can fill this role may shift regularly...


Group Nesting

Now that we have discussed the different group types, let’s chat about group nesting. Why do this? There is a dramatic shift towards easy administration, exposing the most features to the users (with the least amount of work). Here is a brief example, if we take an ED Department with Nurses, LPN’s, Patient Care Techs, a Charge Nurse, a Nurse Manager, and Physicians:

ED (department)
|_ ED Nurse -----> ED Charge Nurse
__|_ ED LPN

|_ ED Patient Care Tech

|_ ED Physician
__|_ ED Resident

|_ ED Nurse Manager

  1. The ED departmental group would only have listed on the members tab in the Vocera Administration Console: ED Nurse, ED Patient Care Tech, ED Physician, and ED Nurse Manager.
  2. The ED LPN group may be nested in the ED Nurse group as they would typically be expected to respond to the same broadcasts as an ED Nurse.
  3. As with the ED LPN group, the ED Residents are a member of their own group, but nested in the ED Physician group allowing them to receive calls to the ED Physician group or receive broadcasts.

So, why not nest the ED Charge Nurse group within the ED Nurse group? The members of the ED Charge Nurse group (in this example) are already an ED Nurse or they would not have the ability to add themselves to the ED Charge Nurse group. Nesting the group would not hurt anything structurally or feature-wise, but it is redundant - right?

Also, the ED Nurse Manager is typically NOT nested in (a member of) the ED Nurse group as this is typically a person who has “ascended” to a managerial role and will not be providing direct patient care or wish to receive calls destined for the ED Nurse group. If there is an emergent situation, and an urgent broadcast is initiated, it should be to the greater/larger group ED as these situations usually require multi-modality coordination. Make sense?

By nesting groups (i.e. Letting them be a member of another group) there are a bunch of benefits – granting (or denial) of permissions, and subsequently, access to Vocera features. For example, if we refer to our ED example above, by nesting groups, the ED Nurse Manager group is a member of the ED department group. One could deduce that if our end-user Jane Smith is the ED Nurse Manager, therefore a member of the ED Nurse Manager group, and the ED Nurse Manager group is a member of the ED (department group), then Jane Smith is a member of the ED department group. The associated/built-in logic (and permissions) of group nesting flows all the way through to the lowest group level (if A --> B, and B --> C, and C --> D, then A --> D)...


Call Flows

Group Nesting is a big structural database component, Call-Flow is the movement of calls/information through a particular department. Call-Flows within departments, and sometimes between departments/groups, are crucial for the efficient flow of calls and information as well as typically allowing a call end with a live person. Call flows are in many cases, a foreign concept, and may seem a bit abstract to describe. Ask the question "If I call the group and no one answers, is it alright for that call to go to voicemail?" This question should be asked for almost every group. The answer to that question typically depends on the department being interviewed, but usually the answer is – NO... Default call flows can be easily setup, and will typically meet more than 85% of almost any departments needs – first, some assumptions within our setup:

  • If I call a "group", I most likely do not know who (particular user) I want to connect with (and in general do not necessarily care who it is), but I expect to reach "someone"...
  • If I call a "user" by name, I explicitly know who I want, and it is alright for the call to go to voice-mail...
  • Calls to a department (ICU) explicitly forwards ALL calls to the main desk extension of the department.
  • Every role-based group (ICU Nurse, ICU Tech, etc...) forwards unanswered calls to the main desk extension of the department or the “Charge” role for the department – ask the question...
  • Every dynamic group (ICU Charge Nurse) forwards unanswered calls to the main desk extension of the department.
  • Special role-based groups (ICU Nurse Manager) forwards unanswered calls to that person's desk phone or to the main desk extension of the department. By default, forward unanswered calls to their desk phone if an office extension is provided. This allows voicemail to be checked in one location if desired.
  • Be careful here though - in order to forward unanswered calls for a Special role-based group to a desk phone to work properly, the special role-based group must be a single person, who every day, when they are at work will ALWAYS be that position (i.e. is not a dynamic rotating role...) and use that particular phone extension, or a group of persons that share an extension. If this is not true, and/or the special role-based group contains more than one member that does not share an extension, forward unanswered calls to the main desk extension of the department.



Database Gotcha’s – yep, there’s plenty...

  • Users should ALWAYS be a member of a group, and that group should always be a member of a department.
  • At the very least, all users should be a member of a department – either through explicit membership (no recommended) or by being a member of a group, that is a member of the department (recommended).
  • Nesting a department in another department will produce unexpected results from the voice recognition standpoint. When calling by first name and department (see the point above) - which department?
  • All Groups should forward somewhere – who’s going to check the voicemail for a group that has a couple hundred users?
  • When setting up dynamic role-based groups – pay attention to the group that can add/remove themselves. Make sure it is not the same group that you are editing. For example, when editing the ICU Charge Nurse group, when setting the permission for the group that can add/remove itself to that group, make sure not to select the ICU Charge Nurse group – yes, it will let you do that...
  • Be sparing with the alternate spoken names for users and groups – a tidy database is an efficient database.
  • If not enabling dynamic extensions (no brainer), do not use the same desk extension for multiple users.
  • Call-Flows/group forwarding – the recommendations above suggest a maximum of one “hop” before forwarding to the main extension of the department. This is for a reason – forwarding will not happen indefinitely, there is a “timeout” associated with a call. Forwarding too many times assures that a call will not end with a person some number of times...


The list above is by no means inclusive, nor is this post – but very common things to mess up on...

I hope this post is helpful. I’ve built a bunch of Vocera databases over the years, and have focused a bunch of time on finding the right mix between group nesting (structure) and ease of administration. In my experience, it takes almost twice as long to fix one than to set one up well from the get-go...

Wednesday, December 24, 2008

technical support bulletins

I got the latest tech support bulletin last night in my email and realized there could be people who don't know about them. To view the current bulletin you can CLICK HERE.

A few notes from this one:
  • I have some clients to notify about the need for B2000 firmware updates.
  • The 4.1 Infrastructure Guide is now available.
  • I read through the open issues list and I have not encountered any of them in production, I guess it is Christmas!
If you want to sign up for this bulletin, click the "Vocera Technical Support" link on the right and follow the "E-Notification List" link there.

Sunday, December 21, 2008

catching up the blog

It's been a while since I've had a chance to post to this blog and I have quite a bit to say so I am going to break it up over the next week or so. Most of my time has been spent doing 4.0 -> 4.1 upgrades so good info on that is coming up soon.

I updated the links on the right with several items regarding headsets:
  • Vocera section got links for guidelines and approved headsets
  • Engineer section gets a link for my Plantronics MX 500 headset
  • Badge section got a link for a basic user headset
I use "play test tone" to look for places dropping packets nearly every week. Even at the lowest setting it's too loud to use the speaker for this, and if you put a normal headset in for this it will leave your ear ringing. The MX 500 has an inline volume adjustment, I turn both my badge and headset all the way down, then turn the headset knob until I can just barely hear the tone clearly. It's simple to adjust the volume as you walk from loud rooms to quieter ones, I strongly recommend a headset with this type of adjustment for anyone doing this sort of work.

The hot tip of the week is also about headsets and it comes from one of the Vocera administrators I was working with this last week on an upgrade. He did a little searching for headsets for some users and found Amazon had a huge price drop on PLANTRONICS MX 150 headsets. The price went from around 16$ to 2$ a few days ago, simply insane! One unit did order 25 of them and got 3 DOA out of the bunch, but at that price they seem pretty disposable to me.

One final little admin note to close with. I've been pretty disappointed with my search ranking being so low for this blog, a search for "thom's vocera blog" didn't even hit this in the first ten pages of google results. I took a look at my source code today and found that blogger had put an unhelpful "noindex,nofollow" in my header. I got that cleaned up so hopefully some better visability is coming soon.

Monday, December 15, 2008

D41JCT-LS, the other analog integration

The Problem
I had a little situation come up over the last few days, Vocera support identified a bad analog board in a VTS server that needed to be replaced. An attempt to swap out the formerly working board with a new one failed when the server could not initialize the new card. Down the rabbit hole we go.

The original card had 6 ports on it with the pinouts I described in THIS previous post. It's the D120JCT card I have linked on the right side of the screen. Since only four ports were being used, the replacement card was the D41JCT which is also a supported card in Vocera 4.0.

When the engineer booted up after inserting the new card and loading the driver from C:\dialogic\driver it gave an error saying "no high density card found"

At this point we know we have two problems, somehow we have to get the VTS to recognize the low density four port card, and we have to cut off all our terminations and then terminate those four lines on their own RJ11's.


Drivers
The first hurdle is getting the driver loaded. Since it didn't work by pointing the the correct directory during the first attempt I'd guess uninstalling VTS and reinstalling it after the new board is in the server would be the way to go. I would call Vocera support first, and run that plan past them before I did it. This will help them keep the case notes up to date and update the knowledgebase if needed.


Terminating Cables
Once that's done we need the correct terminations for the new card. So we have this pinout currently:


And when we put in the four port card we need to go back to this:


Once we get to this spot we should have a working telephony integration to rerun our test plan against.


Conclusion
There is another option, instead of rebuilding the VTS and crimping new RJ11's we could just swap the 12 port board for another 12 port board using the same driver and current terminations. Personally I prefer this approach.

Anyone else run into this before? I am also curious is anyone has seen a Dialogic card fail on a VTS before? If so please drop a comment below.

Sunday, December 14, 2008

introductions

When I created this blog I dropped one very short little post and got to work posting more technical items. Someone just pointed out I never really told you who I am or what audience I am looking for.


About me
My name is really Thom, I'm a senior engineer with a technology consulting firm. A few years ago I evaluated a product called Vocera and my company decided to become a Vocera partner.

Prior to working on Vocera I had been installing other systems for my clients for quite a while:
- Cisco routers, switches, wireless, and firewalls (2000-2008)
- Cisco CallManager, Unity, etc (2001-2008)
- Cisco IPCC/UCC Express (2002-2008)
Obviously these skills aligned with Vocera very well, lucky me!

Prior to consulting I paid my sys admin dues on a handful of systems: Novel, NT, Windows 2000, various Unix systems, and AS400 midranges between 1998 and 2000. Given how many systems I was working on during the y2k build up, it's safe to say I knew some of them far more intimately than others. Most of my work was NT/2000 and AS400, but my love for *nix came from this time period. If I ever go off on a Slackware or BSD rant it will be tied to 1999 in some way.

In the late 90's I connected up with a few magazine editors and while I was going through the certification tracks for MS and Cisco. I wrote reviews and articles for several magazines but by 2002/2003 I was too busy to keep up with it and I put writting aside.


About this blog
Vocera has several hundred customers, and thus there should be several hundred sys admins out there. There are also a handful of us engineers who deploy these systems week after week for new customers. All of us get to see a little piece of the Vocera landscape first hand, but none of us get to see all of it. Along the way we all learn little tricks, or create our own tricks out of neccessity and our own creativity.

This blog is a place for me to post links to the things that I need and that others might use, like code 128 barcode scanners, etc. This is also a place to link official Vocera information, like release notes or various specifications. Most importantly it's a place to bind together all the loose ends as it relates to Vocera. When I looked around I found that there are many examples of backup scripts for Windows servers, but I couldn't find one anywhere that told me what files on a Vocera server should be backed up.

Finally, the reason this is a blog and not a static html page is the comments section. Knowing that there are people out there with their own tricks, I really wanted to have a mechanism for everyone to be able to share that knowledge with the entire community.


Conclusion
Since I didn't find a place to say it above I would like to say a few things. I am not compensated by Vocera in any way. I use CDW links but I am not paid commission or advertising by them in any way. Buy your goods wherever you like, CDW provides manufacturer part numbers on each of its pages and that is why I use their links.

The reason I am not publishing my last name or contact info is because I'm not using this as a business vehicle, this is a place for cooperative sharing of information. If you are shopping for a partner please work with your Vocera account team to select one.

Comments? Questions? You know what to do.

Thursday, December 11, 2008

troubleshooting user complaints

Vocera is a wonderful product, I prefer working on it over every other system I install because it performs very well and I have personally encountered very few bugs in it. So if we have a solid system that is correctly configured, and our end users are getting furious about "I'm sorry, I didn't understand" what does that mean? I see the same few things over and over:
1) wireless network issues
2) database issues
3) user training issues
4) hardware issues

To make things just a little more tricky, normally there is some amount of each of these at play. Lets look at each of them individually.

Wireless Network Issues
The very first thing we need to issue a command from the badge to Genie is a wireless network to carry our request. How do we know if we have wireless network problems? Here are the hints:
- Genie responses have gaps in speech
- VRS report on "low recognition by access point" shows some AP's well below the average
- red lights on the top of badges
- badges display shows "Searching for Access Point"

If we have the symptoms what do we do? My usual recommendation is to log into a badge as a user with Administrator rights and "play test tone." Stick a headset in the badge and walk the areas reporting problems. Listen for dropped packets, slow roaming between access points, or completely falling off the network. If you expereince any of these you know you have wireless issues to tackle.

Next step is to run a full wireless survey that includes a spectrum analyzer looking for sources of interference. The old Airmagnet views will show channelization problems, dead spots, misconfigured AP's, etc. The Cognio type of component will help you find the things that are scrambling your frequency range and disrupting your badges from functioning correctly. I won't delve any deeper into wireless, if you have symptoms of wireless problems they must be addressed so we can troubleshoot deeper problems.


Database Issues
It's amazing to me how many database problems I find out in production systems (we are talking about absolute show stoppers.) If you want to sniff out these issues, my suggestion is to start thinking like Genie. When you look at group names, Address Book Entries (ABE) and Alternate Spoken Names (ASN) you look at them phonetically. Make sure that when you sound it out it will sound like what your users will say. Vocera has a built in feature to check out your data integrity, you should run this to look for obvious problems!

When you are ready to check the database, login to the admin console, choose the Maintenance menu, and click the tab for Data Check. Select the items you want to check and click the Check button. Take a good read through the output. Think like Genie and see what names are similar or identical, check ASNs and ABEs for conflicts.


User Training Issues
With a voice grade wireless network in place and a clean database available, we have removed the easily fixed items from the list so now we have to deal with users.

First thing we need to know is who is having problems and how bad the problems are, so I hope you have a Vocera Report Server. Log into your VRS and run a low rec by user for the last week, this will tell you who is having problems, how many commands they have issued and what percentage was successful. After putting on some kevlar it's time to go out and meet the users and ask some questions:
- Do you have more trouble in some places than others? (wireless network)
- Are there people or groups they call that it never seems to understand? (check name and ASNs in the database)
- What commands do they commonly use? (Call? Get? Find?)
- What accessories do they use? How do they wear the badge? If used, how does their headset sound?
- Ask them to recreate some scenario that typically fails and observe.

I found this one recently where "call I C U" never works, the group is in the database as "ICU" with no ASNs. Sometimes you can catch this in your database check but other times it's the users who have to lead you to them. I have one client where many users did "learn a name" for a group that needed an ASN. A year later none of the new people could ever call that group and wasn't taught the "learn a name" command in their onboarding training. All the users knew was that Genie always understood it when someone else said the group name, but was broken when they tried it.

Gotta have thick skin to walk out with the target on your back, but if you've made it through the wireless and database and your users still have problems this is the only way.


Hardware Issues
Sometimes things just break, would we get those damage plans on our cell phones if we knew we would never drop them? We can miminize the risk of badge damage by using accessories, but some of them will be destroyed anyway. When a badge is 1/16th of an inch thick or in lots of little pieces it's easy to know it's bad. When it is cosmetically normal people expect it to work and that is sometimes a mistake. Older badges can be damaged internally and look fine on the outside. I've seen bad badges cycle in and out of shifts where one unlucky roulette player gets to hear "I don't understand" all shift.

Again we have a case for having a VRS and running reports for low rec by badge. If badges are being passed person to person it's very easy to find a bad device. Things get a little slippery when people have a one-to-one badge ratio. It's a GREAT approach for inventory to have people keep their own badge and be responsible for it, but what we then find is that the low rec by user and the low rec by badge gives us duplicate data.

Batteries do wear out and when they do they should be pulled from service. The current shipping batteries have expiration dates on them, not an exacting science, but a rough indicator of when they should be replaced.


Conclusion
Vocera is a great system but it does require ongoing maintenance to be sure the wireless coverage doesn't change over time, the database has all the utterances it needs without conflicting extras, the users are staying sharp with their usage, and the badges/batteries/accessories performing and being used correctly. What else should be on the list?

Thursday, December 4, 2008

service packs and upgrades

Lets do a quick crash course on system maintenance. Vocera versions are usually represented as:

X.Y spZ

X = major release version
Y = minor release version
Z = service pack level

Currently most production systems will be running 4.0 with some level of service pack, and the very newest systems will be 4.1. There has not been a service pack released for 4.1 as of this post.


Service Packs:
Service packs are released quarterly so set a reminder for yourself to log into the support page, click the tab for "Find Knowledge Base Article" and under the section "All Knowledge Base Articles" you will see a category for "Service Packs & Upgrades" lets take a peak in there.

So scanning for 4.0 today I see that latest service pack released for 4.0 was sp8 so I will click that article. First thing I see is a link to the RELEASE NOTES. ALWAYS read the release notes!!! This will tell you what is fixed, what is still open, and HOW TO install the service pack. The other important link will be to download the service pack itself.

Some people feel this sort of patching should be left for consultants, personally I think the sys admin should run these when released and OS patches monthly, my two cents.


Upgrades:
Upgrades are a different bear. New versions (minor or major) introduce new features and often cause a significant restructure of the database and sometimes change the behavior of the system.

These are invasive and have some risk that can be mitigated by training and experience, thus I think this is the sort of work you should bring in your parter to help with. Even if you have someone come in to run major or minor version upgrades you need to make sure you block appropriate downtime and notify users of the upgrade plan well in advance. Your partner should help you plan for the upgrades and let you know what to expect for impact in your particular environment.


Conclusion
Service packs, should be applied by sys admins as they are released.
Upgrades, should be well planned for and performed by people trained to do them.

Questions? Comments? Post them below!

Wednesday, December 3, 2008

easter eggs vs funny genie

I have been asked to explain Vocera easter eggs and funny genie, as well as what administrative controls we have over them. First thing to do is explain what these are and how they are different.


Easter Eggs
WIKIPEDIA tells us that an easter egg is "an intentional hidden message" and one of the most famous ones for Vocera is to issue the command "good bye" to genie to which you will hear the reply "live long and prosper." There are several built into Vocera, but they are all responses to certain commands like the one mentioned above.

If you want to try out a few easter eggs, try commanding genie with these:
- good bye
- beam me up
- beam me down
- shut up

Are we good on what an easter egg is?


Funny Genie
Funny genie is something different, when a user says "turn funny genie on" it enables an alternate set of prompts, instead of Jean or Mark saying "Vocera" you might hear a robotic overlord saying "Vocera, Vocera, Vocera, Vocera" or someone say "Aloha." When funny genie is enabled for a user, that user will be greeted by funny genie every time they press the call button (until they issue the "turn funny genie off" command.)

Sometimes people think this is far to much fun and distracting so they want it all switched off. The good news for those people is that after 4.0sp3 the funny genie can be disabled, good news for the fun lovers is that easter eggs can't be removed or disabled.

If you need to kill funny genie you can read the some old RELEASE NOTES and find the procedure:
  • Fixed: Cannot turn off the funny genie.

    A new SysFunnyGenie property, which enables the funny genie, has been added to the server and is set to true by default. (Issue 5876)

    To turn off the funny genie, add the following property to \vocera\server\Properties.txt on all Vocera Servers and then restart the servers:

    SysFunnyGenie = false


Conclustion
So there you go, if you have any questions (or easter eggs to add to the list) please post them in the comments section!

Tuesday, December 2, 2008

creating hex key for WPA-PSK

Vocera badges can use several different mechanisms for securing communication between themselves and the network. This is basically a two part question; what authentication type do you want to use, and which encryption type? I have seen just about every supported combination attempted at some point in time or another, but the combination choosen most often (for performance reasons) is WPA-PSK with TKIP-WPA.

There is one gotcha to configuring the badges this way, and the reason for this post; many AP's will setup with a ASCII passphrase, but the badges require a hexidecimal key exactly 64 bits long.

Normally I go get a copy of the key generator I keep HERE and open up the zip file. There are two directories, if you take a look in the how_to_use folder you will see the following screenshots and an example file of how to run the utility. It certainly isn't hard, but you can spend more time reading about what it can do than what we actually need it to do.

Lets go ahead and convert a passphrase to a hex key. Extract the wpa_key_gen directory to the root of the C:\ on the BCWS like so:

Using the syntax in our example file:
C:\wpa>wpa_passphrase.exe vocera secretphrase

we can sub in our real SSID where "vocera" is, and the passphrase used for "secretphrase" and hit enter.

In my example we got this
Personally I select the entire thing and copy it to a text file local on the BCWS so if I typo'd the SSID or passphrase I can see it when the first badge doesn't associate. From the text file I copy just the key into my Badge Properties Editor for the B1000's and B2000's as appropriate and then flash a badge to make sure it associates to the network.

Now just backup the files you need, copy them where they need to go, and destroy any data you don't want laying around where it shouldn't be, and you are done!