Emergency Communication Process

This process can be found on Google Docs, as Insight will likely be down if an outage occurs. Please bookmark the following:

https://docs.google.com/document/d/1ZuvSUXResJiNYOtqBslsV_ICvLCctWhqM7zqvgD3k-0/edit?usp=sharing

 

When to Use This Process

If at least 1 server is down, then we'll anticipate it being down for at least 20-45 minutes and should activate this process.

Process Outline

  1. Network admin immediately notifies (and confirms receipt by) support manager of incident, including as much as they know about:
    1. Start time
    2. Impact (down, slow, etc.)
    3. How many clients affected
    4. ETR
    5. Whether DR is being activated
  2. Support manager:
    1. Updates status page: http://status.visioninternet.com <https://manage.statuspage.io/login>
    2. Updates phone menu
    3. Notifies Vision internally
    4. Emails clients through Intercom app

Email Communication Schedule

Notify the appropriate list of affected customers. If it's all Vision-hosted US clients, use the "US Hosted" segment. If it only affects a server (e.g. cms6b) you can quickly segment that list in MailChimp.

Client notification
When (Business Hours)
When (After Hours)
First alert (template 1)
20 mins
45 mins
Updates (template 2)
every hour
every 2 hours
Resolution (template 3)
at time of resolution
at time of resolution
After action report
as deemed necessary
as deemed necessary

Business hours here are defined as follows. Tuesdays are longer due to council meetings.

Day
Hours
Monday
Wednesday
Thursday
Friday
6AM - 6PM
Tuesday
6AM - 10PM

The templates include yellow fill-in-the-blank spots – if we don't know what is causing the outage or have no ETR, omit them.

Updating Phone Menu

  1. Login to TelePacific iPBX Portal
    1. URL: http://voice.dsci-net.com
    2. Username: 126113@visioninternet.com
    3. Password: [ask]
  2. During business hours:
    1. Navigate to: Services > Auto Attendant > Edit > Submenus > Tech (direct link)
    2. Change Personal Greeting from "techSupport_AA.wav" to "techSupport_AA_DOWN.wav"
  3. After hours:
    1. Navigate to: Services > Auto Attendant > Edit > Submenus > AH_Tech (direct link)
    2. Change Personal Greeting from "AH_Tech.wav" to "AH_Tech_DOWN.wav"
  4. Click OK

Sending Email Notification

  1. Login to Intercom
  2. Click "Platform" on left
  3. Filter for only <user type> is Super User
  4. Click Message>>Auto message in top menu
  5. Confirm audience
  6. Select channel>>email
  7. Create message and set to live
  8. Update body (use email templates)

    Vision offers clients a 24-hour emergency support line to contact the Account Management team after-hours.

    The number is (747) 777-6490.

    What callers hear:

    When someone calls the number above, it will typically ring for 25 seconds and then go to the following voicemail message:

    You've reached Vision Internet's after-hours emergency support line. If you are calling on a weekday between 6AM Pacific and 6PM Pacific, please hang up and call our normal support line at 310-656-3100. If you are calling after-hours and your Vision website has an emergency preventing public access, please leave your name and organization, a callback number, and a detailed description of the issue you are reporting. We will look into the situation and respond as soon as possible. Please note that if this is not an emergency related to website access, we may not be able to respond until the next business day. Thank you. *beep*