During any outages we've tethered to our phones which is fine - but not great. Using O2 as our mobile provider means that we're still at the mercy of Virgin Media as, now they've merged, there's a potential that O2 mobile data will be sent over VM backhaul: https://community.virginmedia.com/t5/Tech-Chatter/Could-Virgin-Leverage-O2s-backhaul-to-add-new-POPs-for-broadband/td-p/4905489
This is just speculation though, I couldn't find anything concrete.

Both BT and Vodafone offer "unbreakable" wifi which consists of a 4G backup dongle which is plugged into the router they provide you with. From their respective websites:

I don't like how vague this is - what's a "short time" or a "total loss of service"? How will it deal with sporadic routing issues or test the quality of your connection?

My home network is modular - I prefer to have specific devices fulfilling their own roles, to make it easier to replace faulty equipment or upgrade one piece when new technologies come out. I use a Ubiquiti EdgeRouter ER3-Lite as my router, connected to the Virgin Media SuperHub in modem mode. Connected to the router is a TP-Link PoE switch, and connected to that is a Ubiquiti Unifi UAP-FlexHD. This setup has been working really well, and I get full Virgin Media line speed (500Mbps+) over WiFi.

As the EdgeRouter Lite has three ethernet interfaces, and I was only using two, I wondered how difficult it would be to add a 4G modem to one of the ports and use this as an automated failover connection. It turns out, not that difficult!

First of all I had to find a 4G modem which was suited to being on all the time and I found the Teltonika TRB140 - usually used for IoT (Internet-of-Things) applications. I found one on eBay for a reasonable price (~£50), added a power supply and LTE antenna, and put a £5 ASDA Mobile SIM card (which runs on the Vodafone network) with 3GB of data in it just to check everything was working. The web interface is very easy to navigate, and it Just Worked (after receiving the SMS code from ASDA Mobile via the utility in the web interface to activate the SIM card and setting the mobile network APN (Access Point Name) to the correct one).

Running a speed test showed around 12Mbps down, and 20Mbps up. This isn't brilliant, but it's plenty to have one or two concurrent calls and keep connected during a Virgin Media outage. Latency was good and consistent at around 30ms pinging 1.1.1.1.

Once this was tested and working connected directly to my laptop, I changed the local IP of the modem, and connected it to the eth2 interface on the EdgeRouter. I added a static route so that traffic from my local network to the internal IP of the modem went out of the correct interface, and did the same for the SuperHub local IP (192.168.100.1) so that I can still check the status of both devices or modify configuration if needed.

I set the TRB140 to Passthrough mode, which means the EdgeRouter sees the "external" IP address of the modem (which is actually a CG-NAT 10.x.x.x address, not a real public IP) but is still accessible on the local IP where needed.

Once this was done, I needed to configure the EdgeRouter to understand what the two interfaces were for. I didn't want some traffic going out through the 4G modem and some going out through the Virgin Media connection, so I had to do some research. I found this page from Ubiquiti on WAN load-balancing, which makes a passing reference to failover. However, it expects you to use the wizard which will overwrite the existing configuration - I didn't want to do this. I ran the following commands through the CLI to set things up:
(eth0 is the VM SuperHub, eth1 is the switch (local network), eth2 is the 4G modem)

# enter configure mode
configure

# create a PRIVATE_NETS network group (note I didn't add 10.x/8 due to concerns with CG-NAT IPs and LAN addresses, I guess I could've been more specific)
set firewall group network-group PRIVATE_NETS network 192.168.0.0/16
set firewall group network-group PRIVATE_NETS network 172.16.0.0/12

# modify firewall rules for private traffic
set firewall modify balance rule 10 action modify
set firewall modify balance rule 10 destination group network-group PRIVATE_NETS
set firewall modify balance rule 10 modify table main

# modify firewall rules for WAN traffic
set firewall modify balance rule 20 action modify
set firewall modify balance rule 20 destination group address-group ADDRv4_eth0
set firewall modify balance rule 20 modify table main

set firewall modify balance rule 30 action modify
set firewall modify balance rule 30 destination group address-group ADDRv4_eth2
set firewall modify balance rule 30 modify table main

set firewall modify balance rule 110 action modify
set firewall modify balance rule 110 modify lb-group G

# local traffic
set interfaces ethernet eth1 firewall in modify balance

# WAN traffic
set load-balance group G interface eth0
set load-balance group G interface eth2

commit
save

Doing the above got the router using the 4G modem, and running curl https://ident.me  returned a Vodafone public IP. However, this is only one part - any connections from my local network could use the 4G connection. Let's add some availability testing and failover:

# For VM, ping 8.8.8.8 every 20s, 20s after the interface comes up, and count success or failure as 4 failed checks (80s minimum failover time)
set load-balance group G interface eth0 route-test count success 4
set load-balance group G interface eth0 route-test count failure 4

set load-balance group G interface eth0 route-test initial-delay 20 
set load-balance group G interface eth0 route-test interval 20

set load-balance group G interface eth0 route-test type ping target 8.8.8.8

# For 4G, ping 8.8.8.8 every 120s, 5s after the interface comes up, and count success as 4 checks (8 mins) and failure as 3 checks (6 mins)
# these are higher as they're not as important as the VM checks
set load-balance group G interface eth2 route-test count success 4
set load-balance group G interface eth2 route-test count failure 3

set load-balance group G interface eth2 route-test initial-delay 5 
set load-balance group G interface eth2 route-test interval 120

set load-balance group G interface eth0 route-test type ping target 8.8.8.8

# Only use 4G as failover
set load-balance group G interface eth2 failover-only

# Load balance internal traffic
set load-balance group G lb-local enable

# When failing over, flush the connection tracking table
set load-balance group G flush-on-active enable

I had originally set these checks to run way too frequently and got ICMP traffic blocked by 1.1.1.1, which caused my connection to fail over. Oops. But, this highlighted an issue that needed solved - when failing back to the Virgin Media connection, a number of devices were still using the 4G connection. It turned out we need to flush the connection table on a fail back - failing over does this with the flush-on-active enable directive, but seemingly not the other way. I found this very helpful script which I set up at /config/scripts/notification.sh: https://github.com/dennisb1/edgerouter-load-balancing-notification - this gives me an email when the status changes and I also added a small function to flush the connection tracking table:

if [ $INTF = "eth0" ] && [ $STATUS = "active" ]
then                                           
  /usr/sbin/conntrack -F                       
fi  

This was added to the load-balance group by running the following commands:

configure
set load-balance group G transition-script /config/scripts/notification.sh
commit
save

There are some useful commands when diagnosing failover problems:

user@edgerouter:~$ show load-balance status    
Group G
    Balance Local  : true
    Lock Local DNS : false 
    Conntrack Flush: true
    Sticky Bits    : 0x00000000 
 
  interface   : eth0     
  reachable   : true     
  status: active   
  gateway     : <VM Gateway IP>
  route table : 201
  weight: 100%     
  fo_priority : 100
  flows
      WAN Out   : 270K   
      WAN In    : 1359   
      Local ICMP: 10914  
      Local DNS : 0
      Local Data: 77376  
 
  interface   : eth2     
  reachable   : true     
  status: failover 
  gateway     : <CG-NAT IP for 4G>
  route table : 202
  weight: 0%
  fo_priority : 60
  flows
      WAN Out   : 0
      WAN In    : 2
      Local ICMP: 1821
      Local DNS : 0
      Local Data: 0

admin@router:~$ show load-balance watchdog
Group G  
  eth0   
  status: OK   
  pings: 10924 
  fails: 1     
  run fails: 0/4     
  route drops: 0     
  ping gateway: 8.8.8.8 - REACHABLE    
   
  eth2   
  status: OK   
  pings: 1822  
  fails: 51    
  run fails: 0/3     
  route drops: 2     
  ping gateway: 8.8.8.8 - REACHABLE    
  last route drop   : Thu Jan  5 11:16:34 2023     
  last route recover: Thu Jan  5 11:18:35 2023   

I'm happy enough with how it's running, and it has been very stable since setting it up. I'll need to do a proper failover test, maybe by pulling the power out of the SuperHub coax to fibre converter so the interfaces remain up.  I have received a couple of email notifications of brief failover events, and these line up with the Broadband Quality Monitor I have running at ThinkBroadband.

The only change I've made to it since is to swap to a Lebara SIM card which is £6.95/month for 15GB data - this should be more than enough. If our VM connection wasn't as reliable, there are unlimited data SIMs out there for just a bit more money per month.

If I was to do it again, I would spend a bit more on the 4G modem - the TRB140 is great but it only has a single antenna. The RUT240 has two antennas presumably to help with MIMO capability and deliver higher speeds. If I ever need to in the future though, I can easily replace the 4G modem - maybe with a 5G modem!

Traditionally you would create a VPN server to access internal resources on your network, giving client access to whoever needed it and treating users who are logged in as trustworthy. However, this comes with some headaches such as:

• Uptime, patching, monitoring requirements
• Supporting users with connection issues/first time setup
• Revoking access for users with a current session (or disabling users once they have left, if you don't have Single-Sign On (SSO) configured)
• Exposing services on a port to the outside world

Recently I've been trialling Cloudflare Access, which is a completely different way to access internal network resources. Rather than setting up a single server, or a fleet of VPN servers as I have done previously with Pritunl/OpenVPN, Cloudflare Access uses the "Zero Trust" security model. This means that regardless of your logged-in session state, each TCP/UDP connection you make has a policy driven decision whether you can make that connection or not behind it, according to the rules specified by the organisation.
Another benefit is that there is a single UDP outbound connection created by the cloudflared service, so no inbound ports are required to be opened, improving security. Traffic is routed to the nearest Cloudflare edge location to ensure low latency.

I found setup relatively simple:

1. Create an Azure Active Directory tenant (free)
2. Register an App in Azure AD
3. Follow the steps in this article to configure the App in Azure AD and allow Cloudflare Zero Trust to use it as an identity provider (I also created a group in Azure AD to restrict who is allowed to connect)
4. Set up cloudflared (in my case as a Docker container), which sets up a Cloudflare Tunnel with the appropriate network CIDR block allowed to route traffic
5. Follow the steps in this article to configure Cloudflare Access
6. Create a Gateway Policy which allows the appropriate internal traffic to flow
7. Download the WARP client and log in with the Azure AD user account
8. Done!

After a few tweaks (and re-reading the instructions...) I was able to use my connection just the same way as using a traditional VPN, without the hassle of maintaining a server to do so. cloudflared is a very simple service to set up and only needs a token to download its configuration and bootstrap itself.

All of this is free to get started. I'm looking forward to testing Cloudflare Access next, to get rid of SSH key authentication.

As you may have noticed from previous posts, I like Azure but it's nowhere near as good as it could be. I was suprised when I received an email the other day stating that I hadn't claimed my "MCSA" badge on some social network (Acclaim?) which was confusing.

The requirements for MCSA when I took the 70-534 and 70-533 exams to gain the Microsoft Certified Solutions Associate was to pass all three of the 70-{532-534} series of exams.

It looks like the requirements were changed in September 2016 and have applied retroactively, so now I can say that I'm an MCSA: Cloud Platform!

Today marks a massive career goal and personal achievement for me: I passed my final Amazon Web Services exam.

I was set a goal when I first joined Cloudreach as an Engineer in April 2014 to get my AWS Solutions Architect and SysOps Administrator exams (both Associate level) within six months. I managed to do it in six months and 4 days to be exact, passing the Solutions Architect in August and the SysOps in October. As I had passed the SysOps certification I was invited to take part in the DevOps Engineer Professional exam beta, which was tough as it had double the number of questions (over 100!) and you had to wait a few weeks for your result. Thankfully I passed this in February 2015, albeit not with a great pass mark but I was sitting it to get experience to re-sit the final exam - if you failed the beta you were given a free retest.

I then moved on to Azure, gaining my 70-534 then 70-533 by the end of 2015.

I hadn't sat an AWS exam in over a year when a reminder that my Associate level Solutions Architect exam was expiring in August dropped into my inbox. I thought it'd be a good challenge to get the AWS Solutions Architect Professional exam before then. Through a combination of work resources (thanks Alberto!) and watching many Re:invent videos (and failing the practice exam to know where I needed to focus) I managed to pass the exam this month (May 2016) and to complete the set I sat the Developer Associate today.

Here's hoping when AWS release another tier (I think there's supposed to be an Expert tier coming soon) I'll be able to achieve those too!

I might even stick in and get the 70-532 for Azure as well to complete that set...

Thanks for reading!

I've been using Microsoft Azure for around eight months now and have passed both the 70-533 and 70-534 exams. In that time I've had a chance to figure out some of the 'peculiarities' shall we say, compared to Amazon Web Services which I have used for the past three years.

First of all, there are two portals. portal.azure.com, which is called the "Azure Portal" and manage.windowsazure.com which is known as the "Azure Management Console". Recently Microsoft announced the availability of all services in the Azure Portal, featuring deep links to the previous console. This makes management of Azure resources a pain as the previous portal does not feature RBAC (Role Based Account Control) which means co-administrators of the 'subscription' (a way to manage your resources and spend) have full access to the whole subscription and can do anything they want, rather than a more granular approach to permissions as seen in AWS' IAM (Identity and Access Management) service. Some Azure services do not support the new portal (as of 12/01/2016) such as Service Bus which means third parties must have full administrative access to your Azure account which you won't want.

There are two ways of managing resources in Microsoft Azure. Azure Resource Manager (ARM) and Azure Service Management (ASM). The former is more aligned with the RBAC experience of the Azure Portal while the latter is related to the older Management Console, or 'Classic' deployment model. Microsoft need to make the Portal the only place to manage all resources (including Classic resources), but I'm not sure if or when that will arrive.

Another example of a peculiarity is restoring a VM backup. It is impossible to remove an OS disk, for example to roll back an OS drive to a previous version. The VM must be deleted and a new one created in its place, which won't have the same private IP. Frankly this is a crazy design choice as you can happily make this change in Hyper-V, which Azure runs on. You have to either get creative with scripts, or use Azure Backup (and that deserves its own blog post entirely!)

A golden rule I kept telling myself throughout my Azure training was: "Azure is not AWS". Although I have been comparing the two during this post, the best way to learn it is to apply the usual Cloud concepts: scaling out not up, designing and preparing for failure but at the same time treating Azure as a completely new concept. For example, Azure does not have the idea of AWS' Availability Zones, instead you have Availability Sets. This is a way of separating VMs within a single datacenter and ensuring they do not reside on the same physical host, network segment or power supply, thus ensuring your machine will not be taken down during physical host maintenance or an outage (in theory). Availability Zones in AWS are at the datacenter level, so you can deploy instances to multiple physical locations connected by low latency interconnects and have high availability that way. Neither is the right way, they are just different ways of looking at the same problem. Azure does not have a VPC, it refers to a VNet which works differently - NAT is built in and there is no 'private' subnet by default, any VM can access the internet even if it does not have an external IP. These subtle changes can trip you up if you constantly compare it to AWS.

In summary, Azure is a capable platform which many companies are using with no problems. However, it's not as polished as AWS and has many annoying peculiarities as discussed in this post. I still recommend Azure, but you have to keep these differences in mind and weigh up the advantages and disadvantages of either Cloud Platform before deciding which one to use or even a combination of both!

I'd love any feedback on this article, thanks for reading.

It has been some time since my last post - I'm going to try to resolve that!

I've not touched VMware for over a year and a half now, having started at Cloudreach in April 2014. I've gained certification during this time in multiple cloud platforms (Amazon Web Services and Microsoft Azure).

The cloud has changed the face of System Administration and Engineering for the better. No longer do you need to go and buy a server for tens of thousands of pounds and spend two weeks setting it up, upgrading components, stress testing, carrying it to the datacenter and making it live. In the cloud, things are so much easier!

Some key practices to live by:

• automate everything (everything!)
• design for failure
• commit your design to a repository

Why automate everything?

If you automate your environment, all servers (also known as Instances or VMs) will have the same configuration which allows you to use the in-built scaling features of all cloud platforms worth their salt - some examples of the tools you can use to do this are: Chef (my favourite), Ansible, SaltStack and PowerShell Desired State Configuration (DSC). It also means you don't need to log into specific servers to troubleshoot as you can send logs elsewhere for investigation and rebuild the instance with very little or no administrative effort.

Why design for failure?

Cloud platforms are by their nature built with the most fault tolerant, up to date equipment and communication links but that doesn't stop a physical server having an issue or a digger killing a bunch of connections. Amazon Web Services specifically state you should design for failure and expect an instance to become unavailable at any time. This is worked around at an architecture level by having at least two instances per service in two separate locations (Availability Zones - AWS), or in Azure in two "Fault Domains". In fact, in Azure you must deploy two VMs (in an Availability Set) for one service to get an SLA which is 99.95%.

Commit your design to a repository?

When building a cloud environment you can use tools such as CloudFormation in AWS or Resource Manager in Azure. These templates define your infrastructure as a JSON template which you can commit to a source control system just like you would for application code or scripts. This allows your engineers to update the JSON templates for any changes to the infrastructure, upload it to the cloud platform and let it handle deploying the changes. There's no need to manually resize a volume or change the desired number of instances in an autoscaling group. Of course, you could write the template just to deploy the environment and then manage manually if you really wanted to.

I'd be interested to hear from anyone that is having issues with cloud deployments, migrations or your thoughts on this post!

Thanks for reading.

After copying my vCenter server using ovftool, I watched it boot up and could see it said there was no configuration for eth1. When the MAC address changes, it will not update the config file automatically. Here's what to do: First of all go into the vSphere Client and connect to the host which is running the vCenter server. Go to the console, give it time to boot up until you get to the prompt. Enter your root username credentials. Type the following (the second one is all one line):

cd /etc/udev/rules.d/cp etc/udev/rules.d/70-persistent-net.rules etc/udev/rules.d/70-persistent-net.rules.originalvi 70-persistent-net.rules

vi is very old school, so you'll need to press i to get into insert mode, go to the line that says:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}"00:50:56:aa:52:39", ATTR{type}"1", KERNEL=="eth", NAME="eth0"

Comment it out with a # (shift and 3 for our UK keyboards...) and edit the line below so that the NAME changes from eth1 to eth0. Hit escape a couple of times, then type :wq and enter to save. Reboot the machine and all will be fixed.

Hope this helps!

So, I needed to copy my vCenter Server Appliance over to another host. Obviously can't do that while it's switched on, and it takes care of the migrations in vSphere Client so what can we do? Download ovftool (I downloaded the Windows version) from http://www.vmware.com/support/developer/ovf/ and install it on your PC. Go to the installation directory and shift-right-click to choose 'Open a command window here'. Type the following (note it is all one line):

ovftool -ds="Destination Datastore Name" "vi://username@esxihost/VM Name" "vi://username@esxihost"

Don't make the mistake I did of including the VM name at the end of the destination. This is so quick and easy and requires no copying of an OVF template to your PC then back again, it does it straight between the hosts. Hope this helps!

I found a problem when upgrading to ESXi 5.1 that Veeam won't back the servers up giving a "Error: Host with uuid '12345678-1a23-1b23-1234-123456789101' was not found" message. Having found http://tsmith.co/2012/veeam-job-fails-after-upgrade-to-vsphere-5-1/ (thanks Tim) he tells you how to sort it in an older version, but I couldn't figure out where to fix it in Veeam 6.5.

It's in File > Help > License > Licensed Hosts, you need to revoke on each host then run your backup jobs as normal, it will re-add them back to the list. Hope this helps!

Well I say easy, but it's not that easy. Total time was around eight hours including server firmware updates.

1. Upgrade vCenter Server (I used the appliance, so download the appropriate files from the VMware site) - this is done by booting both, new one has DHCP, copying keys in the web interface so they can speak to each other and finally restarting and deleting the old server. Make sure you update the original server through the web interface before continuing! I wasted a good two hours waiting on it upgrading  but it never did so I had to start again.
2. Check you can connect and see all the hosts, and that everything works OK.
3. Shut down all VMs and hosts.
4. I used the easier (in my opinion) option of using iLO to upgrade ESXi, I connected through IE so I could mount the virtual disk (ESXi 5.1 HP specific ISO file) and install. This works great for Essentials or Essentials Plus customers - Enterprise customers should use the Update Manager.
5. Boot into the installer and make sure you choose the upgrade option, when it asks which disk to choose use the one without an asterisk (this is local storage, may be different for SAN) - the one(s) with asterisk(s) are your VMFS partitions containing VM files.
6. Reboot each host, check you can connect to the host, boot your vCenter, make sure you can connect to it and it should see all hosts and be working fine! You may need to update your backup software to account for having a new version of ESXi on the hosts.

As I said in my last post about this, vCenter Converter 5.0 didn't work with ESXi 5.1. VMware released the new version, 5.0.1 on 25 October 2012. It's not fully verified with vCenter 5.1,  but should work much better than the previous version. My SSL post below should still work to speed it up, let me know if it doesn't.

Here's how to get the serial number/service tag or product name from a server or client PC.

Open up command prompt and type:
Serial number:

wmic bios get serialnumber

Product name:

wmic csproduct get name

This was from about a year ago, when we were transitioning internet connections. We had two default gateways and obviously traffic can only go out one/come in one. This was for one of our web servers.

The solution is to tag packets:

echo "10 special" >> /etc/iproute2/rt_tables
ip route add default via <second gateway ip> table special
ip rule add from <second gateway ip> table special

This way, traffic to 6.6.6.6 for example will go in and come out as normal. However if your new IP is 7.7.7.7, the traffic is tagged with "special" and goes out the other gateway.

Hope this helps!

Converter 5.0 is slow, and doesn't work with ESXi 5.1 yet, so don't try it! Converter 5.1 is due for release in November I believe. For slow transfers with ESXi 5.0.1 or below, see the following text.

SSL is the issue, VMware have switched it on by default - for a much higher transfer rate, disable SSL.

Here is how it is done:

Open the converter-worker.xml configuration file. It is located in "%ALLUSERSPROFILE%\VMware\VMware vCenter Converter Standalone" folder for Windows Vista or newer or in "%ALLUSERSPROFILE%\Application Data\VMware\VMware vCenter Converter Standalone" for older Windows versions.

Set the key Config/nfc/useSsl to false and save the configuration file. Restart the "VMware vCenter Converter Standalone Worker" service.

It should look like this:

...
<nfc>
<readTimeoutMs>120000</readTimeoutMs>
<useSsl>false</useSsl>
...