CockroachDB – how to build a 4 node SQL cluster on ubuntu and HyperV

CockroachDB Overview

Description: cockroach is an open source, survivable, strongly consistent, scale-out SQL database. If you wonder where google engineers go when they leave google, they go out on their own and build unbelievably great scalable and distributed open source software. Essentially if you want to run your own fault-tolerant SQL database across multiple datacenters and cloud services, using your own servers, allowing you complete control of your database, without paying hefty licensing fees, then run cockroach. The info in this post is not a review of cockroach, but rather a demonstration of a lab setup and POC.

To get started in our lab, first we want to build around 3 or 4 test clone servers or “nodes”. I use ubuntu on top of HyperV, but you can use any flavor of linux or MacOS you want. It can also run on Windows Docker.

If you’re like me and use Hyper-V on Win10, make 4 x Ubuntu 16.04 “clones” – first build a ‘goldmaster’ image, and clone it 4 times – guide here: https://4sysops.com/archives/clone-a-ubuntu-server-in-hyper-v-2012-r2/ – or use something like virtualboxes.org.

Create 4 virtual machines, each having it’s own IP address:
Node1: inet addr:10.0.10.169
Node2: inet addr:10.0.10.170
Node3: inet addr:10.0.10.171
Node4: inet addr:10.0.10.172

Make sure each node is up to date and has ntp installed and synchronized with the commands:

sudo apt-get install ntp

Use the command

timedatectl

To ensure that…

NTP synchronized: yes

At this point before you install/run cockroach, it’s wise to export each node VM with HyperV as a backup.

On Nodes 1,2,3,4 download the latest binary here https://www.cockroachlabs.com/docs/install- cockroachdb.html with the command:

sudo wget https://binaries.cockroachdb.com/cockroach-latest.linux-amd64.tgz

Extract the binary with the command:

tar -xvf cockroach-latest.linux-amd64.tgz

Move the binary to a location in your PATH or add the directory location to your path. You can learn about your path with the command:

sudo vi /etc/environment

And then move your extracted cockroach to /usr/sbin with the command:

sudo mv cockroach-latest.linux-amd64/cockroach /usr/sbin/

Do a sanity check with the command:

cockroach version

Start cockroach in insecure mode in the background on Node1 (master server) with the command:

sudo cockroach start --background --insecure --host=10.0.10.169

Result should be something like below:

CockroachDB node starting at 2017-03-15 23:16:23.118419329 -0700 PDT
 build: CCL beta-20170309 @ 2017/03/09 16:31:10 (go1.8)
 admin: http://10.0.10.169:8080
 sql: postgresql://root@10.0.10.169:26257?sslmode=disable
 logs: cockroach-data/logs
 store[0]: path=cockroach-data
 status: restarted pre-existing node
 clusterID: 08b6bfe6-4886-466b-a9c6-bc58a3809113
 nodeID: 1

Go ahead and browse to the admin page http://10.0.10.169:8080

On your other nodes:

sudo cockroach start --background --insecure --host=10.0.10.170 --join=10.0.10.169:26257

*where –host=current node ip address you’re having to join with the master server 10.0.10.169

Your results should look something like the following:

CockroachDB node starting at 2017-03-15 23:23:43.783097234 -0700 PDT
 build: CCL beta-20170309 @ 2017/03/09 16:31:10 (go1.8)
 admin: http://10.0.10.170:8080
 sql: postgresql://root@10.0.10.170:26257?sslmode=disable
 logs: cockroach-data/logs
 store[0]: path=cockroach-data
 status: initialized new node, joined pre-existing cluster
 clusterID: 08b6bfe6-4886-466b-a9c6-bc58a3809113
 nodeID: 2

Your web interface should provide you with performance graphs:

Identify the new nodes in the View Nodes List link:

Go on and add the remaining Nodes to the cluster.

???

Profit! – just kidding

Now you can go on to learn about cockroach SQL and create some databases and tables and test how pulling the plug on one of your nodes doesn’t bring down the DB, and how all the data is replicated to all 4 nodes. It’s recommended you don’t run this lab on a single workstation-class system, but something that meets the cockroach DB minimum system requirements. This product is still in beta and features are subject to change. Regardless, cockroachdb is an incredible addition to the open-source community and I’m sure will be very useful to a lot of systems admins and application developers.

Fix ubuntu when the OS will not boot – kernel panic – kernel panic not syncing vfs unable to mount root fs on unknown-block 0 0 – error /boot full remove old kernels from command line

To begin, it will probably take at least 30 minutes resolve this issue…

This fix solved my problem with the “vfs unable to mount root fs” error, but of course your results may vary. As always, first backup your system or do an export of the vm so you have a copy of the system as it existed before you started screwing around with it 😉

After running apt-get update / apt-get upgrade and then a reboot, you may receive the following error: kernel panic not syncing vfs unable to mount root fs on unknown-block 0 0 on ubuntu 16.04.

In many cases this  will be due to the /boot drive becoming 100% full because many updates have been made to the kernel. By default, ubuntu will retain the old kernels and add them to the list of available kernels you can boot into in the Grub2 boot loader menu. You can confirm that your drive is full by issueing the command:

df -h

The result will likely show the following:

In order to resolve this issue and boot successfully, while you’re looking at the error during boot, (you should already be at the console), and restart the vm or computer into the Grub2 menu then choose “Advanced options for ubuntu” view where you can see a list of old kernels you can boot into. Some report you can do this booting with the Shift key held down, or in the event it’s a virtual machine, you should be able to arrow-down in the Grub start screen and choose Advanced options for ubuntu on startup:

Grub2 boot menu.

Once you go into the advanced boot menu you will likely see several kernels listed. Choose the next-oldest kernel from the top/highest version of kernels. In my case I booted into the version labeled Ubuntu, with Linux 4.4.0-57-generic (my boot menu screenshot below is clean, but you’ll likely see several kernels listed).

Cross your fingers and hope you get to your login prompt. From here I jumped on putty and connected from that client, as I prefer it over the console.

Next, login and follow the directions that I found here:

http://askubuntu.com/questions/2793/how-do-i-remove-old-kernel-versions-to-clean-up-the-boot-menu

To save you the search, here are the instructions I used to first list and then remove the old kernels:

Open terminal and check your current kernel:

uname -a

DO NOT REMOVE THIS KERNEL! Make a note of the version in notepad or something.

Next, type the command below to view/list all installed kernels on your system.

dpkg --list | grep linux-image

Find all the kernels that lower than your current kernel. When you know which kernel to remove, continue below to remove it. Run the commands below to remove the kernel you selected.

sudo apt-get purge linux-image-x.x.x.x-generic

Or:

sudo apt-get purge linux-image-extra-x.x.x-xx-generic

Finally, run the commands below to update grub2

sudo update-grub2

Reboot your system.

sudo reboot

As you can see from my terminal history, I had to remove a few:

589  uname -a
 590  dpkg --list | grep linux-image
 591  sudo apt-get purge linux-image-4.4.0-21-generic
 592  sudo apt-get purge linux-image-4.4.0-22-generic
 593  sudo apt-get purge linux-image-4.4.0-24-generic
 594  df -h
 595  sudo apt-get purge linux-image-4.4.0-24-generic
 596  sudo apt-get purge linux-image-4.4.0-28-generic
 597  sudo apt-get purge linux-image-4.4.0-31-generic
 598  sudo apt-get purge linux-image-4.4.0-34-generic
 599  sudo apt-get purge linux-image-4.4.0-36-generic
 600  sudo apt-get purge linux-image-4.4.0-38-generic
 601  df -h
 602  sudo apt-get purge linux-image-4.4.0-42-generic
 603  sudo apt-get purge linux-image-4.4.0-45-generic
 604  sudo apt-get purge linux-image-4.4.0-47-generic
 605  sudo apt-get purge linux-image-4.4.0-51-generic
 606  sudo apt-get purge linux-image-4.4.0-53-generic
 607  sudo update-grub2
 608  dpkg --list | grep linux-image
 609  df -h
 610  sudo apt-get purge linux-image-extra-4.4.0-21-generic
 611  sudo apt-get purge linux-image-extra-4.4.0-22-generic
 612  sudo apt-get purge linux-image-extra-4.4.0-24-generic
 613  sudo apt-get purge linux-image-extra-4.4.0-28-generic
 614  sudo apt-get purge linux-image-extra-4.4.0-31-generic
 615  sudo update-grub2
 616  df -h
 617  sudo reboot
 618  dpkg --list | grep linux-image
 619  uname -a
 620  sudo reboot

After the reboot, you can see my /boot partition returned to a manageable size:

I hope this post helps someone save some time and help them fix their ubuntu boot problems.

Google Cloud Platform Overview

The pace of global cloud computing is continuing to grow exponentially. While Amazon still holds the lion’s share of cloud services, Google’s Cloud Platform has been growing at the fastest pace. In this article, we’ll first examine what makes the Google Cloud Platform different; provide you with a list of its components, solutions, and features; and finish up by discussing pricing.

Cloud Services trends and opinion

According to a Synergy Research Group study: “In terms of year-over-year growth, Google enjoys the lead at 162 percent, while Azure has grown by an even 100 percent. AWS is in fourth with a 53 percent year-over-year (YoY) growth rate.…

Read the rest of the article here…

Google Cloud Platform overview

How to Install ISC DHCP Server on Ubuntu 16.04

The Internet Systems Consortium (ISC) Dynamic Host Configuration Protocol (DHCP) server is free, open-source, and easy to install. Both enterprises and small networks have used ISC DHCP in production for many years.

In this guide, I’ll demonstrate how to locate your current DHCP server and then install and set up an ISC DHCP server. We’ll then move on to gaining control of your new DHCP server, best practices, monitoring the logs, and setting up static address reservations.

Read the rest of the article here:

Install ISC DHCP Server on Ubuntu 16.04

Set up Ubuntu as a domain controller with SAMBA on VirtualBox

If you want to run a domain controller on your network but don’t have access to a Windows Server license, you can use SAMBA, the free open-source software, and VirtualBox, the free virtualization software. We’ll describe the procedure for setting up a virtual server using VirtualBox and netboot.xyz iPXE and move on to setting up your domain controller with SAMBA.

Read my full article here:

Set up Ubuntu as a domain controller with SAMBA on VirtualBox

Clone a Ubuntu server in Hyper-V 2012 R2

Ubuntu runs on Hyper-V perfectly fine, so you may want to run many Ubuntu Virtual Machines (VMs) on Hyper-V Server 2012. R2 This article will show you how to clone or duplicate a single Ubuntu server on Hyper-V with different network interfaces and host names. Cloning Linux servers on Hyper-V is easy and quick when you have the right knowledge and tools.

Read my full article here:

Clone a Ubuntu server in Hyper-V 2012 R2

Clone a Ubuntu server in Hyper-V 2012 R2

Ubuntu runs on Hyper-V perfectly fine, so you may want to run many Ubuntu Virtual Machines (VMs) on Hyper-V Server 2012. R2 This article will show you how to clone or duplicate a single Ubuntu server on Hyper-V with different network interfaces and host names. Cloning Linux servers on Hyper-V is easy and quick when you have the right knowledge and tools.

Read my full article here:

Clone a Ubuntu server in Hyper-V 2012 R2

Raspberry Pi 3 Model B 32 GB Setup and Config

Raspberry Pi 3 Model B 32GB Initial Setup and Configuration

So my first Raspberry Pi 3 Model B 32GB just arrived which is part of a “Canakit” on Amazon which can be found here. Below is a general setup and config step by step guide to get you started with your own Pi. I may follow up this post with the progress of my pie-in-the-sky project to build a miniature hyperloop “pod” and track, controlled by a stand-alone Raspberry Pi. But before I get into powering my Pi with an electrified rail, we need to get the Pi setup and configured. 

  1. Unbox, connect peripherals & power on (I swear, every video and tutorial spends too much time on this topic.) But I will say that the little machine is very quick and responsive, and I’m really happy with it’s performance so far!
  2. Initial power up. The Red LED powers on, and my monitor stays black, hmm, it’s not booting… Oh duh, it will help if I insert the included 32GB MicroSD. Important to note is the SD card does not click-eject or click-insert but is friction-based receptacle.
  3. Ok 2nd power up – it’s alive! Boots to the desktop and looks great – except the resolution. I used the Wifi config to get connected to my WAP easy. But it’s strange that I can’t ping anything. A quick search found this post, and the command:
     sudo chmod u+s /bin/ping

    With that command I can now ping out and about.

  4. Now I can go online with the default web-browser (Web 3.8.2) and find a fix for my monitor’s resolution. To set a raspberry pi 3 monitor resolution, follow the instructions in this post – which told me to first check my monitor’s capabilities with the commands:
    tvservice -d edid
    edidparser edid

    and then seeing that I do have a DMT monitor (group 2) and an hdmi mode 82, I edited /boot/config.txt with the new editor named Geany. I started geany with sudo so I could edit the protected file with the command:

    sudo geany
    

    Which brings up the Geany file editor running as sudo, and then I could open and browse to the File System > /boot/ directory where I could edit the file config.txt to include the following lines:

    hdmi_group=2
    hdmi_mode=82

    After a restart my monitor’s resolution is correct.

  5. So now to make sure my OS is up to date. By default the version I installed is
    Linux raspberrypi 4.4.9-v7+ #884 SMP Fri May 6 17:28:59 BST 2016 armv7l GNU/Linux
    

    So it is time to run an update. It’s so nice that the distro is debian-based and the familiar apt package management system is installed. No dist-upgrade necessary.

    sudo apt-get update
    sudo apt-get upgrade

    I’ll be following up soon with more findings and initial configuration, but for now I’ve got enough to be able to branch out and explore the world of Raspberry Pi!