CockroachDB – how to build a 4 node SQL cluster on ubuntu and HyperV

CockroachDB Overview

Description: cockroach is an open source, survivable, strongly consistent, scale-out SQL database. If you wonder where google engineers go when they leave google, they go out on their own and build unbelievably great scalable and distributed open source software. Essentially if you want to run your own fault-tolerant SQL database across multiple datacenters and cloud services, using your own servers, allowing you complete control of your database, without paying hefty licensing fees, then run cockroach. The info in this post is not a review of cockroach, but rather a demonstration of a lab setup and POC.

To get started in our lab, first we want to build around 3 or 4 test clone servers or “nodes”. I use ubuntu on top of HyperV, but you can use any flavor of linux or MacOS you want. It can also run on Windows Docker.

If you’re like me and use Hyper-V on Win10, make 4 x Ubuntu 16.04 “clones” – first build a ‘goldmaster’ image, and clone it 4 times – guide here: https://4sysops.com/archives/clone-a-ubuntu-server-in-hyper-v-2012-r2/ – or use something like virtualboxes.org.

Create 4 virtual machines, each having it’s own IP address:
Node1: inet addr:10.0.10.169
Node2: inet addr:10.0.10.170
Node3: inet addr:10.0.10.171
Node4: inet addr:10.0.10.172

Make sure each node is up to date and has ntp installed and synchronized with the commands:

sudo apt-get install ntp

Use the command

timedatectl

To ensure that…

NTP synchronized: yes

At this point before you install/run cockroach, it’s wise to export each node VM with HyperV as a backup.

On Nodes 1,2,3,4 download the latest binary here https://www.cockroachlabs.com/docs/install- cockroachdb.html with the command:

sudo wget https://binaries.cockroachdb.com/cockroach-latest.linux-amd64.tgz

Extract the binary with the command:

tar -xvf cockroach-latest.linux-amd64.tgz

Move the binary to a location in your PATH or add the directory location to your path. You can learn about your path with the command:

sudo vi /etc/environment

And then move your extracted cockroach to /usr/sbin with the command:

sudo mv cockroach-latest.linux-amd64/cockroach /usr/sbin/

Do a sanity check with the command:

cockroach version

Start cockroach in insecure mode in the background on Node1 (master server) with the command:

sudo cockroach start --background --insecure --host=10.0.10.169

Result should be something like below:

CockroachDB node starting at 2017-03-15 23:16:23.118419329 -0700 PDT
 build: CCL beta-20170309 @ 2017/03/09 16:31:10 (go1.8)
 admin: http://10.0.10.169:8080
 sql: postgresql://[email protected]:26257?sslmode=disable
 logs: cockroach-data/logs
 store[0]: path=cockroach-data
 status: restarted pre-existing node
 clusterID: 08b6bfe6-4886-466b-a9c6-bc58a3809113
 nodeID: 1

Go ahead and browse to the admin page http://10.0.10.169:8080

On your other nodes:

sudo cockroach start --background --insecure --host=10.0.10.170 --join=10.0.10.169:26257

*where –host=current node ip address you’re having to join with the master server 10.0.10.169

Your results should look something like the following:

CockroachDB node starting at 2017-03-15 23:23:43.783097234 -0700 PDT
 build: CCL beta-20170309 @ 2017/03/09 16:31:10 (go1.8)
 admin: http://10.0.10.170:8080
 sql: postgresql://[email protected]:26257?sslmode=disable
 logs: cockroach-data/logs
 store[0]: path=cockroach-data
 status: initialized new node, joined pre-existing cluster
 clusterID: 08b6bfe6-4886-466b-a9c6-bc58a3809113
 nodeID: 2

Your web interface should provide you with performance graphs:

Identify the new nodes in the View Nodes List link:

Go on and add the remaining Nodes to the cluster.

???

Profit! – just kidding

Now you can go on to learn about cockroach SQL and create some databases and tables and test how pulling the plug on one of your nodes doesn’t bring down the DB, and how all the data is replicated to all 4 nodes. It’s recommended you don’t run this lab on a single workstation-class system, but something that meets the cockroach DB minimum system requirements. This product is still in beta and features are subject to change. Regardless, cockroachdb is an incredible addition to the open-source community and I’m sure will be very useful to a lot of systems admins and application developers.

 

Dell Latitude 3450 cannot install windows 7 with samsung se-208 DVD driver missing

So I recently had problems installing Windows 7 SP1 with an original certified Dell installation DVD using a Samsung thin profile SE-208 external USB DVD/CD.

Upon booting to the Windows 7 installation, after telling the Windows 7 installer to go ahead, it said that the DVD/CD ROM drivers were missing.

I also could not install with a bootable Windows 7 USB key that I created by first ripping the Dell DVD to an ISO with IMG Burn, and then creating a bootable USB drive with rufus-2.12.exe. The same error – no drivers detected.

After finding this post here, it came to me that I had only tried the external DVD drive on the USB port that is on the right-hand side of the laptop (USB 3.0).

I instead connected the external USB DVD drive into the Left-hand side of the laptop USB port (USB 2.0) and booted the DVD into the installation and proceeded normally.

So if this happens to you, connect your bootable device to only a USB 2.0 port, when trying to install Windows 7 on a newer PC or laptop that has both USB 2.0 and USB 3.0 ports!

Windows 7 networking basics – How to map a drive between two computers in a WORKGROUP – not joined to a domain

This how-to procedure for mapping network drives pertains to Windows 7 PC’s that are not joined to a domain, but are members of the same network Workgroup. This how-to map network drives is not the same as using the Windows “HomeGroup” feature – this tutorial is a little more advanced – but the method works for me consistently.

  1. Make sure that both computers are on the same network and subnet. This should be already done in most cases as your computers should pick up IP addresses and network settings from a DHCP server/router/modem. Things might get weird if each PC trying to reach one-another are on different connections, i.e. one is on WiFi and one is on an Ethernet cable. Essentially both machines should have IP addresses that look similar, something like 192.168.0.5 and 192.168.0.6. You can find your IP address by right-clicking on the Network icon in the taskbar, down by the clock, or go into the Control Panel > Network and Internet > and choose: Network and Sharing Center > Change Adapter Settings > Right-click on Local Area Connection (the adapter which is connected to the network and internet) > Status > Details… button > IPv4 Address.
  2. Each computer should be able to ping one another by IP address and by hostname. In a command prompt (Start button > All Programs > Accessories > Command Prompt) you can test this with the following command to ping by IP address:
    ping 192.168.0.5

    You should receive “Reply from 192.168.0.5…” and not “Request timed out.” Again, this should be done from each computer to the other.

  3. Next determine what the computer name is for each computer. Do this by Right-clicking on the Computer icon in Windows File Explorer and choose Properties.

    Windows 7 Computer Properties

    Look for “Computer name:” – use this computer name to do another ping test. From a command prompt, type in:

    ping computername

    where you replace ‘computername’ with the name of the other computer you want to ‘talk to’. Again, you should get ‘reply from…’, not ‘error/no host/time-out’. If you get replies when you ping the IP address but not the computer name, then you can still map the drive/share from your computer, but it won’t look pretty.

  4. If you can ping by IP address but not computer name, for testing purposes, make sure the Windows Firewall is turned off (temporarily), and that both computers are members of the same Workgroup. In the same Computer Properties as above, where you found the Computer name: … you should see the “Change settings” link to the right of the Computer Name:. under the Computer Name tab, click on the “Change…” button > select Workgroup: and then enter the same workgroup name on both of the computers that will share files. Most people do Workgroup: WORKGROUP. Once you’re able to ping each computer from one another (at least by IP address), you should be ready to share out a folder and then map a drive letter to that share.
  5. Next, make sure that the folder you want to access on, for example, Computer Name: PC1 is actually shared out by the PC1 computer. In PC1 Windows file Explorer, Right-click on Computer and choose Manage.

Computer management

6. In the management tool, expand Shared Folders and then click on Shares:

Shared folders

7. If you don’t see the folder you want to share listed, click on More Actions > New Share > follow the wizard (don’t worry about the offline settings.) Typically if you’re in an environment where you trust everyone, you can set the share to be accessible (read/write/execute) by Everyone (Everyone is the name of an actual user group that resides in all Windows computers). Do this by selecting “Customize Permissions” then place check-marks in Allow: Full Control, Change, Read > OK:

Windows 7 share permissions

If you’re wondering what the $ is for in the shares I have on my machine, the $ is used to hide a file share. If the folder name has a $ at the end, it’s hidden from people browsing the computer’s IP address or UNC name (explained later), but since you know it’s there you can still get to it. For example, in my shares screenshot above, I could browse to the share by typing in \\jasonPC\jcshare$ . But had I only typed in \\jasonPC\ then it would not be displayed.

8. Now that the share is available, from PC2 you can browse to the share by the UNC computer name (Universal Naming Convention used by all windows computers – in Apple/Mac’s it’s actually weird and to browse to a share on a Mac you would use smb://jasonPC/share.)

Open Windows File Explorer, in the address bar, type in the computer name that has the share you want to connect to preceded by two back-slashes (\\) and then followed by another backslash. So for example \\JASONCWKS\ and then hit the enter key. In the event if you could not ping the other computer by the UNC computer name, you can do the same action but replace the computer name with the IPv4 address, for example: \\192.168.05\ .

9. You will see a list of shares available on the computer. Next, right-click on the share and choose Map Network Drive.

Browse by UNC computer name

10. Now provide the drive letter you want, place a check-mark on Reconnect at Logon and then Finish

Map network drive

That’s about it! Your other computer should now have the drive mapped with full read/write permissions.

Cannot connect to Server 2008 R2 with RDP broken – Interactive Logon Initialization Process has Failed

ESXi 5.5 – recently I tried to RDP into my Server 2008 R2 machine without success. Looking at the console, I get the message: ” Interactive logon process initialization has failed. Please consult the event log for more details. ” My first reboot of the machine did an automatic check disk. I can no longer log into the machine either via RDP or on the console. I’d like to not have to rebuild this system as it is my only stand-alone DC in my home lab. I’m going to bring up a separate DC, then do a DCPROMO, however, below are the steps I took to resolve the issue, albeit unsuccessfully – some of these steps may work for you. I was hoping I could mark this process as “Solved” but I haven’t gotten there yet…

Event log says:

  • Event 4005 Winlogon – The windows logon process has unexpectedly quit
  • Event 33 SideBySide – Activation context generation failed for “C:\Windows\system32\LogonUI.exe”. Dependent Assembly Microsoft.Windows.Common-controls.Resources,language=”*”,processorArchitecture=”amd64″,publicKeyToken=”6595b64144ccf1df”,type=”win32,version=”6.0.0.0″ could not be found. Please use sxstrace.exe for detailed diagnosis.

These errors all seem to have started on 8/8/16 when the following events occurred:

Error: 36888 – Schannel – The following fatal alert was generated: 10. The internal error state is 1203

Event 56 – The terminal Server security layer detected an error in the protocol stream and has disconnected the client. Client IP: 223.x.x.x which is a Hong Kong IP. Prior to that are many Event 1012 – Remote session from client name a exceeded the maximum allowed failed logon attempts. The session was forcibly terminated.

I pulled the machine off the internet and the network in case it’s been compromised. On second thought I should probably kill this machine with fire!

Things I have tried that did not resolve the problem:

  • Boot into safe mode
  • Edited RDP settings on my workstation to use less resolution and video bit depth
  • Increased Virtual Machine’s system memory size as suggested, went from 3GB to 7GB
  • Booted to SystemRescueCD and replaced c:\windows\system32\LogonUI.exe – did this by booting the VM to a SystemRescueCD.iso (startx) and then mount the NTFS file system with the instructions here. Next I changed the root password with passwd and then connected to the VM with Filezilla on port 22 and renamed/moved the files.
  • Hotfix 437977 – Windows6.1-KB2615701-v2-x64.msu – because I cannot get into the system to run this, I started Task Scheduler on the remote server from my workstation, and attempted to have the .msu run but get error: the application has failed to start because its side-by-side config is incorrect.
  • Booted the VM to a Server2008R2.iso, Repair your Computer > command prompt,
    sfc /scannow /offbootdir=c:\ /offwindir=c:\windows

    didn’t work. Then tried command prompt > “cd sources” > StartRep.exe didn’t help.

Any suggestions? Has anyone gotten past this error? I can still connect to the machine via any means (mmc terminals such as event viewer, and task scheduler) other than a GUI console.

Edit: I gave up on trying to fix the issue – this is the reason we backup our VM’s and in this case I’m glad I had a good working backup. If you find a way to resolve, please drop a note – otherwise, make sure you have good backups of your DC’s: The VM as a bare metal backup and the System State (Separately.)

How to add a bulk list of Domains into Symantec Cloud MessageLabs Encryption TLS Business Partners

If your email servers support, or are required to enforce transport layer security (TLS), then you can use Symantec TLS Business Partners to configure policies to enforce encryption between your mail servers and the broader public, as well as Symantec’s Email Security Services infrastructure. You can also configure policies to enforce TLS encryption between your domains and the domains of third-party business partner organizations. This article will provide a procedure that demonstrates how to generate a report with Symantec PBE Advanced Domains that displays mail which failed a TLS policy. We then go on to process that report and relax the enforcement of Certificate Validation with the following steps:

  1. Scrub the Symantec Report for just email addresses of the intercepted mail.
  2. Remove the first part of the email addresses, leaving only a list of the domain names.
  3. Compare the domains from Step 2 to already-existing TLS Business Partners and leave only those domains that are unique.
  4. Create a new text .txt file with the unique domains, add ‘test@’ to the beginning of each domain to create an email address that can be tested at checktls.com.
  5. Batch test the domains for certain TLS conditions.
  6. Scrub and sort the checktls.com report for only those domains that meet conditions in Step 5.
  7. Upload the list of unique, tested, and scrubbed domains into the Symantec Cloud TLS Business Partners.
  8. Use Symantec to again test the domain for Certificate Validation, enable the domain, and enforce the encryption of traffic to that domain.

Often times, TLS encryption requirements will force end-users and recipients of TLS encrypted email to log into a secure mail portal to authenticate and retrieve messages. Depending on your organization’s policies, you may want to ‘relax’ the level of enforcement of TLS requirements so that users can receive mail in ‘plain text’ as a normal message in the user’s mailbox. These different levels of encryption are scored by traits including the availability of TLS encryption, TLS negotiation, the SSL/TLS Certificate trustworthiness, and other factors.

To a large degree, TLS email encryption has not quite become the standard for many mail servers out there. Many large mail providers such as earthlink.net do not support TLS encrypted email. Some mail servers will allow TLS encyrption (TLSAvailable=100), negotiate a TLS transaction (NegotiatedTLS=100), but not have a valid certificate (CertOK >= 33). Often times, the CertOK value will not reach 100 simply because the certificate on the receiving server is a self-signed certificate.

You can check the support of an email address or domain by visiting checktls.com, enter an email address into the address test, and hit the ‘Try It’ button. For example, when we test [email protected], our results are as follows:

TLS Receiver Test

In this case, if Symantec /MessageLabs were to try to send a message to the recipient, the message would not be delivered to the recipient mailbox, but quarantined in a secure mail portal. The recipient would then be required to create a secure portal account, log in, and retrieve the message.

If our results above were to demonstrate the passing of all checks with the exception of the CertOK column, then we may want to relax the enforcement of certificate validation in Symantec. But in order to find all of the domains that meet these conditions, (excluding those domains we already have validated and trust), we need to generate reports, scrub the data, and test for the conditions we accept. Once we find the unique domains, we’ll add them into our list of TLS Business Partners.

While the following procedure does not have long-winded explanations and exact step-by-step instructions, the information and screenshots should be enough to get you through the process, hopefully saving you the hours of research and note-taking that I endured. This article does not go over the initial setup of your domain’s email encryption with Symantec. The prerequisites for this guide are that you already have TLS email encyption setup with Symantec Cloud / MessageLabs.com. Also be sure to have Excel 2013 or newer and NotePad++ installed on your computer. Lastly, you may be required to purchase an account with checktls.com to run batch tests.

Lets get started!

We’ll first run a report in the Symantec Cloud / clients.messagelabs.com to find emails intercepted by the service. Starting in the Dashboard, go to Reports > Report Request > Request a new Report > Request name: PBEAdvancedDomains-Date > continue > email summary report (PDF) >

The data we will include in the report are:

  • Email Summary Report (PDF) advanced settings
  • Emails intercepted by the service
  • Data Protection volume
  • Data Protection volume by domain
  • Top 20 Data Protection recipients
  • Email Detailed Report (CSV) advanced settings
  • Data Protection

Message Labs Report 1

MessageLabs Report 2

MessageLabs Report 3

MessageLabs Report 4

Download the Report.zip

Extract to Desktop\TLSReports\RepDomainsDate

Open the EmailDataProtectionDetail.csv

Copy the column “Email To” to a new spreadsheet named 1stRunRemoveNameAndAtSymbol.xlsx

In 1stRunRemoveNameAndAtSymbol.xlsx, delete the 1st row “Email To”

Go here to learn how to remove the username from the email address http://pakaccountants.com/excel-split-extract-username-domain-email/ but essentially you put the function in row B1 and then copy the function down the column.

=RIGHT(A1,LEN(A1)-FIND(“@”,A1))

Excel wants you to type out the formula (instead of copy/paste) in the first cell B1, then apply formula down the column.

Apply formula to an entire column or row without dragging by Fill feature

In Excel, Select the entire Column B, and then click Home > Fill > Down.

Apply formula fill down
Apply formula fill down

Copy and Paste (Paste – Special->ValuesOnly) the resulting Domains (only) to a new sheet and name

1stRunDomainsOnlyDate.xlsx

Now you should have 3 files:

1st Run Files

In Symantec Message Labs go to Services > Encryption > existing TLS Business Partners

Download All -> get TLSBusinessPartners.csv > rename to TLSBusinessPartnersDate.csv copy to same folder, so you should have 4 files. In the 1stRunRepDomainsOnlyDate.xlsx you may want to remove all the empty cells with #VALUE! From the bottom of the file.

Create a new folder named Compare, and copy 1stRunRepDomainsOnlyDate.xlsx and TLSBusinessPartnersDate.csv into the folder:

Compare domains files

Remove all domains in TLSBusinessPartners file that are current Partners (as well as those existing in previously completed procedure runs.) – To do so,

Open 1stRunRepDomainsOnlyDate.xlsx. Create a new Sheet 2.

Open TLSBusinessPartners.. file and copy the column with only the domain names Column labeled “Business Partner Domain” into Sheet 2

Run compare and remove all domains that exist in TLS.BP.Date from 1stRunRDODate.

Rename Sheet1 “CleanTheseOfDups”, Rename Sheet 2 “Test”

Scrub the domains of duplicates. Copy all the Business Partner Domains AND be sure to include all of the domains listed in TLS Enforcement page into Test sheet (selecting with the mouse, copy from website, and paste into excel works, just remove the hyperlinks).

Populate test worksheet with existing domains

Next, select Sheet 1 then go to the Data tab and click Remove Duplicates:

Remove duplicate domains

Click OK

Duplicate domains removed

Add a new row to A1 top level and add header named: “CleanThese” for Column A and “Business Partner Domains” for Column B

Copy existing Business Partner unique domains in Test sheet to B1

Should now look something like this:

Clean domains of existing domains

Now Filter for Unique values:

  1. Click on the Column Header to highlight Column A. Select the Conditional Formatting button on the Home tab.
  2. Select the New Rule option from the Conditional Formatting button drop-down list. Select “Use a formula to determine which cells to format" as a rule in the New Formatting Rule window.
  3. Proceed by entering the following formula in the “Format values where this formula is true" box: =countif($B:$B, $A1)
  4. Click on the Format button and specify the format you want to set. It is possible to set a different cell color for duplicate values by clicking the Fill tab and selecting a background color. (Orange)
  5. After having specified the format you want click OK.

Credit: http://spreadsheetpro.net/comparing-two-columns-unique-values/

After comparing the two columns using the guide above, your spreadsheet should look something like the following:

Compare two columns in excel

6. Right click on for example A3, and choose Filter> By Selected Cell’s color.

Copy only those cells NOT HIGHLIGHTED WITH COLOR results in column A to new file named 2ndRunScrubbed.txt and place in folder…

Second run scrubbed

Now we need to add ‘test@’ to beginning of each line. To do so, Open the 2ndRunScrubbed file in Notepad++

One method that doesn’t involve find/replace (but only works if you want to do it on every line, including blank lines in your block), is this:

    • Move your cursor to the start of the first line
  • Hold down Alt + Shift and use the cursor down key to extend the selection to the end of the block, Then type in test@

This allows you to type on every line simultaneously.

You should now have a list of email addresses with all of your unique domains that you want to test at CheckTLS.com

Normalize domains as email addresses for testing

Save the resulting file as 2ndRunScrubbed1-9-17_TestTLSReady.txt

Log into checktls.com and run checks, get weights, and add the new business partners and enforcements.

http://checktls.com/tests.html#Batch

CheckTLS batch test 1

CheckTLS batch test: Once you’ve uploaded the file, hit R(u)n Test Now to initiate the testing. One domain takes 5 minutes, so 4000 domains can take 4 hours or so. An estimate from checktls.com will be given.

CheckTLS batch test 3

Once the report arrives via email, copy the data in email to a new file named TLSCheckReportResults1-9-17.xlsx

Next, find CertOK is a value of 33 or 0, but the TLSAvailable=100 and NegotiatedTLS=100

To do this, do Ctrl+A then Sort & Filter > Custom Sort > Sort by CertOK > Sort on Values > Order Custom List: 33, 0 > OK > OK

(Column H is Cert OK) so Select only those that are anything other than 100, copy the rows to a new spreadsheet named

TLSCheckReportResultsCertOK33-Date.xlsx

Sort the TLSCheckReportResultsCertOK33 file by TLSAvailable=100 and NegotiatedTLS=100

To do this, first remove all columns except EMailAddress, TLSAvailable, CertOK, and NegotiatedTLS=100

Do another custom sort; select columns labeled TLSAvailable, Cert OK, and NegotiatedTLS, and Sort by Column B (TLS) then by Column D (SenderOK/Sec) Largest to Smallest:

Excel custom sort columns on values largest to smallest

Select only those rows that have at least TLSAvailable = 100 … CertOK(%) … NegotiatedTLS = 100; where CertOK % is less than 100:

Excel custom sort results

Copy these values to a new .xlsx file and folder named:

TLSCheckSortTLSAvailable100Negotiated100-Date

Copy custom sort results above into a new .xlsx file and rename the filename the same as the folder name.

Remove everything except the domain name (remove ‘test@’) from this file. Follow instructions above – delete columns B, C, D, then formula =RIGHT(A1,LEN(A1)-FIND(“@”,A1))  – then fill down (select column first)

Copy resulting domains to a new .txt file named:

TLSCheckSortTLS-DomainOnly-Final-Date.txt

Now we need to add the tested domains to the Business Partner for the week. Eg. RepDomains-Date in Symantec.

Log into Symantec Cloud,

Services > Encryption > TLS Business Partners > Add new BP > RepDomains-Date > Upload New Business Partner Domains > Uncheck “Enabled” > choose file > select TLSCheckSortTLS-DomainOnly-Final-Date.txt > Save

Upload New TLS Business Partner Domains

Now that all of the new unique domains have been imported, we need to test them within Symantec and change Certificate Validation from Strong to Relaxed if required.

Click on the domain name, hit the TLS Test button, and if the TLS connectivity fails, hit Close, change the Certificate Validation to “Relaxed” and then Save.

Relaxed certificate validation

Go through each of the newly uploaded domains, and when finished, make sure to ENABLE TLS Encryption enforcement in the TLS Enforcements tab.

Take those in the list that test correctly and are enforced and add to Data Protection Exceptions. Services > Data Protection > PBE Advanced TLS the Portal Trigger Template > (scroll down) Recipient Exclude Domain List  > Edit List >  Add List Items > Add

It’s a good idea to repeat this procedure every so often so that you catch mail that is being delivered to the secure portal unnecessarily. Hopefully, this procedure will help someone save some time if they have the same task, or at the least help someone sort and scrub email addresses and domains.

Fix ubuntu when the OS will not boot – kernel panic – kernel panic not syncing vfs unable to mount root fs on unknown-block 0 0 – error /boot full remove old kernels from command line

To begin, it will probably take at least 30 minutes resolve this issue…

This fix solved my problem with the “vfs unable to mount root fs” error, but of course your results may vary. As always, first backup your system or do an export of the vm so you have a copy of the system as it existed before you started screwing around with it 😉

After running apt-get update / apt-get upgrade and then a reboot, you may receive the following error: kernel panic not syncing vfs unable to mount root fs on unknown-block 0 0 on ubuntu 16.04.

In many cases this  will be due to the /boot drive becoming 100% full because many updates have been made to the kernel. By default, ubuntu will retain the old kernels and add them to the list of available kernels you can boot into in the Grub2 boot loader menu. You can confirm that your drive is full by issueing the command:

df -h

The result will likely show the following:

In order to resolve this issue and boot successfully, while you’re looking at the error during boot, (you should already be at the console), and restart the vm or computer into the Grub2 menu then choose “Advanced options for ubuntu” view where you can see a list of old kernels you can boot into. Some report you can do this booting with the Shift key held down, or in the event it’s a virtual machine, you should be able to arrow-down in the Grub start screen and choose Advanced options for ubuntu on startup:

Grub2 boot menu.

Once you go into the advanced boot menu you will likely see several kernels listed. Choose the next-oldest kernel from the top/highest version of kernels. In my case I booted into the version labeled Ubuntu, with Linux 4.4.0-57-generic (my boot menu screenshot below is clean, but you’ll likely see several kernels listed).

Cross your fingers and hope you get to your login prompt. From here I jumped on putty and connected from that client, as I prefer it over the console.

Next, login and follow the directions that I found here:

http://askubuntu.com/questions/2793/how-do-i-remove-old-kernel-versions-to-clean-up-the-boot-menu

To save you the search, here are the instructions I used to first list and then remove the old kernels:

Open terminal and check your current kernel:

uname -a

DO NOT REMOVE THIS KERNEL! Make a note of the version in notepad or something.

Next, type the command below to view/list all installed kernels on your system.

dpkg --list | grep linux-image

Find all the kernels that are lower than your current kernel version. When you know which kernel to remove, continue below to remove it. Run the commands below to remove the kernel you selected.

sudo apt-get purge linux-image-x.x.x.x-generic

Or:

sudo apt-get purge linux-image-extra-x.x.x-xx-generic

Finally, run the commands below to update grub2

sudo update-grub2

Reboot your system.

sudo reboot

As you can see from my terminal history, I had to remove a few:

589  uname -a
 590  dpkg --list | grep linux-image
 591  sudo apt-get purge linux-image-4.4.0-21-generic
 592  sudo apt-get purge linux-image-4.4.0-22-generic
 593  sudo apt-get purge linux-image-4.4.0-24-generic
 594  df -h
 595  sudo apt-get purge linux-image-4.4.0-24-generic
 596  sudo apt-get purge linux-image-4.4.0-28-generic
 597  sudo apt-get purge linux-image-4.4.0-31-generic
 598  sudo apt-get purge linux-image-4.4.0-34-generic
 599  sudo apt-get purge linux-image-4.4.0-36-generic
 600  sudo apt-get purge linux-image-4.4.0-38-generic
 601  df -h
 602  sudo apt-get purge linux-image-4.4.0-42-generic
 603  sudo apt-get purge linux-image-4.4.0-45-generic
 604  sudo apt-get purge linux-image-4.4.0-47-generic
 605  sudo apt-get purge linux-image-4.4.0-51-generic
 606  sudo apt-get purge linux-image-4.4.0-53-generic
 607  sudo update-grub2
 608  dpkg --list | grep linux-image
 609  df -h
 610  sudo apt-get purge linux-image-extra-4.4.0-21-generic
 611  sudo apt-get purge linux-image-extra-4.4.0-22-generic
 612  sudo apt-get purge linux-image-extra-4.4.0-24-generic
 613  sudo apt-get purge linux-image-extra-4.4.0-28-generic
 614  sudo apt-get purge linux-image-extra-4.4.0-31-generic
 615  sudo update-grub2
 616  df -h
 617  sudo reboot
 618  dpkg --list | grep linux-image
 619  uname -a
 620  sudo reboot

After the reboot, you can see my /boot partition returned to a manageable size:

I hope this post helps someone save some time and help them fix their ubuntu boot problems. Please leave a comment if this helped resolve your issue or if there is a smarter/faster way to fix this problem.