Cannot connect to Server 2008 R2 with RDP broken – Interactive Logon Initialization Process has Failed

ESXi 5.5 – recently I tried to RDP into my Server 2008 R2 machine without success. Looking at the console, I get the message: ” Interactive logon process initialization has failed. Please consult the event log for more details. ” My first reboot of the machine did an automatic check disk. I can no longer log into the machine either via RDP or on the console. I’d like to not have to rebuild this system as it is my only stand-alone DC in my home lab. I’m going to bring up a separate DC, then do a DCPROMO, however, below are the steps I took to resolve the issue, albeit unsuccessfully – some of these steps may work for you. I was hoping I could mark this process as “Solved” but I haven’t gotten there yet…

Event log says:

  • Event 4005 Winlogon – The windows logon process has unexpectedly quit
  • Event 33 SideBySide – Activation context generation failed for “C:\Windows\system32\LogonUI.exe”. Dependent Assembly Microsoft.Windows.Common-controls.Resources,language=”*”,processorArchitecture=”amd64″,publicKeyToken=”6595b64144ccf1df”,type=”win32,version=”6.0.0.0″ could not be found. Please use sxstrace.exe for detailed diagnosis.

These errors all seem to have started on 8/8/16 when the following events occurred:

Error: 36888 – Schannel – The following fatal alert was generated: 10. The internal error state is 1203

Event 56 – The terminal Server security layer detected an error in the protocol stream and has disconnected the client. Client IP: 223.x.x.x which is a Hong Kong IP. Prior to that are many Event 1012 – Remote session from client name a exceeded the maximum allowed failed logon attempts. The session was forcibly terminated.

I pulled the machine off the internet and the network in case it’s been compromised. On second thought I should probably kill this machine with fire!

Things I have tried that did not resolve the problem:

  • Boot into safe mode
  • Edited RDP settings on my workstation to use less resolution and video bit depth
  • Increased Virtual Machine’s system memory size as suggested, went from 3GB to 7GB
  • Booted to SystemRescueCD and replaced c:\windows\system32\LogonUI.exe – did this by booting the VM to a SystemRescueCD.iso (startx) and then mount the NTFS file system with the instructions here. Next I changed the root password with passwd and then connected to the VM with Filezilla on port 22 and renamed/moved the files.
  • Hotfix 437977 – Windows6.1-KB2615701-v2-x64.msu – because I cannot get into the system to run this, I started Task Scheduler on the remote server from my workstation, and attempted to have the .msu run but get error: the application has failed to start because its side-by-side config is incorrect.
  • Booted the VM to a Server2008R2.iso, Repair your Computer > command prompt,
    sfc /scannow /offbootdir=c:\ /offwindir=c:\windows

    didn’t work. Then tried command prompt > “cd sources” > StartRep.exe didn’t help.

Any suggestions? Has anyone gotten past this error? I can still connect to the machine via any means (mmc terminals such as event viewer, and task scheduler) other than a GUI console.

Edit: I gave up on trying to fix the issue – this is the reason we backup our VM’s and in this case I’m glad I had a good working backup. If you find a way to resolve, please drop a note – otherwise, make sure you have good backups of your DC’s: The VM as a bare metal backup and the System State (Separately.)

How to add a bulk list of Domains into Symantec Cloud MessageLabs Encryption TLS Business Partners

If your email servers support, or are required to enforce transport layer security (TLS), then you can use Symantec TLS Business Partners to configure policies to enforce encryption between your mail servers and the broader public, as well as Symantec’s Email Security Services infrastructure. You can also configure policies to enforce TLS encryption between your domains and the domains of third-party business partner organizations. This article will provide a procedure that demonstrates how to generate a report with Symantec PBE Advanced Domains that displays mail which failed a TLS policy. We then go on to process that report and relax the enforcement of Certificate Validation with the following steps:

  1. Scrub the Symantec Report for just email addresses of the intercepted mail.
  2. Remove the first part of the email addresses, leaving only a list of the domain names.
  3. Compare the domains from Step 2 to already-existing TLS Business Partners and leave only those domains that are unique.
  4. Create a new text .txt file with the unique domains, add ‘test@’ to the beginning of each domain to create an email address that can be tested at checktls.com.
  5. Batch test the domains for certain TLS conditions.
  6. Scrub and sort the checktls.com report for only those domains that meet conditions in Step 5.
  7. Upload the list of unique, tested, and scrubbed domains into the Symantec Cloud TLS Business Partners.
  8. Use Symantec to again test the domain for Certificate Validation, enable the domain, and enforce the encryption of traffic to that domain.

Often times, TLS encryption requirements will force end-users and recipients of TLS encrypted email to log into a secure mail portal to authenticate and retrieve messages. Depending on your organization’s policies, you may want to ‘relax’ the level of enforcement of TLS requirements so that users can receive mail in ‘plain text’ as a normal message in the user’s mailbox. These different levels of encryption are scored by traits including the availability of TLS encryption, TLS negotiation, the SSL/TLS Certificate trustworthiness, and other factors.

To a large degree, TLS email encryption has not quite become the standard for many mail servers out there. Many large mail providers such as earthlink.net do not support TLS encrypted email. Some mail servers will allow TLS encyrption (TLSAvailable=100), negotiate a TLS transaction (NegotiatedTLS=100), but not have a valid certificate (CertOK >= 33). Often times, the CertOK value will not reach 100 simply because the certificate on the receiving server is a self-signed certificate.

You can check the support of an email address or domain by visiting checktls.com, enter an email address into the address test, and hit the ‘Try It’ button. For example, when we test [email protected], our results are as follows:

TLS Receiver Test

In this case, if Symantec /MessageLabs were to try to send a message to the recipient, the message would not be delivered to the recipient mailbox, but quarantined in a secure mail portal. The recipient would then be required to create a secure portal account, log in, and retrieve the message.

If our results above were to demonstrate the passing of all checks with the exception of the CertOK column, then we may want to relax the enforcement of certificate validation in Symantec. But in order to find all of the domains that meet these conditions, (excluding those domains we already have validated and trust), we need to generate reports, scrub the data, and test for the conditions we accept. Once we find the unique domains, we’ll add them into our list of TLS Business Partners.

While the following procedure does not have long-winded explanations and exact step-by-step instructions, the information and screenshots should be enough to get you through the process, hopefully saving you the hours of research and note-taking that I endured. This article does not go over the initial setup of your domain’s email encryption with Symantec. The prerequisites for this guide are that you already have TLS email encyption setup with Symantec Cloud / MessageLabs.com. Also be sure to have Excel 2013 or newer and NotePad++ installed on your computer. Lastly, you may be required to purchase an account with checktls.com to run batch tests.

Lets get started!

We’ll first run a report in the Symantec Cloud / clients.messagelabs.com to find emails intercepted by the service. Starting in the Dashboard, go to Reports > Report Request > Request a new Report > Request name: PBEAdvancedDomains-Date > continue > email summary report (PDF) >

The data we will include in the report are:

  • Email Summary Report (PDF) advanced settings
  • Emails intercepted by the service
  • Data Protection volume
  • Data Protection volume by domain
  • Top 20 Data Protection recipients
  • Email Detailed Report (CSV) advanced settings
  • Data Protection
Message Labs Report 1
MessageLabs Report 2
MessageLabs Report 3
MessageLabs Report 4

Download the Report.zip

Extract to Desktop\TLSReports\RepDomainsDate

Open the EmailDataProtectionDetail.csv

Copy the column “Email To” to a new spreadsheet named 1stRunRemoveNameAndAtSymbol.xlsx

In 1stRunRemoveNameAndAtSymbol.xlsx, delete the 1st row “Email To”

Go here to learn how to remove the username from the email address http://pakaccountants.com/excel-split-extract-username-domain-email/ but essentially you put the function in row B1 and then copy the function down the column.

=RIGHT(A1,LEN(A1)-FIND(“@”,A1))

Excel wants you to type out the formula (instead of copy/paste) in the first cell B1, then apply formula down the column.

Apply formula to an entire column or row without dragging by Fill feature

In Excel, Select the entire Column B, and then click Home > Fill > Down.

Apply formula fill down
Apply formula fill down

Copy and Paste (Paste – Special->ValuesOnly) the resulting Domains (only) to a new sheet and name

1stRunDomainsOnlyDate.xlsx

Now you should have 3 files:

1st Run Files

In Symantec Message Labs go to Services > Encryption > existing TLS Business Partners

Download All -> get TLSBusinessPartners.csv > rename to TLSBusinessPartnersDate.csv copy to same folder, so you should have 4 files. In the 1stRunRepDomainsOnlyDate.xlsx you may want to remove all the empty cells with #VALUE! From the bottom of the file.

Create a new folder named Compare, and copy 1stRunRepDomainsOnlyDate.xlsx and TLSBusinessPartnersDate.csv into the folder:

Compare domains files

Remove all domains in TLSBusinessPartners file that are current Partners (as well as those existing in previously completed procedure runs.) – To do so,

Open 1stRunRepDomainsOnlyDate.xlsx. Create a new Sheet 2.

Open TLSBusinessPartners.. file and copy the column with only the domain names Column labeled “Business Partner Domain” into Sheet 2

Run compare and remove all domains that exist in TLS.BP.Date from 1stRunRDODate.

Rename Sheet1 “CleanTheseOfDups”, Rename Sheet 2 “Test”

Scrub the domains of duplicates. Copy all the Business Partner Domains AND be sure to include all of the domains listed in TLS Enforcement page into Test sheet (selecting with the mouse, copy from website, and paste into excel works, just remove the hyperlinks).

Populate test worksheet with existing domains

Next, select Sheet 1 then go to the Data tab and click Remove Duplicates:

Remove duplicate domains

Click OK

Duplicate domains removed

Add a new row to A1 top level and add header named: “CleanThese” for Column A and “Business Partner Domains” for Column B

Copy existing Business Partner unique domains in Test sheet to B1

Should now look something like this:

Clean domains of existing domains

Now Filter for Unique values:

  1. Click on the Column Header to highlight Column A. Select the Conditional Formatting button on the Home tab.
  2. Select the New Rule option from the Conditional Formatting button drop-down list. Select “Use a formula to determine which cells to format" as a rule in the New Formatting Rule window.
  3. Proceed by entering the following formula in the “Format values where this formula is true" box: =countif($B:$B, $A1)
  4. Click on the Format button and specify the format you want to set. It is possible to set a different cell color for duplicate values by clicking the Fill tab and selecting a background color. (Orange)
  5. After having specified the format you want click OK.

Credit: http://spreadsheetpro.net/comparing-two-columns-unique-values/

After comparing the two columns using the guide above, your spreadsheet should look something like the following:

Compare two columns in excel

6. Right click on for example A3, and choose Filter> By Selected Cell’s color.

Copy only those cells NOT HIGHLIGHTED WITH COLOR results in column A to new file named 2ndRunScrubbed.txt and place in folder…

Second run scrubbed

Now we need to add ‘test@’ to beginning of each line. To do so, Open the 2ndRunScrubbed file in Notepad++

One method that doesn’t involve find/replace (but only works if you want to do it on every line, including blank lines in your block), is this:

    • Move your cursor to the start of the first line
  • Hold down Alt + Shift and use the cursor down key to extend the selection to the end of the block, Then type in test@

This allows you to type on every line simultaneously.

You should now have a list of email addresses with all of your unique domains that you want to test at CheckTLS.com

Normalize domains as email addresses for testing

Save the resulting file as 2ndRunScrubbed1-9-17_TestTLSReady.txt

Log into checktls.com and run checks, get weights, and add the new business partners and enforcements.

http://checktls.com/tests.html#Batch

CheckTLS batch test 1

CheckTLS batch test: Once you’ve uploaded the file, hit R(u)n Test Now to initiate the testing. One domain takes 5 minutes, so 4000 domains can take 4 hours or so. An estimate from checktls.com will be given.

CheckTLS batch test 3

Once the report arrives via email, copy the data in email to a new file named TLSCheckReportResults1-9-17.xlsx

Next, find CertOK is a value of 33 or 0, but the TLSAvailable=100 and NegotiatedTLS=100

To do this, do Ctrl+A then Sort & Filter > Custom Sort > Sort by CertOK > Sort on Values > Order Custom List: 33, 0 > OK > OK

(Column H is Cert OK) so Select only those that are anything other than 100, copy the rows to a new spreadsheet named

TLSCheckReportResultsCertOK33-Date.xlsx

Sort the TLSCheckReportResultsCertOK33 file by TLSAvailable=100 and NegotiatedTLS=100

To do this, first remove all columns except EMailAddress, TLSAvailable, CertOK, and NegotiatedTLS=100

Do another custom sort; select columns labeled TLSAvailable, Cert OK, and NegotiatedTLS, and Sort by Column B (TLS) then by Column D (SenderOK/Sec) Largest to Smallest:

Excel custom sort columns on values largest to smallest

Select only those rows that have at least TLSAvailable = 100 … CertOK(%) … NegotiatedTLS = 100; where CertOK % is less than 100:

Excel custom sort results

Copy these values to a new .xlsx file and folder named:

TLSCheckSortTLSAvailable100Negotiated100-Date

Copy custom sort results above into a new .xlsx file and rename the filename the same as the folder name.

Remove everything except the domain name (remove ‘test@’) from this file. Follow instructions above – delete columns B, C, D, then formula =RIGHT(A1,LEN(A1)-FIND(“@”,A1))  – then fill down (select column first)

Copy resulting domains to a new .txt file named:

TLSCheckSortTLS-DomainOnly-Final-Date.txt

Now we need to add the tested domains to the Business Partner for the week. Eg. RepDomains-Date in Symantec.

Log into Symantec Cloud,

Services > Encryption > TLS Business Partners > Add new BP > RepDomains-Date > Upload New Business Partner Domains > Uncheck “Enabled” > choose file > select TLSCheckSortTLS-DomainOnly-Final-Date.txt > Save

Upload New TLS Business Partner Domains

Now that all of the new unique domains have been imported, we need to test them within Symantec and change Certificate Validation from Strong to Relaxed if required.

Click on the domain name, hit the TLS Test button, and if the TLS connectivity fails, hit Close, change the Certificate Validation to “Relaxed” and then Save.

Relaxed certificate validation

Go through each of the newly uploaded domains, and when finished, make sure to ENABLE TLS Encryption enforcement in the TLS Enforcements tab.

Take those in the list that test correctly and are enforced and add to Data Protection Exceptions. Services > Data Protection > PBE Advanced TLS the Portal Trigger Template > (scroll down) Recipient Exclude Domain List  > Edit List >  Add List Items > Add

It’s a good idea to repeat this procedure every so often so that you catch mail that is being delivered to the secure portal unnecessarily. Hopefully, this procedure will help someone save some time if they have the same task, or at the least help someone sort and scrub email addresses and domains.

Fix ubuntu when the OS will not boot – kernel panic – kernel panic not syncing vfs unable to mount root fs on unknown-block 0 0 – error /boot full remove old kernels from command line

To begin, it will probably take at least 30 minutes resolve this issue…

This fix solved my problem with the “vfs unable to mount root fs” error, but of course your results may vary. As always, first backup your system or do an export of the vm so you have a copy of the system as it existed before you started screwing around with it 😉

After running apt-get update / apt-get upgrade and then a reboot, you may receive the following error: kernel panic not syncing vfs unable to mount root fs on unknown-block 0 0 on ubuntu 16.04.

In many cases this  will be due to the /boot drive becoming 100% full because many updates have been made to the kernel. By default, ubuntu will retain the old kernels and add them to the list of available kernels you can boot into in the Grub2 boot loader menu. You can confirm that your drive is full by issueing the command:

df -h

The result will likely show the following:

In order to resolve this issue and boot successfully, while you’re looking at the error during boot, (you should already be at the console), and restart the vm or computer into the Grub2 menu then choose “Advanced options for ubuntu” view where you can see a list of old kernels you can boot into. Some report you can do this booting with the Shift key held down, or in the event it’s a virtual machine, you should be able to arrow-down in the Grub start screen and choose Advanced options for ubuntu on startup:

Grub2 boot menu.

Once you go into the advanced boot menu you will likely see several kernels listed. Choose the next-oldest kernel from the top/highest version of kernels. In my case I booted into the version labeled Ubuntu, with Linux 4.4.0-57-generic (my boot menu screenshot below is clean, but you’ll likely see several kernels listed).

Cross your fingers and hope you get to your login prompt. From here I jumped on putty and connected from that client, as I prefer it over the console.

Next, login and follow the directions that I found here:

http://askubuntu.com/questions/2793/how-do-i-remove-old-kernel-versions-to-clean-up-the-boot-menu

To save you the search, here are the instructions I used to first list and then remove the old kernels:

Open terminal and check your current kernel:

uname -a

DO NOT REMOVE THIS KERNEL! Make a note of the version in notepad or something.

Next, type the command below to view/list all installed kernels on your system.

dpkg --list | grep linux-image

Find all the kernels that are lower than your current kernel version. When you know which kernel to remove, continue below to remove it. Run the commands below to remove the kernel you selected.

sudo apt-get purge linux-image-x.x.x.x-generic

Or:

sudo apt-get purge linux-image-extra-x.x.x-xx-generic

Finally, run the commands below to update grub2

sudo update-grub2

Reboot your system.

sudo reboot

As you can see from my terminal history, I had to remove a few:

589  uname -a
 590  dpkg --list | grep linux-image
 591  sudo apt-get purge linux-image-4.4.0-21-generic
 592  sudo apt-get purge linux-image-4.4.0-22-generic
 593  sudo apt-get purge linux-image-4.4.0-24-generic
 594  df -h
 595  sudo apt-get purge linux-image-4.4.0-24-generic
 596  sudo apt-get purge linux-image-4.4.0-28-generic
 597  sudo apt-get purge linux-image-4.4.0-31-generic
 598  sudo apt-get purge linux-image-4.4.0-34-generic
 599  sudo apt-get purge linux-image-4.4.0-36-generic
 600  sudo apt-get purge linux-image-4.4.0-38-generic
 601  df -h
 602  sudo apt-get purge linux-image-4.4.0-42-generic
 603  sudo apt-get purge linux-image-4.4.0-45-generic
 604  sudo apt-get purge linux-image-4.4.0-47-generic
 605  sudo apt-get purge linux-image-4.4.0-51-generic
 606  sudo apt-get purge linux-image-4.4.0-53-generic
 607  sudo update-grub2
 608  dpkg --list | grep linux-image
 609  df -h
 610  sudo apt-get purge linux-image-extra-4.4.0-21-generic
 611  sudo apt-get purge linux-image-extra-4.4.0-22-generic
 612  sudo apt-get purge linux-image-extra-4.4.0-24-generic
 613  sudo apt-get purge linux-image-extra-4.4.0-28-generic
 614  sudo apt-get purge linux-image-extra-4.4.0-31-generic
 615  sudo update-grub2
 616  df -h
 617  sudo reboot
 618  dpkg --list | grep linux-image
 619  uname -a
 620  sudo reboot

After the reboot, you can see my /boot partition returned to a manageable size:

I hope this post helps someone save some time and help them fix their ubuntu boot problems. Please leave a comment if this helped resolve your issue or if there is a smarter/faster way to fix this problem.