The SolarWinds hack, how does it affect you?

Background

As if this year has not brought enough turmoil and chaos to the country and world, this hack shows that 2020 had at least one more trick up its sleeve. In case you have not heard already attackers were successful in inserting a trojan into a widely used enterprise IT management called Orion from the software company SolarWinds. We may never know the full scope of this hack but any company that installed or updated Orion in the last 6-9 months is potentially at risk. It is very likely before this week you have never heard of Solar Winds so how bad can this be? Solar Winds has a large customer base as shown by the screen capture below from their now removed webpage.

How Widespread is it?

It is estimated over 18,000 entities used SolarWinds Orion product suite. Over the next days, weeks, and months you will probably hear of more and more companies and government bodies that have been breached. This hack has affected local governments from Pima County to highly sensitive Federal government agencies such as the Department of Energy’s National Nuclear Security Administration as well as the U.S. Treasury and Commerce departments. Several organizations such as Cox Communications, Microsoft, Ford, and many others have been affected.

How bad is it?

The attackers were not out to cause chaos and destruction, that would have been a waste of their efforts that would have brought this hack to light very quickly limiting its effectiveness. Rather they built some intelligence into the trojan such as not executing until several weeks after installed. They took efforts to make sure the files were signed and trusted like all other SolarWinds software. The trojan’s communication with the hackers was disguised to look like the SolarWinds traffic in order to obfuscate it and prevent detection. This hack was probably months or years in the making and there is currently no information on how it was accomplished. Some of the likely ways were; directly hacking into SolarWinds systems, using reused credentials, or paid an employee to help with the attack. It was most likely a nation-state actor such as agents of the Russian government from the current indicators.

Is my company at risk?

From what is currently known, only companies that had SolarWinds Orion with specific versions are affected. Orion is a product that is typically only used by larger companies as it is very expensive and is used to manage large computer networks. If you did not have this product installed on your network you are not at direct risk from this attack, most likely you did not. Any company that does or did have this product installed in the last year should immediately take steps to investigate, remediate and recover. Luckily Microsoft and others have taken steps to take down the command-and-control network used by the attackers, but that does not mean they did not get alternate access to breached networks and time is of the essence in situations like this.

Even without the product installed you and your organization can still be at risk. The highest risk will be from your information being used by the attackers. Being they accessed companies, ISP’s, hospitals and government agencies any sensitive information they had could be out in the wild. There is little to no direct risk to you or your organization from being hacked through any of these vectors. Except for in very care cases, your ISP such a Cox Communications being breached, did not and does not give the attackers inside access into your networks. Most businesses use a router/firewall which is not managed by Cox or other ISP’s and would make their network being breached no different than any other internet attack against your company.

With that said the attackers wanted high value targets and would not have had the resources or want to risk their efforts being caught by going after what would be a low value target to them. Government agencies, large companies, defense contractors, hospitals, Universities and ISP’s are their most valuable targets.

What now?

If your organization uses or recently used Orion, I hope you are not still reading this and have already contacted your IT department, Lawyers and a good cybersecurity incident response company such as Cynet, FireEye (yes that is another story in itself), CrowdStrike, SecureWorks or others.

For the rest of us the best thing we can do is take steps to increase our security posture. It is not possible to stop all threats but good cybersecurity hygiene and best practices will protect us from most attacks reducing the likelihood of a breach. Many organizations already have compliance standards they might need to follow based on their industry, such as HIPAA for health care providers, PCI for retail businesses, NIST 800-171 for Department of Defense contractors, SOX for publicly traded companies, Gramm-Leach-Bliley Act for financial institutions, just to name a few.

For companies that do not fall in any of the mentioned regulations, the NIST Cybersecurity Framework is a good starting point for processes and procedures to harden and company against cyber-attacks.  This framework is a scaled down version of the NIST SP800-53 which is cost prohibitive for most small companies. The CSF has guidelines to help a company Identify, Protect, Detect, Respond and Recover as shown in the graphic below.

Final Thoughts

Cybersecurity is more than just implementing technological safeguards. It is technology along with ongoing administration and processes with the goal of reducing risk. This will require some internal change and, in most cases have financial costs associated with it. The cost of not taking steps to protect the organization could be much higher in the terms of reputation, loss of business, government fines or lawsuits.

Java issues administering HP printers and network equipment

With newer versions of Java and older firmware on HP equipment you may run into Java errors trying to administer the devices through the web interface.  Adding the IP or FDQN to the Java exceptions alone will not work now that High security is the lowest level allowed in the Java control panel. Below are a few fixes I have found.

For an HP ProCurve 2810 switch there needs to be an additional entry in the java control panel in the form of http://x.x.x.x/classes/agent.jar

2016-06-17 23_36_10-ScreenConnect - CONTROLLER-PC - Connected

For an HP 4650 printer I found the java.policy file needs a line to allow the specific port used by the printer interface, this case 161, that is not a normal HTTP/HTTPS port. The file was located in “C:\Program Files (x86)\Java\jre1.8.0_77\lib\security” which could be different based on your JAVA version. A line similar to ‘permission java.net.SocketPermission “x.x.x.x:port#”, “connect, accept”;’ will need to be put in the grant section of the file. This is in addition to the http://x.x.x.x in the Java control panel

2016-06-17 23_37_49-C__Program Files_Java_jre1.8.0_91_lib_security_java.policy - Notepad++ [Administ

I hope this helps some other people that run into the same issues. If you have any other fixes I would love to hear them.

QuickBooks 2015/2016 Unrecoverable Error

As any IT person can probably attest to QuickBooks can be a challenge sometimes, it is a widely popular accounting package used by  millions. Most issues arise when it is used in a multi user environment, from simple issues like firewall ports for the DB server, user error like leaving it in single user mode or making their workstation the host, to crazy issues like QB no longer liking the file name and and the only solution is to rename the file. Well this time it turned into tens of hours of frustration and talking to the Office of the President….

Haha, this is only at Intuit we are talking about, not the White House, now back to the problem.  The issue first came up when trying to move from QuickBooks Online to QuickBooks Desktop, every time the export process would fail. I called QBO (QuickBooks Online) support, they were very kind and helpful yet after about an hour of trying still the same result. Their next step was to send me a QB portable file that they created, later that day I got the file, restored the file and QB crashed with an unrecoverable error. Tried a few things with no change, so I decided to try another computer. Well same result once again, another unrecoverable error. Was there a problem with the QB file or was this a bigger issue?  I tried creating a new company file and also opening a sample file, yet again the dreaded unrecoverable error.

2015-10-14 08_42_47-qbunrecoverableerror.png - Windows Photo Viewer

Doing some research it appeared to be a .Net related issue, I uninstalled and reinstalled .Net and repaired QB installation. No change, still getting the error.  I installed QB on another 3 computers and they all had the same issue. I uninstalled some programs that were the same between computers and still no change. Not wanting to get too bogged down by this issue and hoping their support could use the QB error logs, I thought I would call QB support. After an hour or so of working with Tier 1 support I asked to be escalated to Tier 2. In all I worked with 4 different people over about 4 days with Tier 2 support, then with a Supervisor. Quickbooks 2016 came out during this and still had the same issue. Ultimately the supervisor told me to reload Windows on all of the machines. I was not happy with this solution so I emailed the VP of Small Business at Intuit who is the head boss over QuickBooks.

Within an hour I got an email that the Office of the President at Intuit would contact me the next business day. The next day I got a call from someone in the local Tucson Intuit location. I rehashed everything that had been gone through and we exchanged some emails. I was then told that they would send someone onsite. Now a week later a person from the local site shows up, and we try many of the things that had been done with Tier 2 support. At this point I have blood running down my forehead from banging my head on the wall at the absurdity of doing the same steps that did not work before over and over again. In talking with this person I glean that while I had sent many of these error reports in that nobody had even looked at any of them. The next step was to have development look at the error logs that have been going into limbo at Intuit somewhere. So now I wait again.

After about 5 days I get a email that they have learned something from the logs and have a fix they would like to attempt, so I arrange to do a remote support session in the morning. It turns out one of the 3rd party programs on my suspect list, IBM iAccess 7.1, might be the culprit even though uninstalling it did not fix the issue. This program adds a bad entry into the machine.config file for .Net off to edit the file at ‘C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config’ and ‘C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config’

iaccessQBissue2

The iAccess software had added an extra line to the machine.config file which is XML and needs to be properly structured. Once the <DbProviderFactories /> line was removed from the machine.config files we attempt to open Quickbooks and it now works. I tried this on the rest of the machines and they all work now. Lesson learned on this one, delete all files even though .Net was uninstalled. I am grateful that Intuit cared  enough to stick through this and help find the resolution rather than just insist on reloading Windows on all of the machines. I am also investigating this support article from IBM to see if it is related to the problem http://www-912.ibm.com/n_dir/nas4apar.nsf/ALLAPARS/SE45767

Cotap Targeted Spam

Cotap, a company that offers ‘secure’ team messaging for businesses has gone an unethical route and is using targeted SPAM emails to solicit customers. You get an email saying you have been added as a contact for the company, but oddly enough nobody in the organization has used or heard of Cotap’s service.  From searching the internet and seeing posts in the forums at Spiceworks it looks like this service has been doing this for a few months. I would suggest that you mark the emails as spam and add their cotap domain to your spam service blacklist.

cotap

Exchange Server 2013 – Blank ECP page

I was asked to look into an issue with Exchange 2013. Report was that users were not able to send emails internally or externally. This had been happening since around 2PM and it was now 5PM when I was asked to intervene. I asked my usual questions such as what changes had there been. Specifically anything in Active Directory such as user accounts, groups and so forth, asked if there were any network changes or virtualization reconfiguration. The answer given to all of these was no. The exchange servers had already been rebooted multiple times.

Opened OWA just fine, yet no mail since around the affected time, outlook was connecting on the client PC. Then tried to connect to the Exchange Administration Console website ECP and no dice, just a blank screen with no error. Logged into the client access server, all services are running, no major errors in the event log, no errors in the IIS log for the default site.

Next up was to try the Exchange powershell console. When launched it came up a bit slow, after a minute it gave errors of  “…winrm client cannot process the request access denied…” against all three servers in the organization. Tried the exchange console on the other two servers, and one of them worked the other had the same error of access denied. Verified that AD was syncing properly, and that DNS was working. I looked into this error online which lead me down many paths, none of which were of any luck.

I called my contact back and probed some more. Turned out that the network time was changed on the primary domain controller role holding server, early in the morning, to adjust for the users complaining the time on their computer was not close enough to their cell phones. With this new information I logged into all 5 domain controllers and found 2 of them had time that was  10 minutes off due to the earlier time change. Fixed the time on the domain controllers, rebooted all 3 exchange servers and now email was flowing and the ECP page worked fine.

Cognos – Formatting Output

In creating reports that are usable by people sometimes there is a necessity to change computer given output in order to make it more reader friendly.  In this accounting package there is limitations on the digits available for job numbers and it is up to a 6 digit field with no special characters so a job number might look like 14248. Unfortunately when creating reports most of the people in the organization would expect the format to be 14-248 instead. To achieve this output I used Cast to break apart the values and add hyphen symbol after the second character.

cast(substring([Job#],1,2)+'-'+substring([Job#],3,3),varchar(6))

Basically this takes the 5 digit codes, grabs two characters starting from the first (hence the 1,2) and adds the – symbol followed by grabbing the last there characters starting at item 3 ( 3,3) and then casts this into a varchar that is 6 digits long.

Cognos trials and tribulations – Data Modeling

Having to learn Cognos on the fly has been a challenging yet rewarding experience.  I had done some work with Crystal Reports and Microsoft SSRS in the past and this is a very different beast all together. For one in the environment for my work the Cognos server is hosted by the application provider and Cognos allows a data model to be built, and in this case I do not have direct DB2 SQL access but rather data the way the vendor has modeled it for me. In this case trying to convert from some old reports in a legacy reporting tool for the LOB to a Cognos report proves challenging. Not having direct access to the DB and field names that do not always correlate can make it difficult. Using the ‘Lineage’ tools has helped me to find what I am looking for. To access this, right click on your data object, and choose lineage.

lineage

From here you can choose the ‘Technical View’ tab and see what underlying object in the database this object was derived from.

techview

Office 365 connection, calendar and delegated account issues

I have seen many issues lately where Office 365 has connection issues affecting setting up new Outlook profiles, odd certificate popups, seeing delegated email accounts and shared calendars.  Turns out it is due to Outlook resolving to the root domain such as domain.com rather than autodiscover.domain.com.  This is an issue when hosting providers have mail servers that are listening on that domain and it causes Outlook to get the response and fails the connection.  The quick way to test this is to import a registry file that will change this behavior.  Copy the data below, save to a text file, rename to rootdomainfix.reg and import into the registry.

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\15.0\Outlook\AutoDiscover]
"ExcludeHttpsRootDomain"=dword:00000001

 

Java 8 WMI query

In the past using a WMI query to filter computers with Java 7 the query looked like

Select * From win32_Directory where (name="c:\\Program Files\\Java\\jre7" or name="c:\\Program Files (x86)\\Java\\jre7")

which fairly quickly returned results.  Now with Java 8 the folder is in a format of c:\Program Files\Java\jre8.0_xx which causes the query to run for about 10 minutes.  In searching for a better way I came up with

Select * From Win32_Product where name like "Java 8%"

which is reasonably fast, since I wanted both Java 7 and 8 my final query was

Select * From Win32_Product where (name like "Java 7%" or name like "Java 8%")