Tuesday, February 23, 2010

(Editorial): Clearing the Air On the SSL Issue

Much has been made recently of ZDNet’s so-called “expose” (really a smear piece) about our software. And one of the most disturbing accusations is that we were exposing user data by having our tracker agent communicate over the unsecured TCP Port 80. As evidence, ZDNet blogger Jason Perlow offered up network traffic sniffer data showing our client communicating in the open and without encryption.

After an extensive investigation on our end, we have concluded that the use of Port 80 by the ZDNet client was an isolated incident, the result of a registration script error on our end. Simply put, we were not prepared for the sudden disappearance of our co-branding partner, InfoWorld (ZDNet registered with us the same day that IDG pulled the plug), or their normal registration process, which they controlled. As a result, when ZDNet registered its test VM with us, they were inadvertently redirected to an old test version of the script and it input the incorrect values for the ASPScriptPort and ASPScriptProtocol fields in our console configuration table.

Note: Other clients that were connected at the time were indeed communicating over SSL/Port 443. In fact, if ZDNet bothered to check back now they’d find that the agent now defaults to this mode for all future connections.

Much has also been made about our SSL certificate, which expired in September. To this accusation we have no response other than to say that we screwed-up. It was on our to do list and we missed the reminder notices from our issuer. Fortunately, our client agent is configured to ignore certificate errors when dealing with our server - a precaution we built in when a malformed certificate tripped us up several years back. So while the certificate may have been invalid, our client agents were still capable of connecting to the server using an SSL-secured connection. Regardless, we will be renewing the certificate shortly.

Bottom Line: The DMS Clarity Metrics Tracker agent is an SSL-secured program that does not expose user’s data – when properly configured. Obviously, the events of this past weekend – specifically, the sudden removal of the Windows Sentinel code from InfoWorld’s web site (an act that was in violation of our hosting contract with IDG) – caused a few ruffles in what had heretofore been a fully secure service with an unblemished operating record spanning nearly four years.

But we don’t expect Mr. Perlow and his hired hit-cronies at ZDNet to pay any attention to above disclosure or how it might explain their “alarming” experiences with our agent. They have their marching orders, and all signs point to Redmond as the real impetus behind the whole sordid affair.

Make no mistake: Online IT Journalism is dead, folks, the victim of greed, bias and an unquenchable thirst for page views…

RCK

14 comments:

Japhrg said...

"Fortunately, our client agent is configured to ignore certificate errors when dealing with our server - a precaution we built in when a malformed certificate tripped us up several years back. So while the certificate may have been invalid, our client agents were still capable of connecting to the server using an SSL-secured connection."


Wait a second--while I can understand mistakenly not renewing your cert (stuff happens), your client ignores certificate errors??!?!

Just date related cert errors? How about revocation? Do you check the cert chain? How bout self signed certs with your name on them? Certs created in the future? Certs that don't apply to your domain?

Seriously, there are very, very good reasons why cert errors should be paid attention to. Performance data isn't as critical as say financial data, but you do have financial customers, right? How do they know they are connecting to the 'real' xpnet? Encryption is kinda moot if your agent is talking to the wrong server.

Randall C. Kennedy said...

Jeremy,

It's configured to do so only if it's communicating with our server - a process that requires a significant degree of handshaking just to get started. The chances of somebody being able to fake our server's communications syntax *and* spoof our domain name *and* return the necessary configuration values to make Tracker run at all, are infinitesimal.

And assuming somebody *did* manage to pull all of the above off, they'd come away with...drum roll please...a bunch of useless raw metrics data.

Again, this is a simple, free hosted performance monitoring service. We're not conducting e-commerce or collecting anybody's credit card info. If we were, we might worry more about making it airtight.

However, right now we're more concerned with balancing the need for a *reasonable* degree of security with the need to scale to tens of thousands of users - which, when dealing with the volume of data we're collecting, is a tall order for any server to handle when SSL is involved.

Next question...

RCK

quux said...

Much has also been made about our SSL certificate, which expired in September. To this accusation we have no response other than to say that we screwed-up.

(Emphasis mine.)

How many employees does Devil Mountain have?

Anonymous said...

Actually you are opening a potential security hole on users computers. Your client does not adequatly verify the identity of the server on the other end. Furthermore, when a cert is found to be invalid, it then defaults to a insecure communications method. This would be fairly easy to compromise for a user. At that point your client is open to direct attacks.

Given that it runs as a service and on most XP machines is likely an admin, you would then be opening their machine to a potential takeover remotely. Since you have admitted to several bugs so far in your code in the course of these threads, I am concerned that you have effectively given the keys to a nice little botnet to anyone willing to take the time to do a little packet sniffing on your client(since its freely downloadable, it wouldn't be difficult to do).

If you were responsible, you'd shut this down immediatly.

Randall C. Kennedy said...

ReflexVE,

Actually, I never said the client defaults to an insecure communication method. If it's configured to use Port 443 via an SSL-encrypted path, then that's what it will use until instructed otherwise. The only reason it became mis-configured for Port 80 on the ZDNet systems was that it received the incorrect parameter from our server. And that, in turn, was the result of an unexpected switchover from InfoWorld. Had we known they were going to pull the plug that quickly we would have been better prepared. But I guess when I told them to stuff their offer to stay on, they took their frustration out the only way the knew how.

RCK

Japhrg said...

Randall, I understand the vector is pretty minimal, but us in the security community are trying *really, really* hard to make sure little issues like this don't turn into big ones.

Yes, it probably won't cause anyone to get their ID stolen or compromise some valuable resource. But ignoring cert errors that are there for a very, very good reason just can't be encouraged--just like pushing data into a memory buffer that isn't checked can't be encouraged (smashing the stack for fun and profit, anyone?).

i guess all i'm saying is this: ignoring cert errors is a slippery slope, please don't go down that one.

Randall C. Kennedy said...

Jeremy,

I agree, which is why we'll be fixing this in the next version of the Tracker agent. Thanks for the input!

RCK

Anonymous said...

Randall -

You just stated that the problem is fixed. Then you stated that it will be fixed in the next version. Which is it?

Also, what method do you use to force an update on the existing installations to close this potential security hole? And how do you prevent a third party who has sniffed your network communications from using it to push their own updates to the system?

Finally, your existing version that is installed on 20,000+ PC's will default to port 80 as it stands today, unless you have already pushed down an update. I don't know why you keep saying they are not doing this since it can be demonstrated that they are.

And please stop saying that IDG begged you to stay on. Had they done that they would have lost the little bit of credibility they had left. Proof or it didn't happen.

Randall C. Kennedy said...

ReflexVE,

You misunderstand our architecture. When the Tracker agent starts, it queries our server for a configuration profile, including which Port and Protocol to use. This happens each time the service starts, so the value is automatically updated if the user reboots or otherwise manually stops/restarts the service.

Also, we have to ability to remotely reconfigure the running client from our server. Which is how we can push out changes to this and other configuration parameters on the fly.

So, for the few users who might have been affected by the outdated script I mentioned (or, in Ed Bott's case, by an old console group record from years ago), the situation has been corrected remotely - both for clients that were running at the time the update was pushed out and for those that subsequently reconnect to the server after reboot or similar stop/restart event. ZDNet has now acknowledged as much in an update to the original article.

Right now, there should be nobody with a Tracker installation talking to our server over Port 80. If you know of an exception to this, let me know and we'll correct it immediately.

Finally, as for the fixed vs. fixed in next version discussion, the former referred to us forcibly reconfiguring any mis-configured agents via the process I detailed above, while the latter refers to our decision to change the certificate checking behavior to make it more airtight in the next release.

RCK

Anonymous said...

Why do you keep saying 'our'? So far as anyone can determine, this is a one man operation. Who else works for you? I cannot find any mention of anyone else anywhere with your organizations on their resume, either past or present, via any social or business network I check.

I'm fairly well connected. If you had a real staff, especially one with connections to Intel's labs(who I have worked directly with on numerous occasions and am personal friends with several people for more than a decade) wouldn't I at least have heard of *someone* who has worked for your various company names?

If its a one man show, just say "I" instead of "our". Right now it comes off highly disingenuous.

As for your fixes, I can think of several attack vectors using those routes, including spoofing your server, DDOS, etc, that would make your architecture as highly suspect.

Randall C. Kennedy said...

ReflexVE,

It's a corporation. And I'm not the owner, so...

As for the rest, then why don't you list some suggestions for how to fix our vulnerabilities and we'll see about implementing them. So far, you're just commenting to be destructive. Try being constructive for a change and help us make xpnet.com a better site.

RCK

Anonymous said...

So who else is involved? And I told you before, the reason I am coming after you is that you have made my life difficult with many of your reports in the past. Why should I go easy on you now?

As for your offer, I am a very experienced test engineer who actually has worked on many of the products you have evalutated at a low level. I even know of the performance measurement hooks that can be used to truly measure system performance. If your offering me a job, let me know and I'll forward a resume. I don't work for free, nor do I have any of your purported altruistic goals around your project.

I make shit better. And I do so by finding problems in code, in design and in methodology. And yes, that means I am *highly* critical, I get paid good money to be.

Anonymous said...

Hey, you haven't yet given evidence that your corporation has other people involved, specifically the techs you claim are former Intel employees. I've done a bunch of checking and I cannot find anyone on LinkedIn, Spoke or other such networks who has your company in thier history, and as I stated before, no one I know who works for Intel has heard of anyone working for you.

Can you address this claim? Its not a corporate secret to state who works for you. If anything, it would lend you some credibility.

Randall C. Kennedy said...

ReflexVE,

OK, I'll bite. Who at Intel have you spoken with? Pat Gelsinger? Paul Otellini? Carl Klotz? Glen Begis? Nick Mavrick? Tom Harper? Laurie Rader? Jim Woodruff? Scott Nipper? Narayan Manepally?

These individuals most definitely know who I am and what I did for their company. Of course, this was 8-12 years ago, before Devil Mountain was even formed, when I was still working under the umbrella of my first corporation, Competitive Systems Analysis, Inc. (Simply "CSA" to most Intel people), so...

But hey, I've got an idea: Why don't you show us some of *your* bona fides? I've named names - even in instances where I probably shouldn't have - and I can back up my claims in a court of law, if necessary. Why don't you tell us a little bit about yourself? You can start with your real name and place of work, and we'll go from there. Should be fun!

RCK