Thursday, November 19, 2009

Moving on to Scala... Hopefully

Ah, Scala, I became infatuated with this language some time ago when I saw the following video presentation from fosdem 2009.

This language is such an evolutionary step over what was there before that's it really makes you think what's coming up in the future. By that, I mean what else will come from this field which was once thought to be done, where original research was basically considered to be all over.

So, why hopefully? Well, the problem with Scala, much like any new language, is tooling. The good news however is that Scala produces Java Bytecode instead of starting from scratch.

Meaning that's it's still a pain to use your favorite IDE, Netbeans in my case, with this language. However, leveraging the existing Java platform means that it doesn't suffer from things that would normally suffer any nascent language:
  1. Uses the JVM, so performance is what you would expect. It's fast and works well.
  2. Tons and tons of libraries because it can access any Java library out there.
  3. Continuous integration servers and other tools that you would want to complete your development life cycle are all readily available since it's Java based
  4. Mature application servers. Code produced from the compiler is Java Bytecode, hence any Java EE server, like Glassfish, will just work.
Unfortunately, for now, until the Netbeans support matures, I won't be using this. However, I would recommend developers still have a look at that video. It's hard to look at this video without being in total awe of what's going on.

Thursday, October 22, 2009

Kicks & Giggles with Windows 7


256 core system with 1TB of RAM. Windows 7 supports up to 2TB of RAM. You'll need Windows Server 2008 R2 (Windows 7 Server) if you need more however.


Netbeans and GWTTestCase

In the "yet another problem I couldn't find an answer on Google for" unofficial series.

The problem lies in the fact that the GWT compiler operates on Java source code and not Java bytecode. Unfortunately, a standard WAR project on Netbeans will exclude the Java sources from a WAR file.

The first step is to go edit your project and go to "Build" and in "Exclude From WAR File:" (their spelling here not mine) field remove the value there which by default is "**/*.java,**/*.form".

The second thing you need to do is go to the run section and specify heap options in the "VM options" field. This is needed because tests run outside the application server you're using.

Finally, just hit "Test Project" and see your GWTTestCase's go. However, I would be remiss if I didn't mention that the vast majority of your GWT test code should not use GWTTestCase. Simply put, if your views are properly segregated, the majority of your GWT client code will run as regular Java code, so just use regular JUnit test cases for those.

Saturday, October 3, 2009

Capturing UIWebView Touches

I’ve searched everywhere on how to do this and everywhere the same result.  Either use a transparent overlay that you put over the web view to capture touches but you then lose the ability to click on HTML anchors, pinch and move around or subclass UIWebView and disable user interactivity.

First, a word of caution, the technique I describe here is not for the faint of heart nor is it for the inexperienced.  If you mail me for help, most people do instead of leaving comments for some reason, I will silently discard your email with a smile on my face.

The first problem is that UIWebView internally uses many undocumented views.  The iPhone has many very useful, undocumented views.  Unfortunately, using them means an automatic rejection from Apple’s AppStore.  The UIWebView is a composite of HScroller, UIWebDocument, UIImageView and other more minor views.  Out of all of those, only UIImageView is documented. 

Furthermore, the UIWebView documentation states that you should not subclass this view.   Considering how completely useless doing that is, it’s actually sound advice since it will not hurt your AppStore submission approval process.  UIWebView delegates all its functionality to internal components, overriding any method found in there doesn’t buy you anything, just lost time.

The first thing you need to know is that UIWebView is a very narrow view of the underlying WebKit engine.  Apple probably did this to make its system secure but also keep people from doing too many things with it too.  If you’re on Android, just smile and be happy you don’t have to do business with these people.

The second thing you need to know is that it’s OK to reference undocumented views as long as you use documented API’s to get them and you store such references in a documented class with the obvious choice being UIView.

OK, so let’s describe what we’re going to do before we show some code.  The steps are the following:

  1. Define a new protocol that extends the UIWebViewDelegate protocol, we’ll call this UIWebViewDelegateEx for now.  Define a new method called “tappedView”.
  2. Define a view, not a view controller, that is completely transparent.  This view holds the reference to the UIWebView you want to capture information from.  This view implements the UIWebViewDelegate protocol in full and holds a reference to your custom UIWebViewDelegateEx object.
  3. We define a couple of flags in this view, basically didMove and didShouldStartLoadViewRequestGetCalled.
  4. We define a timer in this class.
  5. The overlay should be above web view.  Completely transparent.  It doesn’t have to be over all of it but at least the section you want to get events from.
  6. We find the view that we need that was touched by the user by using hit testing.  Again, only documented API’s are used to get this view.
  7. We define a timer that we start when we detect a tap.  If this timer fires before the UIWebView delegate method “shouldStartLoadWithRequest” is called, we consider this a tap, otherwise, we consider that you activated an HTML child object of some kind, most likely an anchor.

So, at the end we have something that looks like this:

#import <UIKit/UIKit.h>

@protocol UIWebViewDelegateEx<NSObject, UIWebViewDelegate>

/**
* Called when the view was touched by the user and wasn’t an anchor.
*/
- (void)tappedView;

@end

/**
* Intercept any touch events by displaying a transparent
* overlay on top of a web view.
*/
@interface WebOverlayView : UIView<UIWebViewDelegate> {
        /**
         * The view that we are monitoring, i.e., the view that we will
         * possibly steal events from
         */
        UIWebView *webViewComposite;
        NSObject<UIWebViewDelegateEx> *delegate;

@private
        BOOL didMove;
        BOOL didShouldStartLoadViewRequestGetCalled; 
        NSTimer *timer;
}

@property(nonatomic, retain) IBOutlet UIWebView *webViewComposite;
@property(nonatomic, retain) IBOutlet NSObject<UIWebViewDelegate> *delegate;
@property(nonatomic, retain) NSTimer *timer;

@end

From here, we have to filter what we want, the strategy is basically is that if the user touches the view and the UIWebViewDelegate method “shouldStartLoadWithRequest” does not get called, we can assume that the user touched the view without triggering an HTML object like an anchor, this is where the timer comes in.  If after the elapsed time, “shouldStartLoadWithRequest” has not been called, we call our custom “tappedView” message.

#import "WebOverlayView.h"

@implementation WebOverlayView

@synthesize webViewComposite;
@synthesize delegate;
@synthesize timer;

#pragma mark -
#pragma mark NSObject

- (void)dealloc {
    [webViewComposite release];
    [delegate release];
    [timer invalidate];
    [super dealloc];
}

#pragma mark -
#pragma mark WebOverlayView

- (UIView *)findViewToHitForTouches:(NSSet *)touches
        withEvent:(UIEvent *)event
        inView:(UIView *)v {
    UITouch *touch = [touches anyObject];
    CGPoint pt = [touch locationInView:v];

 
    return [v hitTest:pt withEvent:event];
}

- (UIView *)findViewToHitForTouches:(NSSet *)touches withEvent:(UIEvent *)event {
    return [self findViewToHitForTouches:touches withEvent:event inView:webViewComposite];
}

- (void)timerFired:(NSTimer *)t {
    timer = nil;
    if (didShouldStartLoadViewRequestGetCalled == NO)
        [delegate tappedContent];
}

#pragma mark -
#pragma mark UIView

- (void)touchesBegan:(NSSet *)touches withEvent:(UIEvent *)event {
    [[self findViewToHitForTouches:touches withEvent:event] touchesBegan:touches withEvent:event];
    didMove = NO;
}

- (void)touchesMoved:(NSSet *)touches withEvent:(UIEvent *)event {
    [[self findViewToHitForTouches:touches withEvent:event] touchesMoved:touches withEvent:event];
    didMove = YES;
}

- (void)touchesEnded:(NSSet *)touches withEvent:(UIEvent *)event {
    [[self findViewToHitForTouches:touches withEvent:event] touchesEnded:touches withEvent:event];
    if (didMove == NO) {
       [timer invalidate]; 
       didShouldStartLoadViewRequestGetCalled = NO;
       timer = [NSTimer scheduledTimerWithTimeInterval:0.75
                target:self
                selector:@selector(timerFired:)
                userInfo:nil repeats:NO];
    }
}

- (void)touchesCancelled:(NSSet *)touches withEvent:(UIEvent *)event {
    [[self findViewToHitForTouches:touches withEvent:event] touchesCancelled:touches withEvent:event];
    didMove = YES;
}

#pragma mark -
#pragma mark UIWebViewDelegate

- (void)webViewDidStartLoad:(UIWebView *)webView {
    [delegate webViewDidStartLoad:webView];
}

- (BOOL)webView:(UIWebView *)webView shouldStartLoadWithRequest:(NSURLRequest *)request
        navigationType:(UIWebViewNavigationType)navigationType {
    didShouldStartLoadViewRequestGetCalled = YES;
    return [delegate webView:webView shouldStartLoadWithRequest:request navigationType:navigationType];
}

- (void)webViewDidFinishLoad:(UIWebView *)webView {
    [delegate webViewDidFinishLoad:webView];
}

- (void)webView:(UIWebView *)webView didFailLoadWithError:(NSError *)error {
    [delegate webView:webView didFailLoadWithError:error];
}

@end

If this code was useful to you, drop me a line in the comments (not e-mail).  Obviously, you need to apply the technique to your own application but at least it shows it’s doable.

Wednesday, September 23, 2009

The Insanity of Google Chrome Frame

So Google is fixing Internet Explorer for Microsoft. Great, fine. You know what really pisses me off however? Is that Google decided this was a priority when you have people like Ray Ryan at Google I/O 2009 telling people we don't have infinite resources to get GWT 2/UiBuilder out the door.

Or the same speech about various AppEngine features. Like, for example, how about a JPA implementation that is actually worth a damn in AppEngine? Like instead of giving us a big speech about de-normalizing data on AppEngine, how about you use the current JPA annotations to do it for us, you have enough information in those annotations to do it for us.

Or a browser on Android that is actually good, you know with multi-touch and pinching. And don't give me that crap about Apple patterns. Apple is at war with Google, Google is just too dumb to realize it yet. Also, people, don't call the Android browser Chrome Lite, it's not fast enough to be deserving of the name Chrome just because it's WebKit based. The architecture of these 2 browsers are very, very different.


Monday, August 24, 2009

Ending Series on C++0x Feature Focus

Considering that the "x" in C++0x now appears to be an hexadecimal number, I'm no longer interested in continuing my series on this topic.

Sunday, June 28, 2009

WebKit: Bridging the Great Mobile Divide

I hate Objective C and I hate Apple's Xcode and I think I hate Apple's AppStore policies even more but the people do love their iPhones. Furthermore, if you've had to deal with development on many different smartphones on the same project, you know how painful this is. You code one version for the iPhone, then you port it to BlackBerry, then Windows Mobile, then Nokia N60, and if you're lucky, Android.

So, if you want to develop for iPhone you either have two choices: develop a web application or go native. Unfortunately, even if Mobile Safari is a fantastic browser, it doesn't yet implement all of the HTML 5 features. Assuming iPhoneOS 3 here, Safari supports the following:
  • The new video/audio tags. Albeit using only mp4 containers, h264 video and AAC audio. Furthermore, it does so by opening the video player application, in effect exiting the browser
  • It supports app cache. With the proper manifest, your application's assets will be stored locally. Opening the door for the application to run without connectivity
  • Support for HTML 5 databases. First is the assets, then it's the data. With both, your application can potentially run without connectivity.
  • Canvas. It supports canvas for more complex rendering.
  • Workers. Thread workers so that long running JavaScript code doesn't need to lock up the user interface
The only big in-draft HTML 5 (the file desktop API isn't in the draft yet) feature it doesn't support is geolocation. That being said, what if you need access to the accelerometers? Or the iTunes music library? Sorry, you're back to native and coding in Xcode and Objective-C.

Well, a funny thing happened to me the other day while watching this Google I/O session. I was expecting an introduction to the C/C++ kit and that's it. Turns out that was pretty boring. What wasn't boring however was the idea of using a web view and injecting custom JavaScript element in the document's DOM.

That concept introduces a new, more sane, alternative to coding in Objective C and Xcode and supporting many different platforms. Basically, you write an HTML 5 application using GWT. So you're writing with Java 5 syntax in the IDE (and debugger) of your choice for the majority of the application. You only write a small shell containing web views in your iPhone application.

So how about porting to other phones? Well consider the following phones that have WebKit in some variant or another:
  1. BlackBerry. Ships with a WebKit based browser since 4.6. That means the Storm/Bold, won't cover most of your users but eventually it will. Please note however that JavaScript is disabled on BlackBerry phones by default.
  2. Android. Ships with a WebKit based browser.
  3. Nokia S60. WebKit is now the default browser engine in S60 phones. Write a shell using C++ and Qt.
  4. Windows Mobile. WebKit is nowhere to be found. Again, not a problem since the Qt port to this platform has WebKit, write a shell using C++ and Qt.
  5. Pre. Ships with WebKit. Not only, but Palm only wants web based, HTML 5, apps to be written for its phone using WebKit. They even introduce some new DOM elements to access other services in their implementation.
  6. Finally, iPhone, write a shell in Objective C embedding WebKit.
The strategy is the same, use a native application so that you're not jailed by the browser but keep most of your code portable by embedding a web view using WebKit. Furthermore, thanks to GWT's deep JavaScript integration and GIN, you could write interfaces that get bound to the proper native/platform specific implementations automatically at runtime.

So now we actually have a platform for sharing a lot of code between the major smartphones out there. And what if you need those iPhone like visual effects? Well, it's absolutely possible with JavaScript. Consider gwt-fx, it has all the standard iPhone visual effects and more and when you're stepping out of what the standard iPhone controls do out of the box, is actually easier and less verbose to use in terms of code. For your enjoyment, gwt-fx has an effect playground available here.

Ah, NSObject reference counts, how will I miss thou? Well, not at all I must say.

Monday, June 22, 2009

Windows 7 Exploit Source Released

In an effort to get Microsoft to change their mind on UAC's default behavior in Windows 7, the source code for the UAC injection technique has been released. Considering this system has been covered here many times, I won't rehash anything here but you can find the source and information about the tool here.

Saturday, June 6, 2009

Microsoft Research For the Win?

Experience in reviewing Microsoft counts. Paul Thurrott is one of the most credible Microsoft technology reporter in the business.

In his latest podcast with Leo Laporte, episode #110, he rants very humorously at Project Natal. All I can say is ouch and thanks for putting this technology in perspective.

Tuesday, June 2, 2009

Microsoft Research For the Win

Just a light post to say how impressed I am with "Project Natal".

EnGadget has the details here.

Saturday, May 30, 2009

Move Construtors

For simplicity sake, we assume the name space 'std' is used here. Given the following:

vector<vector<string> > v;

We now have an expandable array of string arrays. Now I don't want to poke inside the possible different implementations the string class could have here. We assume the worst, that string is implemented using a single C array of characters and leave it at that.

However, also for the sake of this discussion, we assume that the vector template class capacity is always at least bigger than its size. So we can assert the following:

assert(v.size() <= v.capacity());

Where the capacity holds numerous empty slots at the end of the array for future allocations. So, if we have the following:

vector<string> v1, v2;

v.reserve(32);
v.push_back(v1);
v.push_back(v2);

We are can safely state that the insertion of v2 inside of v did not cause v1 to be re-created, copied and destroyed.

The problem comes from the following:

v.push_back(v1);
// v.push_back(v2..v(n - 1) where n == capacity == size
assert(v.capacity() == v.size());

The next time we insert into this vector, the capacity will be expanded, most implementations will double the size to avoid any extra copying that might be caused by being over-conservative.

This means we have the following algorithm going on inside v the next time we insert:

1. Save the capacity to a local variable and double it, we call this n
2. Allocate a new array of vector of size n.
3. Copy construct a copy of every element found in the original array
4. Destroy every element found in the original array
5. Delete the original array
6. Capacity is now equal to n
7. The new array replaces the old one
8. The new element is inserted at the position "size" and size is incremented by 1

The problem here are steps #3, #4 and #5. Vector simply can't move the pointers found in the array because ISO C++98 dictates that vector (and all standard C++ containers) be safe for storing objects, i.e., data types that have a virtual pointer table. In fact, in this case, vector can't even hold pointers to its elements because the standard also dictates that vector, like all other standard based containers, is value based for maximum flexibility.

So the problem is that all these vectors found inside 'v' are copied element by element, string by string and then destroyed.

Now, considering that capacity is usually expanded by 2 every time this limit is reached, this shouldn't be too bad right? Well, unfortunately no, without any compiler optimizations, returning the value from a function or method is sufficient to trigger a copy. So simply put:

v = do_something(v);

where do_something is defined to be:

vector<vector<string> > do_something(vector<vector<string> > v) {
return v;
}

is sufficient to trigger 2 copies, one by the value argument, and one by returning the value. Suffice it to say this is not very efficient especially considering how widespread the standard containers have now become in the world of C++.

The new ISO C++0x standard introduces a new feature to address this shortcoming, a move constructor. In the previous example, our vector template class defined a copy constructor using the following syntax in order to perform copies:

template<typename T>
vector<T> (const vector<T>& other) {
// perform the copy
}

However, we can now complement this constructor with a move constructor following this new syntax:

template<typename T>
vector<T> (vector<T>&& other) {
// OK, let's steal some memory, so for example
m_rawArray = other.m_rawArray;
other.m_rawArray = 0;
m_size = other.m_size;
other.m_size = 0;
}

Here, the move constructor only gets called when a copy is performed and the previous copy is also known to be at the end of its life. So that means it's OK for us to "steal" memory that belonged to our forebearer.

This is a simple feature that will address the main performance concern of the standard C++ based containers. Even if end users are unfamiliar with this new feature, they will still reap the benefits of it the next time they compile since most standard C++ library implementations (and the Boost libraries) will be enhanced to use move constructors. By using a move constructor here, vector can now avoid all the expensive copy operations it needed to perform in the previously showcased algorithm.

Monday, May 25, 2009

C++0x Feature Focus

Starting a new series of blog posts related to language changes found in the upcoming C++0X standard. These will focus on core language changes and not on any libraries. You will need gcc from the head branch or the Visual Studio 2010 beta 1 to compile any of the upcoming samples. Even then, some of the samples will not build even with these compilers since some features have yet to be implemented.

I want to highlight the following new features:
  • Move Contructors
  • Perfect Forwarding
  • Variadic Templates
  • Closures
Closures, a.k.a lambdas, are already implemented today in the libraries but the ISO committee decided it was time to put this in the core to have a unique syntax across all systems.

Friday, April 24, 2009

A Peek at the Compatibility Future

Users enjoy the broad compatibility Windows has. Even 20 year old 16-bit DOS based applications still run on Windows 7. However, the flip side of this is that it has made Windows very hard to maintain for Microsoft and very hard for developers to use.

That is, targeting multiple version of Windows is very difficult. Bugs present in one version and not another meant different run-time behavior for the application. Add to that the complexity that earlier versions of Windows were also fundamentally different depending on which version of Internet Explorer was installed, even the kernel was touched, meant you add a significant number of operating systems to test and debug. Ultimately, the best solution for portable applications for Windows was to use some kind of layer between your application and the operating system, most notably Java. Sun, or whoever, does the heavy lifting and you target one platform.

If you're familiar with Microsoft however, there's one very interesting technology that you just know is the future. The future of compatibility that is. That technology is called MED-V. Yes, it's a stupid name but the technology is far from stupid. MED-V is based on Virtual PC but instead of running a different operating system with a different desktop it runs your applications side by side.

What is stupid however, other than the name of course, is that this technology is currently only for enterprise customers who have volume licensing. This is a very sad state of affair considering that this would allow for a much more streamlined user experience for people who only want the latest and greatest and much greater compatibility for those who do not. What I mean by that is that ever since Vista shipped, Windows is a considerably more modular operating system with the capability to turn various features on or off. The potential still isn't realized today but Windows 7 goes a step further by allowing various applications to be "turned off".

Now, with MED-V, you could go that extra step, on your brand spanking new machine, you could, say, no 32-bit application, so that entire part of the OS goes missing. No resources no nothing. It also means that even if you're running these compatibility modules, it doesn't mean they're running all the time or at the same time. Furthermore, since these are full virtual environments, they're actually a copy of the target OS that the legacy application was designed for. What's also very interesting is even if these are virtual environment, they're running on the bare metal which means you still have the full power of your actual machine unlike actually using a virtual box.

Well everyone who knows about MED-V knows it's the future, well, at least the technology if not the name. This technology would allow for Windows to grow at a faster pace and cut off deprecated APIs and SDKs without breaking compatibility with older applications.

Which brings me to today, Microsoft has unveiled Windows XP Mode (XPM) for Windows 7. This is the same technology that powers MED-V but what it means is that every copy of Windows 7 will ship with a fully licensed copy of Windows XP SP3 for compatibility mode. So if you're a large enterprise with poorly written line of business (LOB) applications, all of them really, you'll be able to depend on this Windows 7 module to run all those applications. If you just happen to not care about this and just consider it bloat, that's fine too, just turn it off. In fact, if you don't really care about any of this, XPM doesn't actually run unless you use something that needs it.

Reviews of pre-release versions of Windows 7 praise how lean it is compared to its predecessor. That you agree with that or not doesn't really matter, thanks to XPM and eventually the full MED-V, you'll be able to turn off huge, gigantic portions of the operating system and developers and users alike will be able to enjoy new versions of Windows that are free from binary compatibility with older applications but nonetheless offer full compatibility which will mean a much leaner, more secure operating system for all.

Monday, April 13, 2009

Google Updater

Google has been very busy this past week making new release after new release. The release however of Google Updater as open source is definitely very interesting.

One of the most annoying aspect of the modern Windows XP/Vista desktop is keeping your applications up to date. That is, you have an auto-update for Windows and other Microsoft software. You have one from Adobe for your Adobe applications, one from Mozilla for Firefox and the same software but yet another instance for Thunderbird. You have one for Vuze, you have one for any number of applications. Keeping all your software up to date today just requires too much work.

It was hoped that with Windows Vista, Windows Update would be opened to 3rd parties beyond the driver space but that hasn't happened.

The availability of a high quality open source application updater for Windows doesn't guanrantee it will become the de-facto standard nor that all companies that end up using it will share the same instance/server on the back end but it may pressure Microsoft to finally open up Windows Update to 3rd parties.

Anyway, I see this as good news and just hope that we will finally see some progress in this area.

Wednesday, April 8, 2009

Let there be Light! (or Java)

Google has just added Java support to their AppEngine framework. Obviously, the main selling point here for using this system besides free resources for up to 5 million page views a month is the database.

AppEngine uses the Big Table database which is a fully distributed, replicated database and you just get it for free.

Now, if Google would just support NetBeans out of the box already.

Monday, April 6, 2009

Secure IPC in OS X - Part 2

I wrote the first of what was intended to be a set of Secure IPC in OS X posts. Instead, Part 2, will be The End.

The idea is fundamentally flawed. The easiest way to see this is to consider that what I was trying to achieve was an escalation to root privileges for a particular operation without any user interaction. This is a security hole with a few easily exploitable vectors:

First, the dynamic validation of code signatures does not work as you might expect - it is only shorthand for validating the files on disk.

Second, I was assuming that the IPC system is secure, as pjulien points out here, any process can inject arbitrary code into any other process of the same privilege level. This means that your process can, without needing to modify any files on disk, be made to execute code that was not written by you.

Third, assuming dynamic validation worked like I wanted it to, then the first two problems would be solved. The third problem is that this option is exposed to a user via a GUI. The GUI of any application can be programatically controlled (for instance, by a unit testing tool).

I'm sure there are other attack vectors that I haven't thought of. The point is, when escalating privilege, you need to implement some kind of barrier to entry that asks a user for permission, otherwise it can circumvented programatically.

It turns out this type of mistake is not uncommon. Microsoft recently committed a similar error with their more friendly UAC system for Windows 7. This article explains how the UAC whitelist in a Windows 7 Beta can be trivially subverted by the second problem (code injection).

Wednesday, March 11, 2009

Reading and Writing to Another Process

I had a conversation with a friend the other day about this topic, he wasn't aware that this was indeed possible on OSX. As a side note, if your operating system supports debugging, and more specifically, attaching and detaching a debugger to an already running process then the operating system has to have some support for this. Just as a reference, here's a rundown of the functions you can use to read or write bytes to other processes on Linux, Windows and Mac OS X.

On Linux, you need to look no further than open(), read(), write() and close(). You just find the PID you are looking for under the special "/proc" file system, in there, change to the directory that corresponds to your PID. Inside, there is a file named "mem". If you have access, you can just open this file and read and write to it.

On Windows, given a PID, get a process handle using the OpenProcess() function. From there, you can reserve space in the other process by using VirtualAllocEx(). You can use WriteProcessMemory() and ReadProcessMemory() to write and read to this other process.

On Mac OS X, you first obtain a Mach task using task_for_pid(), once you have that, you only need to use vm_read() and vm_write() to read and write to the other process.

All of the above assume that the operating system is clamping your access to other processes based on your credentials otherwise you get something like this.

Tuesday, March 10, 2009

Secure IPC in OS X - Part 1

This is the first in a series of posts about a specific implementation of a secure IPC solution for OS X.

First, what do I mean by secure IPC? IPC is the communication between two or more processes on a single computer. There are many methods for doing IPC, some are applicable to most operating systems and others can be quite specific. On OS X, the most obvious choices for IPC are:
In each of these cases, lets assume there is a server process running with elevated privileges that handles requests from one or more client processes. In a typical IPC setup, the server waits for instructions from the client to perform a task. None of the three methods have the ability to establish the sender's identity "built-in." Thus, a malicious process could communicate with the server and get it to do undesirable things. Identifying the sender will be covered later in the series.

You might be thinking there are well-documented industry standard ways to secure client-server communication, in use in important applications like online banking. These methods employ public-key cryptography to ensure the privacy of the communication as well as the identities of the communicating parties. This hinges upon a malicious user not having access to the private key, in other words it is security based on what-you-know. On a single computer, what-you-know security fails because the programmer has to assume that no secrets will remain secret for very long if they are installed on a user's machine (for instance, an embedded private key can be extracted from the binary). The alternative is security based on who-you-are.

For my purposes, the data being transferred between the processes did not need to be encrypted, only the identity of the communicating parties needed to be confirmed. Both OS X and Windows have the ability to establish the identity of an application using code-signing. Thus, the goal of secure IPC, for my purposes, can be thought of as, retrieving and validating the code-signature of the processes participating in IPC.

Part 2 will deal with determining the identity of these processes at run time.

Using code-signing in OS X - 10.6 update

I wrote a few days ago about the private code-signing API in OS X, and expressed some doubt as to whether applications using it will get burned when 10.6 is released.

There is some good news. I have tried the latest developer seed build, 10A286, and the codesign tool, and my code, works!

Thursday, March 5, 2009

theothermike.printf("hello\n") && lessons(XmlSerializer)

This is my first article here, so I figure I'd do a little introduction first. I'm a professional software developer/geek, with expertise mainly in Perl and *nix environments. I've been toying around with Java and C# in the last couple of years, and more recently, I've started using C#/.NET at my day-job in a specific project.... which brings me to my first post.

While working on making a SOAP request and using the XML serialization facilities in .NET, I spent the better part of a day tracking down why all of a sudden my code had started throwing runtime exceptions when changing a class member from being an ArrayList to a generic List<SomeClass>

Thinking at first that the serialization services didn't actually work with generic collections, I had found several examples that proved otherwise..

Now keep in mind that I'm fairly new to Visual Studio (this project is in 2005), and IDEs in general as I had mainly been using Emacs in a terminal window before. So, when I was presented with the exception dialog, I was extremely annoyed at how many 'levels' I had to dig into the dialog to actually get to the exception information I needed. For some reason, it doesn't word wrap either, and even stretching the dialog across my 2 monitor setup doesn't reveal everything in the exception.

Almost ready to give up using generic collections, I started modifying the code to use ArrayList and arrays[], but cleaner than it had been before.

Now for some reason, the GUI was STILL throwing an exception... but at least it 'bubbled' up to a different exception dialog that would actually let me see the text of the entire exception*.

Way down in the stack, I saw the error message causing my problems wasn't related to the fact that I was using generic collections, but rather due to the fact that the class used in the generic collection (ie. SomeClass) didn't have a default constructor anymore since I had created another constructor to take parameters for convenience. This does make perfect sense after I realized my mistake, but it didn't even occur to me as I'm still fairly new to C# (I had assumed the default constructor would still be there even though I didn't define it)

Adding a default constructor of course took care of the problem, and I reverted back to using List instead of ArrayList.

Applying some tips from pjulien, I also made the class 'sealed' and exposed the List using IList instead ..... much nicer:)

... live and learn

* of course you can see the entire exception in the 'output' window of VS, but this I learned after the fact

Using code-signing in OS X

OS X 10.5 includes code-signing support. Unlike Windows, code-signing in 10.5 doesn't really affect the user experience, but that will almost certainly change in future versions. The only code-signing aware features in OS X are the firewall and parental control systems. Both of these systems use the code-signature to identify an application independently of its location on disk and of its version, therefore, firewall preferences can be saved accross upgrades.

OS X 10.5 allows you to sign your applications and verify their on disk and runtime signatures. Here's the catch; Apple only support doing this via the codesign utility.

But what if I want to make use of code-signing in my application, for instance, checking the validity of a process I'm about to do IPC with? If you are like me, you don't like the idea of executing codesign directly and picking up the return code, you'd prefer an API. Well one exists and its really nice. Here is the code to verify the runtime signature of a process:

SecCodeRef code = 0;
SecCodeCreateWithPID(pid, kSecCSDefaultFlags, &code);
SecCodeCheckValidity(code, kSecCSDefaultFlags, 0);

Obviously, you'll have to check return codes and such, but 3 lines for code-signature validation is pretty nice. But all it checks is that the code hasn't been tampered with, no identity checks here. Adding your own extra requirements (like an identity check) is also easy (see man csreq for details on requirement strings):

SecRequirementRef requirement = 0;
SecCodeRef code = 0;
SecRequirementCreateWithString("certificate root H=\"abcdef12345\"", kSecCSDefaultFlags, &requirement);
SecCodeCreateWithPID(pid, kSecCSDefaultFlags, &code);
SecCodeCheckValidity(code, kSecCSDefaultFlags, requirement);


For brevity, I lied about being able to pass a char* to create the requirement (it should be a CFString) and I didn't check my return codes. Replace "abcdef12345" with your public key's SHA1 and you can verify that the code is signed by you. Pretty nice. Much better than the 100 or so lines of CryptUI code I had to write to do the same thing. As always though, there is a downside, the API is private, and codesign doesn't work on the preview builds of Snow Leopard, which makes me think they may have changed the something. Bummer.

Still, if you're interested download libsecurity_codesigning from the Darwin source code page and enjoy.

New Contributor

I'm going to starting contributing to this blog. I will mostly post about things I find interesting as I learn my way around OS X and Objective-C, from the perspective of a person coming from Windows/Linux and C++.

By way of introduction, a bug report. NSFileManager reads past the end of allocated memory. The details can be found at rdar://6636446, Apple have fixed the issue for some future release (10.5.7 or 10.6?).

I came across this by running my application with Guard Malloc. For the Linux developer, Guard Malloc is basically DUMA for Mac. Like Linux, it is easily used via the Mac (pseudo-)equivalent of LD_PRELOAD, DYLD_INSERT_LIBRARIES, or if you can stand using XCode long enough, it can be enabled at the bottom of the "Build" menu.

Wednesday, March 4, 2009

Windows 7 Shadow Copies

Since Windows Server 2003, Windows has had a feature called shadow copies. Basically, this allows you to get back to previous versions of existing or deleted files. The feature was later enhanced in Vista to make the "system restore" feature. Namely, the system takes snapshots of important system files your computer needs to boot by creating "restore points". At any point in the future you can revert to such a restore point. This can be because your system stop rebooting or you just noticed that game demo included SecuROM and is still installed even if the game demo no longer is.


The feature works has advertised however, there was no way in Vista to know how much disk space this feature was using or could use. Windows 7 includes new configuration options that control these settings.

shadow1shadow2

JavaFX

Now that the 1.0 release has been superseded by the 1.1 release, I decided it was time to look at JavaFX and see if there was anything there.

The first thing that hits you when looking at the samples is how smooth everything is on screen. That's good because Sun really needs to impress with these samples if it has any hope of getting developers to use them.

The second thing that hits you is the claim about there is no JavaFX mobile, there is only JavaFX. Code you write in JavaFX will work over "all the screens of your life". It's certainly possible to write a JavaFX application that would work over a phone, BD player, TV, browser and your desktop. Unfortunately, you wouldn't want too, the effort required to get that going wouldn't be worth it. The problem doesn't arise when you look at what's there but what isn't. Most basic controls like a text box or a list are imported from Swing verbatim or are simply missing. In other words, you have to use Java/Swing to add the missing functionality in your own application.

Now, if you're planning on deploying to all of the above platforms. All these Java stacks have swing with the exception of Java ME/CLDC. The CLDC environment happens to be the one that sits on those 2 billion phones that Sun keeps talking about and that's an issue. It's possible but highly unlikely that your application will not require any user interaction beyond flashy effects. The real problem with the CLDC however, is that it doesn't have AWT either. So beyond writing a complete replacement for these controls in CLDC, which is what the LWUIT project is doing, there isn't much you can do here. Is it possible that Sun will finally start to push Java ME/CDC for smart phones? I don't know, time will tell. However, the BlackBerry is still the most popular business oriented smart phone out there and it's based on CDLC. I say "based", because that virtual machine has some serious issues that I feel invalidates it as a Java virtual machine in the first place. Meaning getting JavaFX to run on a BlackBerry is still a big question mark.

The other big news is that Sun has reported they now have done 100 million installs of JavaFX on the desktop. I was really impressed by this at first because I thought this was people going to javafx.com, downloading the bits and installing them. What else could it be right? There wasn't a new Java 6 update so they couldn't have been piggy backing on the regular Java runtime from Sun to deploy JavaFX right? Wrong. Java 6 update 12 is still the latest release, but if you go to java.com right now and inspect the current installer, you can see that the installer binary has been modified to include JavaFX. 100 million desktop installs is still very impressive, Silverlight just recently passed this milestone and it has been out for far longer. However, let's face it, JavaFX is getting deployed because of Java Update, not because end users are excited about it and getting the bits themselves.

JavaFX on the desktop is still interesting. Thanks to the massive work that was done in Java 6 update 10, it can leverage the completely overhauled applets and web start systems in that release. Which means you can deploy your FX code as applets or web start applications or a combination of both. For example, most of the samples on javafx.com run in the browser, however installing on the desktop just requires the user to press "Alt" and drag the applet outside the browser to have the application live on your desktop. What's also good about this process is that the install is per user, so there are no UAC escalation prompts so anyone can install these applications that they be an administrator or not.

More good stuff:
  • Sun decided that it wasn't going to re-invent a completely new set of tools for content designers. Instead, it wrote plugins for Adobe's creative suite.
  • Media codecs. Finally, video comes to the Java platform. JMF (Java Media Framework) is dead and buried, it was last updated in 2003. JavaFX features On2 codecs and plugs into DirectShow on Windows and QuickTime on OSX.
  • JavaFX isn't a virtual machine, it runs on top of an existing JVM. That's good news indeed because in conjunction with Java Web Start, if your application is started by someone that still doesn't have JavaFX, just Java, the application will still load just fine. The JavaFX jar is just another jar which means web start will just grab it just like any other jar that is part of your application. Furthermore, if your users don't have Java, you can still use Sun's JavaScript script for Java deployment to get it to end users.
  • Updatable runtime even for phones. This is a departure from how Sun has been doing business on phones. Having the runtime updatable over the air is a really a good thing. Even if you bought a first generation iPhone, you haven't been left out in the cold, your system has been upgradable to every release that has been put out there. Sure, it's not over the air, you need a computer to install it but this is still a good thing.
Now some bad stuff:
  • Applets still take too long to start. Java 6 update 10 really helps but it still isn't good enough. It needs to be instantaneous if Sun has any hopes of replacing Flash with this technology. When you get to a web page, your banner ads have to be showing immediately, they can't be showing the Java spinning logo.
  • The Java system tray icon and bubble advertisement. Are you kidding me here? Really, really annoying.
  • No Android support outside the traditional model. That means Android handset developers need to license it from Sun and it doesn't come over the air. It follows the traditional model of what ships with the phone, dies with the phone. Of course, Android unfortunately hasn't exactly caught on fire, might be a non-issue here.
  • Not all JavaFX versions have been released. Namely Linux.
  • No visual editor. I've said many times over now that NetBeans' excellent matisse editor needs to move to the next level, beyond just Swing/AWT support. Please note that visual editors are available from 3rd parties.
All in all, I hope Sun keeps working really hard on this. The 1.1 release came mere months after the 1.0 release and I think it needs to keep doing that. Release very often so that it can patch holes and missing functionality bit by bit, forget the huge releases here, if it does that, it will be in a position where JavaFX 1.1 is competing against Silverlight 3. JavaFX's scope, i.e., the total number of classes, already doesn't compare favorably to Flex or Silverlight, let it gain ground bit by bit instead of waiting till the platform is abandoned before making another release.

Next up, Silverlight...

Windows 7 DST Heads Up isn't New

Turns out this feature is in Vista also.

Tuesday, February 3, 2009

Windows 7 Editions

One thing that Linux and MacOS X users enjoy over their Vista counterparts is that they don't have to worry about choosing an edition of their operating system or upgrading to the same operating system but to a different edition, e.g., from Home Premium to Ultimate works but Home Premium to Business doesn't.

The Windows SuperSite today has an article about the editions that are being removed. Also, each edition is a super-set of the previous one, so upgrading is much simpler since there aren't any distinct branches anymore in the editions.

That being said, I still think they are still too many versions. There should be only 3, Starter, Home and Professional with the starter edition being targeted at netbooks only.

Sunday, January 25, 2009

Most Recently Used Files From the Start Menu

Instead of a global menu item that shows a pot pourri of recent documents for all applications, the Windows 7 menu is smarter on how it shows you recent documents.


Recent documents are displayed and bound next to any compliant application. Pictures are worth a thousand words here:

















Saturday, January 24, 2009

Windows 7 Hidden Features

Starting a small series of posts about new features in Windows 7 that don't have a lot of flash, hence are unlikely to be covered in the mainstream, but are still neat and interesting.

Finally, Windows can burn ISO images directly from its shell. It's about time Microsoft did this considering the APIs have been there since Windows XP. I'm excited about this since most vendors usually bundle some crapware in order to provide this feature to their customers. Hopefully, this will mean a cleaner hard drive on purchase.

Now, if Microsoft would just get around to finally supporting virtual desktops. The APIs have been in there since at least Windows NT 4.

Windows 7 has the ability to mount Virtual PC VDC disk images. You can see the mounted disks like any other drive in the "Computer" view. This would have come very handy a while ago, I have a collection of virtual machines from Windows 95 up to Vista. With Windows NT 4 and earlier, it was really hard to download updates and/or service packs since:

a) Out of the box, they either don't come with any browser or come with Internet Explorer version 1. If you try to use Internet Explorer version 1 on mainly any site, including Mozilla, it doesn't work, the browser errors out.
b) You can't use a shared folder because you can't install the guest additions until you've updated to the latest service packs/updates but you have no means to get them on there.

This feature would have made it easier to just put the latest updates on these deprecated operating systems and get to the point where I could just install the guest additions. Ultimately, I had to put all the updates on an FTP site and use the text based ftp command that comes with NT versions 4 and earlier. I was still looking for a solution for Windows 95 and this is what finally made it possible to update. My 95 disks are original Windows 95 floppy disks, so I didn't have the FTP command on this install either.

Windows 7 Codec Support

In addition to the codecs supported by previous versions, Windows 7 supports an impressive list of new codecs and media containers out of the box.

New containers:
  1. MP4
  2. MOV
  3. 3GP
  4. AVCHD
  5. ADTS
  6. M4A
  7. WTV (new Windows Media based format for PVR based recordings)

New codecs:

  1. H.264
  2. MPEG4-SP
  3. DivX
  4. Xvid
  5. MJPEG
  6. DV
  7. AAC-LC (regular AAC)
  8. LPCM
  9. AAC-HE

You can get all these codecs and more with the K-Lite Codec Pack, however, this is the 32-bit version. The 64-bit version of K-Lite doesn't offer many codecs at all.

Now, if Microsoft can just add support for the Matroska file format, we'll be all set.

Monday, January 19, 2009

The Nokia N96

Just got my brand new toy. The Nokia N96, I honestly don't see the mass market appeal for such a device but I don't think I've felt this young in quite some time. This device is a whole lot of fun.

When I say fun, I don't mean this phone has a touch screen or a full keyboard, more fun like it supports live TV and has a PVR, a pretty damn good web browser, tons of codecs for all your media, good podcasting software, etc, etc. An unlocked version of this thing will set you back $800 USD. I have no idea what the target market for this device could possible be but I like it.

That being said, the N97, if not for the price tag, would be a much more serious product. Touch screen, full screen keyboard... and yes TV and PVR among other things.

Anyway, the N96 is a very good product, everything on this phone just works. The Java support is top notch with support for SVG based interfaces even which makes for some really interesting user interfaces. Again, however, budget conscious users might want to stay away here.

A video review is available here.

Microsoft SkyBox

The net is abuzz today about the rumored new user interface for Windows Mobile based devices. You can see some more information here.

However, that's not the major pain point when developing for Windows Mobile. Yes, the user interface is the major pain point for users when dealing with Windows Mobile but not for developers.

The real issue for developers is deployment. That's right, when developing for Windows Mobile, you have basically 3 options:

  1. Java ME
  2. .Net Compact
  3. Native
Native is a pain due to the many different CPUs you'll encounter when dealing with Windows Mobile devices. Furthermore, many third party libraries just haven't been certified/tested on this platform. Here, I'm thinking mostly about Boost. Additionally, you'll either have the choice of using MFC or using the C Windows API. Just fun all around. Deploying a cabinet file for these devices if you can straigthen out all these issues is straight forward but getting there is a long road.

Java ME, when present, is usually the CLDC variant. Meaning it's a pain, it's light, missing many important classes, has pre-Java 5 syntax which means you can't use the majority of 3rd party libraries on it, open source or otherwise, and well, usually, not even loaded on the phone to begin with. Furthermore, if you don't have it on your phone, there isn't a convenient way to get it or upgrade it. What ships with the phone is what dies with the phone. Again, deploying a JAD file isn't terribly difficult, that is, if the phone has a Java VM to begin with and that Java VM is of decent quality. As it turns out, most Java VMs that ship on Windows Mobile aren't worth the bytes they fill up.

The last choice is .Net Compact. This one isn't too bad except for the fact that most phones don't ship a recent version of .Net Compact. Usually, you'll find .Net Compact 2 with no service packs, which is a total mess, or .Net Compact 1 ranging from service pack 3 to no service pack at all. The minimum that is actually usable is .Net Compact 2 SP2. So the problem you have now is upgrading .Net compact. The only official supported way of doing this is to download an MSI package to Vista/XP and the next time your phone connects to your PC, your desktop will upgrade your phone. Not very useful if you want to deliver applications on the show room floor.

So keep going Microsoft, keep going, and maybe someday you'll look back on this and say, what were we thinking?

Saturday, January 17, 2009

The Last Cylon

Seems I was wrong, Ellen is the final Cylon. The actress tells the LA Times she's been sitting on the secret for over two years.

Friday, January 16, 2009

Battlestar Galactica

The season premiere just ended and I don't want to spoil anything for anyone. However, talk about living up to the hype. The show has been on hiatus for a year now courtesy of the writer's strike.

I always liked sci-fi but not sci-fi shows, e.g., Star Trek and Star Wars to name a few, just placeholders, something to fill the void but not particularly enjoyable. This show, however, goes beyond its genre.

The premiere was beyond anything I could have hoped for. I believe this show will live on as a pillar of American television. A milestone, a new standard of what quality should be and what it needs to be to keep audiences engaged.

Now, spoiler/theory alert. Do not keep reading if you don't want to know what I think.

There is no way in hell that Helen is the last Cylon. However, that doesn't mean she isn't a Cylon of course. We now know, thanks to Saul Tigh, that a Cylon can grow older. I believe Helen is an aged #6 model.

Who knows what Starbuck is at this point, or Dee, or anyone for that matter. It just seems to point to that everyone is a Cylon or at least a descendant of one.

Sunday, January 11, 2009

Windows 7 Beta

Still in the early stages of use but I have to say, an apology is needed here. I've been laughing at all the glorious reviews Windows 7 has been getting. Considering Windows 7 is just some spit and polish of Windows Vista, obviously if these people liked Windows 7, then they must have liked Windows Vista.

Well, having used 7 for a couple of days now, I have to admit, that this is some serious spit they used here.

What can I say other than "wow". It just shows that in this release, having nailed all the architectural underpinnings in Vista, Microsoft was free to concentrate on the user interface, be it the shell or the bundled applications. All I can say is that this is one serious calculator program.

So far, I only have 2 bad things to say about the system:
  1. The return of the "My" prefix, so "my music", etc. Shoot the person in the head who thought bringing this back was a good idea already.
  2. UAC is toned down. I always felt UAC was a good thing. Vista got a lot of heat for pointing out bad programs to users. Seriously, how would Linux users have reacted to a program that needed constant write access to /user/bin? The levels are good since it should stop some people from disabling it completely. However, I think the highest level should be maintained as the default. I think if Windows Explorer wouldn't show you a dialog that an operation is going to need escalation, hence 2 escalation dialog warnings, and would just put the escalation shield icon directly in the menu item, this would solve the problem most users had with UAC to begin with.
Anyway, all this to say, I apologize for laughing at all these glorious reviews of Windows 7. Windows 7 is really quite something.

However, not that I want to finish on something negative, I can't help myself to take a shot at Windows Mobile. How is it possible that the same company that is producing something so good like Windows 7 is also producing, at the same time, something that is so mediocre, namely Windows Mobile?