Thoughts about evolutionary algorithms

WARNING: This post is likely to be rather long, full of musings with very little concrete data to back them up. I hope it will interest you, but I don't promise anything...

Today I was playing with my little girl, Tamar (6 months), when some questions started to form in my head. When you think of the ultimate goal of each living species, it's rather simple: ensure the existence of the species for as long as possible. This is obtained by "mechanisms" such as reproduction, survival of the fittest, randomization mechanisms (e.g. mutations, weather changes, etc.) and numerous other techniques Mother Nature has to offer.

Scientists have been trying to simulate some of these mechanisms for various purposes for half a century, with variant degrees of success. Genetic Algorithms are the most widely known of them, and have been extensively researched in the past 20 years or so. The basic idea is that you have some objective function you want to optimize, and a large set of parameters to tune. The assumption is that the time it takes to compute every possible set of parameters is computationally prohibitive (unless you're willing to wait 10,000 years until you get the answer). The tuning of the parameters is done by applying operations known in genetics, such as reproduction or crossover (taking some parameters from one solution and some from another - creating a new solution in its own), mutation (changing some parameter(s) through some randomized mechanism in order to create a new solution), selection (each iteration keeps the best solution sets from previous iterations), and variants thereof. The main issue I would like focus on, is that there is always one objective function, and every solution (i.e. set of parameters) is judged by the score it acquires at that objective function.

Now let's get back to my little girl. As I said, as far as Nature is concerned, my daughter's ultimate purpose, like every human being, is to ensure the human race will persist for eternity. However, even Mother Nature knew that this is too big a goal to be of any concrete value. Therefore, it has set an infinite amount of sub-goals in our life, each of which bringing us closer to the ultimate global goal of Human Kind (be it in a minuscule amount). These sub-goals are, for example, to reproduce, which requires (as sub-sub-goal) to meet and get acquainted with males, which requires among other thing, to know how to walk (that's already a sub-sub-sub-goal), which has a precondition of being able to crawl (sub-sub-sub-sub-goal I think).

So Tamar's objective function, for our discussion, is to learn how to crawl. Not thinking about any of these things, I knew she needs to learn how to crawl (and frankly I wanted her to get tired, but that's even more besides the point), so I set her on a matress, with a blinking toy in front of her to attract her attention.

(I told you it would be long... don't lose patience, we're getting there...)

If we get back to Genetic Algorithms, the objective function of our problem, is to learn how to crawl (say we can give a score for the crawling quality). However, to reach this goal, I created a new objective function, apparently completely unrelated to the real goal - reaching this annoying blinking toy. She could end up reaching it in a million of ways, or never being able to reach it on her own (which is actually what happened, since she kept crawling backwards), but at the end of the day, her mind and body would have learned some new things that will, eventually, serve her to reach that real goal (crawling). In other words - I created a synthetic objective function, with only some vague relationship with the real objective function, with the express intent of improving the chances of, on the long run, optimizing the real objective function (crawling).

Now my question is - how could this approach be applied to Genetic Algorithms in general? How can we synthesize, throughout a GA's run, new objective functions, which will help the GA to reach the ultimate objective function?

Another rather different question that comes to mind - Tamar didn't reach the blinking toy, because she started crawling backward. I'm no specialist in these matter, but I allow myself to assume that at some level, she has learned something from this experience. Somewhere in her brain, something has marked that the movements she did were wrong (to reach her goal, i.e. get the toy), and that she should find another way. She will do the same movements once, twice, maybe thirty times - in the end she will grasp that the solution must be found elsewhere. As far as I know, Genetic Algorithms (and all their variants) are always concerned with "good" solutions (i.e. solutions that get good scores on the objective function). I don't think there is any algorithm that knows how to keep track of "bad" solutions, in order to avoid getting back to solutions close to that. A possible approach could be to run a "positive" GA, which strives to improve the objective function, but always keep track of the "bad" solutions. Whenever one the new generation's items is too similar to one of the bad solutions (or some representative of the bad guys), it is disposed of without even checking the objective function, and a new one is created instead. Of course, there is the risk of missing some good solutions that are accidentally similar to the bad ones, but in case that computing the objective function is an expensive task, it may be worth the tradeoff.

Trying Photobucket

I'm sick of not being able to upload pictures to my blog. So I've decided to try out Photobucket. We'll see how it goes...

Tamar doing a face

Tamar

Gal dancing at her 3rd birthday

Leasing

If you live in Israel, and especially if you're a high-tech professional, you are probably aware of the big noise around the expected change in taxes for leasing cars.

Globes just published a great read - an article that compares this issue to some other, more understandable yet completely virtual scenario. Even if you don't subscribe to the point of view that the taxes should indeed be increased, it's still fun to read.

Another interesting read on this subject is a site by the same author (and others), which explains why the taxes should indeed be increased.

If, like many others, you're trying to figure out the actual monetary value of using a leased car, check this calculator out.

Lastly, although I understand all the reasons why the taxes should indeed be increased, I'm still a big fan of leasing cars. I'm a complete car-dummy, and the feeling that whatever the problem, there is someone who will come within a couple of hours to fix it, is worth a lot more than money. So IMHO, if indeed there is a glitsh in the current tax calculation, then justice must be done and the taxes should be updated. I still, however, want to keep the option of using a leased car (as an employee), without being pointed at as if I were a danger to society.

SQL Server Execution Plan - Rantings...

Let me say it outloud: SQL Server Execution Plan SUCKS!

In essence, SSEP (I'm too lazy to keep writing the whole name) is supposed to help you analyze how the SQL Engine performed a query (more specifically what optimization decisions were taken) in order to either alter the query and/or the tables to improve performance. As such, SSEP provides you with a nice graphical interface, that shows every element of the execution along with it's relative cost. You can then see more properties for each element, which supposedly will help you better understand it.

SSEP in SQL2005 has gone through very few changes since SQL2000, and it's a pitty. With a small query, SSEP can help you out - you can easily find the most costly element, analyze it and kill the culprit. With long and complicated queries, however, it's a nightmare. The most prominent issue here is USER EXPERIENCE!!! Let me explain:

In most cases, the first thing you do when you run SSEP is to search for the most costly sub-query and in that you look for the most costly query element. When you have a long and complicated query (which sometimes is inevitable), you can find yourself "surfing" the execution plan for many long minutes, just searching for the elements of interests, never being actually certain you got them all. You just keep scrolling up/down left/right and back again, trying to get a grasp of what's going on. There is a zooming feature, but it just isn't enough! There is no easy way to navigate through the data and search for specific elements. I mean - come on, would it have been so difficult to add, for example, a list of elements with the main properties (say object name, cost and physical and logical operation), allowing you to sort it and jump from that list to the relevant element in the execution plan?

Another USER EXPERIENCE issue is that of the data that comes along with each element. Granted, SSEP is aimed at advanced users (usually DBA level), but would it hurt someone to provide some more explanative descriptions?

Lastly, if you really want to improve the USER EXPERIENCE, you should guide the user (without forcing her hand) to find the real problem as fast as possible. For instance, you could use colors for the elements, such that costly operations would stand out (color table scan in red, index seek in green). Note that these colors should also stand out in the list mentioned above. You could even propose some solutions ad-hoc. For instance, if you see a very costly Bookmark (used to link between indexes and actual data in the tables), you can propose to add an index that will include all the fields required, effectively removing the need for a bookmark (because the engine will read the data directly from the index). There are tons of things you could do to make the user's life easier, and help her finish her job faster, so why not?!?

Spam the spammer!!!

I just got a spam mail that made me REALLY angry.

It's from a company that calls itself Mailmedia. The message itself, however, was sent by a user at the lombardisoftware.com domain (I'm pretty sure that the genuine Lombardi company has nothing to do with this...).

What upset me the most with this mail, was the nerve - it was no less than a promotion for businesses to use their services to send spam mails. Yes, that's right - I got a spam mail that is a promotion for a spam company!

But that means that there should be some way to communicate with them, right? Unfortunately, there is no website, or email address - I would have been more than happy to spend a few hours setting up my own spamming machine to kill their website or their email address. I would, really! But they weren't that stupid - they only provided an Israeli mobile phone number:

050-5281978

(Note, I've searched this phone number up in Google and found only one entry in a Gays forum. This guy seems to be searching for a fatherly type... I have no idea whether this is genuine, or just another frustrated spamee who wants payback.)

Now, my request to every Israeli reading this post - please please please - spam this phone number with as many calls as you can. Let them FEEL how annoying it is to be spammed all day long. Let them LOOSE potential clients because the line will always be busy. Make them SUFFER as much as we do from their never ending spam mails.

Be smart: don't forget to HIDE the calling number when you call them (for Orange you must preceed the number with #31#, for HOT you must preceed with *43).

DISCLAIMER: Although I really want as many people to join the spammers spamming effort as possible, I take no responsibility whatsoever to any miscomfort, legal or other issues you may encounter as a result of doing as I just asked.

On the superfluousness of the Singleton pattern in .NET

We all know the Singleton pattern. It's intent is to "Ensure a class only has one instance, and provide a global point of access to it." (from GOF). There are many possible implementations of it, some are good, others are to be ashamed of. You can find a good review by Jon Skeet, or on Patterns and Practices where you can also find a successful implementation using double-locking mechanism. By the way, the implementation in "Design Patterns in C#" by Steven John Metsker was utterly dissapointing, to say the least.

But this is all assuming you indeed need a Singleton pattern. If you read the Intent, as defined by the GOF carefully, there is nothing in simply using a static class that does not meet this intent. At the time of writing the Design Patterns book, the GOF didn't have C# or .NET. Back then, they were mostly relying on C++ and Smalltalk. I'm not proficient in Smalltalk, but I know that with C++ there were problems with static and global variables. For instance, it wasn't clear what their order of instantiation would be, nor when the destructors would be called. So when you're working with good old C++, you just have no choice but to use the Singleton pattern. With .NET, however, things are different. Instantiation of static classes have a clear and deterministic behavior. With destruction, of course, it's different, but most cases where you need a Singleton you'll keep it alive as long as the program is running anyway.

In the Design Patterns book, there are some other "consequences" which could end up as reasons for using the Singleton pattern instead of simply a static class:

1. Permits refinement of operations and representation - or in other words - a Singleton class can be extended by means of inheritance. Most cases I have encountered, there is no inheritance going on with the Singleton, and if there is, then the actual Singleton is the last in the hierarchy. Even more so - many implementations use sealed classes either for performance optimizations or simply because the pattern wouldn't work with inherited classes.
2. Permits a variable number of instances - or in other words - one day, some 5 years ago, some freaky newby will invent a reason to use multiple instances of the Singleton and you should make it easier for the idiot to do the change. One word: YAGNI !

3. More flexible than class operations - same as the 2 previous items

So, why would you use the Singleton pattern in .NET instead of just a static class?

1. Inheritance - I said in most cases there is no need for it. There is always the exception... You can see one on Jon Skeet's blog.

2. Destruction - when you want to keep a reference counter of the number of consumers, and somehow destroy it when it's no more in use. Complicated, especially given the non-deterministic destruction of objects in .NET.

3. Other static functionality - if you need some static functionality that should not cause the actualy 'Singleton' to be created (any call to a static method will cause the ctor to be called). Note that this is also a problem with most of the pattern's implementations, because they rely on a static field/ctor.

4. Parameterized contruction - this seems to me like the most likely reason to use the Singleton pattern instead of a static class, but here again there is a problem because most of the cleanest implementations rely on the static constructor. The idea here is that sometimes you want the Singleton to be initialized with some parameters, so you can't rely on the paramterless, static ctor.

Now that that's settled, and given the name of my blog - why not using the Singleton pattern?

1. As a general principle, I like using patterns when they are needed, no more. So the main reason why not, IMHO, is that in most cases it's simply superfluous - you end up spending design/coding/unit testing/qa/debugging/etc time on something you don't need. Smart.

2. Some of the implementations are really clean and neat (see, for example, the 4th in Jon Skeet's list, which is the one I used to use before I understood I don't need it).

3. Inertia - Most of us are still in the C++ state of mind that there is no other way to really "Ensure a class only has one instance, and provide a global point of access to it." .

I don't know what this guy is on - but I want some...

This is really hallucinating. If this thing is for real, I want to know what this guy takes - and want some for myself...

Oren Zarif is one of these people who claim to have exceptional, supernatural and X-Files-like powers. Apparently, he likes to get it really hard. According to YNET, he has contacted Mike Tyson, proposing a fight. Zarif , weighting 68 kgs, as far from being a boxer talent as one can be, claims to be able to box with his eyes closed. He proposes a one-on-one, with his eyes completely covered. If Zarif loses, he would pay Tyson 5M$. If he wins, Tyson will have to donate the same amount to the US government for the victims of global terror.

For my English readers, the letter sent to Tyson, in passable English, can be found here. Source - YNET.

WTF ?!?!?

A Million Dollar Contest - For Super-Geeks

Netflix is a very large (largest?) DVD rental company. One of their most important assets is Cinematch, an in-house developed movie recommendation system. Cinematch's purpose is to predict, based on a user's previous rating of movies, the rating a user would give to other movies. The result, of course, is a recommendation of movies that are most likely to suite the user's taste.

Apparently, Cinematch works pretty well, but the guys at Netflix would like it to work even better. So they came up with the Netflix Prize: encourage developers and researchers to come up with an algorithm that improves the quality of rating prediction of Cinematch significantly enough (at least by 10%), by promising a prize of 1,000,000$. To make it even more interesting, and since the whole contest spans over at least 5 years, there is a yearly prize of 50,000$ which will be given to the best solution each year.

Being a Machine Learning freak, I find this contest GREAT! I also like the rules of the contest a lot. Basically, the winning algorithm will have to be made publicly available at the end of the contest.

This kind of initiatives are so great because they encourage the development of new concepts and algorithms, which even if they won't win the first prize, might very well be helpful for various other purposes. I think the best comparison is Fermat's Last Theorem. In his will, Paul Wolfskehl initiated a prize of then 100,000 marks to whomever would be able to prove or disprove Fermat's Last Theorem. This generated a huge interest in the subject, which resulted in an incredibly rich amount of new ideas and whole new areas of mathematics being discovered and researched to this day. I doubt that the Netflix Prize will have the same effect, but I do believe it will give Machine Learning a well-deserved boost.

As far as Netflix is concerned - they can only win from it. The news about this contest should inevitably increase their exposure. If the contest succeeds, and someone manages to provide significantly better results - it will be worth much more than 1M$ for them. If nobody manages to win the contest, then they can heartily claim to be using the best movies matching algorithm human brain could come up with do date. Either way, it's a win-win situation for them.

It's a very difficult task, but I think I'm going to give it a try, as far as time permits...

If you're interested, following are the Terms and Conditions in a Nutshell:

Contest begins October 2, 2006 and continues through at least October 2, 2011.

Contest is open to anyone, anywhere (except certain countries listed below).

You have to register to enter.

Once you register and agree to these Rules, you’ll have access to the Contest training data and qualifying test sets.

To qualify for the $1,000,000 Grand Prize, the accuracy of your submitted predictions on the qualifying set must be at least 10% better than the accuracy Cinematch can achieve on the same training data set at the start of the Contest.

To qualify for a year’s $50,000 Progress Prize the accuracy of any of your submitted predictions that year must be less than or equal to the accuracy value established by the judges the preceding year.

To win and take home either prize, your qualifying submissions must have the largest accuracy improvement verified by the Contest judges, you must share your method with (and non-exclusively license it to) Netflix, and you must describe to the world how you did it and why it works.

For more elaborated information, check out the Netflix Prize page.

Yahoo! Mail to become unlimited in size?

According to YNET, Yahoo! announced that as of May 2007 they will gradually update all Yahoo! Mail accounts to unlimited mailbox sizes. I couldn't find any additional source mentioning this, but I trust YNET didn't make this up.

This isn't surprising, of course. Ever since Google introduced GMail it was obvious that at some point they would announce that the mailbox size has become unlimited. Maybe the surprising thing is that the first company to come with such an announcement is Yahoo!, who's been lagging behind Google with most of the previous advances.

This is, of course, a very welcome announcement, hopefullly to be followed by all the other main webmail suppliers.

Cool!

P.S: I think that with this new development, it's time that the ISP's decide whether they want to continue supplying email services to their customers. If they do - then they should dramatically improve the level of their service (speed and mailbox size). Otherwise, they're really making fools of themselves.

Joke By Code

I haven't blogged much lately, I know, sorry. Between work, kids, buying a house (!!!!!) and a million other things, I just couldn't find a minute.

Anyway, today I was searching for some good books about SQL Server 2005 programming. I have a lot of experience with SQL 2000, but it's about time I get to know the little baby (who's way past the 'baby' stage and would better be called a teenager) a little more.

So I came accross Murach's SQL Server 2005 for Developers and like everyone, I started skimming through the 6 customer reviews. All reviews were really good, except the last one, which seems to have been written by particularly dissapointed customer. So I decided to have a look at the 3 comments to his review. Here is what I found (quote):

SELECT OneHalfBrain
FROM Name
WHERE LName = `Husain'

results: NULL

It made me laugh (the reviewer is called Munawer Husain, so I assume it was meant personally against him, and not in some stupid racist direction).

Multithreaded Computation Support in Matlab - FINALLY!!!

A couple of weeks ago, Mathworks released a new version of Matlab (R2007a). They have, at last, added support for multithreaded computations.
A LOT of entries to my blog come from searches like "matlab dual core", which surprisingly puts a post of mine very high in the results list. The thing is that especially now with the increased usage of dual and quad-core machines, running heavy Matlab computations with a single thread is just a waste of resources. Apparently, the guys at Mathworks were listening to the users, and added support for multi-threaded calculations. It seems to be managed under the hood somehow, and requires changes to the preferences. From the release notes:

"If you run MATLAB on a multiple-CPU system (multiprocessor or multicore), use a new preference to enable multithreaded computation. This can increase MATLAB performance for element-wise and BLAS library computations.
By default the preference is not set, so you must set it to enable multithreaded computation. With the preference enabled, MATLAB automatically specifies the recommended number of computational threads, although you can change that value. On AMD-based Linux platforms, MATLAB supports multithreaded computation, but requires an extra step to change the default BLAS."

Doesn't sound a very nice way to do it, and certainly lacks user control, but it's a start. I didn't upgrade to the new version yet, so I can't talk from experience. Also, the new release seems to include improvements to the Distributed Computing Toolbox, which sound also very interesting.

Google Image Search API updated (again...)

What can I say - the html format was changed again.
You can download the complete code from my article on CodeProject.
If you just want the update - I just had to change the Regex file, so you can simply replace its content with the following:

imagesRegex: (dyn\x2EImg\x28\x22[^\x22]*\x22,\x22[^\x22]*\x22,\x22(?<code>[^\x22]*)\x22,\x22(?<imgurl>[^\x22]*)\x22,\x22(?<width>[^\x22]*)\x22,\x22(?<height>[^\x22]*)\x22,[^\x29]*)
dataRegex: (?<width>[0-9,]*)\s+x\s+(?<height>[0-9,]*)\s+(pixels\s+){0,1}-\s+(?<size>[0-9,]*)(k)
totalResultsRegex: (?<upperLimit>upperLimit>(\s)*)(?<lastResult>[0-9,]*)([^=])*=(?<maxLimit>maxLimit>(\s)*)(?<totalResultsAvailable>[0-9,]*)

Is it the end of Internet Freedom? A possible workaround...

Politicians are trying to add more and more limits to the freedom and anonymity we enjoy over the Internet. A few months ago, they wanted to force talkbackers to identify themselves - death sentence to talkbacks. Now, they are seriously talking about forcing users to identify themselves when using adults websites.
It's a pitty really. Instead of searching for ways to improve the quality of the service they provide us, they try to castrate the Internet into something more manageable (for them). Also, the philosophy behind this law they are trying to pass, is so anti-democratic and anti-privacy that it makes me want to cry.

Anyway, there might be a way around this. Check out Tor . It's a technology that basically routes web requests (or any other TCP-based communication) through various nodes in an encrypted form, making it almost impossible to track down the original user.

A Personal Letter to Everyone I Know, Knew, Will Ever Know, Think I Know, Think S/He Knows Me - To EVERYONE!!!

 

Dear family, friends, ex-friends, future-friends, acquaintaces, whatever,

That's it, I've had it!

STOP SENDING ME CHAIN LETTERS !!!

When you think AOL/Microsoft/IBM/whatever is monitoring emails, and you forward them in hope for earning a few bucks - I understand you have as much common sense as my 3-year old (or less).

When you think you can save the life of some poor person with cancer who needs AB blood - it's obvious you haven't been around the Internet for the past 10 years.

When you think luck will come to you by spamming others' mailbox - I truely hope you are never going to meet any Hari-Krishna, because you're soo easy to fool.

When you think you make me feel loved and happy by forwarding some 10-pages long 15-year old load of BS - you're oh! so wrong!

So please please please, I don't want to offend you. If you have something to write to me - be my guest, I'd be glad to read it. If it's something you got from someone who got it from someone who... don't think - leave me out of it! I'd better loose one good joke than have yet another 5 spam (sorry - chain) mails in my inbox!!!

Yours sincerely,

Ilan

P.S: If you really can't help it, and still want to forward this junk (not to me!), I suggest you check out the following first:

http://breakthechain.org/
http://www.snopes.com/
http://info.org.il/irrelevant/  (Israeli)

Wrapper for svmpredict.exe

% predictions = libsvmpredict(test_set, model, varargin)
% options:
% -b probability_estimates: whether to predict probability estimates, 0 or
% 1 (default 0); for one-class SVM only 0 is supported
%
% NOTE1: This function actually executes LibSVM's svmpredict tool. For more
% info check out: http://www.csie.ntu.edu.tw/~cjlin/libsvm/
% NOTE2: Although LibSVM supports more types of labels, this wrapper is
% limited to integer labels only.
% NOTE3: This function assumes that svmpredict.exe is located in a directory
% that lies in your system's PATH.
function predictions = libsvmpredict(test_set, model, varargin)
if isempty(varargin)
options = struct([]);
else
options = varargin{1};
end;

% Dump the test set to a temporary file
testfile = tempname;
fid = fopen(testfile, 'wt');
dumpData = zeros(size(test_set,1), 2*size(test_set,2) + 1);
format = '%d %d:%f';
for i=1:size(test_set, 2)
dumpData(:, 2*i) = repmat(i, size(test_set,1), 1);
dumpData(:, 2*i+1) = test_set(:, i);
if i < size(test_set,2)
format = sprintf('%s %%d:%%f', format);
end;
end;
format = sprintf('%s\n', format);
fprintf(fid, format, dumpData');
fclose(fid);

% Dump the model to a temporary file
modelfile = tempname;
fid = fopen(modelfile, 'wt');
fprintf(fid, model');
fclose(fid);

% Run svmpredict with the given model and write the predictions to a temporary
% file
predictionsfile = tempname;
command = 'svmpredict';

if isfield(options, 'b')
command = sprintf('%s -b %d', options.b);
end;

dos(sprintf('%s %s %s %s', command, testfile, modelfile, predictionsfile), '-echo');

% Load the predictions temporary file and return its content
predictions = dlmread(predictionsfile);

return;

Wrapper for svmtrain.exe

% model = libsvmtrain(training_set, options)
%
% options:
% -s svm_type : set type of SVM (default 0)
% 0 -- C-SVC
% 1 -- nu-SVC
% 2 -- one-class SVM
% 3 -- epsilon-SVR
% 4 -- nu-SVR
% -t kernel_type : set type of kernel function (default 2)
% 0 -- linear: u'*v
% 1 -- polynomial: (gamma*u'*v + coef0)^degree
% 2 -- radial basis function: exp(-gamma*|u-v|^2)
% 3 -- sigmoid: tanh(gamma*u'*v + coef0)
% 4 -- precomputed kernel (kernel values in training_set_file)
% -d degree : set degree in kernel function (default 3)
% -g gamma : set gamma in kernel function (default 1/k)
% -r coef0 : set coef0 in kernel function (default 0)
% -c cost : set the parameter C of C-SVC, epsilon-SVR, and nu-SVR (default 1)
% -n nu : set the parameter nu of nu-SVC, one-class SVM, and nu-SVR (default 0.5)
% -p epsilon : set the epsilon in loss function of epsilon-SVR (default 0.1)
% -m cachesize : set cache memory size in MB (default 100)
% -e epsilon : set tolerance of termination criterion (default 0.001)
% -h shrinking: whether to use the shrinking heuristics, 0 or 1 (default 1)
% -b probability_estimates: whether to train a SVC or SVR model for probability estimates, 0 or 1 (default 0)
% -wi weight: set the parameter C of class i to weight*C, for C-SVC (default 1)
% -v n: n-fold cross validation mode
%
% NOTE1: This function actually executes LibSVM's svmtrain tool. For more
% info check out: http://www.csie.ntu.edu.tw/~cjlin/libsvm/
% NOTE2: Although LibSVM supports more types of labels, this wrapper is
% limited to integer labels only.
% NOTE3: This function assumes that svmtrain.exe is located in a directory
% that lies in your system's PATH.
function model = libsvmtrain(training_set, labels, varargin)
if size(training_set,1) ~= size(labels,1)
error('The training_set and labels should have the same number of rows.');
end;
if isempty(varargin)
options = struct([]);
else
options = varargin{1};
end;

% Dump the training set to a temporary file
datafile = tempname;
fid = fopen(datafile, 'wt');
dumpData = zeros(size(training_set,1), 2*size(training_set,2) + 1);
dumpData(:, 1) = labels(:, 1);
format = '%d';
for i=1:size(training_set, 2)
dumpData(:, 2*i) = repmat(i, size(training_set,1), 1);
dumpData(:, 2*i + 1) = training_set(:, i);
format = sprintf('%s %%d:%%f', format);
end;
format = sprintf('%s\n', format);
fprintf(fid, format, dumpData');
fclose(fid);

% Run svmtrain with all given options and write the model to a temporary
% file
modelfile = tempname;
command = 'svmtrain';

if isfield(options, 's')
command = sprintf('%s -s %d', options.s);
end;

if isfield(options, 't')
command = sprintf('%s -t %d', options.t);
end;

if isfield(options, 'd')
command = sprintf('%s -d %f', options.d);
end;

if isfield(options, 'g')
command = sprintf('%s -g %f', options.g);
end;

if isfield(options, 'r')
command = sprintf('%s -r %f', options.r);
end;

if isfield(options, 'c')
command = sprintf('%s -c %f', options.c);
end;

if isfield(options, 'n')
command = sprintf('%s -n %f', options.s);
end;

if isfield(options, 'p')
command = sprintf('%s -p %f', options.p);
end;

if isfield(options, 'm')
command = sprintf('%s -m %f', options.m);
end;

if isfield(options, 'e')
command = sprintf('%s -e %f', options.e);
end;

if isfield(options, 'h')
command = sprintf('%s -h %d', options.h);
end;

if isfield(options, 'b')
command = sprintf('%s -b %d', options.b);
end;

if isfield(options, 'wi')
command = sprintf('%s -wi %f', options.wi);
end;

if isfield(options, 'v')
command = sprintf('%s -v %d', options.v);
end;

dos(sprintf('%s %s %s', command, datafile, modelfile), '-echo');

% Load the model temporary file and return its content
fid = fopen(modelfile, 'r');
model = fread(fid, '*char')';
fclose(fid);

% For some reason, the data read has superfluous newlines (it seems to have
% doubled the newline characters or something like that).
model = regexprep(model, '\r', '');

return;

LibSVM Matlab Wrappers

As I mentioned in previous posts, I've been using SVMLight for quite some time now. It's been good while it lasted, but I'm now in the search for a stronger SVM library, especially one that handles multiple labels well. I've been introduced to LibSVM, which seems very promising. In theory, they have libraries for various systems and languages, including Matlab. However, I've had quite some trouble making it work with Matlab, so instead of breaking my head into solving the problem, I've decided to work around it in a quick-and-dirty manner (my favorite...).

What I did, is simply wrap the LibSVM's executables in a simple way, so it can be executed using Matlab functions (given the executables are located somewhere in your PATH).

The following two posts will include a wrapper for svmtrain.exe and svmpredict.exe

Please let me know if you encounter any problems using them.

Ah, of course, they are provided AS IS, no warranty or anything alike. I also don't take any credit for it, nor any responsibility as to how/who/when/where/etc. it can be used. For this - please check with the authors of LibSVM.

NTFS File encryption does not like virtual drives

I like the NTFS file encryption feature - you simply define a folder as being encrypted, and any susequent file you add to it is automatically encrypted. This means that, depending on permissions, other users may see that the file exists, but they won't be able to get the content. Even the system Administrator won't be able to read the file's content!
(There are various other goodies related to encrypted files, such that you can't copy them and such, but that's besides the point).

But I also like virtual drives. For example - my 3 PC's have a virtual drive called Z: that maps to a shared folder in one of the machines (called C:\Shared - to be precise).

So I had a couple of files in an encrypted folder (accidentally on the machine that hosts the folder mapped to by Z), and wanted to transfer them to another machine. I sent them to Z, and then I wanted to decypt them so I would be able to retrieve them from my second machine.
Just to make sure the scenario is clear: I'm connected with the correct user (the one who encrypted the files), and simply want to decrypt the files. The only thing is that I want to do it from a virtual drive (Z:).

 

BABOOM!!!

"An error occurred applying attributes to the file:
....
The system cannot find the path specified."

Of course, once I tried to decrypt from C:\Shared instead of Z:\ - everything went fine.

Not nice, really not nice.

My daughter can do magic !!!

Purim is getting near, and this year, our eldest daughter Gal (almost 3) is going to dress up as a fairy. So yesterday we decided to try out the costume - pink dress, purple skirt on top of it, purple wings, silver crown and, inevitably - silver star-tipped wand.
Once she was completely dressed she started playing with her wand, in it went more or less like this:

Gal: Hocus, pocus, bili-bili-bocus: Grandma and Grandpa.
(nothing happens)
Gal: Where are Grandma and Grandpa ?
Ma&Pa: Huh?
Gal: Where are Grandma and Grandpa ???
(our little brains try to understand why she suddenly starts asking about her grandparents. after a minute or so, the smart of us gets the point)
Ma: Were you expecting Grandma and Grandpa ?
Gal: Yes - I said "Hocus, pocus, bili-bili-bocus" and Grandma and Grandpa didn't come!!!

(At this point I understand what's going on. So I step aside, secretly call Grandma and Grandpa, and ask them to come, and give us a call when they're outside so she can conjure them again, and this time succeed).

Ma: You know, you need to practice a lot for this kind of spells to succeed. Come on, let's practice...

(After half an hour, and a phone call from Grandma and Grandpa telling us they are outside)

Ma: Come on, let's try conjuring Grandma and Grandpa again
Gal: Hocus, pocus, bili-bili-bocus: Grandma and Grandpa.

The face she did when she saw them entering the house, was worth all the gold in the world...

Reflector 5.0 is out - awesome!!!

Usually, you don't feel much difference when Lutz Roeder releases a new version of his can't-live-without-it tool. You open it, it says that there is a new version, you click OK, and you don't feel any difference. You're using a new version of Reflector, but don't see where the big deal was.

Today, however, things went completely differently. I downloaded the new version (using the Automatic Update feature which works like a charm), and was already surprised that the downloaded files was so big (1 MB).
Then I opened it... This guy is a genious:

1. Integrated help from MSDN - when you look at a method's decompilation, you automatically see, at the bottom of the screen, the MSDN information about that method. Even cooler - the links inside that pseudo-MDSN pane work inside Reflector, and navigate to the correct place showing you the decompilation automatically!
2. Expand Methods - When you look at a class, there is a link at the bottom called "Expand Methods" which will expand all the methods in one screen, effectively providing you with a complete decompiled class.
3. Enhanced Analyzer feature - I LOVE the "Assigned By" feature, which shows you which methods change a field!
4. Integrated search in MSDN and Google (I wonder how long it will take for Microsoft and Yahoo! to go complaining about the lack of configurability of the search engine...)
5. Shell integration - run reflector.exe /register in a command prompt. Do it! NOW!!! You did it? Sure?!?! Well, assuming you're smart enough to have taken this advice - from now on, when you right-click on a managed dll, you'll have a "Browse with .NET Reflector" option, which will automagically open the dll with, huh, well Reflector of course. Actually, you don't even have to go through right-click - it's set as default when you double-click the dll...
6. More features here. I wouldn't be able to put it better - The Best Tool Ever Got Even Better!!!

Cool!

Thanks Yossi for pointing it out.

Microsoft Office errors are soooo descriptive....

Source: http://www.microsoft.com/products/ee/transform.aspx?ProdName=Office&EvtSrc=OfficeErrorMessages&EvtID=10046&LCID=1033&ProdVer=11.0

 

Details

Product:
Office

Event ID:
10046

Source:
OfficeErrorMessages

Version:
11.0

Message:
Links to additional information about the error message you saw are available below.

---

Currently there are no Office Assistance articles available for this error message.

What's up with YNet ???

For the past few days, every once in a while, when I open YNet, my browser takes up 100% CPU and I have to kill my whole iexplorer session to continue working properly. I've installed IE7 only a few weeks ago, so I don't know whether it's IE7 or YNet's fault, but it's starting to really get on my nerves!!!

4th Israeli bloggers dinner

If you're interested, make sure to let Omer know about it.
I really hope I'll be able to make it...

Anyways - kuddos to Omer for setting this up, again. It also looks like the deal he managed to get is a good one - I'm hungry only from thinking about it...

Political confession

I'm generally a left-wing voter - the two last elections I voted Meretz. I can't say I'm happy with their behavior, nor that I agree with all their opinions, but it's as close as it gets.

Anyway, I must admit that in the 18+ years I've been living in Israel, there has been no minister as good at his job as Ariel Atias, our minister for communications - from Shas - quite the oposite of my political beliefs. This guy is AMAZING - he is 100% focussed on improving service to the citizen, and doesn't mind fighting all the big communications, in most cases ending with the upper hand.
Just to name a few:

• Force companies to make it very clear to the customer, while signing a contract, what happens at the end of the contract (not just in small letters).
• Limit the ability of companies to force customers to buy their phones through them.
• Forced Bezeq to provide a free phone book over the internet (144).
• Asked the companies to give reductions to people living in the north, during the war.
• Still working on making our phone numbers company-free (i.e. changing company doesn't force you to change your number). This has been postponed several times, but I hope it will succeed in the end.
• Opened the stationary phone market to concurrence.
• Force companies to provide 3 non-paid seconds before being routed to a voice message box.
• His new baby - force companies to announce customers BEFORE they increase their prices.

And that's just part of it. A good article about him was recently published in YNet. Give us a handful ministers and politicians like him - and this could really be a nice place to live.

I was born on the same day as Albert Einstein

Yep, that's right, just 96 years later. You can see it all here, Together with Billy Crystal, Sir Michal Cain, Johan Strauss, Telemann and Jerry Greenfield - you can say I'm in good company.
Cool.

House security flaw

My neighbor's alarm is working - drives us nuts. I went to see if they are connected to some security center, and saw 4 or 5 placards of the same company. I called them, gave the neighbors' full name and address (either would have been sufficient). They told me that the neighbors are not customers of theirs. So how come there are so many of your signs all around the house? - I asked. They answered that probably my neighbors installed their alarm system through the security company, but they are not active client of the phone service ("moked"). Hmmm, so basically, if I were a prospective burglar, I could call each of the phone services in the area giving them the details of the house I want to break into, and in a few minutes I would have a pretty good notion of the level of security I would have to deal with...

You may ask - why is it the service provider's fault? What should they do different? Well, for starts, they could say something like "we will check whether they are our customers, and if not we will contact the police.", leaving me no way to know which it will be.

Visual Studio 2005 is *NOT* a LIAR!!!

UPDATE: I didn't think of it, but apparently the Memory Usage I see on VS indicates the amount of memory used by Resharper. I'm sorry if I offended anyone by this post - I don't mind picking on people and things here and there, but not when I'm wrong...

Did you ever notice the note on the lower left side of VS2005 window? At the moment of writing this post it says: "Memory Usage: 53.1 Mb". Sounds good, doesn't it? A quick look at the Windows Task Manager for the devenv process tells me the true story:
Mem Usage: 273,028
VM Size: 231,884 (note that this is the better measurement to assess how much memory is used by your process).

LIAR !!!

Sacha Barber

I've never met this guy, but Sacha Barber keeps writing article after article on The CodeProject about AI, .NET and what's in between. In January only he published no less than 10 (!!!) high-quality articles (his rating averages around 4.5/5 for most articles).

I don't know how he manages to do this, but I sure hope he'll be able to keep them coming. Great work!

Musings: Why I think Google missed a big opportunity and ISP's are stupid

Huh? Why I what? Let's take it one step at a time:

GMAIL - a big (huge?) opportunity missed by Google

For almost a decade, there was one de-facto leader in the free webmail industry: Hotmail. Yeah, yeah, I know, there were others, most importantly Yahoo! Mail, but still - Hotmail was the first and until GMail came around (2004) I think it's fair to say Hotmail was the most prominent player. A monopoly? - certainly not.
At Google, some clever people understood that the free webmail industry is lacking some important features, and they took the challenge: GMail offered an inconceivably large mailbox (they started with 1GB, now it's almost 3GB), backed with Google search facilities, slick simple web interface, POP3 access, etc. They did one big mistake - left it in Beta phase for 3 years, based on invitation-only registration (at the beginning each user had very limited invitations available). I remember that when it started, people were searching with every lost friend whether he may have an invitation left in order to register. And it remained this way for long - much too long.
Why? Maybe it's because it was developed as part of Google's 20% time each employee can use to develop whatever she likes - and it took too long for management to give it proper priority? Maybe because they were afraid of getting out of disk-space if too many users would register? Bandwidth? Spam fear? Bugs that needed to be resolved?
Whatever the reason may have been, it's in the past and they missed the opportunity to become THE email world-leader, almost as they are the leader in web search. Because now, everyone (except ISP's, see below) offers large mailboxes with Ajax interfaces, and it really doesn't matter whether you get 1,2 or 5GB - most people never use more than 100MB, and by the time they will need more, storage cost will drop enough for it to be unworthy noting.
A last thing about GMail. Actually, it's about Google Accounts - there is a small security flaw... I recently created an account for a family member. She wanted the password to be some date. I know, it's bad security practice, but bear with me for a second, ok? Now, let's say the date required was 31 January 1999. When I typed as password "31011999", the passwords strength indicator says it's "Good". When, however, I typed as password "01311999" (i.e. MMDDYYYY instead of DDMMYYYY), the indicator says it's "Strong". Do I need to say more?

ISP's - why are they so stupid?

I don't know how it goes in other countries, but here in Israel, the default mailboxes you get from your ISP is extremely small (around 5MB). The result is that people don't use their ISP's mailbox for any important purpose, and prefer using their free webmail, such as GMail, which can also be accessed through POP3. There is even a movie that shows how to do the configuration, if you're not a techie.
Why do I think they are stupid? Let's look at the mobile phone analogy. In Israel there are several cellular companies, each with its own prefix. If I want to change my mobile provider, I know it will incur changing my phone number - so I will do it only if it's really really really worth it. I don't have to tell you the headache and costs incurred: service providers, clients, business cards, invoices, friends, etc. - all must be updated with the new number. Even the Israeli minister of communication understands the problem, and is actively trying to force the various phone companies to support changing provider without changing the number - until now without success.
With mail addresses it's similar - people hate changing mail addresses. Granted - it's much easier to change your email address than to change your phone number, and cheaper (keeping a forwarding account will cost you less than 1$/month), but still, after a couple of times you get sick of it and will try to avoid it as much as possible. Also, it is much more complicated, if not impossible, to force an ISP to support emails that belong to another provider. So basically, ISP's could bond their customers by giving them excellent email services (large mailbox, fast access, search facility, web access, etc), but they just don't see it. I'd even go further - they should provide FREE emails to everyone, not only to customers. Why? Well if I were a potential customer of some ISP, and I would see that most emails around me belong to that ISP, I would assume it's a big company that gives good services - and would be more likely to be drawn by it.
Ah, one last stupidity - I recently had to create an account at Netvision for a friend of mine. They still limit the usernames to 8-characters (only alpha-numeric I think). Hmm, WAKE UP !!!!

YNet's encyclopedia

YNet expose a pretty interesting encyclopedia. Of course, I usually first search in the Hebrew Wikipedia. However, sometimes the value I'm searching is missing, or incomplete, and I want an answer. Their encyclopedia costs 19NIS/month. It's not much, but if I need to use it, say, only once every other month on average, it's a little expensive. What I would suggest is to provide a similar paying mechanism as they do with the cars pricelist - you call a special phone number, get an access code over the phone, and can keep using the site as long as the communication is not disconnected. This allows for people like me, who only need the service occasionally, to be part of their paying customers, without too much hassle and for reasonable prices.

GoogleFight

Today I was introduced to a Google goodie (not affiliated to Google) I wasn't aware of - GoogleFight. It's a simple site that lets you compare the number of results Google returns for two different queries. So you can easily compare the popularity of things (or people) - apparently, despite SVM's superiority in most cases, Neural Networks are much more popular than Support Vector Machines. "Madonna" is 4 times more popular than "Britney Spears". There is much more "good" than "bad" in the world. You get the idea...
Alternatively, you can check expressions and improve your writing - the American way of writing "color" is so much more popular than the English "colour". It's better to "speak english" than to "talk english", etc.
Nice...

.NET WTF - InstallUtil.exe gets corrupted ?!?!?!

I'm currently playing with a service - installing, uninstaling, connecting from remote machine, etc. Suddenly, out of the blue, when I tried uninstalling the service (by the book, using InstallUtil.exe) I get the following error:
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\InstallUtil.exe is not a valid Win32 application.
(in the command prompt I even get a message "Access Denied")

Being a restart-freek, I first tried to restart the machine - no change.
So I started googling my way, found several people complaining about this with no concrete reason/solution, until I found someone saying that replacing the installutil.exe file solved the problem. I looked at the .NET folder, and sure enough, InstallUtil.exe lies there, with size 0 and Date Modified - a few minutes ago !!!

?!?!?!     W  T  F     ?!?!?!

Of course, replacing the file with an uncorrupted one solved the problem (lucky for me I have two machines with .NET installed).

Cusbara - amazing parodic site!!!

If you're Israeli, or just concerned about the situation in Israel, you MUST visit Cusbara. As an example, check out their song about Israeli-made corruption. It's so sadly true you have to laugh, otherwise you won't stop crying...

Google Image Search API Fixed but...

I've updated the API to comply with the latest changes by Google. However, since the change required some code changes, in addition to the changes to the regular expressions, we will have to wait until the guys at CodeProject upload the new version to the site. I've sent them an email with a request to do it asap - usually it takes a few days. Hopefully this time it will be a little bit faster.

Update: I checked the article - it is now up-to-date. Please let me know if there are any more problems.

What's wrong the Blogger search???

It's not the first time this happens to me, and it's starting to get on my nerves. Many of the things I write in my blog are mostly done for my own personal benefit - easily retrieval of data from anywhere I am.
Today I wanted to find my recipe for my lentil soup. I have it on paper somewhere, but not at my fingertips. So I opened my blog and did a blogsearch for "soup" inside my blog. Sounds reasonable, doesn't it? Well, my recipe was nowhere to be found, even when I tried other searches ("lentil", "rich", "Soup" with capital S, etc.).
The funny thing in all that is that when I do a simple search on Google, I get my recipe immediately.

Problems with the Google Image Search API

My Google Image Search API is currently broken. Apparently Google completely changed the format of the HTML they return, so the API doesn't work. I gave it a quick look and it seems that, unlike previous changes which tended to be rather limited, this time they completely changed the structure of the result, so I have to build the regular expressions from scratch. I'll try to do it in the next few days and post an update in 1 week or so.

Integrating Matlab and TFS

It IS possible to integrate Matlab with Team Foundation Server. Here are the steps you must take (I'm assuming Matlab and TFS Explorer are already installed on your machine):

• Install the Visual Studio 2005 Team Foundation Server MSSCCI Provider.
• In the registry, search for the providers under:

HKEY_LOCAL_MACHINE\SOFTWARE\SourceCodeControlProvider\InstalledSCCProviders

• Copy the existing registry entry for "Microsoft Team Foundation Server MSSCCI Provider" and give the new copy a shorter name, e.g., "MS Team Foundation Server".
• In Matlab preferences (File => Preferences), select the "Source Control" subsection under "General"
• Choose "MS Team Foundation Server" and click Apply.

That's it. From there on you can use TFS much like you would have used VSS. I suppose most new TFS features are not available, but at least you can do simple checkin/checkout, view history, compare files, etc.

Reference (requires a Mathworks login): http://www.mathworks.com/support/bugreports/details.html?rp=347563

אני לא ליצן

אבא: גל, תפסיקי לאכול את השקית
גל: אבל אני לא אוכלת אותה
אבא: גל, אני לא רוצה שתשימי את השקית בפה, גם אם את לא אוכלת אותה. אני לא צוחק איתך
גל: אבא, אתה לא מצחיק - אתה לא ליצן!

Did you know - Installing Team Foundation Server is a nightmare!!!!!!!!!!!!!!!!

Just to name some issues:

1. Prerequisites (or Things You Should Install Yourself Because The TFS Setup Team Was Too Lazy To Make The Process Smooth And Automatic)
1. IIS
2. SQL 2005
1. + Some hotfixes
3. .NET 2.0
1. + Some hotfixes
4. SharePoint Services
5. Team Foundation Server
6. Visual Studio Team Foundation Server Quiescence GDR
7. Visual Studio 2005 SP1
2. The setup requires some 3-4 users to be already defined on the domain (that is - you don't have the option to add these during the installation process).
3. TFS includes a license for SQL 2005 Standard Edition. But nobody tells you where you you can find the required Product Key. So we're now on the phone with Microsoft to figure that out...

I can understand problems related to the conversion of existing source management systems to TFS (up to some extent). But I NEVER thought ANY software could be THAT cumbersome to install. We didn't even reach the point of converting the data, and frankly if the setup is so complicated, I shiver at thinking of the complications we will encounter at converting...

AI to solve networking problems

Take a look at this excellent article on CodeProject. It describes the use of Ant Colony Optimisation algorithm for network load balancing.
Ever since I took the Anat Bremler-Barr's Advanced Networking course I've had the feeling AI algorithms could significantly help networking problems (not only routing, also various network attack detections for example). I've always been more attracted by Machine Learning algorithms rather than Ant and the like, but still, it's in the good direction...

Why aren't RSS readers smarter???

The blogosphere (both readers and writers) can be roughly split into two groups:

• Those who believe bloggers must have an agenda and stick to it. These people believe that a blogger know to be "technical" shouldn't throw too many stuff about his private life at his readers, political views, etc. Some bloggers take it to the extreme and manage several blogs, each on a different subject (or various RSS feeds).
• Those who believe bloggers should feel free to write whatever they feel like, and it's up to the readers to choose which post to read and which to skip. I'm in this group...

Well, it's fairly easy to merge these two groups. The only thing needed is some smart RSS reader, where you could define "rules" either per blog or general, which describe the content you are interested in and/or the content you don't even want to see. So if people reading my blog don't want to read my soup recipes, they can add a negative rule to my RSS feed and still enjoy (I hope) the rest of my posts. Alternatively, people who are only interested in stuff I write about search engines, could add a strongly positive rule, which will disregard the other posts. Of course, it would work better with blogs that are better organize than mine and also include tags...

As a side-note, this is a good example of applications that can be made much smarter than they are, but people just don't think hard enough how this could be done. Technically, it's no biggy. The only problem is to think about it. I think in many cases, the problem is that people designing the required features either don't understand software well enough, to see the wealth of possibilities, or are too strongly connected to the way developers think, and don't see what would really help the "simple" users.

The Angle of Health must be pissed off

I'm not a believer. Actually, I'm as far of a religious believer as one can be. But if there is some power orchestrating it all, it must have got something really strong against my family. If indeed there is such a power - then it's completely reciproqual and I'm really pissed off.

It started with all of us getting sick, for 1-2 weeks. That's not very nice with a 2-months old baby, but it's winter, so we can't really complain.

Then my wife's grandpa got sick and hospitalized. More on this later.

Then by big daughter, Gal, got sick again. This time, with a tenacious 40 degrees Celsius we just couldn't fight for several days.

Then my wife's grandpa lost the battle. He died last Thursday. I hope to be able to write a little more about this great man in a few days.

Now Gal is sick again. This time, she's got an extremely rapid fever. It can go up or down by more than 2 full degrees within the hour!! Most importantly, last night she had a (second) febrile convulsion. It's one of the most horrible experience a parent can have to endure. The most horrible thing is to hold your child in your hands for 5 full minutes, the whole body in seizure, not having anything to do to stop it. Once this nightmare is over, the whole body enters some kind of apathic state, and no matter how well you know it's going to fade away, you've got the conviction half her brain got fried by the seizure. Only after 10 more minutes did she start crying, and talking a bit, letting us know she's OK (the ambulance took some 20 minutes to arrive, of course).

So if you were wandering why I haven't been posting much lately, now you know ...

WinSnap - makes screenshots better

WinSnap - makes screenshots better.
That's their motto, and it's o so true!!! I don't remember where I heard about it for the first time (maybe the Daily Grind? I'm not sure). Anyway, it's an amazingly cool little screenshots maker, which gives you literally all you need to do the task. When it loads, it automatically takes a screenshot, so it really takes a single click (or keyboard shortcut) to create a cool-looking screenshot.

Highlights (in my opinion):

• Simple, no fuss. One click to create the snapshot, another to directly send it by mail/copy to clipboard/save image locally.
• Various coloring and shadowing options.
• Various capturing modes:
• Full screen
• Application
• Window
• Client
• Popup menu
• Region selection
• Contour, Watermark, simple image manipulations.
• Small.

Since I downloaded it I've made several tens of screenshots, some to be saved locally, others directly sent by mail. I wish all software I used was so simply tuned to fit the needs it was aimed to fullfil as WinSnap.

Sole problem I encountered: it's slightly slower at taking snapshots than I would expect (around 1 second). I'm greedy - I want it to be quasi-instantaneous (say 1/5 second).

Rating on Ilan scale: 9/10

UPDATE: I discovered in the Settings that there is a default delay of 1 second when you take a snapshot while the application is open. Once you change that to 1 ms you get the kind of performance I was expecting.
This changes the rating in my eyes to a perfect 10/10 !

Misradia.co.il - online shop (Israeli) review

I've been working from home for more than 2 years now. Every now and then I need some office stuff: recordable media, computer peripherals, printer paper, envelopes, etc.
Ever since I've started buying at Misradia I've been trying to do all my office purchases from there. The prices are good, they have a wealth of stuff, everything is always on stock (to my experience), very good service (via Skype). I made an order yesterday at 1PM. It arrived today at 11AM!!!

Pros:

• Good prices
• All office stuff available in their catalog and it's all on stock
• Extremely fast delivery
• Good service
• Simple and intuitive site
• Free door-to-door delivery (for large orders)

Cons:

• Sometimes the packages are not sufficiently well organized and some things arrive crumpled. Usually it's only the carton box that contains the goods.
• Some articles lack more thorough details on the site. Some lack good pictures.
• There is a fairly high minimum order to get a door-to-door delivery (I think it's 223 NIS). So you should calculate your orders carefully to fully enjoy this benefit.

Rating on Ilan scale: 9 (of 10)

Making public transportation easier - an application idea

Tonight I tried helping my sister to figure out how she could get to Jerusalem, in the middle of the night, through public transportation.
For my non-Israeli readers - in Israel there is one major inter-city bus company (Egged) with a few minor ones. Egged also has most intra-city busses, except for Tel-Aviv where it's the Dan company. For inter-city traveling there is also an ever-growing train company. There are no subways of any kind. Most information for these companies is available online.
Now, from where she is, there are several ways to travel to Jerusalem (mainly Afula, Haifa, Tel-Aviv). The thing is - how do you find the best path?
All these companies provide a means to retrieve very straight-forward information. For example, given a bus number - show me the hours of departure. Or given a source and destination. However, none help you actually plan the trip, when you need more than one means of transportation. Let's not even speak of plans that include both busses and trains.
The thing is that it should be fairly easy to do it - there are relatively few companies (2 main ones), all providing an online search form. Writing such an application shouldn't be complicated, and would obviously help a lot of people.

Just an idea...

iPhone

"iPhone combines three products — a revolutionary mobile phone, a widescreen iPod with touch controls, and a breakthrough Internet communications device with desktop-class email, web browsing, maps, and searching — into one small and lightweight handheld device. ..."

And when you peek at the technical properties, you really start druling like a Rotweiler. Some goodies:

• Wide screen iPod (4/8 GB)
• GSM Quad band (for those wondering - works in Israel)
• Wi-Fi, EDGE, Bluetooth
• 5 hours Talk/Video/Browsing, 16 hours audio playback
• 115 x 61 x 11.6 mm
• 135 grams
• 2 MP camera

Take a look!!!

More info:

http://www.gadgetnutz.com/modules/news/article.php?storyid=272
http://blogs.pcworld.com/staffblog/archives/003462.html
http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/01-09-2007/0004502993&EDATE=

(Thanks to my friend Gil for telling me about this little gem)

Things that make me hate good software

I've been using Foxit Reader instead of Acrobat for over a year, and I must say that I am generally very pleased with it. I really love it - it's fast (way faster than Acrobat), relatively low memory footprint, supports all PFD files I read (there were some issues with the previous version but the new one really works like a charm) and doesn't even require installation.

But ...

A few days ago I wanted to use their PDF printer add-on. Never mind the fact that it didn't work and got completely stuck - I managed without it in the end. But today I wanted to print something from Outlook, and guess what - they automatically replaced my default printer to be the Foxit PDF Printer!!!

Now THAT's something that can make me really HATE, otherwise EXCELLENT software!!!

Excellent security blog

Alik Levin has started posting very seriously (31 posts last month). His blog is very much focussed on security issues, and it's a real gem. His post are concise and thorough at the same time, each post focussing on a very special topic each of us has/will encounter at some point.
Highly recommended for anyone working with .NET and/or SQL Server.

I may not be cool, but I think this sucks!!!

A new Israeli site provides users a means to retrieve peoples' name and address given their phone number.
Now imagine this scenario: your 16-year old daughter sat for a nice cup of coffee with her friend at some coffee shop and forgets her mobile phone. A guy who noticed her in that coffee shop and found her good looking takes the phone and starts going through her numbers in memory. He finds the "Home" number, and through this despicable site, knows exactly where to locate her.....
This site should go down - the sooner the better! I know that in many cases the database is outdated, but I managed to find the names and addresses of many people very close to me.

(through YNET)

Article updated

I've updated my API for Google image search article. It now compiles with VS2005 and loads the regular expressions from an external text file, instead of hard-coded.

The Circle of Life in the eyes of a 2.5-year old

Gal: They are going home.
Dad: Who's going home?
Gal: The rain! (pointing at the raindrops on the car window)
Dad: Where is their home sweetie?
Gal: Underneath the trees! (said in a what-kind-of-stupid-question-is-that tone)
Dad: Who told you that's their home?
Gal: I did.
Dad: Hmm, if that's their home, then where are they coming from?
Gal: From underneath the trees! (same tone)
Dad: So the raindrops originate from underneath the trees, and then go back home underneath the trees?
Gal: Yes!

A goose laying golden eggs?

While reviewing my blog statistics I discovered that, obviously, the highest hit rates occur during periods where I post more and more frequently.

>> No surprise here - where's the gold?

The more interesting bit, is that almost every post is followed by one or more clicks originating in the Bloggers' "Next Blog" button (look at the upper right corner of this page). I always thought this button refers the reader to some random blog on Bloggers' blogosphere. Well, that's not exactly the case. I don't know exactly how it works, but basically, you have a chance of being referred to by this button, if you posted something a short while ago.

>> OK, OK, now didn't you promise gold?

Here it comes - AdSense, and all other advertisement that are based on either clicks or views (CPM or Cost Per Mille (1000) impressions). With AdSense, they work with batches of 1000 views (I don't know how much you get paid for these - I'm not advertising on my blog in any way, as you see). So, with a little program that would write a new post every second (you could split this into several blogs, of course), you get 86400 views per day (assuming each post results in one single random viewer, through the "Next Blog" button), which I suppose should be beneficial...

>> Nice... But aren't they using CAPTCHA's to avoid misuse of the blogs?

Yes and no - you need to prove, for a while, that you're up to no bad. Then you can send Bloggers an email requesting to be exempted from the annoying CAPTCHA. Once they've done that, you're free to go. Also, with the advent of Windows Live Writer and other similar tools, I guess they are getting looser and looser with this issue.

>> Now really - how much could I earn this way?

Well, it depends on a lot of factors. Let's assume you get 5$ for each 1000 views. With one post/second, you get 432$/day or 12960$/month. And once you have the infrastructure up and running, there is very little work to do, and you may be able to increase it by adding more and more blogs to it. The main problems you may encounter, IMO, are:

1. How to get CPM deals? I suppose in many cases the advertisers will want to really know the sites on which they will be publishing. If that's the case for all CPM programs, you'll have to rely on clicks, in which case the revenues will be substantially lower (if any).
2. How to remain below the radar? Obviously, the more blogs and posts, the more chances you will stand out.
3. Generating random posts - it wouldn't hurt if you managed to make those posts look genuine. Somehow...

Summary:

1. Create several blogs on Bloggers. I would even go for several thousands, in order to have an average of only a few blogs a day, to remain below any radar.
2. For a while, post reasonably valid stuff on all these blogs. It's a lot of work, but it's going to be worth it.
3. For each of these blogs, ask Blogger to be exempted from CAPTCHA.
4. Create an AdSense account for each of these blogs. You may want to try other advertisement tools as well, why not?
5. Write a program that automatically generates random posts. I would make them using concrete English phrases, maybe citing some scanned book or some news website (beware of legal pitfalls). Automaticaly generate a post to one of these blogs every second.
6. Start counting the money...

DISCLAIMER: What I've written above is the result of some random thoughts, while I was trying to solve some completely unrelated problem. I have no idea whether it would indeed work or not - I didn't try it and don't intend to. This should also not be viewed as a recommendation to act this way - I have no idea whether it is legal or not, and am not responsible for anyone who tries it and finds himself behind bars because of that. If you have any information that could shed some light as to whether or not this is feasible, drop a line, to fulfill my academic curiosity ;-)

 

You call this medical treatment???

My wife's grandpa, over 80, got hospitalized a couple of weeks ago due to hyponatremia. As this problem implies, it was probably a result of various other illnesses he has been suffering from lately. While at the hospital, he catched pneumonia and above all he got a very violent bacteria, which reached his heart and even harmed his artificial pacemaker. The only way to save his is surgery, to remove the pacemaker, but his condition is so bad that they're currently trying to figure out how to do it, and in any case the chances of success are very slim.
Had he stayed at home, eaten better and drink a lot, he'd probably had gotten rid of the problem in a few days.

And that, ladies and gentlemen, happened in one of Israel's finest hospitals (Rambam).

VS2005 SP1 - I'm so glad I'm not an early adopter

Because my heart wants me to be an early adopter, I'm always extra careful not to be one (hope this made sense...).

I haven't installed Vista yet, nor Office 2007. Now that Visual Studio 2005 SP1 is out, I almost installed it immediately, but I managed to restrain myself and force myself to wait for a few days.

Well, patience pays well, they say. Apparently there are serious problems with the running time, disk space used and disk I/O in general during the SP1 installation. Thankfully, Jon Galloway posted a way to overcome it.

I think I'll wait a few more days to see if anything else shows up and then I'll take my chances :-)

(through The Daily Grind)

Update: Oren Eini has gone through a similar ordeal when he installed the service pack. Here are some other useful links from his blog:

http://blogs.msdn.com/astebner/archive/2004/11/10/255346.aspx

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1042560&SiteID=1

( http://connect.microsoft.com/VisualStudio/feedback/Vote.aspx?FeedbackID=226009 )

The telephone evolution

Once upon a time, you only had a simple phone, with a single 10-digit dial ring. Period. That was a huge leap from the Telex and those phone communications that had to go through some central. It worked, everybody was happy.

Then came the more fancy, digitized phones. Still 10 digits, but push-buttons instead of a dial ring. Some even came with no wire (and usually very short-life batteries).

Then they added all kinds of features, with memory and coffee making. Well, it never made coffee, but usually saved the 10 numbers you recorded upon purchase correctly, but you never remembered whether number 3 was Mom or Mother-in-law. Bummer!

Then came the cellular phones - like the latter, just mobile. You felt safe, that no matter where you went, you'd always be able to ask for help. You ended up being reachable even at the least convenient moment, and getting low connectivity or no more battery juice when stuck with a flat tire in the middle of the Nevada desert at 3AM. Murphy rules!

Now look at this review in YNet about the Nokia 6131 (Hebrew). I won't get into the details, but instead I'll just translate the subtitles of the article, in the order in which they appear:

• Design
• Screens (that's no typo - this little baby has 2 teeth, er..., screens)
• Media (referring to camera - video and stills - , music, flash disc)
• Connectivity
• Conversation quality and battery
• Summary

I wander what Darwin (or Alexander Graham Bell) would have to say about this...

Fighting spam mail collectively

(Note: this post may describe an already existing technology. If it doesn't exist, then it's high time it would...)

As far as I know, there are two major techniques used by spam filters:

1. List - the filter has a list of indicators that allow the classification of a message as spam. This list is typically updated periodically, and usually the users can add items to it when they mark a mail as spam. Outlook 2003's Junk Mail Filter is a good example, which, as far as I know, is primarily based on domains and email addresses.
2. Adaptive - the system starts with an initial classification mechanism, sometimes as simple as allowing all emails. Then, whenever a mail is marked by the user as spam, the classifier is updated to inhibit this rule. These kinds of techniques are usually based on some machine learning algorithm, often some kind of derivation of the Bayes classifier. The advantage of these techniques is that they are more likely to discover spam mail of a completely new format. On the other hand, it may result in weird cases of false negatives and even false positives.

The limitation of these methods is that:

1. The first one depends on how well your provider knows about new types of spam mail
2. The second one requires, for each type of new spam, a set of samples it can learn from. So you will inevitably need to mark some mails as spam until your classifier is correctly tuned.

What I'm proposing, is that whenever you manually mark a mail as spam, the whole mail would be sent to the provider. This will give the provider a huge amount of recent spam mails, making it possible to update the filters in a matter of minutes. Then, each client that reads mails thereafter will already have a filter that will know how to filter these new types of spam mails (assuming the clients update frequently, such as once a day).

Problems:

1. This could, in itself, generate a huge amount of traffic, only due to the constant data sent to the provider.
1. Solution: Data could be sent in bulks to the provider (once every few hours), and no necessarily in email format
2. Solution: If the clients are frequently updated, this shouldn't be a problem, because then fast enough all clients will have an updated filter and stop sending data to the provider.
2. A spammer could buy several licenses and start pumping the provider such as to make him generate completely useless filters.
1. Solution: Data sent to the provider will be encrypted and include recognizable information from the user (based on the license information).
2. Solution: It should be fairly easy to detect such spamming clients and rule them out.

The bottom line is that using the strength of collectivity, it should be possible to create much more accurate spam filters, much faster.

Martial Arts Against Rapists

Some time ago I advocated we should include some kind of martial arts in the school curriculum, to help preventing violent crimes. Well, here is a concrete proof of this idea (Hebrew). A 16 year old girl has been attacked by a potential rapist. Fortunately, the girl is proficient in Karate and with a couple of strikes managed to break the bastards nose and make him run away for his life. As simple as that - she knew how to defend herself, didn't freeze out of panick, and managed to avoid one of the most awful experiences a girl could endure.
I'm just sorry she didn't manage to break his legs, so he could get caught, or maybe his neck, which would have solved the matter completely...

בחנוכה אוכלים סוכריות ולחם

גל: אבא, איזה חג קרב ובא?
אבא: נו גל, איזה חג?
גל: חג החנוכה!!!
אבא: ומה אוכלים בחנוכה?
גל: מה?
אבא: מה אוכלים בחנוכה?
גל: מה?
אבא: מה אוכלים בחנוכה? ... סו ... סו
גל: סוכריות !!!
אבא: לא, אוכלים סופגניות. מה עוד אוכלים בחנוכה? ... ל... ל
גל: לחם !!!

Rob's done it again...

Check this out - An Open Letter to the Software Managers of the World - oh, so true...

Buffering hacks

Check this out: http://blogs.microsoft.co.il/blogs/applisec/archive/2006/12/04/Buffer-Overflow-_2F00_-Overrun-examples.aspx

A truely amazing pantomime...

Just got this by email - really amazing: http://www.dailymotion.com/visited/search/jerome%20murat/video/xf9oo_jerome-murat=

More missing things in C#

Something I'd like to add to my C# wish-list:

Why is it that when you use "using", you can only put a single parameter inside it. Often enough you need to use multiple resources at once, and want to get hold of them and release them at the same time. In such case you need to write something like this:

using (StreamReader reader = new StreamReader(inputFileName))
{
    using (StreamWriter writer = new StreamWriter(outputFileName))
    {
        
    }
}

What I would really have liked better would be something like this:

using (StreamReader reader = new StreamReader(inputFileName),
       StreamWriter writer = new StreamWriter(outputFileName))
{
    
}

 Wouldn't you agree?

More ISP ranting...

[Update: Upon request, I've added the names of the companies I was referring to as well as some updates at the end.]

I'm not happy with my Internet. There are, at least for a few more months, 5 major ISP's in Israel (which will unfortunately drop to 3 soon enough). There are more, but I'm only considering the largest ones.
Anyway, of these 5 ISP's, one (Internet Gold) I don't like for personal reasons (I'd rather not elaborate), another (012 Golden Lines, the smallest of the lot) has a reputation of being technically inferior and yet another (Bezeq International) has a very bad reputation of taking money off customers without prior notice, making it very hard to get it back (experienced by people close to me).
This leaves me with 2 choices. I had been customer of one of these 2 options (Barak) for several years, being pretty happy with it (service, support and quality). Yet, during the past few months I had had a need to work more intensively with a cross-atlantic VPN connection. This connection got constant short disconnections due to packet losses somewhere in the middle. After having tried to fix this problem together with my ISP for a couple of weeks, I came to the conclusion that that's the best they can offer, and decided to move to the second of my optional ISP's (Netvision).
Yet, after a while I started to feel the download times are significantly lower than with the previous ISP. I'm talking HTTP download (I don't use any P2P) from major US-based websites. I talked to their support, and after some junior support person finished proposing me stupid solutions (workarounds that would barely make my granma happy, if she were alive), I finally got to talk to a senior support guy. This was the worst support conversation I ever had - he was rude, didn't let me finish my sentences, constantly made me feel like I'm a bone-head, and made me decide to move back to my previous ISP - at least they were polite!
Customer service tried to convince me to stay, and since I have technical problems with both, I'm now paying for 2 ISP's, checking the differences between them every day or so, hoping I'll be able to decide in the next couple of weeks.

I am now several weeks in this test. In general, there is no significant difference in download time between the two providers (Barak and Netvision). When one of them is slow, the other seems to be as slow (with sometimes an advantage for Barak). One MAJOR difference is the download time of emails (I think it's most promiment with my 2 GMail accounts). There Netvision can get stuck for several minutes (especially mails with large attachements), whereas Barak works like a charm.
I am supposed to receive in the next few days a router that will do the VPN work instead of the software installed on my machine. I'll then check the VPN connection more thoroughly, and make my final decision. Due to the quality of the support team and the email speed, I really hope I will be able to choose to get back to Barak and get this done with.
In any case, I assume the decision between these two will have only a short-term impact, since the two companies are due to be merged during the year of 2007.

Where is the regulator?!?!

Until recently, the Internet market in Israel was a pretty good example of the free market. You had various suppliers, each with their pros and cons, and as a customer you could move relatively freely between them. Also, due to the harsh competition, they were eager to please - to our (the customers') benefit: prices kept going down, service quality kept increasing, etc.

A couple of months ago, Netvision and Barak have announced a merger that will take place by the beginning of 2007.

Today, Internet Gold and 012 Golden Lines have announced a similar merge.

Thus, from 5 medium-sized competitive companies (Netvision, Barak, Internet Gold, 012 Golden Lines and Bezeq International), we're left with 3 strong behemoths (Bezeq Int'l was pretty large to begin with), significantly harming the service and prices for the customers.

BTW - it's not only Internet we're talking about - the exact same companies also provide international communication services, and some are also involved in either the cabling or the sattelite television.

REGULATOR - WHERE ARE YOU ?!?!?!

A project I'd like to know more about

Sam Gentile speaks about a successful 14-months Agile/XP project w/ Wilson OR/M and other goodies. He promises to provide us more info - and I'm looking forward to reading it.

Sanity Tests

Continuing my QA posts (Testers are from Venus, Programmers are from Mars, QA Day), I'd like to talk about Sanity Tests.
Regardless of your development methodology (Waterfall, Agile, Scrum, WhatEver...), at some point or another you have to forward releases to the QA team's inspection. Much too often, the QA team receives a version of the software, starts testing it, and after a while (one day, 3 days, sometimes a week), they realize that some basic functionality has been harmed too significantly to be able to continue their tests. So the release goes back to the programmers to fix the problem, after which the testers must start everything all over again.
An excellent way to avoid this, is to use a layered testing methodology. The idea is to create testing procedures that, instead of investigating a specific feature to its smallest details, investigates the whole system up to some level of detail. Then each test procedure is actually a "drill-down" into more specific parts in the whole system. The idea behind this methodology is to be able to provide as profound a test as time permits, without ever missing a single feature.
Yet, many organizations and QA teams are not build for this kind of methodology, or simply don't believe in it. Also, it has some inherent overlapping of work, which could make the whole thing completely irrelevant in many cases. Imagine a feature which, to be tested, requires a whole day or preparations. With layered testing you would need to setup this test several times - once for each "layer".
So if we can't use layered testing, what can we do to avoid the problem stated above?

Well, here comes the Sanity Test...

The purpose of a Sanity Test is to test all the basic functionality of the system, without getting into the specifics of any of them. For example, if you're working on a customer support system, it is completely inacceptable to get thrown out of the application whenever you try to open a new support case. On the other hand, you might want to continue testing the system, even if some of the reports don't open properly. So the idea is to test the system end-to-end by making sure that at its base, all features work, without focusing further into any of the features. From my experience, for a team of approximately 4 developers, the sanity test should take around 1-2 working days tops (that's for development cycles of ~1 month - Agilists would get other figures). The most important thing here is that: a build that does not pass Sanity Test is not fit to be transfered to QA!

I've been working with Sanity Tests at 3 separate organizations, twice were my initiative. It's not easy to make everybody understand how important it is. Most of the times, the developers are those responsible to do the testing (after all, although it's acceptable to have bugs, the developers are still responsible to provide a workable release), and they don't like it one bit. Project managers get p--d off when a release is not transfered on time to QA because it didn't pass the test. Some QA teams don't know how to enforce the rule that they should not accept non-sane builds. And, of course, creating the Sanity Test Procedure can be quite tricky - you want it to be such that it completes fast, but tests all features well enough to know they are more or less OK.

Despite the problems it incurs, Sanity Tests have proven temselves to be very useful in creating more reliable software on time. The developers become more responsible about their code, testers are more productive and the whole development process runs more smoothly.