Testers are from Venus, Programmers are from Mars

I don't know how this works in other parts of the world, but here in Israel it is very common for wannabe programmers to start with a job as testers. Usually it's some part-time job they do in their last year of studies, with the hope of being appointed a fix job as programmer when they graduate. I didn't go through this path, but many programmers I know have.

My impression from this process, as an outsider who has worked with experienced QA Engineers as well as Software Engineers from various backgrounds is simple - it's WRONG WRONG WRONG!!!

Why?

Well, first, it emphasizes a common misconception that QA Engineering doesn't require specific qualities, something most person with enough brain to eat with fork and knife would be able to do. I think it's degrading one of the most important parts of software development. Being a good QA Engineer is hard, really hard even. It requires specific qualities very few people actually have. When you take inadequate people to do a difficult job - don't be surprised that the results are sub-optimal (to say the least).

Second, I think that there are elementary differences between the character traits required from good QA Engineers and from good Software Engineers. Actually, the qualities required are so contradicting that I don't know any person who would be able to be good at both.

Let me elaborate:

• Patience - programmers are usually rather impatient people. When they feel their computer isn't working well, they'll go poke around the registry and whatever, sometimes making things even worse, just because they don't have the patience to wait for an IT person to help them out. A good tester needs a tremendous amount of patience. They may find themselves doing hundreds of tests, each slightly different from the other, for several days in a row.
• Feedback - programmers like to get immediate feedback, to see results quickly. That's why add-ons like Resharper give you the little green light on the side - so you know up-front your code will compile. Before that, most programmers used to hit the 'compile' button every few minutes, because they want to know - now! Testers can find themselves working on a good release, finding very few bugs. So there is really no feedback for a long time - the fact that they didn't find a bug, is it because the software is indeed so good, or maybe they missed something?
• Ego - programmers are very fond of their ego. We're not like sales-persons, but still, we like being appreciated (hope I didn't offend anyone here...). Good testers are rarely adequately appreciated - if they find many bugs, programmers won't like them (which is a mistake, of course, but that's life). If they don't find bugs, then they inevitably will show up in production, which is even worse. So testers need to be able to swallow their pride often, very often. Something most programmers I know are not capable of.
• Order - finding bugs is not enough. Being a good QA Engineer means you should be able to work in a very orderly fashion, documenting each step in order to be able to reproduce the bug (and give the developer as much information about it as possible). I know very few programmers who are good at this - it's sad, but it's a fact of life we must accept and learn to live with.
• Creativity - both tasks need a healthy amount of creativity. Yet, there is an important difference in the type of creativity required. Programmers need a contructive creativity - finding better ways to build software. Testers, on the other hand, need to find new ways to break the appplication. The difference is subtle, but significant.

I'm sure I'm missing still many other characteristics, but I think the idea has passed. When I think of the really good QA Engineers I know - none of them would make a good Programmer. Same goes the other way - I know of no good Programmer who would be able to be a good QA Engineer.

I strongly believe QA is one of the most important aspects of Software Development. In some sense, it's the most important one - it's the last phase the application goes through before it gets to the clients, the last chance to improve it before you make a fool of yourself. To my great dissapointment, many software companies I know fail to understand this. With companies that are not 100% software (ISPs, networking, applicances, etc.) the situation is even worse - some don't see why they need a QA Team, if they hire good enough programmers...

As I said, QA Engineering is hard. Doing this good is really hard. Finding good QA Engineers is hard. It's important to understand this, and accept it, if you really want your software to be good!

In this spirit I intend to post a couple more posts in the next few days (unless my wife gives birth before I get the chance). I intend to talk about "QA Day" and "Sanity Tests". If you don't know what these are - stay tuned, these are very effective tools for improving the whole development process.

SQL Injection Attacks

Scott Guthrie posts about Guarding Against SQL Injection Attacks. He also points to a great post by Bertrand Le Roy on the exact same subject. Oren Eini tried to tried to create a HQL injection with no results so far.

I think that of all, what fascinated me the most was Rocky Heckman's webcast, where he displays a step-by-step SQL injection attack, which cut the air of my lungs the first time I saw it. Be sure you don't miss it!

Moving to feedburner!

I'm moving my RSS feed to feedburner. Please make sure you register to: http://feeds.feedburner.com/WhyNotilanAssayagsBlog . I think I succeeded in redirecting feeds from the old RSS/Atom urls to feedburner automatically. If you experience any problem - please let me know.

The Multi-Tasking Myth - Revisited

Jeff Atwood strongly recommends to avoid multi-task working. He even backs it up with studies and citations from people who have already proved they know what they're talking about.

In principle, I agree with him, but I would like to set one exception (in French they "it's the exception that proves the rule"). The exception refers to research projects. Very often, I find myself immerged in some aspect of a research. I encounter some problem, but no matter how hard I try, I just can't find a way out. I'm stuck! When this happens, the best way out (at least for me), is to step back. I give my brain a rest, letting it work on other things. This could be anything - some simple programming task, reading a book, whatever. The important thing is to do anything that's as far as possible from my initial task. Then, after a couple of hours, it hits me (usually when I'm in the shower...) - I suddenly understand what I've been missing, and what I must do to get out of my current situation and move on.

So as far as I am concerned - when I have a complicated research project to work on, I always have a backup, low-priority task, preferably a very simple one, to which I can switch in case of stall. The switch costs, of course, but at least I don't find myself investing a lot of energy getting nowhere.

In all other cases, context switching is a huge time-waster and usually causes more harm than good, no doubt about that!

P.S: I'm not talking about literally doing 2 things together (talking on the phone and surfing for example). This is something that always degrades the quality of both actions. I admit - sometimes I do that, but then I'm willing to pay the price...

Wireless Security - the difference between Alfred the Optimist, Luba the Ostrich, Uninformed Max and Sacha Paranoia

Many people use wireless routers at home. It lets you use all your computers without the hassle of having cables all around the house, it's cheap, easy to install, and usually works pretty well. Very few people, however, think seriously about security issues regarding their home wireless networks. Even those who do think about it, tend to think their network is made secure by using some relatively-easy-to-hack tricks. Others, not concerned by security issues, might have half their bandwidth used up by their neighbors without them knowing it. The most shocking part of all this, is that many enterprise IT managers do the same mistakes with their enterprise wireless networks. My purpose today is to shed some light on the vulnerabilities and how they can and are being handled.

First, let's remember that a wireless network is, well - it's a network and it's wireless (duh!). Being wireless means that as long as you are in range, you can read packets sent from the access point and send packets to it. Whether you'll be able to understand the packets or whether the access point will do something with packets sent to it, is another problem. Yet remember - it's all up there in the air!

Second, let's look at the possible vulnerabilities:

1. Bandwidth "theft"
2. Network intrusion
3. Denial Of Service (DOS)

Bandwidth "theft"

Refers to other people (usually neighbors) using your wireless network to access the internet. Since most ISPs charge by bandwidth, if somebody is using your internet connection at the same time as you are - then you get less bandwidth for yourself. I "stole" bandwidth several times during the war when I was staying at other people's houses (at some point I started wandering around the country with my own router to stop doing that). As far as I know, it's not really called "stealing", since the air is public domain, so you can't claim anything about packets floating around. But still, if you have someone constantly using your bandwidth, it can become pretty annoying.

Network intrusion

All browsers, firewalls, etc. use the notion of "trusted" and "untrusted" zones. The idea is that in your trusted zone you have computers you trust will not try to do you any harm, whereas the untrusted zone (everything else) can consist any computer in the world, including some that might be very mean to you, if you just give them a chance. Before the wireless era, things were pretty simple. Anything inside the company/house is trusted, anything else is not. Yet now, when you can connect to the network without being physically even inside the building, it's more complicated. You could, obviously, configure your firewall to treat your wireless network as an untrusted zone. This is good, and is actually the way to do, if you're using a publicly available wireless network service (say at the coffee shop or in the mall). However, if you are at home, and have your printer connected to one computer and want to print from another computer, it's much simpler to have all your home network as a trusted zone (and that's just one example, of course). But this means, that any computer connected to your home wireless network will be trusted, even your neighbor downstairs (unless you protect yourself, see below).

Denial Of Service (DOS)

Refers to a general way of attacking a target, such that some important resource would become unavailable. The idea is basically to perform a huge amount of communication with the target, taking up all it's resources for yourself, thus denying them from other, legitimate users of the resource. There are various means of performing DOS attacks on a wireless network, resulting in disrupting legitimate usage of the network. Actually, wireless networks are very poorly protected against such attacks. Anyway, I'll keep this issue out of the discussion for today.

 

OK, so we're focussing on bandwidth theft and network intrusion. How do we avoid them? Well, let me introduce you to a few friends of mine: Alfred the Optimist, Luba the Ostrich, Uninformed Max and Sacha Paranoia...

Alfred the Optimist

Alfred believes people are fundamentally good. He might not even realize there are any vulnerabilities in using a wireless network - why would anyone want to do something bad to his network? Why would someone use his bandwidth, if that person could pay for it himself? Alfred installed the router on his own, keeping the factory settings without changing anything except what was absolutely necessary to actually connect to the internet. He has been experiencing some slowness in his internet connection, especially at night, when he knows his neighbor likes to download illegal music files. But, heck, you know, that's how it goes with the internet - sometimes it's faster, sometimes is slower. Yeah, yeah, he's got occasional annoying pop-ups and he has to reboot 3 times a day otherwise it takes 7 minutes to load notepad, but who hasn't a few problems once in a while? (Note: Most Alfred's don't use any firewall for the exact same reasons, and don't understand how the salesperson managed to convince them to buy that anti-virus license...)

Luba the Ostrich

Luba is a computer programmer. She's not an expert in networking, nor in security, but she's got her BSc and understands both the problems of bandwidth theft and network intrusion. She knows there are various protection method, but she's smart enough to know everything comes with a price - if you're going to secure everything on your network, you will have to pay in performance! And Luba doesn't like to pay in performance!!! So she pokes in her router's configuration options and sees it's possible to disable broadcasting of the SSID. She searches a bit about that, maybe tries it out, and then understands that once the SSID is not broadcasted, her network cannot be found. Hey, that's cool! If nobody knows my network exists, they won't be able to connect to it - so I'm safe. Luba knows, of course, that if this was enough a solution, then there wouldn't be so many other options in the security tab of her router. But as we said, she's an ostrich, so she keeps living in denial. One day, Luba's conscience starts to bother her - maybe my network is not secure enough? So she goes back to poking around at her router's configuration settings, and then she discovers you can configure an access list - a fixed list of MAC addresses that may connect to the network. Coooool - she quickly runs "ipconfig /all" on all her computers, writes down all her wireless adapters' MAC addresses and fills the list. That's it - she can now sleep peacefully, certain she's protected against both threats.

The truth is that in most cases, Luba will be fine with either of these options (especially with both). Yes, you read correctly, in most of the cases, not all. And here is why:

1. Not broadcasting the SSID doesn't do anything to prevent someone from connecting to the network. All routers have a default SSID name. If Luba didn't change her router's SSID, then there is a good chance that many people have networks with the same SSID as she has (all aother Luba's out there). So if I used to be connected to a network with that SSID, and I find myself in the area of Luba's house, my laptop will automatically connect to her network, although she doesn't broadcast her SSID.
2. Even if she did change her SSID, that it is not broadcasted doesn't mean that it is not visible. Each packet Luba sends from her own PC holds her SSID (unencrypted). So if I use some kind of wireless sniffer (links at the bottom), I can easily discover any wireless network currently in work, including Luba's.
3. Similar to the SSID, in an unsecured wireless network, the MAC addresses are also transfered unencrypted, so once I've catched a valid packet with my sniffer, I can very easily spoof the MAC address and use hers instead of mine in my packets.

We must admit - there is no real chance anyone will do much effort to run a sniffer and then spoof her MAC address just to steal some bandwidth. So as far as bandwidth theft is concerned, with both SSID broadcasting disabled and the use of an access list, Luba is practically immunized. However, is she protected against intrusions? Well, partly - if the intruders just try to enter any wireless network to create heavoc - they won't be willing to do much effort (there are too many Alfred's out there to waste their time on Luba). However, if she's concerned about people trying to get specifically into her network, then they are most likely to be ready to make the effort, and then Luba is in trouble...

Uninformed Max

Max used to be an Ostrich, like Luba. One day, when he had his head deep under the ground, a big fat bull came around. The bull was horny, and, well you know... Anyway, since then Max has become a little bit more cautious (and has some difficulties sitting down...). He uses a specific SSID, changed his default administrator password on his router, has disabled SSID broadcasting and uses a MAC access list. In addition, he wants his network to be secure. Looking at all the possibilites, he chooses the one that seems the simplest, yet secure - WEP. He feels good an cosy with his own secured wireless network.

The problem with Max, is that nobody told him that WEP is sooo not-secure that it's just a waste of time and energy. Look at this and this and please don't miss this...

Sacha Paranoia

Sacha is one of a kind. He changes his SSID once a week and uses a 127 character password for his administrator account that changes each time he logs in. Asking him whether he disabled SSID broadcasting or uses MAC access lists could cause him to rip your head off just due to the insinuation that he might have missed that. Sacha knows everything there is to know about WEP's shortcomings. Until last year he was using WPA. Now he uses WPA2. Although his network is as secure as currently possible, he's had problems sleeping at night, imagining minuscule ET's wandering around his precious network. Last week he almost had a heart-attack, when only one of his 3 firewalls succeeded in blocking an attack. Actually, it wasn't a real attack - for some reason his firewall thinks his fax is a malicious software enemy. So Sacha disconnected his router completely. Actually, he's disconnected his computer from the internet altogether. From the power source as well - to be on the safe side. He is now working in his garden, watering his flowers. His hard-disk in his back-pocket, just in case...

 

Interesting software

Netstumbler - For mapping active wireless networks

Airsnort - Can be used to extract the WEP encryption key

Ethereal - Network protocol analyzer

Working at Google

Joel Spolsky points to Steve Yegge's view on Agile. I've personally never had the chance to work using Agile methodologies - it was inconceivable in the companies I used to work for, and now I'm mostly involved in long-term research projects, where development is minimal.

Anyway, I think that the subject of Steve's post is a little misleading. In addition to depicting why he thinks Agile is mostly bad (comparing it to good and bad cholesterol), it also provides a very interesting insight of how it is to work at Google. Like many of the commenters, I think in most companies it's not a realistic approach. Nevertheless, I'm pretty sure every company could learn a few things from them, and at least adopt some of it.

Anyway, I really enjoyed reading that post - I usually avoid reading long posts, but I'm happy I didn't skip this one :-)

Hypocrites

I'd like to add my 2 cents to Roy's post about U2U's decision not to work with Israelis.

The way I see it, politics and business are two very different think, and mixing them together is wrong. The base for this statement is that:

1. That's not the way to settle things. If anything, I think that good business, which leads to good human interaction, could be much more helpful in promoting political ideas than not doing business at all.
2. It's a collective punishment, much like the same things Mr. Uyttersprot is trying to go against.

China is one of the most oppressive countries against its own people - would you stop making business with all Chinese companies, stop buying anything Made in China, because of that?

The American use of Guantánamo Bay detainment camp has been controversial to say the least - would you stop making business with all American companies because of that?

Saudi Arabia's government states that all citizen must be Muslim. The religious minorities are not allowed to have their churches or temples or pray in public. There are also very harsh laws oppressing women. I suggest you'd stop buying all petroleum-based products...

Nike has admitted abusing its workers in the Asian continent. Not buying their products would make MUCH more sense, since it would directly hurt the abuser in question. Yet, I'd bet you'll find at least one pair of Nike's in Mr. Uyttersprot's closet.

I'm not saying Mr. Uyttersprot is right with his political claims. All I'm saying is that there are many aspects to this war, every Israeli agrees with some of the decisions taken during the war and disagrees with others. Like Roy, I believe it's almost impossible to pass a judgement when you haven't been there, let alone when you don't know all the facts. In any case, and even if you do subscribe to the political views of Mr. Uyttersprot, I don't think it's wise to mix business with politics. And if you do - please, have the decency to remain consistent with your claims: stop doing business with China, USA, Saudi Arabia and almost any other country on the globe (including Belgium, which has a history it would rather forget, oppressing the people in Kongo). And if you're not, then you're just a hypocrite.

[Note: I was born in Belgium, live in Israel and hold both nationalities.]

Search enhancement - take 2

Yesterday I suggested a way to leverage the properties of blogs to enhance searches. This made me think of some additional enhancements possible, if you try to use the type of content at hand.

Let me explain:

Current search engines employ 3 basic information sources to retrieve the most relevant results:

1. The actual text in the web page.
2. The structure of the text (i.e. headlines vs. simple content, various HTML tags, etc).
3. The structure of the web - we all know about PageRank.

What is common to all 3 sources is that they don't seriously differentiate between various types of web pages (enterprise vs. private homepages, blogs, newsgroups, news channels, e-commerce, etc.). This isn't completely accurate, since it is possible to perform searches that only search in specific sources of information (newsgroups, blogs, etc.), but that's not the point. 

What I feel is missing is an intelligent usage of the structure of each type of web page. 

Some examples:

1. Blogs, news channels - why not implement a voting mechanism (similar to PageRank or other) that takes into consideration the number of talkbacks, the number of registered RSS clients, etc.? Even if a webpage has a low PageRank, if it has a large number of commenters or many RSS subscribers, from many different places, it may indicate that the site is much more important than it may seem.
2. Newsgroups - number of threads, number of users, etc. I don't see many links to newsgroups on the web in general. Yet, some are very active. Then why not use additional, newsgroups-specific parameters to measure a newsgroup's relevance?
3. e-commerce - Many price-comparison web sites allow their users to rate products and write comments about them. I'm sure that the more products are being rated and the more raters there are, the more chances there are that the price-comparison site is a good one. Even more so - products/vendors with many/high rating accross price-comparison sites should be promoted.
4. Professional Magazines - Many give users the possibility to rank and write comments about products (like CNET), others give users the possibility to give feedback about the quality of the articles (like MSDN). Why not use that as part of the retrieval process?

What do you think?

Leveraging blogs for search enhancement

Here is an idea - any remarks/suggestions would be welcome.

One of the most important differences between a blog and a simple web site, is that a blog changes all the time. The same person could post 10 different posts on 10 different subjects. Each would be related to that person, yet the only place where they get all connected together is in the blog.

Let's imagine I met someone some time ago, we had a nice little chat, and I gave him my business card. Unfortunately, when he sent his pants to dry-cleaning, he forgot to take out the card (sounds familiar?). He remembers my first name is Ilan, remembers I live in the northern part of Israel, that I work on my thesis and that I like making soups.

Had he remembered my family name is Assayag, things would have been fine - a search for "ilan assayag" on Google brings my blog as one of the first entries. Yet he doesn't remember my family name...

What would be really cool, would be to search for everything you know about the person, and get some kind of aggregated results. So in this case, that person could search for "ilan soup thesis north israel". The search engine would know that search results from blogs should be aggregated in an intelligent manner. A simple solution could be to look at ALL the post from the same blog as if they were ONE single document (although I suppose you could come up with better solutions). If this would be possible, then I'm pretty sure that a search like "ilan soup thesis north israel" would return the correct answer, even though each element of the query can be found on another post within the same blog.

Note: I know that it would be simpler to search for "ilan blog", which, in this case, would be sufficient, although I'm pretty sure that if my name was something else, such a simple query would just not be enough.

Three Ways to Inject Your Code into Another Process

Look at this amazing article!!! (by Robert Kuster)

Matlab R2006a impressions

Until recently I was using Matlab 6.5 (R13) for both my academic and professional research projects. I tried using Matlab 7 (R14) several times (university, friends, etc.) but the UI was so slow I never seriously considered upgrading.

Along came Matlab R2006a, which I am now using a lot. The UI is still much slower than 6.5, but not slow enough to become a showstopper. Here is a short summary of my impressions of R2006a, with emphasis on the new features interesting to my work:

1. Overal performance:
1. Calculations - I really don't know. Supposedly, some of the functions were optimized (I ran into at least one or two) but I really couldn't tell I felt a difference (though it doesn't mean there isn't)
2. UI - Version 6.5 (R13) was very stable, fast and with a relatively low memory footprint. Version 7.0 (R14) was such a nightmare in terms of UI performance that it was mostly unusable. Version R2006a is much better than version 7. It's not as responsive as 6.5 (especially startup), but it's absolutely usable. Also, it has some cool added features (enhanced debugging, tabbed windows, etc) that makes it worthwhile.
2. Support Vector Machines - It's the first version with an SVM library. It's extendible and rather simple to use, but is slower than SVMLight. Also, it seems to me that in terms of accuracy it is also inferior compared to SVMLight (although I'm sure it depends a lot on the problem at hand). I have the feeling that their SVM support is still at its infancy, and I dare assume it will evolve with their next versions. For now, it's just a set of function in the Bioinformatics toolbox... Anyway, if you want a simple way to use SVMLight from Matlab, try this out .
3. Genetic Algorithms - The GA library is a real kick! It has all the features you can find in other libraries available online, and more. Also, the UI is very nice and easy to configure. It also generates graphs out-of-the-box that are much more informative than those I've been using so far. The downside is that the documentation sucks (especially when using bit strings) and that I had to debug and solve 2 bugs in the original Matlab code (!!!) to make it work properly with bit strings (I may post on this some time in the future). This being behind me, I think it's a very good library in all important ways: seems to do a good exploration of the search space (not too susceptible to local minima), fast execution, easy configuration, cool graphs, etc.
4. Distributed Computing - I only had a short time to test it at a remote location. Yet I couldn't make it work. The architecture is very simple, but the various components kept getting stuck and loosing connection for no apparent reason.

P.S: I'm posting this using Windows Live Writer - interesting to see how this turns out...

Google Image Search API - now available in Chinese

I'm happy to say that my little article on CodeProject has been translated to Chinese. I know several languages, but Chinese is not one of them, so I have no way of judging the translation :-)
Anyway, I'd like to thank hidecloud for the efforts and time put in this, and I hope it will now be helpful for some more people.

Windows Updates Has Got Some Nerve !!!

Windows Update is a great, indispensable, tool. With all the threats around the Internet, using a non-updated machine is simply stupid.
Yet, it is lacking one huge problem - configurability !!!
The configuration options are so limited I don't know whether to cry or to laugh.

I am using a server to run various CPU/memory/time-intensive tasks. Last week I was running a task that was taking 100%CPU for about a day and a half (and it had several more days to run), when suddenly "poof", the server rebooted by itself. I looked at the monitor in the morning, unable to understand why I get the logon screen, when I knew I had this quite heavy task running. A quick look at the Event Viewer showed me that the computer rebooted at 3:30 AM. Right before that, I see the Windows Update Agent, prouding himself of having finished downloading updates and being ready to install them. Since this happened to me already a long time ago, I knew exactly what happened - my server is (sorry - it WAS) configured to run the updates automatically, and some of the updates required to reboot the computer. So Mister Windows Update decided, without asking my opinion, to simply reboot and kill everything that's on its way. Mister Windows Update is so vain that He doesn't even need to let you know He did it - if you really must know, just figure it out implicitely from the Event Viewer.
BTW, when I said it happened to me a long time ago - it wasn't really to me. It happened at a customer (very large investment bank), suddenly their server rebooted in the middle of the night, without prior notice. Since the server was dedicated to the application I was responsible of, I had to figure out, what was going on remotely (I was in Tel Aviv, the server in London, and many security constraints in the middle). It took me much longer back then, since I had no clue what was going on, and like a polite developer my first assumption was that something is really wrong with my software. It took several iterations, me feeling guilty and stupid not to know how bad my own software can behave, until I found the real culprit.

Why can't we configure things like this (each, of course, should have many possibilities):

• Reboot only for high-risk security updates
• Announce the reboot X time in advance, both on the machine and by mail/SMS/whatever
• Download automatically only updates related to X,Y,Z applications - the rest doesn't interest me
• Install automatically only security updates - the rest leave to me
• Download at time X, send email/SMS/whatever in order to give the administrator the time to run the install in an orderly fashion, if he doesn't then install at time Y
• Before installing, send a message to the currently logged-in user, to let him overrule (i.e. postpone the install), send an email/SMS/whatever
• Install/reboot only if the computer has been idle for some time
• If you are configured to reboot, and really have to kill the rest of the process population, have the decency to let a note behind, in the form of mail/SMS/Event Viewer/whatever

I can go on forever, but you get the point.

Some interesting search engines

A couple of weeks ago I posted about a way to get more results from your search engines (i.e. beyond the number of maximal results for a query). I then referred to Ask Jeeves, which has been along for quite some time, and provides a nice query refinement interface.
I was asked whether I know of any more search engines, which provide more features than the features available by the 3 giants (Google/Yahoo!/Live). Well, here's my personal list (in no particular order):

A9 - Simultaneously runs your search on various data sources. For example, you can enter a query and it will search for images, books, web and wikipedia all at once. The results are then presented, all on one page, but in a separate columns for each data source.
Vivisimo - Clustering of query results into groups of related results.
iBoogie - Clustering of query results into groups of related results.
Mooter - Clustering of query results into groups of related results. The clusters are presented in a graphical form, and when you drill-down you get a similar tree-like interface like the others.
DogPile - Simultaneously searches all major search engines (Google, Yahoo!, MSN, Ask Jeeves and more). The results are presented in aggregated form, with an indication of which engines returned it.
KartOO - Clustering of query results into groups of related results. The clusters are presented in a graphical form. Their UI may seem somewhat complex, but it's pretty cool once you get used to it.
WiseNut - Groups related results into categories. It generates much less categories than other similar engines.
Infonetware - Clustering of query results into groups of related results.

If you have anything to add, please let me know :-)

What is my mother tongue?

I was born in Antwerp, Belgium. Although the local formal language is Flemish, at home and with my friends I spoke exclusively French. Most lessons at school were in Flemish, and so was the language we had to use on the streets.
At the age of 13 we moved to Israel. With my family I kept speaking French, but everything else was Hebrew. Well, except for my baby sister, who was 3 when we moved to Israel, so with her I speak a mix of French and Hebrew. All my studies from high-school to Masters Degree were in Hebrew.
For 10 years now I have been working with computers, all the scientific and other professional material I read is in English, 99% of my mail communication is in English, I like to read books in English, etc.
I type at least 10x faster in English than any other language.
I read the fastest in English, then French and rather slowly in Hebrew.
There is no language I can (hand) write such that other people than me would be able to decipher what I wrote. And that's when write slowly.
When I write (or type) in French, the mistakes I make suit a 13-year old.
I dream usually in either French or Hebrew, but once in a while I even dream in English.
I speak Hebrew with my wife and French with my child.
People say I have a Hebrew accent in French, a French-Belgian accent in Hebrew and a mixed French/Hebrew accent in English.

Now you tell me - what is my mother tongue?

Finding Great Developers

Joel has once again published a very interesting article, this time about Finding Great Developers.
The most interesting part, to me, was his description of their internship and how they guide it to result in perfect recruitments.
For my part I have been responsible of recruiting several employees in the past. In the company I worked for, this kind of long-term pipeline was not an option - I usually had only a few weeks or a few months to find the person I needed, and I always needed somebody with experience.
In my experience, and in contrast with Joel's, employee referrals have always proved to be the best source. The most important part, though, is to interview the referring employee thoroughly, before even starting the process with the candidate. I have always seen that when you talk to someone who knows the candidate, if you ask the right questions, you can get a very accurate idea of what to expect. The clue (and it's IMHO the base for everything related to recruiting) is to ask open questions, that will force the referrer to tell the things you want to hear.
I agree that there is a conflict of interest, when you give a bonus to your employee for a recruit, but if you know your employee, trust her, and interview her thoroughly about the referred friend, you should be OK. Also, although non-compete agreements are sometimes used here in Israel as well, as far as I know, the legal situation here makes it very difficult to enforce them (they are usually overruled by a law for free employment).

Getting more results from your search engine

Most search engines limit the total number of results they return per search string. With Google it's 1000. With Yahoo!, if it hasn't changed lately, it's 5000.
I've been asked a few days ago whether I know of any workaround that makes it possible to get more results per search string. My first answer was - "sorry, no can do". I did find a way to circumvent the limitation imposed by the Google API that limits the number of queries that can be executed per day (which is accidentally also limited to 1000). This workaround is a side-effect of my Google Image Search API. Yet, this does not provide a means to get more than 1000 results per search string.
After giving it some thought, I could figure out at least one way to increase the number of results per query. It's not a very accurate solution, but it's better than nothing. The idea is to use the various search engines that perform query refinement (some call it clustering of results). A good example is Ask Jeeves. When you perform a search on these engines, they also give you a list of suggestions to narrow or expand your search. That is, if you search for "apple", the narrowing suggestions are things like "Apple the fruit", "Facts about Apples", "Apple Tree", "Macintosh", etc.
When you work with this kind of engines (or with a "simple" search engine and one with narrowing capabilities together), you can start out by running the original search (apple) and retrieve all the results available for that. Then you can iteratively retrieve the results for all narrowing queries as well (up to 1000 for each), and keep drilling down as much as you like. Of course, there will most probably be a substabtial amount of duplicates in the results, which you will have to handle. Also, the more you drill down, the farther you'll get from the original query (i.e. query drift). Another problem is that of ranking - say your original query was "apple", how do you define the ranks between the results for "apple tree" and "Macintosh". So this still raises quite a few questions. Yet, in the end, you can end up with a much larger number of results that are to some extent related to the original query.

You may ask - why would someone need more than 1000 results per search string? Besides, the further you go down the ranking, the less their relevance to the original search string. In most cases - you're right. Yet, for some research purposes, not only would you need more than 1000 results - you might even prefer getting these than the "good" results returned in the first few pages.

Can anyone come up with some other (better?) idea to work around this limitation??? If you have an idea - please drop me a line!

(Note: I'm using the term "search string" to indicate a complete search, regardless of the number of results pages you get. The term "query" refers to what retrieves one single results page, since the query also includes the result index at which the results page should start. In other words, all "search results" for a single "search string" are achieved by sending multiple "queries" - if you have 100 results in each results page you need to execute approximately 10 "queries" to retrieve all the results Google provides for that "search string")

Setting default share permissions

If you use shared folders often, you probably know that Windows XP defaults share permissions to "Everyone" (with full access). If you don't know that - shame on you!
I don't know exactly where this default is stored, but if you want to change it, you can do with with the Tweak UI PowerToy.
In the tree on the left, select "Access Control". Then choose the "Default share permissions" in the combobox and click "Change".
Enjoy :-)

Working with Source Safe over the web

I am working (from Israel) for a company from the US (I haven't finished my thesis yet, but my grant is dry and I still need to feed my family...). Since I'm working on applications related to trading, there is a great emphasis on security so every file transfer is done over a VPN.
Lately I needed to work directly with their VSS database. I agree with most that Source Safe is really something that should be left in the past, but it will take some time before I can convince them to switch and in the mean time work must continue. Everybody who has tried to use VSS over an Internet connection knows it's just impossible to work this way. Add to that the cross-Atlantic delay and a VPN link and you get to wait 10-20 minutes only to open a tree in the viewer (that is, if you're lucky enough not to get link errors, which I'm currently investigating with my ISP). So I've been looking for applications that help accessing an existing VSS database over the Internet.
The major tools I found were as follows:

SourceAnyWhere (SAW) - http://www.dynamsoft.com/products/SAW_Overview.aspx

SourceOffSite (SOS) - http://www.sourcegear.com/sos/index.html

SourceXT - http://www.acorden.com/sourcext/features.htm

VSS Remoting - http://www.sourceremoting.com/index.htm

VSS Connect - http://www.vssconnect.com/VssConnect.htm

The prices vary around 150$-250$ per user, except for VssConnect which costs only 30$.

They all work with some kind of web server and have a way to encrypt the data, but in our case it's irrelevant since we are working over a VPN link anyway.

SAW and VSSRemoting have a special feature that supposedly improves file transfer significantly, by transfering only the parts of the files that are different.

We didn't want to waste too much time reviewing each, so we decided to start by checking out the one that looks the most mature (SAW).

In general, I am very happy using it. It works really fast, integrates with Visual Studio 2005 and does the work well. I did encounter a few issues, though:

1. Switching with the same solution between SAW and VSS didn't work well for me. I contacted DynamSoft's support and they tried to help me, but it didn't work. In my case I don't care much, because 99% of the time I'll be using SAW anyway. If you plan on switching between them often - I suggest you check it out thoroughly.

2. File comparison application dissapointing:

a. Compares lines and not words/characters (like VSS)

b. No option to ommit blanks, so even if the difference is just an irrelevant space, you will see the whole line marked as different (like VSS)

c. The GUI doesn't work properly - when there is only one difference, the arrows to jump to the difference are disabled when there is only one.

(NOTE: According to Support, this issue should be fixed soon - in the next release).

3. Sometimes, when I just do something on the solution with no need to interact with VSS, it starts performing all kinds of synchronization operations with the server. During that time I can't do anything with the solution. Since SAW works really fast with the server, it's not the end of the world, and usually this process is over in 10-20 seconds. It also doesn't happen a lot, but still - it's annoying and shouldn't happen at all.

(NOTE: According to Support this is initiated by the IDE and not by the SAW integration client directly)

4. When trying to perform something with too many files (~15+) I get socket errors. The link to the server isn't broken, but the specific operation is aborted. This is a very painful issue, since it requires a lot of manual workarounds. I'm still investigating this issue with both DynamSoft and my ISP - I have reasons to believe it may be caused by problems in my Internet connection. I would still have hoped SAW would be able to cope with minor connection problems, though...

Despite these, we are probably going to purchase it - especially if I manage to fix the socket errors problem by fixing the Internet connection.

As a last note, Support told me that in the next month they are about to release their next version (5). I asked whether, if we purchase SAW before the release, we would be entitled to an automatic upgrade when the next version will actually be released (like JetBrains did when we purchased Resharper 1.5, little before the release of 2.0). I was dissapointed by their answer: they haven't decided yet about the upgrade policy. Therefore, even if I manage to fix the socket errors problem, we will still have to wait for the release of the next version before purchasing (or for them to tell me we will get the next version for free anyway).

The War

On July 16th I had to flee my house – and came back after having wandered around the country with my family for 30 days. If you want to get a glimpse of what the war has done to my personal life, here you go.
I live in a small village in the Izrael valley. It’s in the northern part of Israel, but quite far from the border with Lebanon (about the same distance as Haifa). I have a magnificent little girl, a beautiful wife 7 months pregnant and a crazy dog. In the proximity of our village is an important air-force base. In normal days, the sound of the planes is not very pleasant, but you learn to live with it. We knew, before moving to that village, that if a war was to happen, our little heaven would be troubled, especially due to the proximity to the base, who is an obvious target. We never figured how much …
To keep things simple, I’ve decided to summarize the impact of this war on our day to day life as follows:

• We don’t have any shelter in our house – so from the beginning we had to flee in order to be safe. So we slept 30 days at other people’s who have been kind enough to open their house for us (6 different places).


• Rockets landed a couple of hundred meters from where I was, and more importantly from my girl’s kindergarten (in the town next-by).


• The first time the rockets landed close by, all phone lines in the neighborhood of the kindergarten crashed. From the road we took to the kindergarten it took a while to understand that the smoke came from behind the kindergarten and not from there exactly…


• At some point my wife decided to go back to work (we were staying at our in-laws at the time who unlike us have a shelter). The sirens caught her when she was in the parking lot, ready to go back home. It’s an open lot, with no cover at all. At first she simply dropped to the ground, in order to try and avoid the deadly bullets. When the rockets started to fall she tried to find something to she could shelter under. She found a place with a 3 millimeter roof where a few other people were taking shelter. The funny thing was that the roof was the last of their concern – there were many gas tanks pilled up right next to them … When the attack was over, she literally flew home. Two hours later there was another attack in the same area. 3 of the rockets landed right in the path my wife uses back home.


• One specific attack was particularly scary – we were at my in-laws and when the siren started my mother in law was in the shower. She didn’t make it to the shelter on time. Suddenly the rockets started falling – REALLY close. My wife, who’s 7 months pregnant sat next to me, in the shelter. At that moment, while we felt the whole house tremble and her mother wasn’t answering our calls – I thought we were about to loose our baby.


• Before that same attack, my 2-year old girl was looking at a DVD of the Teletubies. For those who don’t know what this is – they are little creatures who are happy with everything and laugh for anything. You could rip their heads off and they would still find a reason to laugh and be happy and nice. Anyway – from that moment on, my little girl doesn’t stop telling me that “the Teletubies scared her”. She has become moody, winy, and can’t stay more than a few minutes without seeing us both (don’t even think of leaving her with somebody else). She often wakes up screaming shortly after having fallen asleep, probably due to nightmares.


• Until now, when I hear an ambulance my heart misses several beats – my first reaction is that it’s a siren again. Any strong noise (even a door being slammed) makes me fear a rocket has fallen. When I’m with others, we usually exchange looks and it’s clear to all that we all experience the same thing.


• The thing is – we are among the lucky ones. None of our friends and relatives got killed or seriously injured. Just to show you how lucky we are indeed - a friend of us lives in another village next to us. Two of her nephews who live in the same village got hit by those horrible bullets being propelled by the rocket when it explodes. They each got 2 bullets in the arm. In the past two weeks they endured together more than 10 surgeries and they are not over yet. Imagine if the bullets had hit the abdominal region or the head…